1/ 28

Calibration procedure for estimates obtained from

posterior approximation algorithms, with application

to domain-level modeling

Julie Gershunskaya 1

1U.S. Bureau of Labor Statistics (OEUS Statistical Methods Staff)

FCSM Conference - October 2023
2/ 28

Outline

Motivation

Resampling based approach

Simulations

Small domain model used in CES

Summary and Recommendations
3/ 28

Motivation

▶ Current Employment Statistics (CES) survey publishes monthly

estimates of employment at detailed levels by industry and

geography

▶ In small samples, direct sample-based estimates can be

unstable. We use Small Area Estimation (SAE) modeling

techniques to produce better estimates for small domains

▶ To formulate SAE models, we choose a Bayesian approach for its

flexibility and the ability to handle complicated models

▶ CES produces estimates monthly and has tight production

schedule: It is essential to use fast and efficient model fitting

algorithm.
4/ 28

Motivation (cont’d)

▶ We use Automatic Differentiation Variational Inference (ADVI)

algorithm implemented in Stan modeling language. The mean field

approximation of the ADVI is relatively fast and efficient.

▶ Small domain model was tested on historical CES series: estimates

showed better performance, compared to alternative models

▶ However, it has been reported in the literature (Yao et al. 2018)

that ADVI (in particular, the mean field approximation) may

produce inaccurate uncertainty measures of point estimates

▶ The goal of the current research: to develop a practical tool for

the evaluation of the fit and correction of a bias in the posterior

variance of the ADVI-based point estimator
5/ 28

Outline

Motivation

Resampling based approach

Simulations

Small domain model used in CES

Summary and Recommendations
6/ 28

Setting the stage: Fay-Herriot Model

Available data for each of i = 1, . . . , N domains of interest:

▶ yi

: direct sample-based estimates

▶ vi

: variances of direct estimates yi

▶ xi

: vector of covariates

Assume the following two-level model:

yi

ind∼ N (θi

, vi) sampling model

θi

ind∼ N



x

T

i β, τ 2u



linking model

β and τ

2

u

are unknown model parameters.

Fit the model and obtain θi using ADVI algorithm.
7/ 28

Resampling based approach, general outline

Assume, the model is correct but the fitting algorithm may produce

systematic errors in uncertainty measures.

The goal is to evaluate and potentially correct the fit.

We focus on the first two moments of the distribution of model fitted

parameters

Steps:

1. Fit the model using the original data.

2. Extract multiple samples from the posterior predictive distribution.

3. Refit the model for each of these ”bootstrap“ samples.

4. Evaluate results against originally fitted parameters.

5. Adjust (if needed) original estimates.
8/ 28

Resampling based approach, details

Example: FH model

yi

ind∼ N (θi

, vi) sampling model

θi

ind∼ N



x

T

i β, τ 2u



linking model

Steps:

1. Fit the model using ADVI algorithm on the original data.

2. Extract random draws (θ

(α)

i

, y

(α)

i

) from the posterior distribution of

θi and the posterior predictive distribution of yi, α = 1, . . . , A

3. Refit the model for each re-sampled dataset α = 1, . . . , A using the

same ADVI algorithm. Obtain posterior means m(θ

(α)

i

) and

variances v(θ

(α)

i

) for respective parameters.
9/ 28

Pivotal quantity

▶ Form pivotal quantity

T

(α)

i =

m(θ

(α)

i

) − θ

(α)

q

i

v(θ

(α)

i

)

.

▶ If m(θ

(α)

i

) is unbiased for θ

(α)

i

and v(θ

(α)

i

) is consistent estimate of

its variance, then T

(α)

i ∼ (0, 1).

▶ However, we assume our model parameters are estimated with an

error, so the moments of T

(α)

i would have to be corrected.
10/ 28

How to adjust the pivot?

Suppose, true posterior variance of θiis

Var(θi) = v(θi)ci,

where

v(θi) is the posterior variance of θi obtained from the original run using

some approximation algorithm;

ciis a shift in scale due to the approximation algorithm.

1. Based on our assumption, in the “bootstrap world”, v(θi) = v(θ

α

i

)ci.

Thus, to correct the variance, we would have to divide pivot T

α

i

by

√

ci

2. Since we draw “bootstrap” samples from a posterior distribution

with biased variance v(θi), we would have to divide pivot T

α

i

by √ci

(again!), to bring it up to true posterior variance Var(θi).
11/ 28

Estimation of the adjustment

Variance of thus adjusted pivot is

Var(c

−1

i

T

(α)

i

) = 1 ⇒ c

2

i = Var(T

(α)

i

)

Thus, we can estimate ci from the bootstrap as

ci =

r

A−1XA

α=1

(T

(α)

i − T ̄

i)

2,

where

T ̄

i = A

−1XA

α=1

T

(α)

i

.
12/ 28

I. Pivot-based Confidence Intervals

The adjusted pivot is

T ̃

(α)

i =

T

(α)

i − T ̄

i

ci

∼ (0, 1).

The calibrated CI for area i is

Ci(γ) = hm(θi) + pv(θi)cit ̃i,γl, m(θi) + pv(θi)cit ̃i,γr

i

,

where t ̃i,γland t ̃i,γrare quantiles of T ̃

(α)

i

over the bootstrap

distribution, for a given nominal level γ.
13/ 28

II. Re-scaled Confidence Intervals

Adjust each draw θ

∗

i

from the original posterior distribution of θi:

 ̃θ

∗

i =

θ

∗

i − m(θi)

p

v(θi)

p

v(θi)ci +  ̃m(θi),

The adjusted CIs are obtained by computing percentiles over the

adjusted draws  ̃θ

∗

i
14/ 28

Outline

Motivation

Resampling based approach

Simulations

Small domain model used in CES

Summary and Recommendations
15/ 28

Simulation setup: FH model

▶ Consider N = 150 domains. Set: β = 1, τ

2

u = 1, σ

2

i = 1.

▶ Generate:

xi

ind∼ Unif(0, 2) covariates

ui

ind∼ N



0, τ 2

u



random effects

εi

ind∼ N



0, σ2

i



random errors

▶ True domain values: θi = x

T

i β + ui

“Direct” domain estimates: yi = θi + εi

Suppose variances of yi are measured exactly (vi = σi) and known.

▶ Run the model and extract S = 200 simulation datasets from the

posterior distribution of θi and posterior predictive distribution of yi.

▶ Repeat re-sampling algorithm A times on each of S datasets.
16/ 28

Results: FH model

Table: Coverage properties for model fitted m(θi), 50% nominal, over 150

domains and S = 200 simulation runs, using A = 500 re-samples

Orig Fitted Rescaled Pivot

Coverage 0.531 0.493 0.492

Length 1.019 0.935 0.933
17/ 28

Coverage of Point Estimate, 50% nominal, FH model

0.00

0.25

0.50

0.75

1.00

0 50 100 150

domain
18/ 28

Length of 50% nominal CIs, FH model

0.92

0.93

0.94

0.95

1.010 1.015 1.020 1.025 1.030 1.035

orig
19/ 28

Outline

Motivation

Resampling based approach

Simulations

Small domain model used in CES

Summary and Recommendations
20/ 28

Co-modeling of variances and co-clustering model (CCFH)

(yi, vi) are observed data, where yi direct sample-based estimates and

vi are direct sample-based estimates of variances of yi.

(θi, σ2

i

) are model parameters

yi

ind∼ N



θi, σ

2

i



sampling model for θi

θi

iid∼

XK

k=1

πkN



μk + x

T

i β, τ 2u



linking model for θi

vi

ind∼

XK

k=1

πkG



ani, anibkσ

−2

i



sampling model for σ

2

i

σ

2

i

ind∼

XK

k=1

πkIG 

2, exp z

T

i γk

 linking model for σ

2

i

In CCFH, vi are modeled along with the point estimates yi.
21/ 28

Simulation setup: CCFH model

▶ Used real CES data for the initial run.

▶ Extract S = 200 simulation datasets from the posterior distribution

of θi and posterior predictive distributions of yi and vi.

▶ Repeat re-sampling algorithm A = 500 times on each of S datasets.
22/ 28

Results: CCFH model

Relative residuals of original and adjusted variances of θi

−2

0

2

4

 OrigVarFitted AdjVarFittedY

(EstimatedVar − TrueVar)/TrueVar

Over 166 domains and 200 simulations.
23/ 28

Results: CCFH model

Table: Coverage properties for model fitted m(θi), 50% nominal, over 166

domains and S = 200 simulation runs, using A = 500 re-samples

Orig Fitted Rescaled Pivot

Coverage 0.625 0.559 0.549

Length 0.618 0.537 0.528
24/ 28

Coverage of Point Estimate, 50% nominal, CCFH model

0.00

0.25

0.50

0.75

1.00

0 50 100 150 200 250

RespUI
25/ 28

Length of 50% nominal CIs, CCFH model

0.2

0.4

0.6

0.8

0.3 0.4 0.5 0.6 0.7 0.8

orig
26/ 28

Outline

Motivation

Resampling based approach

Simulations

Small domain model used in CES

Summary and Recommendations
27/ 28

Summary and Recommendations

▶ Inaccurate uncertainty measurements of some approximation

algorithms have been reported in the literature

▶ We considered a resampling based evaluation and adjustment

methods

▶ The methods rely on the assumption that the model is correct.

Hence it is important to conduct thorough model checking

▶ Bias adjustments may be needed only if there is indications of a

significant bias; otherwise, we may only be adding noise

▶ Although our main target was just the first two moments of the

distribution of the fitted parameters, for the considered models, the

procedure also lead to CIs with nearly nominal coverage properties

▶ The pivot-based method gives slightly shorter but more variable CIs.

It is also less practical as it requires larger number of bootstrap

samples.
28/ 28

Thank you!

▶ CONTACT INFORMATION:

▶ Gershunskaya.Julie@bls.gov