Bayesian Nonparametric Multivariate Spatial Mixture

Mixed Effects Models

Scott H. Holan1,2, Ryan Janicki2, Andrew M. Raim2, and Jerry Maples2

Department of Statistics

University of Missouri1

and

U.S. Census Bureau2

October 27, 2022

This report is released to inform interested parties of ongoing research and to encourage discussion of work in progress. The

views expressed on statistical issues are those of the authors and not those of the U.S. Census Bureau. The DRB approval

number for this project is CBDRB-FY19-490.

Scott H. Holan FCSM – 2022 October 27, 2022 1 / 35
Outline

Overview of the project goals, description of the datasets, and

discussion of difficulties with the data

Introduction of the multivariate spatial model

Summary of model-based predictions of counts within counties and

comparison to direct estimates

Discussion of how the model can fail

Extension of the multivariate spatial model to a mixture of

multivariate spatial models

Conclusion

Scott H. Holan FCSM – 2022 October 27, 2022 2 / 35
American Community Survey (ACS)

Nationwide survey collecting information about, for example, race,

citizenship, employment, and other demographic and housing unit

characteristics

3.5 million households sampled each year

Billions of estimates produced

1-year estimates available for areas with a total population of 65,000+

5-year estimates available at geographies down to the block group

Scott H. Holan FCSM – 2022 October 27, 2022 3 / 35
Examples of Requested ACS Special Tabulations

Household income to poverty ratio

Proportion of the population in poverty

Total population by age, race/ethnicity, sex

Counts of housing units by Tenure by Household Size by Age of

Householder by Household Income

These tables are needed at different geographic levels, such as by State,

County, Tract, or American Indian and Alaska Native (AIAN) area.

Scott H. Holan FCSM – 2022 October 27, 2022 4 / 35
ACS Tabulations Used

Dataset 1 (AGE): Counts of children by age (0–1, 2–3, and 4–5) in

counties in Minnesota

Dataset 2 (AGE x RACE): Counts of children by age (0–1, 2–3, and

4–5) by Race (White Alone, Black Alone, Asian Alone, Native

Hawaiian or Pacific Islander Alone, American Indian and Alaska Native

Alone, Other Alone, or two or more races) in counties in Minnesota

Scott H. Holan FCSM – 2022 October 27, 2022 5 / 35
Issues with the Data

Small or zero sample in some counties for some cells

Skewed or multimodal distributions of direct estimates

A large number of cells with direct estimates of zero

Some sampling variances are either zero or undefined

Multivariate dependence between the observations within county and

multivariate spatial dependence between counties

Limited predictor variables available

Scott H. Holan FCSM – 2022 October 27, 2022 6 / 35
Data

0

500

1000

1500

2000

2500

0e+00 1e+05 2e+05

Direct Estimate

count

Histogram of the ACS 5 year estimates

 of total children ages 4 − 5

 in all counties in the U.S.

0

50

100

150

200

0 5 10

Log of Direct Estimate

count

Histogram of the log of the ACS 5 year

 estimates of total children ages 4 − 5

 in all counties in the U.S.

Scott H. Holan FCSM – 2022 October 27, 2022 7 / 35
Direct Variance Estimates

Scott H. Holan FCSM – 2022 October 27, 2022 8 / 35
Notation

Z

∗

i

is the direct estimate for the ith observation. For ease of

presentation, i indexes the counties and demographic variables.

Zi = log(Z

∗

i + 1)

x

T

i

a vector of predictors, which includes the log of the county

population size, and dummy variables corresponding to the

demographic cross classifications

Di = Var(Zi) are the estimated (design-based) sampling variances

Scott H. Holan FCSM – 2022 October 27, 2022 9 / 35
Multivariate Spatial Model

Cressie and Wikle (2011); Bradley et al. (2015):

Data Model:

Zi = Yi + εi

εi ∼ N(0, Di)

Process Model:

Yi = x

T

i β + s

T

i η

η ∼ Nr



0, σ2K



Goal: Prediction of Y

∗

i = e

Yi − 1

Scott H. Holan FCSM – 2022 October 27, 2022 10 / 35
Remark About the Choice of Process Model

A popular choice of process model used in the spatial literature is

Yi = x

T

i β + s

T

i η + ξi

,

where

x

T

i β captures trends in population and demographic cross

classifications, as well as multivariate spatial dependencies

s

T

i η accounts for residual multivariate spatial dependence (random

effects)

ξi represents fine-scale variability

Note that including ξi resulted in overfitting to our datasets

Scott H. Holan FCSM – 2022 October 27, 2022 11 / 35
Basis Functions

The matrix S, with rows consisting of the vectors s

T

i

, are constructed via

Moran’s I basis expansion (Hughes and Haran, 2013):

Let PX = I − X(X T X)

−1X T

Let A be the adjacency matrix of observations

The columns of S are the first r eigenvectors of PX APX

We used 50% of the available basis functions: eigenvectors

corresponding to the largest positive eigenvalues

Scott H. Holan FCSM – 2022 October 27, 2022 12 / 35
Construction of K

The matrix K is constructed to

induce multivariate dependencies

induce spatial dependencies using a conditional autoregressive (CAR)

structure

be reduced rank compared to a CAR process

Random effects may be shared across observations, inducing

multivariate dependencies

Scott H. Holan FCSM – 2022 October 27, 2022 13 / 35
Construction of K Cont.

Let u

T = (u1, . . . , un) be an intrinsic conditional autoregressive (ICAR)

process, with precision matrix 1

σ2Q, so that

ui|uj, j 6= i, σ2 ∼ N





X

j∼i

j

ni

,

σ

2

ni



 ,

where niis the number of neighbors of area i. Then

K = arg min

C








Q − SC −1S

T










F

,

where the minimization is over the space of r × r positive definite matrices.

Scott H. Holan FCSM – 2022 October 27, 2022 14 / 35
Estimation of the number of children by

age in counties in Minnesota using the

multivariate spatial model

Scott H. Holan FCSM – 2022 October 27, 2022 15 / 35
Prediction of the Number of Children, Ages 0 – 1, 2 – 3, 4

– 5, in Counties

Our target is predictions of counts in counties in Minnesota.

We fit the model to counties in Minnesota and adjacent states (Iowa,

North Dakota, South Dakota, Wisconsin)

We wrote code in R and STAN (Carpenter et al., 2017) to fit the

multivariate spatial mixture model to each dataset

Scott H. Holan FCSM – 2022 October 27, 2022 16 / 35
Comparison of Direct Estimates and Predicted Values

0

10000

20000

30000

Count

0

10000

20000

30000

Count

(a) (b)

Figure: (a) Direct estimates of the number of children, ages 0–1.

(b) Model-based predictions of the number of children, ages 0–1.

Scott H. Holan FCSM – 2022 October 27, 2022 17 / 35
Comparison of the Standard Errors of Direct Estimates and

Standard Errors of Predicted Values

0

100

200

300

400

500

Count

0

100

200

300

400

500

Count

(a) (b)

Figure: (a) Standard errors of direct estimates of the number of children, ages

0–1. (b) Posterior standard error model-based predictions of the number of

children, ages 0–1.

Scott H. Holan FCSM – 2022 October 27, 2022 18 / 35
Min Q1 Median Q3 Max

96 78 74 68 41

Table: Percent reduction in coefficients of variation

Scott H. Holan FCSM – 2022 October 27, 2022 19 / 35
Estimation of the number of children by

age and race in counties in Minnesota

using the multivariate spatial model

Scott H. Holan FCSM – 2022 October 27, 2022 20 / 35
Example of Varying Spatial Patterns in the Data

5000

10000

15000

Count

Direct estimates of the number of White children,

ages 2 − 3, in counties in Minnesota

0

200

400

600

Count

Direct estimates of the number of

American Indian or Alaska Native children,

ages 2 − 3, in counties in Minnesota

Scott H. Holan FCSM – 2022 October 27, 2022 21 / 35
Great results fitting the multivariate spatial model to a wide variety of

ACS special tabulations

Can have poor results when fitting to more complicated datasets with

more demographic cross classifications

Issues:

Varying spatial patterns for different Race/Ethnicity groups.

The spatial model can be too aggressive in smoothing the data

Possible solutions:

Manually break up difficult datasets if we know the clustering

structure, and fit separate models on each cluster of data

Use a model-based approach to simultaneously cluster the data, and

predict counts

Scott H. Holan FCSM – 2022 October 27, 2022 22 / 35
Multivariate Spatial Model with Dirichlet Process Mixing

Data Model:

Zi = Yi + εi

εi ∼ N(0, Di)

Process Model:

Yi = x

T

i βi + s

T

i ηi

θ

T

i =



β

T

i

, η

T

i



| G

i.i.d. ∼ G

G ∼ DP (αG0)

α is a concentration parameter and G0 the base measure

Scott H. Holan FCSM – 2022 October 27, 2022 23 / 35
Multivariate Spatial Model with Dirichlet Process Mixing

Cont.

Parameter Model:

G0 = Np(0, σ2

β

Ip×p)Nr(0, σ2

ηK)

α ∼ Gamma(aα, bα)

σ

2

β ∼ InvGamma(aβ, bβ)

σ

2

η ∼ InvGamma(aη, bη)

With this parameterization we have conjugacy. However, the Gibbs

sampler can be slow to converge.

Scott H. Holan FCSM – 2022 October 27, 2022 24 / 35
Stick-breaking Representation of the Dirichlet Process

Prior

Sethuraman (1994):

G (·) = X∞

j=1

πj δθj

(·)

θj

i.i.d. ∼ G0

Vi

i.i.d. ∼ Beta(1, α)

π1 = V1

πi = Vi

Y

k≤i

(1 − Vk )

Implementation: Gibbs sampler in R/C++ (Neal, 2000)

Scott H. Holan FCSM – 2022 October 27, 2022 25 / 35
Multivariate Spatial Mixture Model – Finite-dimensional

Approximation

We can truncate G to get a finite-dimensional approximation

Data Model:

Zi ∼

X

K

j=1

πjf (zi| Yij , Di),

where f (· | Yij , Di) is the normal density with mean Yij and variance

Di.

Process Model:

Yij = x

T

i βj + s

T

i ηj

ηj ∼ Nr



0, σ2

j K



Scott H. Holan FCSM – 2022 October 27, 2022 26 / 35
Multivariate Spatial Mixture Model Cont.

πj are the probabilities of belonging to cluster j

The number of clusters is not random

If the truncation level is sufficiently large, many clusters will be empty

Nearly identical results using a Dirichlet process prior or a stick

breaking prior, as long as the truncation is sufficiently large.

Faster convergence using the stick breaking prior.

Scott H. Holan FCSM – 2022 October 27, 2022 27 / 35
Estimation of the number of children by

age and race using the multivariate

spatial mixture model

Scott H. Holan FCSM – 2022 October 27, 2022 28 / 35
Comparison of Direct Estimates and Predicted Values of

the Number of White Children, Ages 2 – 3

0

5000

10000

15000

20000

Count

Direct estimates of the number of

White children, ages 2 − 3,

in counties in Minnesota

0

5000

10000

15000

20000

Count

Model−based predictions of the number of

White children, ages 2 − 3,

in counties in Minnesota

Scott H. Holan FCSM – 2022 October 27, 2022 29 / 35
Comparison of the Standard Errors of Direct Estimates and

Standard Errors of Predicted Values

0

100

200

300

400

500

Count

Standard errors of direct estimates of the number of

White children, ages 2 − 3,

in counties in Minnesota

0

100

200

300

400

500

Count

Standard errors of model−based predictions of the number of

White children, ages 2 − 3,

in counties in Minnesota

Scott H. Holan FCSM – 2022 October 27, 2022 30 / 35
Comparison of Direct Estimates and Predicted Values of

the Number of American Indian and Alaska Native

Children, Ages 2 – 3

0

200

400

600

Count

Direct estimates of the number of

American Indian or Alaska Native children,

ages 2 − 3, in counties in Minnesota

0

200

400

600

Count

Model−based predictions of the number of

American Indian or Alaska Native children,

ages 2 − 3, in counties in Minnesota

Scott H. Holan FCSM – 2022 October 27, 2022 31 / 35
Comparison of the Standard Errors of Direct Estimates and

Standard Errors of Predicted Values

0

20

40

60

Count

Standard errors of direct estimates of the number of

American Indian or Alaska Native children,

ages 2 − 3, in counties in Minnesota

0

20

40

60

Count

Standard errors of model−based predictions of the number of

American Indian or Alaska Native children,

ages 2 − 3, in counties in Minnesota

Scott H. Holan FCSM – 2022 October 27, 2022 32 / 35
Min Q1 Median Q3 Max

57 30 20 7 -70

Table: Percent reduction in coefficients of variation for counties with direct

estimates with standard errors greater than 50

Scott H. Holan FCSM – 2022 October 27, 2022 33 / 35
Conclusions

We were able to produce model-based estimates with greater

precision than the corresponding direct estimates, despite the lack of

strong covariates

A multivariate mixed effects spatial model can be used for a wide

variety of ACS special tabulations

For more complicated datasets with varying spatial and/or

multivariate characteristics, we developed a multivariate spatial

mixture model which provides greater flexibility

The model is of independent interest and is applicable to a broad set

of subject-matter scientific problems

Scott H. Holan FCSM – 2022 October 27, 2022 34 / 35
Thank you!

holans@missouri.edu

Scott H. Holan FCSM – 2022 October 27, 2022 35 / 35
Bradley, J. R., Holan, S. H., and Wikle, C. K. (2015). “Multivariate spatio-temporal

models for high-dimensional areal data with application to Longitudinal

Employer-Household Dynamics.” Annals of Applied Statistics, 9, 4, 1761–1791.

Carpenter, B., Gelman, A., Hoffman, M., Lee, D., Goodrich, B., Betancourt, M.,

Brubaker, M., Guo, J., Li, P., and Riddell, A. (2017). “Stan: A Probabilistic

Programming Language.” Journal of Statistical Software, Articles, 76, 1, 1–32.

Cressie, N. and Wikle, C. K. (2011). Statistics for Spatio-Temporal Data. John Wiley &

Sons.

Hughes, J. and Haran, M. (2013). “Dimension reduction and alleviation of confounding

for spatial generalized linear mixed models.” Journal of the Royal Statistical Society,

Series B, 75, 1, 139 – 159.

Neal, R. M. (2000). “Markov chain sampling methods for Dirichlet process mixture

models.” Journal of Computational and Graphical Statistics, 9, 2, 249 – 265.

Sethuraman, J. (1994). “A constructive definition of Dirichlet priors.” Statistica Sinica,

4, 639 – 650.

Scott H. Holan FCSM – 2022 October 27, 2022 35 / 35