A model-based allocation approach

for estimation of adult competency

in small domains

Andreea L. Erciulescu, Tom Krenzke, Leyla Mohadjer, 

Robert Fay, Westat

Weijia Ren, Meta; Jianzhu Li, FINRA

Federal Committee on Statistical Methodology (FCSM)

Research and Policy Conference 2022

October 25, 2022

The views presented in this paper are those of the author and do not represent the views of any Government Agency/Department or Westat. 1
Contribution: Official Statistics

Model-based domain-level literacy and numeracy estimates of average scores and

at-level proportions

I Proficiency levels: At or Below Level 1 (0 - 225), At Level 2 (226 - 275), At or

Above Level 3 (276 - 500)

I Domains: county/state x age/education group, county, state

Source: U.S. Skills Map: State and County Indicators of Adult Literacy and Numeracy [National Center for Education Statistics at the

Institute of Education Sciences]

2
Contribution: Resources

3
Data

I Sparse survey sample at fine levels: 2012/2014/2017 U.S. Program for the

International Assessment of Adult Competencies (PIAAC)

County-level sample summaries for age and education groups

Age/Education Number of counties* Number of respondents

group with sample data Minimum Median Maximum

16-24 172 1 10 82

25-34 177 1 11 96

35-44 180 1 9.5 56

45-54 178 1 10 72

55-64 180 1 10 45

65-74 176 1 6.5 31

Less than high school 175 1 8 67

High school diploma or GED 180 1 16 86

Some college 182 1 16 114

Bachelor’s degree or higher 179 1 13 115

*There were 185 counties with sample data.

I Auxiliary information: American Community Survey (ACS)

4
Overview of the Modeling Process

I P1 is the proportion At or Below Level 1

I P2 is the proportion At Level 2

I P3 is the proportion At or Above Level 3

I MEAN is the average score

5
Model-based Allocation

Y

(b)

g,jk := Y

(b)

g,j

Y

(b)

jk

Y

(b)

j

I g = 1, ..., 10 is an index for the age/education groups

I j = 1, ..., 51 is an index for the states

I k = 1, ..., 3, 142 is an index for the counties

I b = 1, ..., 4, 500 is an index for the posterior samples

I Y is either P1, P3, or MEAN

I Y

(b)

j

, Y

(b)

jk , Y

(b)

g,j

, Y

(b)

g,jk are the posterior samples for the county, state,

state-by-group, and county-by-group quantities, respectively

6
Selected Results: Shrinkage

Change in county-level estimates for literacy proportions of less than high school population

7
Selected Results: Comparability

County-level estimates for literacy proportions of less than high school population At or Below

Level 1 versus ACS variables

8
Selected Results: Uncertainty

Credible interval (CI) widths and coefficients of variation (CV) for county-level model-based

estimates for At or Below Level 1 literacy proportion of less than high school population

Set of Domains / Metric Percentile of Empirical Distribution

20 40 50 60 80

All domains

95% CI width (%) 21.1 23.4 24.5 25.9 29.4

CV (%) 10.7 12.1 12.9 14.0 17.7

Domains with sample data

95% CI width (%) 19.6 21.5 22.7 24.0 27.9

CV (%) 10.3 11.6 12.3 12.8 16.3

Domains without sample data

95% CI width (%) 21.2 23.5 24.7 25.9 29.5

CV (%) 10.7 12.1 13.0 14.1 17.7

9
Summary

I Model-based estimation at fine levels of aggregation

I county by group, in addition to previous developments at state, state by group, and

county

I implausible results carried over from county and state estimation

I Consistency across levels of aggregation

I county by group as allocation of state by group according to county-to-state ratio

I no benchmarking

I Precision

I county by group coefficients of variation mostly below 20%

I Official statistics

I Skills Map

Remark. Modeling at the unit level or at the lowest level of aggregation (e.g., county by group) would

provide unified frameworks for constructing estimates at all levels of aggregation (e.g., county by group

aggregation

−−−−−−−→ county

aggregation

−−−−−−−→ state, and county by group aggregation−−−−−−−→ state by group). However, such

approaches would require reliable aggregation weights, covariates, or survey direct estimates, which

may not be available in practice.

10
Selected references

Krenzke, T., Mohadjer, L., Li, J., Erciulescu, A., Fay, R., Ren, W., Van de Kerckhove, W., Li, L., and

Rao, J.N.K. (2020). “Program for the International Assessment of Adult Competencies (PIAAC):

State and County Estimation Methodology Report (NCES 2020-225).” U.S. Department of

Education, National Center for Education Statistics. Washington, DC: U.S. Government Printing

Office. Retrieved [10-26-2020] from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2020225.

Li, J., Krenzke, T., Ren, W., Mohadjer, L., Fay, R., and Erciulescu, A. (2022). “Program for the

International Assessment of Adult Competencies (PIAAC): State-level Estimation for Age and

Education Groups Methodology Report (NCES 2022-050).” U.S. Department of Education, National

Center for Education Statistics. Washington, DC: U.S. Government Printing Office. Retrieved

[9-2-2022] from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2022050.

Erciulescu, A., Ren, W., Li, J., Krenzke, T., Mohadjer, L., and Fay, R.(2022). “Program for the

International Assessment of Adult Competencies (PIAAC): County-Level Estimation for Age and

Education Groups Methodology Report (NCES 2023-004).” U.S. Department of Education, National

Center for Education Statistics. Washington, DC: U.S. Government Publishing Office. Retrieved

[10-3-2022] from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2023004.

11
Thank you!

AndreeaErciulescu@westat.com

12
Auxiliary Data Selected for County-level Models

Percentage of population ...

I age 25 and over with less than high school education

I age 25 and over with more than high school education

I below 100 percent of the poverty line

I Black or African American

I Hispanic

I civilian noninstitutionalized who has no health insurance coverage

I age 16 and over with service occupations

12
Auxiliary Data Selected for State-by-group-level Models: Education Groups

Percentage of population ...

I civilian noninstitutionalized who has no health insurance coverage

I below 100 percent of the poverty line

I age 16-24

12
Auxiliary Data Selected for State-by-group-level Models: Age Groups

Percentage of population ...

I with less than high school education

I with more than high school education

I below 100 percent of the poverty line

I age 16 and over with service occupations

12