Utilizing Occupational Employment and Wage Statistics (OEWS) to Improve Occupational Requirements Survey (ORS) Small Domain Estimation (SDE)

Xingyou Zhang

2021 FCSM Research & Policy Conference

Washington, DC (Virtual Conference)

November 3, 2021

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Co-authors

Statistical Methods Group (SMG)

Office of Compensation and Working Conditions (OCWC)

Bureau of Labor Statistics (BLS)

Erin McNulty

Ellen Galantucci

Patrick Kim

Joan Coleman

Tom Kelly  

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Here is an overview of what we will be covering in today’s presentation. 

First, we will talk a little bit about the changes in the ORS sample design for the second wave and what is driving the need for new weighting methodology. 

Then we will cover a little bit on all of the different weighting components including:

Final Establishment Weights

Final Occupational Weights and some of the weights and factors that are new.

Sample Group Factor

Benchmark Final Weights

And last we will provide some future research and recommendations for ORS Weighting.

2

Overview

Background (ORS and OEWS)

ORS small domain estimation objective

Small Domain Estimation with ORS

Modeling Framework

Basic steps

Preliminary Results

What is next?  

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Here is an overview of what we will be covering in today’s presentation. 

First, we will talk a little bit about the changes in the ORS sample design for the second wave and what is driving the need for new weighting methodology. 

Then we will cover a little bit on all of the different weighting components including:

Final Establishment Weights

Final Occupational Weights and some of the weights and factors that are new.

Sample Group Factor

Benchmark Final Weights

And last we will provide some future research and recommendations for ORS Weighting.

3

Occupational Requirements Survey (ORS)

A survey conducted by BLS on behalf of the Social Security Administration (SSA) and collects data to measure the requirements of work in the national economy in four areas: 

1) Physical demands; 2) Environmental conditions; 3) Education, training, and experience; 4)Mental and cognitive demands

Sample design

A two-stage stratified sample of establishments and occupations within selected establishments

Annual sample size of 10,000  

8,500 (85%) private industry establishments

1,500 (15%) State and Local Government establishments

ORS721&722 data collected from 2018-2020

‹#› — U.S. Bureau of Labor Statistics • bls.gov

ORS Small Domain Estimation Objective

Objective: producing reliable estimates at 6-digit 2018 Standard Occupational Classification (SOC) level for 844 target occupations 

Only 310 out of 844 SOCs met minimum ORS estimation criteria 

Model-based Small Domain(Area) Estimation

A small domain (area) is defined as any domain if the domain-specific sample is not large enough to support direct survey estimates of adequate precision

Small domain estimation is to making estimates for small domains with adequate precision via borrowing information statistically from the survey and other relevant data sources

Borrow information from OEWS 

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Occupational Employment and Wage Statistics (OEWS)

A semiannual survey designed to produce estimates of employment and wages for specific occupations, with an annual sample size of nearly 400,000. 

OEWS Nov 2017-May 2020 had sampled about 1 million establishments 

OEWS and ORS occupation sample size distribution

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Multilevel Regression and Poststratification for Small Domain Estimation

ORS Survey Data

Target outcome: 

Auxiliary variables:  

Cluster information: 

Sample Frame (OEWS)

Predicted outcome:

Auxiliary variables:  

Cluster information:

Multilevel Regression Model

Fitted Multilevel Regression Model

Model Specification

Model Fitting

Model Prediction with OEWS

Poststratification

aggregate outcome: 

By 6-digit SOC domains

Small domain estimates

at levels of  6-digit SOC

Validation by reliable direct survey estimates

Estimate SEs via Bootstrapping

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Step 1: Obtain the ORS direct survey estimates by SOC  

When n is too small, we could not use ORS data to obtain reliable estimates () directly and  could also be very biased. 

Binary outcome: personal protective equipment (PPE) use (Yes vs No)

For example, one occupation of SSA interest had only two quotes in ORS721&722  (n=2) while OEWS had a sample of 2,880 6-digit SOCs (UNK: Unknown)

(UNK = Unknown)

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Step 2a: Use ORS data to construct and fit a multilevel model   

This step is to use multilevel model to borrow information across entire ORS dataset to obtain the model parameter estimates  (total 45,199 job quotes)

: the known ORS outcome of interest, such as PPE use. 

: the known fixed effects and β: their unknown regression coefficients 

establishment ownership; 

employment size

ORS sampling geographic areas (9 census divisions plus 15 CSAs/MSAs)

industry (2-digit NAICS)  

: the known random effects and : their unknown regression coefficients 

detailed occupation groups (6-digit SOCs)  

The model fitting process is to estimate what are the most likely values of model parameters ( and ), given the known data from ORS (y, x, and z). 

This is the statistical learning process from ORS data

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Step 2b: Apply the fitted multilevel model    to OEWS data to obtain the predicated values for the outcome of interest ()

Model prediction space is the set of small domains based on all possible combinations of model predictors 

24 ORS sampling geographic areas, 4 employment size groups, 20 industrial groups, 3 ownership groups, plus 844 6-digit SOC occupation groups, 4,861,440 small domains in total 

Although OEWS does not have data on the outcome of interest (y), we could conveniently have a predicted value for the outcome () for each OEWS quote (total 9,827,696 job quotes), since ,  are known in OEWS,  and  are known after model fitting

The modeling prediction process is to estimate what are the most likely predicted values of outcome of interest (), given the known data (X, Z) from OEWS and the known model parameters ( and ). 

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Step 3: Use the predicted () in OEWS and we could calculate the populated estimates of interest:  

 is the OEWS job quote final weight 

 is the predicted value for PPE use for one OEWS job quote

 is model-based estimate for small domains of interest

For example, OEWS has a sample of 2,880 6-digital SOCs for one occupation of SSA interest for summary with predicted PPE use

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Step 4: obtain the variance estimates associated with small domain estimates that account for both model-based uncertainties and OEWS sampling uncertainties via bootstrapping 

 , where  =1, 2, ..., 1,000 

With a sample of 1,000 , we could conveniently to produce summary statistics for 

Mean, median or any other percentiles, 90% or 95% Confidence intervals (CIs)

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Step 5: Compare reliable estimates from both ORS and OEWS to check our model validity

In general OEWS sample size for an occupation is much larger than its corresponding ORS sample size ()

When n is large enough for a reliable estimate based on ORS, we should expect both estimates (ORS direct survey estimate,  and OEWS model-based estimate,  ) are equivalent if our multilevel model is valid or appropriately specified. 

Comparison of  and  

National level  

Occupation level  

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Preliminary Results

Outcome: Personal protective equipment use (Yes vs No)

National Level Comparison ORS direct survey estimate and model-based estimate of PPE use

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Comparison ORS direct survey estimate and model-based estimate of PPE use

6-digit SOC Occupation Level 

Occupations that passed minimum number of observations based on ORS publication criteria

‹#› — U.S. Bureau of Labor Statistics • bls.gov

What is next? 

Cross-validation 

Drop all quotes for one occupation with a large sample size from model fitting process, then compare its model-based estimate with its direct survey estimate  

116 SOCs with at least 100 quotes from ORS  

Test model predictive power for those occupations without ORS samples at all 

Reduce sample sizes for one occupation with a large sample size for model fitting, then compare its model-based estimate with its direct survey estimate  

116 SOCs with at least 100 quotes from ORS  

Test model predictive power for those occupations with small ORS sample sizes

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Acknowledgement

Office of Employment and Unemployment Statistics (OEUS)

Julie Hatch

Laurie Salmon, David Byun and  Andrea Wagoner.

Occupational Employment and Wage Statistics (OEWS) program team 

Office of Compensation and Working Conditions (OCWC)

OCLT: DCDAP&DCDE 

SMG ORS Team

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Here is an overview of what we will be covering in today’s presentation. 

First, we will talk a little bit about the changes in the ORS sample design for the second wave and what is driving the need for new weighting methodology. 

Then we will cover a little bit on all of the different weighting components including:

Final Establishment Weights

Final Occupational Weights and some of the weights and factors that are new.

Sample Group Factor

Benchmark Final Weights

And last we will provide some future research and recommendations for ORS Weighting.

17

Xingyou Zhang

 Division Chief, Statistical Methods Group

Office of Compensation and Working Conditions (OCWC)

www.bls.gov/ors

202-691-6082

Zhang.Xingyou@bls.gov

Contact Information

‹#› — U.S. Bureau of Labor Statistics • bls.gov

# image6.png

# image7.png

# image8.png

# image9.png

# image10.png

# image11.png

# image12.png

# image13.png

# image14.png

# image15.png

# image16.png

# image17.png

# image18.png

# image19.png

# image20.png

# image21.png

# image22.png

# image23.png

# image24.png

# image25.png

# image1.jpeg

# image2.png

# image3.png

# image4.png

# image5.png