Occupational Requirements Survey 

Sample Design Evaluation 

Gwyn R. Ferguson1, Erin McNulty1, and Chester Ponikowski1

1Bureau of Labor Statistics, 2 Massachusetts Ave. NE, Washington, DC 20212 

Abstract 

The Bureau of Labor Statistics (BLS) is working with the Social Security Administration 

(SSA) to carry out a series of tests to determine the feasibility of using the National 

Compensation Survey (NCS) platform to accurately and reliably capture data that are 

relevant to the SSA's disability program. The proposed new Occupational Requirements 

Survey (ORS) is envisioned to be an establishment survey that collects information on the 

vocational preparation and the cognitive and physical requirements of occupations in the 

U.S. economy as well as the environmental conditions in which those occupations are 

performed. This paper builds on the evaluation that was presented at the 2013 Federal 

Committee on Statistical Methodology Research Conference that described the advantages 

and disadvantages of integrating the sample design of ORS with the sample design of NCS. 

This paper presents an analysis of the sample design options that were considered for this 

proposed new survey and describes the sample design being used in the pre-production test 

for ORS. It also describes the design issues that need to be resolved before full survey 

implementation. 

Key Words: survey design, occupational data, integrated surveys, survey cost 

1. Introduction 

In the summer of 2012, the Social Security Administration (SSA) and the Bureau of Labor 

Statistics (BLS) signed an interagency agreement, which has been updated annually, to 

begin the process of testing the collection of data on occupations. As a result, the 

Occupational Requirements Survey (ORS) was established as a test survey in late 2012. 

The goal of ORS is to collect and publish occupational information that will replace the 

outdated data currently used by SSA. More information on the background of ORS can be 

found in the next section. All ORS products will be made public for use by non-profits, 

employment agencies, state or federal agencies, the disability community, and other 

stakeholders. 

An ORS interviewer attempts to collect close to 70 data elements related to the 

occupational requirements of a job. The following four groups of information will be 

collected: 

 Physical demand characteristics/factors of occupations (e.g. strength, hearing, or 

stooping) 

 Educational requirements 

 Cognitive elements required to perform work 

 Environmental conditions in which the work is completed 
Field testing to date has focused on developing procedures, protocols, and collection aids 

using the NCS platform. These testing phases were analyzed primarily using qualitative 

techniques but have shown that this survey is operationally feasible. Now it is time to turn 

our attention to the survey design to ensure that we have the best possible sample design 

to meet the needs of the ORS. This paper presents an analysis of the sample design options 

that were considered for this proposed new survey and describes the sample design being 

used in the pre-production test for the ORS. It also describes the design issues that need to 

be resolved before full survey implementation. 

2. Background Information on ORS 

In addition to providing Social Security benefits to retirees and survivors, the Social 

Security Administration (SSA) administers two large disability programs, which provide 

benefit payments to millions of beneficiaries each year. Determinations for adult disability 

applicants are based on a five-step process that evaluates the capabilities of the worker, the 

requirements of their past work, and their ability to perform other work in the U.S. 

economy. In some cases, if an applicant is denied disability benefits, SSA policy requires 

adjudicators to document the decision by citing examples of jobs the claimant can still 

perform despite restrictions (such as limited ability to balance, stand, or carry objects) [1]. 

For over 50 years, the Social Security Administration has turned to the Department of 

Labor's Dictionary of Occupational Titles (DOT) [2] as its primary source of occupational 

information to process the disability claims. SSA has incorporated many DOT conventions 

into their disability regulations. However, the DOT was last updated in its entirety in the 

late 1970’s, and a partial update was completed in 1991. Consequently, the SSA 

adjudicators who make the disability decisions must continue to refer to an increasingly 

outdated resource because it remains the most compatible with their statutory mandate and 

is the best source of data at this time. 

When an applicant is denied SSA benefits, SSA must sometimes document the decision by 

citing examples of jobs that the claimant can still perform, despite their functional 

limitations. However, since the DOT has not been updated for so long, there are some jobs 

in the American economy that are not even represented in the DOT, and other jobs, in fact 

many often-cited jobs, no longer exist in large numbers in the American economy. For 

example, a job that is often cited is “envelope addressor,” because it is an example of a 

low-skilled job from the DOT with very low physical demands. There are serious doubts 

about whether or not this job still exists in the economy. 

SSA has investigated numerous alternative data sources for the DOT such as adapting the 

Employment and Training Administration’s Occupational Information Network (O*NET) 

[3], using the BLS Occupational Employment Statistics program (OES) [4], and 

developing their own survey. But they were not successful with any of those potential data 

sources and turned to the National Compensation Survey program at the Bureau of Labor 

Statistics. 

3. Overview of Potential Survey Design Options 

To develop sample design options for the ORS, BLS reviewed the sample designs for the 

NCS [5] and other BLS surveys, including the OES and Survey of Occupational Injuries 
and Illnesses [6]. Since the ORS will be collected by trained Field Economists who also 

collect the NCS data, potential coordination with the NCS sample design was a key factor 

of consideration. So we identified four basic categories of ORS survey designs allowing 

for different potential levels of coordination with NCS: 

1. Fully Integrated Survey Design – where the NCS establishment sample would be 

a subsample of the ORS establishment 

2. Independent Survey Design – where the ORS establishment samples would be 

selected using a design appropriate for SSA’s needs, the NCS establishment 

samples would be selected using the current NCS sample design, and there would 

be no control on the amount of establishment sample overlap between the samples 

selected for the two surveys 

3. Separated Survey Design – where the NCS establishment sample would be 

selected from the frame, the selected NCS establishments would be removed from 

the frame, and an independent ORS establishment sample would be selected from 

the rest of the frame 

4. OES-ORS Integrated Design – where the ORS establishment sample would be 

selected as a subsample of OES establishment sample 

The next four sections of this paper will describe each of these options in more detail. For 

each option, we will describe several pros (reasons to implement the design) as well as 

some potential drawbacks from implementation. For the first two design options, we will 

also describe the results of some empirical evaluations conducted using sample 

simulations. Although the sample size for ORS has not been finalized yet, we assumed that 

the ORS sample would need to collect data from 10,000 establishments annually for all 

sample simulations and evaluation work. Section 8 of the paper describes the sample design 

being used for the pre-production test of ORS that will start in the fall of 2014. Section 9 

of the paper wraps up with some conclusions and recommendations for future work. 

4. Fully Integrated Survey Design Options

Under this design, we would select an ORS establishment sample each year and then select 

the NCS establishment sample as a subset of the bigger ORS sample. This approach was 

used by the NCS program when a locality wage sample was a part of the integrated NCS 

sample design. We would collect the ORS data from all establishments in the NCS 

initiation sample. If a current NCS update establishment is selected for the ORS portion of 

the sample, we would need to collect the ORS data elements for the existing NCS jobs. 

For simulation purposes, we assumed an annual ORS sample size of 10,000 establishments. 

Since the current NCS sample size includes approximately 3,200 establishments to be 

initiated each year, we assume that we would collect the ORS data elements (and not the 

NCS data elements) from the 6,800 ORS establishments not included in the NCS 

subsample. 

This design option poses some challenges that will need to be overcome in order to 

implement it. The primary challenges are listed below. 

• How should we select the samples to meet the needs of both sets of desired outputs 

– NCS and ORS? 

• NCS samples State & Local government establishments once each decade. Will 

this approach work for ORS? If not, how will we handle this segment of the 

economy for ORS? 
• Will the same sample rotation strategy work for both? 

Some of the reasons we would want to use a fully integrated survey design option include 

the following: 

• Ability to generate linked outputs such as wage percentiles by portion of the day 

lifting/carrying specified weights; 

• Eliminates individual establishment burden associated with having to provide data 

separately for more than one survey program; 

• Lower data collection costs; 

• May be able to use the existing NCS sampling, nonresponse adjustment, and 

benchmarking systems; and 

• NCS has experience doing this. 

Some of the reasons we may not want to use this approach include: 

• Increased interview length for establishments in both surveys is likely to have 

negative impact on establishment response rates for NCS and ORS; 

• Respondent fatigue may result in lesser quality data; and 

• Potential increase in item nonresponse rates. 

For these purposes, testing a design meant that we took a recent sample frame and selected 

at least 150 samples from that frame. We then computed some basic information from each 

of those samples and averaged the data across all the samples. In general, we were looking 

for a sample design that allowed us to select an ORS sample to meet the ORS needs – that 

is a sample that has enough establishments in each industry but not too many and resulted

in the total frame employment when we calculated the sum of the weighted employment 

across all establishments. For NCS samples, we wanted to be able to select the NCS desired 

sample sizes for each industry and geographic area AND have a total weighted employment 

that matches the frame employment. For each option considered, we used the NCS sample 

design and establishment selection methods with only the changes described below. 

4.1 Fully Integrated – All NCS Design 

Under this approach, we used the NCS design and allocation to pick both the ORS sample 

and the NCS subsample. This meant that we selected the ORS sample the same way that 

we pick the NCS sample – just with a larger sample size. So when we oversampled an 

industry in NCS due to wide dispersion of wages and/or lower response, we used the same 

target percentage of sample for the ORS allocation. 

As expected, simulations showed that this design worked relatively well for NCS in that 

the industry distribution was very close to the targeted distribution, the full sample size was 

achieved, and the weighted employment was close to the frame total. However, the ORS 

sample was too big in industries that may not need a particularly large ORS sample, such 

as the Finance, Insurance, and Colleges & Universities industries. Given the unweighted 

results from the feasibility tests conducted in 2013, described in the Phase 1 [7], Phase 2 

[8], and Phase 3 [9] Summary reports, we feel that the NCS industry distribution may not 

be ideal for ORS as it provides too much data in some industries where the occupational 

requirements data do not show a wide dispersion. 
The following table highlights some industries where the ORS sample size was at least 

two times larger under NCS Design than under a proportional to employment design: 

Industry

Average ORS Sample Size 

Using... Percent 

NCS Difference 

Design

Proportional to 

Employment 

Design

Insurance 1,505 506 198%

Finance (Rest of) 2,489 845 194%

Junior Colleges, Colleges & 

Universities 704 259 172%

Utilities 307 136 126%

4.2 Fully Integrated Design with Different ORS and NCS Allocations 

With the current NCS sample design, we explicitly ensure that we select the desired number 

of establishments for each aggregate industry. We distribute the total number to be selected

in each industry to the geographic cells in proportion to the total employment in each area 

of the country for that aggregate industry. Then we implicitly stratify and adjust the 

measures of size to select the establishments in a manner that ensures that we select the 

target number of establishments in each of the detailed industries across the country as a 

whole. 

Under this approach, we allocated the ORS sample size to each industry and geographic 

area stratum in proportion to the stratum employment. This allowed us to set allocations 

for ORS directly in proportion to total employment, reducing the number of establishments 

we sample in some industries compared to the all NCS design. We were satisfied with the 

ORS sample sizes and employment totals that resulted from this approach. The average 

weighted employment totals were very close to the frame employment. 

The following table provides simulation data for the industries with the largest percentage 

difference between the frame employment and the average ORS sample weighted 

employment: 

Detailed Industry

Average Employment Percent 

Frame ORS Sample Difference

Elementary and Secondary Schools 661,077 657,960 -0.5%

Junior Colleges, Colleges & 

Universities 1,164,474 1,167,206 0.2%

Mining 617,765 616,707 -0.2%

Insurance 2,076,306 2,072,942 -0.2%

Other Services (except Public 

Admin.) 3,741,681 3,737,766 -0.1%

However, we were unable to achieve acceptable NCS sample sizes and employment counts 

using this approach. We tried this five different times with slight variations on the process 

each time in an effort to achieve better results. But the results were pretty much the same. 

For all simulations in which ORS was selected with an allocation strictly proportional to 

employment, there are ‘shortages’ of frame units in certain cells targeted by our measure 
of size adjustments. These cells are smaller than the sample cells, so the ORS sample design 

does not ensure enough units in each geographic area. Therefore, the measure of size 

(MOS) adjustments do not help us attain our target sample sizes in all detailed industries. 

The following table highlights the industries where over 10% of NCS sample size was 

lost using MOS adjustment factor approach to subsampling NCS: 

Industry

Average NCS Sample 

Size

Percent 

Difference

Independent 

Design

Subsample 

of ORS

Real Estate, Renting, Leasing 215 162 -24.7%

Mining 86 66 -23.8%

Hospitals 251 210 -16.3%

Total 9,804 9,792 -0.1%

Note that the sampling method allowed much of the “loss” in these industries to be 

absorbed by other related industries (for example, Construction and Manufacturing 

absorbed some of the losses from the Mining industry). However, not all losses could be 

completely accounted for, so the full sample was short of the target size. 

4.3 Fully Integrated Design – Inside Out Option 

For the Inside Out Design, we turned the current NCS sampling cells inside out while 

ensuring that we still selected the target number of establishments in each aggregate 

industry and in each detailed industry. Under this design, we created 24 sampling cells, one 

for each detailed industry including aircraft manufacturing and implicitly stratified within 

each sampling cell for 24 geographic areas. The ORS sample was allocated to each industry 

in proportion to industry employment, and the NCS sample allocations were set to the 

current NCS sample sizes in each industry. 

We were again satisfied with the sample sizes and employment counts of the ORS sample. 

The sample distribution was not substantially different from that under the previous 

approach (see section 4.2). 

The NCS sample was also acceptable under this approach. For private industry, the overall 

NCS detailed industry counts met the NCS targets. NCS counts by the 24 geographic areas 

were close to the area counts using the current NCS sample design. The largest difference 

between the two designs was an increase of 3.5% in Rest of West North Central Census 

Region. Fourteen of the 24 NCS geographic areas had sample sizes that differed by less 

than 1% compared to the current design. We believe that these differences were acceptable. 

But we will need to conduct much further analysis to evaluate regional workloads, 

anticipated response rates, impact on publications, impact on the number of quotes 

available for estimation, etc. before making a choice to implement this design. 

The table below shows the industries with largest percentage difference between the 

sample sizes for the current NCS design and the sample sizes for the design where NCS is 

a subsample of ORS using the Inside Out design: 
Detailed Industry

Average NCS Sample Size

Percent 

Difference

Current NCS

Design

Subsample of 

ORS

Educational Services (Rest of) 58 57 -1.7%

Management of Companies and 

Enterprises

73

74 1.3%

Elementary and Secondary 

Schools

91

92 0.7%

Arts, Entertainment, Recreation 104 103 -0.7%

Mining 86 87 0.6%

This table identifies the geographic areas with the largest percentage difference between 

the sample sizes for the current NCS design and the sample sizes for the design where NCS 

is a subsample of ORS using the Inside Out design: 

Area

Average NCS Sample Size

Percent 

Difference

Current 

NCS Design

Subsample 

of ORS

West North Central Census Division 597 617 3.5%

San Jose-San Francisco-Oakland, CA 

CSA 253 246 -3.0%

Detroit-Warren-Flint, MI CSA 154 150 -2.7%

New England Census Division 224 229 2.3%

5. Independent Survey Design Options 

In this design, we would select a sample of ORS establishments from the sample frame 

using a sample design that works well for ORS. Each year, we would also select an 

independent sample of NCS establishments from the same frame, using the current NCS 

sample design. No special procedures would be implemented to control, reduce, or 

maximize overlap between the two independent establishment samples. Under this survey 

design, BLS will have two separate collection options, one in which ORS data elements 

are collected for all NCS and ORS sampled establishments (i.e. Joint Collection) and one 

in which ORS data elements are collected only for establishments selected in the ORS 

sample (i.e. Independent Collection). With Independent Collection, some NCS 

establishments will be asked to provide ORS data elements if they are also sampled in the 

ORS sample but the rest of the NCS establishments would not be asked to provide ORS 

data elements. 

Under the independent design, the survey scope will match the NCS survey scope which 

encompasses all businesses in the 50 States and the District of Columbia that are Private, 

State Governments, or Local Governments. It will exclude private households and 

agriculture industry. The ORS sample will be selected from the same frame as the NCS – 

primarily the BLS Quarterly Census of Employment and Wages Database which is 

compiled based on Unemployment Insurance filings by businesses across the country. 

Unlike NCS which creates sampling strata based on 5 aggregate industries and 24 

geographic areas, we will stratify by 24 major industry groups. ORS Sample allocation will 

be proportional to employment in each of the major industries with no adjustments for 
response rates or level of accuracy (since these are unknown at this time). If we are using 

the Joint Collection model, the ORS-only allocations will be reduced by the number of 

establishments selected in the NCS sample for each of the 24 industries before we select 

the ORS independent sample. The decision about whether or not to collect ORS data from 

NCS respondents has not yet been made and will be evaluated using the results from a 

feasibility test on joint collection conducted in FY 2014 and from the pre-production test 

results (see Section 8). 

For ORS, we will select 24 independent establishment samples each year, one in each of 

the 24 industry groups. We will implicitly stratify the sample by geographic area to ensure 

a fair distribution of the sample across the country. Each sample will be selected using a 

systematic PPS technique without replacement. For every establishment in the sample, a 

sample of jobs will be selected with input from the respondent during initiation. This 

process is also a PPS selection. When a job is selected, we collect data for all workers in 

the job. The measure of size for each of the two stages of sampling is the employment in 

the establishment or job. 

We know that there will be some overlap of establishments from sample to sample and 

from the NCS to ORS samples under this approach and that the amount of overlap is 

important. So we selected 150 NCS simulated samples, 150 ORS simulated samples 

assuming that ORS data will be collected from the NCS sample (joint collection with an 

ORS sample size of 6,800 establishments a year), and 150 ORS simulated samples 

assuming that ORS data will NOT be collected for every establishment in the NCS sample 

(independent collection with an ORS sample size of 10,000 establishments a year). Both 

sets of samples were split into three groups (Year 1, Year 2, and Year 3) assuming a three 

year rotation. 

In our research of the option under which ORS data are collected from all NCS 

establishments, the amount of overlap between the ORS-only portion of the sample and the 

NCS portion of the sample was minimal. Less than 5% of each NCS sample overlapped 

with ORS at some point during the three-year sample design. 

Amount of overlap in NCS sample between NCS and ORS-only establishments 

Type of NCS-to-ORS 

Overlap

Year of NCS Sample Design

Year 1 Year 2 Year 3

NCS unit is ORS-only unit in 

ANY year of three-year design 4.24% 4.20% 4.20%

NCS unit is ORS-only unit in 

SAME year of three-year 

design 2.24% 2.11% 2.13%

Since the ORS-only portion of the sample is larger than the ORS/NCS portion, there was 

less overlap when looked at from the ORS perspective. 

The amount of overlap between ORS-only establishments and NCS establishments was a 

bit higher under the option where ORS data are not intended to be collected from any NCS 

establishments. The higher amount of overlap for this option was expected, because the 

size of the ORS-only sample was larger at 10,000 establishments a year. The amount of 

overlap is about twice as large for the NCS sample, and about a tenth of each NCS sample 

overlaps with ORS at some point during the three-year sample design. 
Amount of overlap in NCS sample between NCS and ORS-only establishments 

Type of NCS-to-ORS 

Overlap

Year of NCS Sample Design

Year 1 Year 2 Year 3

NCS unit is ORS-only unit in 

ANY year of three-year 

design 10.27% 10.11% 9.95%

NCS unit is ORS-only unit in 

SAME year of three-year 

design 6.51% 6.43% 6.35%

Some of the reasons for using an independent survey design option include: 

 More flexibility to make changes in the future than any variation of the fully 

integrated design option 

 This option gives us the ability change either (NCS or ORS) design without 

changing the other design. 

 This option allows us to make changes to either design at the best point in time 

for that design without forcing us to change both designs at the same time. 

 This option allows us to move from Joint Collection to Independent Collection 

for NCS establishments without changing the sample design, if necessary. 

 In the unlikely event that BLS funding for ORS should go away, there would 

be no impact on the NCS sample design. 

 This option allows us to proceed with an optimal design for ORS to best meet the 

needs of the SSA. 

 This option does not have any impact on the NCS sample design since we can have 

a different sample design for ORS than for NCS. 

 This option allows us to have different rotation patterns for NCS and ORS such as 

3 years for NCS and 3, 5, or 10 years for ORS, if desired. 

 The joint collection option provides us with the ability to generate linked outputs 

such as wages by amount of time sitting each day, if desired. 

 The joint collection option results in lower data collection costs than an 

independent data collection option. 

Some of the challenges to using an independent survey design option include: 

 If we implement the independent collection option, we would not have the ability 

to generate linked outputs. 

 Increase in resource demand during sample selection and systems development 

and maintenance. 

 Increased data collection costs for independent collection option when compared 

to the joint collection option due to the larger ORS sample size. 

 Increase in resource demand during data review and analysis if we are not able to 

collect ORS data from all NCS respondents. 

 Some added complexity to the computation of final ORS weights for estimate 

calculations due to the different sample designs for joint collection option 

 Some added complexity to the ORS variance computation methods due to the 

different sample designs for joint collection option 

Based on the evaluation of this design to date, it appears that the independent design for 

sampling establishments will be the best choice for implementation. However, we would 
still like to evaluate potential response rates for this design, develop expected levels for 

published outputs, and further analyze the most appropriate way to handle large enterprise 

firms with many establishments in the sample frame. This includes evaluating response 

rates for all ORS samples and for establishments in the portion of the sample for which 

both ORS and NCS data elements would be collected. In addition, we need to evaluate the 

potential for nonresponse bias for these samples, especially if establishment response rates 

fall below 80% during the pre-production test described in Section 8. Research is 

proceeding in these areas and will be completed before final long-term production 

decisions are made.

6. Separated Survey Design Options 

Under this design, we would pull the NCS sample from the frame first. Then we would 

remove those units from the ORS frame and draw a sample of ORS units from what is left 

over. We would collect both NCS and ORS data from the establishments in the NCS 

sample. We would collect only the ORS data from the establishments in the Rest of ORS 

sample. So this only works with joint collection for NCS establishments. 

Potential issues with this design: 

 Establishments are sampled independently so one establishment within a large 

enterprise in which all collection occurs centrally (usually at corporate 

headquarters) could appear in the NCS part of the sample and other establishments 

in the same enterprise could show up in the rest of ORS sample. Is there a way to 

sample these large enterprises as a group (for NCS and ORS) to reduce this issue? 

 This approach also requires a more complex weighting scheme for ORS estimation 

than either the Fully Integrated or Independent Sample Design options, but we 

believe that it is doable. 

Due to a lack of resources, this option has not been fully evaluated for use in a production 

environment using sample simulations. While it would eliminate the overlap between NCS 

and ORS selected samples each year while providing flexibility in the design for the Rest 

of ORS portion of the sample, there are several challenges with this option. First, it is quite 

likely that establishments in a Rest of ORS sample could already be reporting data for a 

prior NCS sample so the option does not eliminate all sample overlap. Also, the Rest of 

ORS sample would not reflect a full sampling frame so the weighting process for 

computing estimates would be more complex than under the previous design options. 

Additionally, it is very likely that large enterprises would have establishments in both 

portions (NCS and Rest of ORS) of the sample making is more difficult to collect data for 

these large firms. At this time, the benefits of controlling initiation overlap are outweighed 

by the other issues with the design so it will not be considered further for selection of 

production samples. 

7. OES-ORS Integrated Design 

The Bureau of Labor Statistics conducts another survey which collects occupational 

information from establishments in the United States, the Occupational Employment 

Statistics survey. This is a mail survey which is used to generate mean annual wages and 

employment for detailed occupations in the U.S. economy. Two samples are selected and 

collected for this survey each year with data from the most recent six samples (three years) 

used to compute annual estimates. The samples are drawn in a manner that ensures that 
establishments appear in one and only one of the six samples used in each set of estimates. 

Data on the occupations employed by each business establishment, the number of 

employees, and the wages for those employees is collected via a mail survey. Since this 

survey collects data about the occupational mix in each establishment, it may be 

advantageous to use the collected data from the OES as the sample frame for ORS. This 

could allow ORS to target specific occupations needed for disability determination 

decisions as part of the sample design. However, there is some concern about the age of 

the data and the potential survey error and lack of efficiency that this could introduce. So, 

we have not yet begun an evaluation of this potential design but plan to do so at some point 

in the future. 

8. ORS Pre-Production Test Sample Design 

Beginning in the fall of 2014, BLS plans to conduct a nation-wide pre-production test to 

evaluate ORS survey processes and operations in a possible production environment at the 

request of the Social Security Administration (SSA). Data collection and capture will run 

for approximately six months and will conclude in the spring of 2015. A full evaluation of 

the data elements captured for this pre-production test will be followed by an evaluation of 

the processes, survey design, and other test program elements. In order to fully evaluate 

the potential for implementing the ORS in a production environment, BLS will use the 

Separated Sample Design approach to select the establishments included in the test as 

described below. All of ORS Pre-production’s projected 2,550 sample establishments will 

be collected once for all of the ORS data elements. 

The ORS Pre-production sample will include a combination of both ORS-only 

establishments as well as those that currently exist within the NCS (National Compensation 

Survey). All units will be selected using a 2-stage stratified design with probability 

proportional to employment sampling at each stage. The first stage of sample selection will 

be a probability sample of establishments, and the second stage of sample selection will be 

a probability sample of jobs within sampled establishments. For more information on the 

current NCS sample design as well as factors explored for an ORS sample design, see the 

American Statistical Association (ASA) papers by Ferguson et al titled, “Evaluating 

Sample Design Issues In the National Compensation Survey” [10], “Update on the 

Evaluation of Sample Design Issues in the National Compensation Survey” [11], and “State 

and Local Government Sample Design for the National Compensation Survey" [12] as 

well as the Federal Committee on Statistical Methodology (FCSM) paper by Rhein et al 

titled, “Sample Design Considerations for the Occupational Requirements Survey” [13]. 

Each sample of establishments will be drawn by first stratifying the establishment sampling 

frame by defined industry and ownership. The industry strata for private industry as well 

as state and local government are shown on the next page and are based on the North 

American Industry Classification System (NAICS). 

After the sample of establishments is drawn, jobs will be selected in each sampled 

establishment. The number of jobs selected in an establishment will range from 4 to 8 

depending on the total number of employees in the establishment, except for government 

and aircraft manufacturing units and units with less than 4 workers. In government, the 

number of jobs selected will range from 4 to 20. In aircraft manufacturing, the number of 

jobs selected will range from 4 for establishments with less than 50 workers to 32 for 

establishments with 10,000 or more workers. In establishments with less than 4 workers, 
the number of jobs selected will equal the number of workers. The probability of a job 

being selected will be proportionate to its employment within the establishment. 

The total ORS pre-production sample will consist of approximately 2,550 establishments. 

The private portion of this sample will be approximately 85% (2,168) with one-third (716) 

of that coming from current NCS sample units. The remaining two-thirds (1,452) of the 

private sample will be selected from a national frame not to include any other existing NCS 

sample units. This frame will be stratified by NAICS based on the 24 detailed industry cells 

as defined above. The state and local government sample will be approximately 15% of 

the total sample (382) with one-third (126) of the units selected from the existing NCS 

sample and the remaining two-thirds (256) selected from a national frame not to include 

existing NCS sampled establishments. 

The sample allocation process starts with a total budgeted sample size. Since some of the 

sample for the ORS Pre-production test will be selected from the NCS design, the same 

industry definitions (based on ownership and NAICS as defined in charts below) will be 

used to select both the NCS overlap sample as well as the ORS-only sample. Due to the 

differences in the selection of the original NCS samples, different sampling strata will be 

used for the samples selected from existing NCS samples than from the new ORS-only 

samples. The sample will be allocated proportionally by ownership and industry using total 

employment within each sample cell for ORS-only samples and total weighted 

employment for samples drawn from NCS samples. 

The ORS Pre-production test will select a sample consisting of both NCS sample units as 

well as ORS-only units. The portion selected from the existing NCS sample units will use 

systematic sampling with probability proportionate to measure of size. The measure of size 

(MOS) will be the sample unit employment times its NCS sample weight. 

For the ORS only sample, units will be selected from a frame that excludes all existing 

NCS sample units. This frame will be stratified by ownership and industry as defined 

above, with each sample cell being sorted by area (using the 24 area definitions in the NCS 

design – see below), establishment employment, and establishment identification number. 

These units will be selected using a probability proportional to size approach based on the 

unit’s employment as it was reported to on the state unemployment file. 

Sample weights will be assigned to each of the selected establishments in the sample to 

represent the entire frame. Units selected as certainty will be self-representing and will 

carry a sample weight of one. The sample weight for the non-certainty units will be the 

inverse of the probability of selection. 
ORS Pre-production Stratification for Private Industry 

Aggregate 

Industry Detailed Industry

Included 

NAICS 

Codes

# 

Companies

in Universe

Sample 

Size

Education Educational Services 

(Rest of)

61 (excl 6111-

6113) 78,008 11

Education Elementary and 

Secondary Schools

6111 16,899 14

Education Junior Colleges, 

Colleges and 

Universities

6112, 6113 8,023 15

Finance, Insurance 

and Real Estate Finance (Rest of) 52 (excl 524) 279,462 64

Finance, Insurance 

and Real Estate Insurance 524 185,133 39

Finance, Insurance 

and Real Estate

Real Estate, 

Renting, Leasing 53 349,578 43

Goods Producing Mining 21 34,579 14

Goods Producing Construction 23 744,370 122

Goods Producing Manufacturing 31-33 334,610 224

Health Care, incl.

Hospitals and 

Nursing Care

Healthcare, Social 

Assistance (Rest of)

62 (excl 622, 

623) 1,230,175 194

Health Care, incl.

Hospitals and 

Nursing Care

Hospitals 622 8,419 68

Health Care, incl.

Hospitals and 

Nursing Care

Nursing and 

Residential Care 

Facilities

623 72,659 66

Service Providing Utilities 22 17,130 10

Service Providing Wholesale Trade 42 619,782 121

Service Providing Retail Trade 44-45 1,031,277 308

Service Providing Transportation and 

Warehousing 48-49 225,026 75

Service Providing Information 51 143,541 47

Service Providing Professional, 

Scientific, Technical 54 1,075,999 177

Service Providing

Management of 

Companies and 

Enterprises

55 58,245 40

Service Providing Admin., Support, 

Waste Management 56 485,943 161

Service Providing Arts, Entertainment, 

Recreation 71 127,658 38

Service Providing Accommodation and 

Food Services 72 647,059 240

Service Providing

Other Services (excl 

Public 

Administration)

81 (excl 814) 563,765 76
ORS Pre-production Stratification for State and Local Government Industry 

Aggregate 

Industry Detailed Industry

Included 

NAICS 

Codes

Establishments 

in Universe

Sample 

Size

Education

Elementary and 

Secondary 

Education

6111 62,349 150

Education Colleges and 

Universities 6112, 6113 7,416 39

Education Rest of Education 61 excl 

6111-6113 1,281 1

Financial 

Activities

Other Serviceproducing - Part A 51, 52-53 1,961 2

Goods Producing Goods-Producing 21, 23, 31-

33 6,350 4

Health Care, incl.

Hospitals and 

Nursing Care

Hospitals 622 2,377 21

Health Care, incl.

Hospitals and 

Nursing Care

Nursing Homes 623 1,679 5

Health Care, incl.

Hospitals and 

Nursing Care

Rest of Health 62, excl 

622-623 8,546 9

Service Providing

Trade, 

Transportation, and 

Utilities

42, 44-45, 

48-49, 22 12,764 14

Service Providing Public 

Administration 92 excl 928 107,694 122

Service Providing Other Serviceproducing - Part B

54-56, 71-

72, 81 excl 

814

18,462 15

9. Conclusions and Future Work 

BLS has conducted much testing to evaluate various alternative sample designs and has 

determined that there are statistically viable designs under both the Fully Integrated and 

Independent Design options. Given the additional flexibility of the Independent Design 

and the ability of implement that design with no change in the NCS design, the Independent 

Design is currently being strongly considered for implementation with the first production 

sample selection process for ORS. However, no decision has been made at this time about 

the amount of acceptable overlap between the ORS and NCS samples. Selection of the 

final design for use in an on-going production survey is dependent on many factors such 

as response rates, anticipated survey accuracy levels, the impact of ORS data collection 

from NCS respondents, and survey budget for which we do not yet have data. Response 

rates, anticipated survey variances, and the impact of joint NCS and ORS data collection 
will need to be evaluated using the results of the pre-production test to help guide the final 

design decision. Survey budget numbers also need to be fully evaluated and vetted with 

the Social Security Administration. These activities are scheduled to occur in Fiscal Year 

2015 and will be shared when available. 

References/Footnotes 

[1] Social Security Administration, Occupational Information System Project, 

http://www.ssa.gov/disabilityresearch/occupational_info_systems.html. 

[2] U.S. Department of Labor, Employment and Training Administration (1991), 

“Dictionary of Occupational Titles, Fourth Edition, Revised 1991” 

[3] U.S. Department of Labor, O*Net Online, http://www.onetonline.org/

[4] U.S. Bureau of Labor Statistics (2008) BLS Handbook of Methods, Occupational 

Employment Statistics, Chapter 3. http://www.bls.gov/opub/hom/pdf/homch3.pdf 

[5] U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, National 

Compensation Measures, Chapter 8. http://www.bls.gov/opub/hom/pdf/homch8.pdf 

[6] U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, Occupational 

Safety and Health Statistics, Chapter 9. 

http://www.bls.gov/opub/hom/pdf/homch9.pdf

[7] U.S. Bureau of Labor Statistics (2013), “Occupational Requirements Survey, Phase 1 

Summary Report, Fiscal Year 2013”, http://www.bls.gov/ncs/ors/phase1_report.pdf

[8] U.S. Bureau of Labor Statistics (2013), “Occupational Requirements Survey, Phase 2 

Summary Report, Fiscal Year 2013”, http://www.bls.gov/ncs/ors/phase2_report.pdf

[9] U.S. Bureau of Labor Statistics (2013), “Occupational Requirements Survey, Phase 3 

Summary Report, Fiscal Year 2013”, http://www.bls.gov/ncs/ors/phase3_report.pdf

[10] Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan (2010), “Evaluating 

Sample Design Issues in the National Compensation Survey”, 2010 Proceedings of 

the Section on Survey Research Methods, Alexandria, VA: American Statistical 

Association. 

[11] Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan (2011), "Update on the 

Evaluation of Sample Design Issues in the National Compensation Survey", 2011 

Proceedings of the Section on Survey Research Methods, Alexandria, VA; American 

Statistical Association. 

[12] Ferguson, Gwyn R., Ponikowski, Chester H., and McNulty, Erin (2012), “State and 

Local Government Sample Design for the National Compensation Survey", 2012 

Proceedings of the Section on Survey Research Methods, Alexandria, VA: American 

Statistical Association. 

[13] Bradley D. Rhein, Chester H. Ponikowski, Erin McNulty, (November 2013), “Sample 

Design Considerations for the Occupational Requirements Survey,” FCSM Papers 

and Proceedings, Federal Committee on Statistical Methodology Research 

Conference 

Any opinions expressed in this paper are those of the authors and do not constitute policy 

of the Bureau of Labor Statistics or the Social Security Administration.