Occupational Requirements Survey Sample Design

Gwyn R. Ferguson1and Erin McNulty1

1U.S. Bureau of Labor Statistics, 2 Massachusetts Ave. NE, Washington, DC 20212

Abstract

The Bureau of Labor Statistics (BLS) is working with the Social Security Administration 

(SSA) to establish the Occupational Requirements Survey (ORS), an establishment survey 

that collects information on the vocational preparation and the cognitive and physical 

requirements of occupations in the U.S. economy, as well as the environmental conditions 

in which those occupations are performed. In preparation for the first ORS production 

sample, we studied several potential sample designs and assessed each for practicality and 

coverage of the population. In addition, we considered how and if the ORS sample might 

be coordinated with the sample of BLS’s National Compensation Survey (NCS). In this 

paper, we evaluate each sample design option, describe the sample design that was selected 

for the first ORS production sample, and outline plans for monitoring the effectiveness of 

the selected sample design. 

Key Words: survey design, sample allocation, sample selection

1. Introduction

In the summer of 2012, the Social Security Administration (SSA) and the Bureau of Labor 

Statistics (BLS) signed an interagency agreement, which has been updated annually, to 

begin the process of testing the collection of data on occupations. As a result, the 

Occupational Requirements Survey (ORS) was established as a test survey in late 2012. 

The goal of ORS is to collect and publish occupational information that will replace the 

outdated data currently used by SSA. More information on the background of ORS can be 

found in the next section. All ORS products will be made public for use by non-profits, 

employment agencies, state or federal agencies, the disability community, and other 

stakeholders.

An ORS interviewer attempts to collect close to 70 data elements related to the 

occupational requirements of a job. The following four groups of information will be 

collected:

 Physical demand characteristics/factors of occupations (e.g., strength, hearing, or 

stooping)

 Specific vocational preparation requirements, which include educational requirements, 

experience, licensing and certification and post-employment training

 Mental and cognitive demands of work

 Environmental conditions in which the work is completed

Section 2 of this paper provides additional background information on the ORS. Prior to 

selecting the first ORS sample, which will go into collection in fall 2015, several sample 

design options were studied and compared. Section 3 discusses the major sample design 

options that were considered, including the results of simulations of each of the options. 

Section 4 presents the sample design that will be used for the first ORS production sample 

and explains why it was chosen. Section 5 describes several approaches taken to estimate

the number of occupations for which ORS data will be published under the selected sample 
design. Section 6 describes a modification to the design for the first production sample.

Finally, section 7 outlines plans for using data collected from the first few ORS samples to 

evaluate the selected sample design.

2. Background Information on ORS

In addition to providing Social Security benefits to retirees and survivors, the Social 

Security Administration (SSA) administers two large disability programs which provide 

benefit payments to millions of beneficiaries each year. Determinations for adult disability 

applicants are based on a five-step process that evaluates the capabilities of the worker, the 

requirements of their past work, and their ability to perform other work in the U.S. 

economy. In some cases, if an applicant is denied disability benefits, SSA policy requires 

adjudicators to document the decision by citing examples of jobs the claimant can still 

perform despite restrictions (such as limited ability to balance, stand, or carry objects) [1]. 

For over 50 years, the Social Security Administration has turned to the Department of 

Labor's Dictionary of Occupational Titles (DOT) [2] as its primary source of occupational 

information to process the disability claims. SSA has incorporated many DOT conventions 

into their disability regulations. However, the DOT was last updated in its entirety in the 

late 1970’s, and a partial update was completed in 1991. Consequently, the SSA 

adjudicators who make the disability decisions must continue to refer to an increasingly 

outdated resource because it remains the most compatible with their statutory mandate and 

is the best source of data at this time.

When an applicant is denied SSA benefits, SSA must sometimes document the decision by 

citing examples of jobs that the claimant can still perform, despite their functional 

limitations. However, since the DOT has not been updated for so long, there are some jobs 

in the American economy that are not even represented in the DOT, and other jobs, in fact 

many often-cited jobs, no longer exist in large numbers in the American economy.

SSA has investigated numerous alternative data sources for the DOT such as adapting the 

Employment and Training Administration’s Occupational Information Network (O*NET) 

[3], using the BLS Occupational Employment Statistics program (OES) [4], and 

developing their own survey. But they were not successful with any of those potential data 

sources and turned to the National Compensation Survey program at the Bureau of Labor 

Statistics [5].

3. Potential Sample Designs for ORS

The NCS has a fully-developed survey platform available for potential coordination with 

the ORS. This platform includes its management structure, collection staff, and survey 

procedures, as well as its sample of establishments and occupations. Coordination with the 

NCS sample, in particular, might provide efficient data collection opportunities for the 

ORS. At the same time, the NCS is an established survey that includes the Employer Cost 

Index, a principal federal economic indicator, so the ORS sample design cannot be allowed 

to harm the NCS sample or its outputs. Therefore, we have studied two main categories of 

sample designs: designs in which the ORS and NCS samples are coordinated for sampling, 

and designs in which the ORS sample is selected independently from the NCS sample. 
3.1 Coordinated Sample Design

In a coordinated sample design, the NCS sample would be selected as a subsample of the 

larger ORS sample. All establishments selected for the NCS sample would also be included 

in the ORS. The evaluation of coordinated ORS and NCS sample designs is described in 

detail by Ferguson et al. in “Occupational Requirements Survey Sample Design 

Evaluation” [6]. 

The research described in Ferguson et al. showed that coordinated sampling was possible 

but might be difficult to implement. Multiple sample simulations were carried out with the 

NCS sample selected as a subsample of the ORS sample. It was a challenge to maintain 

NCS sample design with this approach because the preferred sample distribution for each 

survey differs. The NCS sample is allocated among 120 cells that are defined by 5 

aggregate industries and 24 geographic areas. The measure of size for each frame unit is 

adjusted so that sample units are distributed among 24 major private sector industries in

certain proportions. In other words, there are effectively 576 cells for which NCS needs 

sufficient frame units. NCS’s industry proportions, however, differ from the industry 

proportions chosen for the ORS sample. Since the ORS sample is projected to be almost

three times larger than the NCS sample, it was assumed that the NCS sample should be a 

subsample of the ORS sample in a coordinated design. But it was found to be difficult to 

apportion the ORS sample in a way that maintains its own targeted industry proportions 

while providing an adequate frame for the NCS subsample. 

The most promising coordinated sample design required a change to the NCS sample cells. 

With 24 detailed industry sample cells, implicitly stratified by geographic area, the ORS 

sample and NCS subsample were both able to achieve overall target employment and 

sample size goals. Slight shifts to NCS sample by geographic area were noted but not 

considered a serious problem. The sample design also has the advantage of being 

straightforward and easy to explain. However, changing the NCS sample design to this 

degree would require resources that are not currently available, making this sample design 

difficult to implement before the first production sample is needed. 

3.2 Independent Sample Designs

Having studied a range of coordinated sample designs, we investigate independent sample 

designs in more detail in this paper. With an independent sample design, the effect of the 

ORS sample design on the NCS sample design is not a factor, allowing the ORS more 

freedom to tailor its sample design to its needs. To test the effectiveness and utility of each 

design, we followed procedures similar to those used for coordinated designs in Ferguson 

et al. [6]. 

To test each sample design in private industry and to allow comparability, we took a recent 

sample frame – the same frame used to test the coordinated designs – and selected 150 

ORS samples from that frame. We then used those samples to calculate average sample 

sizes and weighted employment totals in various categories of interest. For private industry, 

we also selected 150 NCS samples from the same frame, according to the NCS sample 

design. We calculated the average amount of overlap between each ORS sample and its

corresponding NCS sample. 

For the state and local government sectors of the economy, the NCS sample is selected less 

frequently, so we already have access to the NCS government sample that will be in 
collection during the first few years of ORS production, as well as the frame from which it 

was selected. Therefore, we selected the ORS simulation samples from this frame and

compared the ORS government simulation samples to the actual NCS sample.

Since we were not limited by the NCS sample cells, we used 23 detailed industry strata as 

the ORS sample cells in private industry. We used a proportional to employment allocation 

method to distribute the total sample size among the 23 private industry ORS sample cells. 

Note that the NCS adjusts its measures of size to achieve targeted sample sizes in these 23

industries, but the sample sizes are not guaranteed and can vary by a few units more or less.

See Appendix A for a list of the industry sampling cells for private industry samples.

In government we used 10 detailed industry strata as the ORS sample cells. NCS sample 

cells use 5 aggregate industries and 24 geographic areas for a total of 120 sampling cells. 

For NCS, the 5 aggregate industries are implicitly stratified by the 10 detailed industries

used for ORS. NCS and ORS both allocate the government sample proportional to 

employment. The ORS industry sampling cells for state and local industry samples are also 

shown in Appendix A.

3.2.1 Maximum Overlap Independent Design

We tested two types of independent sample designs. The first type was a maximum overlap 

independent design, in which the ORS sample is selected separately from the NCS sample, 

but ORS data is collected from all NCS units. The total ORS sample comprises the 

independently selected ORS-only sample and the independently selected NCS sample.

To allocate the ORS-only sample using the maximum overlap design, the composition of 

the NCS sample must be taken into account even though it is independent from the ORS 

sample. This is not a concern in the government sample because both ORS and NCS are 

allocated proportional to employment in similar cells. In private industry, however, NCS 

sample sizes for the ORS sample cells are not guaranteed. The NCS sample must be 

selected, not just allocated, prior to allocating the ORS sample, so that the exact number of 

NCS units in each ORS sample cell is known. Then the total ORS sample size, including 

the NCS units, is allocated proportional to employment across the 24 sample cells. The 

sample cell allocations are reconciled with the NCS sample counts; if the NCS sample size 

is greater than the allocation for a sample cell, the ORS allocation is increased to meet the 

NCS sample size and the remaining sample is reallocated among the other cells. The

reconciliation process continues until each total ORS cell allocation is at least as large as 

the NCS sample size for the cell. The NCS sample sizes are then subtracted from the total

ORS allocations to create the ORS-only sample allocations.

Two ORS sample cells, Finance (Rest of) and Insurance, required increases to their original 

allocations due to the size of the NCS sample in the cells, as shown in Table 1. Because 

the NCS sample sizes in these cells were so large, all of their ORS sample units came from 

the NCS sample, and there were no ORS-only units. On the other end of the range, only 

9% of the Health Care, Social Assistance (Rest of) cell came from NCS. Overall, NCS 

units accounted for 38% of the total private industry ORS sample. 
Table 1. Independent Design, Maximum Overlap: Allocation Adjustments for 

Selected ORS Sample Cells (Average over 150 Private Industry Samples)

ORS Sample Cell (Detailed Industry)

Percent Change from 

Original to Final 

Allocation 

Percent of Cell from 

NCS

Health Care, Social Assistance (Rest of) -1% 9%

Accommodation and Food Services -1% 15%

Finance (Rest of) 13% 100%

Insurance 14% 100%

... ... ...

Total ORS Sample 0% 38%

After the ORS-only sample allocations were determined, the ORS-only sample was 

selected using systematic probability proportional to employment selection. The total ORS 

sample clearly overlapped the NCS sample for the 38% that comes from the NCS private 

industry sample (the proportion was nearly the same in government). However, because 

the ORS and NCS samples were independently selected from the same frame, there was 

the possibility of additional, unintentional overlap between the samples. These units would 

be collected in any case, since ORS data is collected from all NCS units under the 

maximum overlap design. However, the unintentional overlap reduces the yield of unique 

ORS data that can be collected. 

Table 2 shows that approximately 1.4% of a single year’s ORS-only sample overlapped

the NCS private industry sample. ORS estimates will ideally be based upon data collected 

from several years of sample, and Table 2 shows that when a three-year span of samples is 

considered, the percentage of overlapping units rose to 3.3%. The three-year overlap 

percentage was not three times the single year’s overlap percentage because there was 

overlap from year to year within a survey. For the ORS-only portion of the private industry

sample, about 6.9% of units overlapped with another ORS-only unit in one (or both) of the 

other years in the three-year span of samples.

The government sample size for the ORS is about 15% of the total planned ORS sample 

size, which is proportional to the share of government employment out of all employment 

(government and private industry). However, government establishments have an average 

of 80 employees per establishment, while private industry establishments have an average 

of 14 employees per establishment. It follows that there were proportionally fewer 

establishments from which to select the government sample, compared to the private 

industry sample. Some establishments had particularly large employment, making it likely 

that they would be selected for any sample, ORS or NCS. As a result, the amount of overlap 

was higher for the government sample. Close to a quarter of ORS-only units overlapped 

with an NCS government unit in any single sample group, and about a third overlapped 

with NCS after three years of sample were combined. The amount of overlap between ORS 

samples within a three year span was also relatively high at nearly 30%. 
Table 2. Independent Design, Maximum Overlap: Percentage of ORS-Only/NCS 

Overlap (Average over 150 Samples)

Span

Type of 

Overlap

Private 

Industry

State & Local 

Government

One year (one sample group) ORS to NCS 1.4% 23.1%

Three years (one estimation group) ORS to NCS 3.3% 33.1%

Three years (one estimation group) ORS to ORS 6.9% 29.2%

3.2.2 Minimum Overlap Independent Design

The second type of independent sample design that we studied was a minimum overlap 

independent design. In such a design, the ORS sample is selected separately from the NCS 

sample, and ORS data is collected only from units selected in the ORS sample. For this 

design, the ORS simulation sample size was allocated across the 24 private industry sample 

cells and 10 government sample cells strictly proportional to employment. There were no

adjustments or other complications, so the resulting simulation sample sizes and weighted 

employment totals met all expectations. 

Under the minimum overlap design, the entire ORS sample is an ORS-only sample. Since 

the ORS would ideally obtain data from the same number of observations regardless of 

design, the ORS-only sample under the minimum overlap design was about 62% larger 

than the ORS-only sample under the maximum overlap design. As a result, though there 

was no intentional overlap between the ORS and the NCS, there was more unintentional 

overlap between ORS-only and NCS units under the minimum overlap design. Table 3 

shows that 1.8% of a single year’s private industry ORS sample overlapped the NCS 

sample, and 3.7% overlapped over a three-year span. These overlaps do not decrease the 

amount of unique ORS data that is collected since the NCS sample is not part of the total 

ORS sample. However, these overlaps do increase respondent burden for the overlapping 

NCS units.

The amount of unintentional overlap between ORS-only units across years was slightly 

lower under the minimum overlap design than under the maximum overlap design. In 

private industry, it was 0.2 percentage points lower, and in government it was 5.3 

percentage points lower. This means that there would be a somewhat larger amount of 

unique ORS data collected.

Table 3. Independent Design, Minimum Overlap: Percentage ORS-Only/NCS 

Overlap (Average over 150 Samples)

Span

Type of 

Overlap

Private 

Industry

State & Local 

Government

One year (one sample group) ORS to NCS 1.8% 17.2%

Three years (one estimation group) ORS to NCS 3.7% 29.0%

Three years (one estimation group) ORS to ORS 6.2% 24.2%

4. Selecting the Sample Design

There was one coordinated sample design that met the sample-related needs of both the 

ORS and the NCS. Also, there were maximum and minimum overlap independent sample 
designs that resulted in satisfactory samples. Any of the three approaches could be used to

meet the sample-related needs of both the ORS and the NCS. Therefore, the choice of a

final sample design hinged upon other considerations: flexibility, staffing, cost, respondent 

burden, and response.

Despite two years of survey field testing, it remains difficult to fully predict the experience 

of a full production collection cycle. Many features of the ORS sample – such as total size, 

industry distribution, and collection timing – require modifications during the first few 

years of ORS production. Flexibility, therefore, is essential. On the other hand, changing 

the sample selection process for the NCS, a stable and well-established survey, must be 

undertaken with care and deliberation. To balance these competing requirements, it was 

first decided that the ORS would implement an independent, rather than coordinated,

sample design. 

The next decision was to determine the desired amount of overlap between the collection 

of the ORS and NCS samples. As discussed in section 3, there would be some overlap 

between the two samples regardless of whether a maximum or minimum overlap design 

was chosen, but the amount of intentional overlap can be controlled. Past field testing 

showed that collecting ORS data from NCS units produced some efficiencies. Established 

relationships with NCS respondents could be used to encourage ORS response. Overall 

staffing needs and associated costs were lowered because the collection of administrative 

data and certain data elements common to both surveys could be combined. For the same 

reasons, the aggregate collection time of a given amount of ORS and NCS data was lower 

than if no units had overlapped, potentially allowing a shorter collection cycle. Aggregate 

respondent burden was also lower, as fewer respondents were contacted to collect the same 

amount of data across the two surveys. 

On the other hand, it was found that the data collection process for a combined ORS and 

NCS unit was lengthy, especially for units with a large number of occupational 

observations. Large establishments can have eight or more occupations. The collection of

ORS items alone for four occupations has typically taken about an hour [7]. A combined 

ORS/NCS sample unit would also involve the collection of all NCS elements: wages, 

hours, and the availability, cost, and provisions of various employee benefits. As a result, 

individual respondent burden is increased, which could discourage existing NCS 

respondents from responding.

The potential danger posed by additional individual respondent burden is the largest 

concern because it could affect the amount of response and quality of collected data for 

both the ORS and the NCS. It was determined that the best way to mitigate this concern is 

to limit overlap between the two surveys and implement the minimal overlap independent 

sample design. This design will be implemented initially for both private industry and 

government, but it was noted that the independent sample design would allow one or the 

other ownership group to modify the amount of NCS overlap in the future. In particular, 

response rates for government units are historically high for the NCS. Given the smaller 

number of available units, unintentional overlap will be higher for the government sector

than for private industry, so it might make sense to increase intentional government overlap

at some point. 
5. Occupation Count Projections

A key feature of ORS estimates is the focus on detailed occupational information at the 8-

digit Standard Occupational Classification (SOC) level. Because of this, it is important to 

collect a large number of unique observations for as many detailed occupations as possible. 

The sample design affects how many occupations will have enough data for publishable 

estimates. 

The sample frame available to ORS contains only establishment-level information and does 

not have occupational data. Therefore, in our selected sample design, industry distribution 

is being considered as a proxy for occupational distribution, and we are assuming that 

occupational variety is roughly proportional to the number of employees in an industry. It 

is difficult to gauge the accuracy of this assumption before collecting occupational data 

under the design, but we have done three studies to get an idea of how many occupations 

ORS will be able to publish. We have studied the former NCS National Wage publication, 

the current NCS estimation sample, and the Occupational Employment Statistics (OES) 

sample.

5.1 NCS National Wage Publication

Until 2011, the NCS program produced a publication of national wage estimates covering 

private industry and state and local government. The estimates were based on a probability 

proportional to employment sample of more than 35,000 establishments, making it similar 

in industry composition and size to the planned ORS sample. Occupations were selected 

using the same probability selection method (PSO) that will be used for the ORS and were 

classified using the Standard Occupational Classification (SOC) codes [9]. For this study, 

NCS wage establishment response rates are assumed to be comparable to response rates in 

the ORS; while wages are more sensitive than most ORS data, a larger quantity of data will 

be requested from an ORS respondent than from an NCS wage respondent. Unlike the ORS 

sample, the NCS wage sample was not a national sample but an area-based sample; data 

were collected from 227 localities, and the unit weights were adjusted with an area factor 

so that the sample represented the nation. Also, six-digit SOC was the most detailed 

occupational level for which data were collected and published. Nevertheless, its 

similarities to the ORS sample make the NCS national wage publication a good source for 

gauging how much occupational data might be available from the ORS sample. 

Wage estimates were published at various levels of detail, from the broad occupation (twodigit SOC) down to the detailed occupation (six-digit SOC). To be publishable, an estimate 

for an occupation had to be based on data from at least three unique responding companies 

(by Employer Identification Number, or EIN) and six unique occupational observations; 

its relative standard error had to be less than 50%; and it had to pass a P-percent dominance

test. These criteria are less stringent than the ORS criteria are likely to be since we have 

less experience with ORS data and will be more cautious with the initial publication criteria 

until BLS has enough experience to determine if the less stringent NCS wage criteria are 

sufficient for protecting ORS respondent confidentiality. Even with these cautions, we 

believe the wage data are a good starting point for assessing how much data might be 

available. 

We counted the number of occupation-specific estimates that were published in the All 

Civilian wage table (covering private industry and government combined) [8]. Table 4

shows that over two-thirds of all possible occupations were published, and over 90% of all 
possible occupations were published at the 2-digit and 5-digit SOC level. (Note that threedigit SOC was used only to group more detailed published occupations, so only a small 

number of the eligible occupations at that level were published even though many more 

could have been published.) At the most detailed level, six-digit SOC, almost threequarters of possible occupations were published.

Table 4. 2010 NCS National Wage Publication (Published May 2011), All Civilian: 

Percent of Occupations Published

Occupational Level Percent Published

2 Digit SOC 95.65%

3 Digit SOC 11.25%

4 Digit SOC* ---

5 Digit SOC 92.57%

6 Digit SOC 72.35%

All Levels 71.61%

* Four-digit SOC is the same as three-digit SOC.

5.2 2014 NCS Estimation Sample

The NCS sample used in estimation in 2014 included about 10,000 establishments. It 

covered both private industry and state and local government units, in roughly the same 

proportions as are planned for the ORS. The NCS sample is being changed from an areabased sample, similar to the national wage sample in section 5.1, to a national sample akin

to the ORS. The 2014 NCS estimation sample was a combination of several annual sample 

groups and included both area-based sample groups and national sample groups. 

As discussed earlier, the current NCS sample is selected probability proportional to 

employment with certain target industry proportions, which results in a different 

distribution of establishments by industry from the ORS. (The NCS national wage sample 

discussed in section 5.1 did not have these target industry proportions.) Occupations were 

selected using the same probability selection method (PSO) that will be used for the ORS. 

Response rates by industry are known and, for the purpose of this study, assumed to be 

similar to those that will be experienced in the ORS, because collection procedures and 

staff will be largely the same. 

We counted the number of unique occupations found in the NCS sample by various levels 

of SOC detail. Also, we counted the number of unique responding establishments by EIN 

within each SOC level. We applied an ownership-based factor to each count to adjust for 

the smaller size of NCS sample compared to the planned ORS sample size of 30,000. We 

determined the proportion of all possible occupations that met several EIN and occupation

criteria levels.

Tables 5 and 6 show that at the most detailed SOC levels, the amount of available data 

quickly falls off as more stringent criteria are added. Requiring 3 establishments and 6 

occupations yields about 80% of possible six-digit SOCs for both private industry and 

government, which is slightly higher than the results from the NCS wage publication. 

Requiring 15 establishments and 30 occupations drops the private industry yield to 53%. 

As might be expected from the smaller government sample, its yield at this level is much 
lower, at 37%; in fact, government yields for all but the most lenient criteria and broad 

occupations are quite a bit lower than the corresponding private industry results.

Table 5. 2014 NCS Estimation Sample: Percent of Private Industry Occupations 

That Are Publishable

Publishability Criteria

Percent of Private Industry 

Occupations That Are Publishable

(by SOC detail level)

6-digit 5-digit 3-digit 2-digit

3 establishments, 6 occupations 80% 85% 90% 100%

15 establishments, 30 occupations 53% 67% 82% 100%

15 establishments, 60 occupations 38% 56% 80% 100%

15 establishments, 90 occupations 30% 46% 75% 95%

15 establishments, 120 occupations 25% 38% 70% 95%

Table 6. 2014 NCS Estimation Sample: Percent of State & Local Government

Occupations That Are Publishable

Publishability Criteria

Percent of State & Local 

Government Occupations That Are 

Publishable (by SOC detail level)

6-digit 5-digit 3-digit 2-digit

3 establishments, 6 occupations 81% 83% 92% 100%

15 establishments, 30 occupations 37% 47% 74% 95%

15 establishments, 60 occupations 23% 31% 66% 95%

15 establishments, 90 occupations 16% 24% 60% 95%

15 establishments, 120 occupations 11% 18% 49% 95%

5.3 2011 OES Sample

The Occupational Employment Statistics (OES) program is a BLS survey program that 

produces wage estimates for over 800 occupations [10]. With OES cooperation, we were 

able to access a version of the 2011 private industry OES sample that the NCS program 

had used for previous research. The OES sample had three years of data covering over 1 

million establishments and included data for all occupations in each of the establishments, 

including the 6-digit SOC code and number of employees for each occupation. The earlier 

NCS researchers had amplified the sample to produce a quasi-frame of establishments with

occupation information.

Since we began with a full private industry (quasi) frame, we were able to apply the 

selected ORS sample design and select a private industry sample of 25,500 establishments

(the private industry portion of the planned ORS sample size of 30,000). We stratified by 

23 detailed industries, combining the aircraft manufacturing and all other manufacturing
industries that had been separate in the studies described in section 3 because the first is 

small subset of the other and is not a particular focus for the ORS. The sample was allocated 

proportional to industry employment. Establishments were selected by systematic 

probability proportional to size selection with employment used as the measure of size. 

Occupation selection followed the planned ORS protocol as closely as possible. The 

number of quotes selected was based on establishment size, and occupations were selected 

using systematic probability proportional to size selection with the occupational 

employment used as the measure of size. 

Each frame establishment was considered to be unique, though there were no EINs 

available to verify this assumption. The number of unique responding quotes was counted 

at several SOC code levels. The actual counts were multiplied by 0.75 to simulate nonresponse. 

Table 7 shows ORS simulation results that are similar to the results from the private 

industry 2014 NCS estimation sample (similar establishment thresholds could not be used 

because of differences in how establishments were identified). The OES-based percentages 

are a bit lower for the most stringent criteria at the most detailed SOC levels; for example, 

requiring 120 occupations for a six-digit SOC yields 16% of possible occupations from the 

OES sample compared to 25% from the NCS sample. The general patterns hold, though, 

so the occupation projection studies provided a rough but consistent idea of the amount of 

data to expect from the ORS.

Table 7. 2011 OES Sample, Private Industry: Percent of Occupations That Are 

Publishable

Publishability Criteria Percent of Private Industry Occupations That Are Publishable

6-digit SOC 5-digit SOC 3-digit SOC 2-digit SOC

6 occupations 78% 89% 94% 96%

30 occupations 40% 60% 89% 96%

60 occupations 26% 42% 83% 93%

90 occupations 20% 34% 77% 91%

120 occupations 16% 29% 68% 91%

6. Year 1 Modification to Design

In addition to the ORS-only establishmentsselected in year one using the minimum overlap 

independent design described above, a supplemental set of establishments from the current 

NCS private sample will be included for collection of the ORS data elements. The 

modification is being made to increase the amount of data that can be published from the 

first production sample given the amount of trained and available field collection staff. The

additional units are all active for the NCS but will be rotating out of the NCS sample after 

October of 2015. The NCS design at the time that these units were selected was a threestage stratified design with probability proportionate to size selection used at each level. 

The first stage was the selection of 152 areas with the second stage being the selection of 

establishments within each selected area. The final stage was the selection of occupations 

within sampled establishments. For more details on this design see “Evaluating Sample 

Design Issues in the National Compensation Survey” by Ferguson et al. [11].
Since the two parts of the year one sample are selected independent of each other and each 

has weights that represent the entire frame, weight adjustment factors will be applied to all 

sample units in order to properly represent the frame. The data from this full sample will 

be used in all estimates generated from the first ORS production sample. Following the 

year one sample, all subsequent ORS samples will be selected independent of the NCS 

sample and will not be supplemented by the NCS. 

7. Future Plans

The first production sample for the ORS has been selected according to the selected sample 

design described in sections 4 and 6. Data collection will begin in the fall of 2015. The 

following year’s sample will need to be selected before all data have been collected from 

the first sample, so we will assess the sample design at intervals throughout collection as 

well as at the end of collection. 

We have assumed for our work so far that establishment (unit) response rates will be similar 

to those in the NCS, but the different subject matter could lead to different response rates. 

The first production sample will allow us to determine whether the sample is less efficient 

in certain industry strata or establishment size classes, which could lead to refining strata 

definitions and/or shifting sample. We will also monitor response rates of individual ORS 

elements (item response) to see if the data yield per establishment differs by industry or 

size class in a way that could be mitigated by the sample design. As part of our detailed 

response analysis, we will evaluate the potential for non-response bias in the ORS survey, 

focusing on the ORS data elements with the lowest item response rates. Similar response

analysis will be evaluated for the 2015 pre-production test sample that was collected earlier 

this year. However, the pre-production test sample was selected using a different design in 

order to meet the test needs of that sample [6]. In addition, staff were just beginning to 

learn how to collect ORS data, so response rates from the test might not accurately reflect

the response that will be obtained during production. 

The number and distribution of responding units will contribute to the amount of data the 

ORS collects for detailed occupations. We will monitor the amount of usable data that we 

collect by occupational SOC code as well as the industries in which SOC codes are 

concentrated. Our studies so far have relied upon the six-digit SOC occupation as the most 

detailed occupation, but the ORS data will allow us to study occupational distribution at 

the more detailed eight-digit SOC level. We will observe how many estimates can be 

published at various levels of detail and also assess standard errors to determine whether 

more sample is needed from certain strata to ensure estimate reliability. 

Researchers and members of the public have commented that it may be necessary to update 

ORS data for certain occupations more frequently than for others occupations. The sample 

designs studied so far do not allow for this flexibility, but we have identified and will 

evaluate additional sample designs that could allow such flexibility. 

Minor changes to the design could be implemented quickly, perhaps as soon as the second 

production sample. Other observations might lead to more extensive research that could 

result in sample design changes further in the future. We will continually gather and assess 

data throughout the collection of all ORS samples. 
8. References

[1] Social Security Administration, Occupational Information System Project, 

http://www.ssa.gov/disabilityresearch/occupational_info_systems.html. 

[2] U.S. Department of Labor, Employment and Training Administration (1991), 

“Dictionary of Occupational Titles, Fourth Edition, Revised 1991”

[3] U.S. Department of Labor, O*Net Online, http://www.onetonline.org/.

[4] U.S. Bureau of Labor Statistics (2008) BLS Handbook of Methods, Occupational 

Employment Statistics, Chapter 3. http://www.bls.gov/opub/hom/pdf/homch3.pdf.

[5] U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, National 

Compensation Measures, Chapter 8. http://www.bls.gov/opub/hom/pdf/homch8.pdf.

[6] Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan. 2014. Occupational 

Requirements Survey Sample Design Evaluation. In JSM Proceedings, Government 

Statistics Section. Alexandria, VA: American Statistical Association.2924-2938.

[7] U.S. Bureau of Labor Statistics (2014), “Occupational Requirements Survey 

Consolidated Feasibility Tests Summary Report: Fiscal Year 2014.” 

http://www.bls.gov/ncs/ors/fy14_feasibility_test.pdf.

[8] National Compensation Survey: Occupational Earnings in the United States, 2010, 

U.S. Department of Labor, U.S. Bureau of Labor Statistics, May 2011, Bulletin 2753, 

http://www.bls.gov/ncs/ncswage2010.htm.

[9] U.S. Department of Labor, Bureau of Labor Statistics, Standard Occupational 

Classification System, http://www.bls.gov/soc/.

[10] U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, Occupational 

Employment Statistics, Chapter 3. http://www.bls.gov/opub/hom/pdf/homch3.pdf.

[11] Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan. 2010. Evaluating 

Sample Design Issues in the National Compensation Survey. In JSM Proceedings, 

Survey Research Methods Section. Alexandria, VA: American Statistical 

Association.2838-2850.

Any opinions expressed in this paper are those of the authors and do not constitute policy 

of the Bureau of Labor Statistics or the Social Security Administration.
Appendix A – ORS Industry Stratification

ORS Production Stratification Private Industry State & Local Govt.

Detailed Industry

Included 

NAICS

Codes

Establishments 

in Universe

Expected 

Sample 

Size

Establishments 

in Universe

Expected 

Sample 

Size

Educational 

Services (Rest of)

61 (excl. 

6111-

6113)

8,671 36 1,251 2

Elementary and 

Secondary Schools 6111 80,655 22 61,828 247

Junior Colleges, 

Colleges and 

Universities

6112, 

6113 17,249 23 7,550 78

Mining 21 35,471 27

Construction 23 746,906 197 6,339 7

Manufacturing 31-33 336,416 385

Healthcare, Social 

Assistance (Rest of)

62 (excl. 

622, 623) 1,277,003 103 8,470 11

Hospitals 622 8,829 311 2,644 34

Nursing and 

Residential Care 

Facilities

623 74,363 149 2,081 7

Utilities 22 17,382 18

12,863 20

Wholesale Trade 42 618,652 184

Retail Trade 44-45 1,036,005 483

Transportation and 

Warehousing 48-49 228,101 138

Information 51 149,018 86

20,678 29

Finance (Rest of) 52 (excl. 

524) 282,821 65

Insurance 524 187,344 109

Real Estate, 

Renting, Leasing 53 357,544 69

Professional, 

Scientific, Technical 54 1,104,197 262

Management of 

Companies and 

Enterprises

55 59,734 68

Admin., Support, 

Waste Management 56 494,313 273

Arts, Entertainment, 

Recreation 71 131,361 74

Accommodation and 

Food Services 72 656,131 405

Other Services 

(excl. Public 

Administration)

81 (excl. 

814) 572,205 127

Public 

Administration

92 excl. 

928 109,075 202