Occupational Requirements Survey

Pre-Production Collection Report

1 

Occupational Requirements Survey 

Preproduction Collection Report 

June 16, 2015 www.bls.gov

Executive Summary

In fiscal year (FY) 2015, the Bureau of Labor Statistics (BLS) completed data collection for the 

Occupational Requirements Survey (ORS) pre-production test. The pre-production test might better be 

described as a “dress rehearsal” as the sample design, collection procedures, data capture systems, and 

review were structured to be as close as possible to those that will be used in production. While the 

feasibility tests in FY 2014 and earlier were intended to gauge the viability of collecting occupational 

data elements and to test modes of collection and procedures, in FY 2015 BLS integrated the prior work 

into a large-scale nationally representative pre-production test.

The intent of this report is to assess the pre-production collection, drawing some preliminary 

conclusions regarding whether BLS is able to successfully capture data on the requirements of 

occupations that will meet the needs of the Social Security Administration (SSA). BLS notes, however, 

that though the collection and review of pre-production ORS data is complete as of May 22, 2015, the 

estimation processes are just beginning. The report on ORS pre-production estimation results is 

anticipated to be released before the end of FY 2015. BLS also notes that because estimation processes 

have not been applied, the numbers presented in this report are unweighted and should be expected to 

differ from the final, weighted numbers that will be available in the estimation report.

Preliminary indications show that BLS is able to successfully collect data on occupational requirements 

that will meet the needs of the SSA. In particular, the overall and item response rates, quality assurance 

program, and feedback from data collection debriefs provide support that survey procedures are 

working well and data on job requirements can be collected by BLS field economists. As BLS moves into 

production, however, there will be continued refinements and testing, particularly for the mental and 

cognitive elements, as described in this report.

Background and pre-production test overview

In the summer of 2012, the SSA and BLS signed an interagency agreement, which has been updated 

annually, to begin the process of testing the collection of data on occupational requirements. As a 

result, ORS was established as a test survey in late 2012. The goal of ORS is to collect and publish 

occupational information that meets the needs of SSA at the level of the eight-digit standard 

occupational classification (SOC) that is used by the Occupational Information Network (O*NET).1

 

The ORS data are collected under the umbrella of the National Compensation Survey (NCS), which uses 

Field Economists (FEs) to collect data. FEs are well-suited for ORS data collection as their training 

(described in the “Quality Assurance” section below) focuses on identifying the appropriate respondent, 

1 The occupational classification system most typically used by BLS is the six-digit SOC (www.bls.gov/soc/), 

generally referred to as “detailed occupations”. O*NET uses a more detailed occupational taxonomy 

(www.onetcenter.org/taxonomy.html), classifying occupations at eight-digits and referring to these as “O*NETSOC 2010 occupations”. There are 840 six-digit SOCs and 1,100 eight-digit SOCs. 
Occupational Requirements Survey

Pre-Production Collection Report

2

probing the respondent to clarify apparent inconsistencies in responses, and following up with 

respondents to ensure data are complete and accurate. FEs generally collect data elements through 

either a personal visit to the establishment or remotely via telephone, email, mail, or a combination of 

modes. 

For ORS, FEs are collecting occupationally-specific data elements to meet SSA’s needs including:

• Physical demands

• Educational requirements

• Mental and cognitive demands

• Environmental conditions in which the work is performed.

In fiscal years 2013 and 2014, several feasibility tests were performed to assess the viability of collecting 

data on occupational requirements using the platform currently used by the NCS. These tests provided 

evidence that the NCS platform could be adapted to ORS data collection, which led to the preproduction test in FY 2015.2

 Unlike the earlier tests, which were small-scale and tested a subset of data 

elements or the viability of different collection methods, the pre-production test was designed as a 

relatively large-scale, nationally representative test of ORS data collection. The sampling, data 

collection, procedures, and review were designed to mimic what will occur during ORS production 

beginning in September 2015. 

This report summarizes the pre-production test from sampling through review, describes future testing, 

and discusses next steps for ORS data collection as BLS moves into production. It is important to note 

that the end of pre-production data collection does not complete the test – estimation, validation, and 

dissemination processes are on-going and a report that covers these topics will be available at the end 

of FY 2015.

Sampling

The ORS pre-production sample was drawn from the same frame as the NCS – the Quarterly Census of 

Employment and Wages, which includes all establishments covered by state unemployment insurance 

laws, and a supplementary file of railroads. The frame contains virtually all establishments in the 50 

United States and the District of Columbia in the private sector (excluding agriculture, forestry and 

fishing and private households) and in state and local governments. The pre-production ORS sample 

contained 2,549 establishments.

Roughly one-third of the ORS pre-production sample consisted of establishments that are also in the 

NCS sample (“NCS-ORS overlap”) and the remainder are ORS-only.3

 Across all establishments (NCS-ORS 

overlap and ORS-only), approximately 15% are government owned and 85% privately owned. 

For each establishment in the ORS sample (establishments are also referred to as “schedules”), jobs are 

selected for data collection through probability selection of occupations; these jobs are referred to as 

“quotes”.

4

 The number of jobs selected within a private establishment varies from 4 to 8, based on 

 2 Copies of earlier test reports can be found at: www.bls.gov/ors/#pretesting, within “Fiscal Year 2014 

Occupational Requirements Testing Results.” 3 Additional information on the survey design can be found as part of the Federal Register Notice, 

www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201403-1220-002. 4 Probability selection of occupations (PSO) is also used to select jobs in the NCS. Additional information on 

probability selection of occupations can be found in the BLS Handbook of Methods, 

www.bls.gov/opub/hom/pdf/homch8.pdf.
Occupational Requirements Survey

Pre-Production Collection Report

3

establishment size, and in government, the number of jobs varies from 4 to 20. It is common for 

multiple individuals within an establishment to have the same job (e.g. elementary school teachers 

within a school/school district), which can result in fewer individual quotes for that establishment. 

Because the quote-level information is tied to the job, not the individual, sampling a certain number of 

jobs within an establishment is not equivalent to sampling a certain number of workers within an 

establishment. 

Sample weights will be assigned to each quote and establishment to represent the entire frame. Sample 

weights were unavailable at the time this report was written, therefore, all numbers presented are 

unweighted. 

Data collection

ORS data collection began in October 2014 and continued until May 2015. At the close of the data 

review process, there were 7,116 quotes collected from 1,851 establishments, slightly more than 4 jobs 

per establishment. These jobs spanned 22 of the 23 unique 2-digit SOCs and 704 unique 8-digit SOCs. 

Though there are 23 2-digit SOCs, one of these, military-specific occupations (55-0000), is out of scope 

for ORS. The 704 8-digit SOCs represent 63.4% of the 1,100 unique 8-digit SOCs. 

A few caveats should be noted about the scope of occupational coverage. First, collection of a unique 

SOC means only that there is at least one job in the data that falls under that SOC; it should not be 

interpreted to mean that an estimate suitable for publication is available for the SOC using preproduction data. A second thing to note is that the employment covered by an individual SOC varies 

considerably by 8-digit code. Therefore, a count of unique SOCs (or the share of total SOCs) is a poor 

proxy for employment coverage. A better measure of scope would be the share of total employment 

covered by the unique SOCs collected in pre-production; however, this requires weighted estimates. 

BLS measures collection response rate as the total number of usable schedules divided by the sum of 

refusals and usable schedules.5

 The collection response rate for the ORS pre-production test was 79.3%. 

This is comparable to the 78% response rate for establishments initiated in the most recent NCS 

sample.6

 Most ORS data elements have both a measure of presence and one of duration. For example, 

FEs ask whether a job involves kneeling as well as how much time is spent kneeling. For data elements 

that measure the presence of a job requirement, the average item response rate was 85%. When we 

include measures of duration as well as presence, the average item response rate was 80%. Generally, 

respondents were able to provide presence and duration of environmental conditions. For certain 

physical requirements, such as gross and fine manipulation, the response rates for presence were high, 

but respondents had difficulty providing data on duration, resulting in a relatively lower response rate 

for those items. Additional information on item-level response rates will be available in the ORS preproduction estimation report.

Interview duration was, on average, 63 minutes. Roughly half (49.4%) of schedules were conducted by 

personal visit and the balance by some other mode (telephone, email, mail, or a combination of 

approaches).

 5 The sum of refusals and usable schedules will not equal the size of the sample initially drawn. Establishments 

may have gone out of business or be classified as out of scope for the survey. 6 A “usable” quote for ORS is defined as a job for which at least one data element has been reported. For the 

Employment Cost Index component of the NCS, a usable quote is one for which information on wages is present. 

For both ORS and ECI, one usable quote makes a schedule usable.
Occupational Requirements Survey

Pre-Production Collection Report

4

Duration tended to be longer, on average, for personal visit appointments (75.6 minutes) than for other 

modes (50 minutes), which is expected since personal visit appointments were more likely to occur for 

establishments with a higher number of sampled jobs. 

The establishment respondents in the ORS pre-production test included human resource 

professionals/workforce planners, workers compensation staff, safety/occupational health/risk 

management staff, and subject matter experts (supervisors, incumbents, department heads). These 

respondents were similar to those who provided data during previous feasibility tests. The feedback 

from FEs in pre-production is that finding the right respondent is key to data collection and that 

respondents with knowledge of job descriptions tended to be most helpful. 

The HR departments were often the starting point for finding respondents, for two reasons. First, for 

establishments that are currently part of the NCS sample, the FEs already have established contacts who

are often in HR. Second, in order to perform the job sampling within an establishment, FEs first need 

information that is typically maintained in the HR department. The FEs reported that often the primary 

respondent would consult with others on staff for responses to data elements. 

Procedures

There are a number of resources provided for FEs to ensure consistent procedures, collection, and 

coding of data elements. FEs are provided with a collection manual (and training on the contents of the 

manual) which covers such topics as securing cooperation from respondents, fundamentals/definitions 

of ORS pre-production data elements (along with practical examples), guidance on how elements might 

be related, and instructions on coding in the data capture system. 

During collection, FEs may submit inquiries to resolve ambiguities that arise. Often these are one-off 

topics, such as whether walkie-talkies would fall under the category of phone communication. The 

procedures staff provides a response to the inquiry, which is then distributed to all FEs. There were 23 

such responses provided during pre-production.

If there is evidence that additional clarification is needed on data elements (i.e. there exists a more 

general source of confusion), then the procedures staff highlights the correct procedure through a 

“procedures alert,” that is distributed to all FEs. An example of this during pre-production was an alert 

that provided additional information on “deviations from the norm” for mental and cognitive elements. 

During pre-production there were five procedures alerts issued.

Debriefs with the FEs were also used during the ORS pre-production test. These were an opportunity to 

share best practices, get feedback on respondent comprehension of data elements, and for FEs to 

highlight any problems they were having with collection. Feedback from these debriefs will inform FE 

training and the materials provided in the collection manual as preparations are made for ORS data 

collection for FY 2016.

ORS review program

The ORS pre-production data were subject to a review program modeled on the NCS that will also be 

used in production. The data review program has components that occur in the field as well as 

components that occur after the data are submitted to BLS headquarters.

The quality assurance process starts with the classroom and on-the-job training of newly hired FEs, 

which takes place over 9-12 months. The classroom training includes survey procedures, collection 
Occupational Requirements Survey

Pre-Production Collection Report

5

protocols, data capture and review systems, and interviewing techniques. Once basic training is 

completed, the certification process begins, which pairs a new FE with a certified FE who will observe 

data collection and review collected data.

Additional review of collected data includes: full schedule review, where all data elements for an 

establishment (also referred to as a “schedule”) are reviewed by experienced field staff; technical reinterview, where respondents are re-contacted and a subset of data elements are re-collected; and 

targeted schedule review, where a subset of elements are reviewed by staff at BLS 

headquarters. Roughly 15% of schedules are subject to full schedule review, 5% to technical reinterview, and 20% to targeted review.

The remaining 60% of schedules are subject only to secondary review, in which data elements flagged 

outside of the primary data capture system are reviewed. This secondary review focuses on verifying 

data elements that are unexpected or inconsistent with other data elements (e.g. physical elements that 

are unexpected given the occupation). This also encompasses review across schedules to identify 

schedules and elements with anomalies. 

The review staff work with the FEs to ensure that all data are correctly coded and that any flagged 

elements have documentation verifying that the elements are correctly coded. Reviewers questioned 

approximately 1% of the data during ORS pre-production. When a field economist receives a question 

from a reviewer, the FE reviews the values coded in the data capture system, reviews their notes from 

the interview with the respondent, may review the procedures and collection manual, and may call back 

the respondent to clarify any information from the interview. Roughly 61% of the time these questions 

resulted in data being changed.

Changes in collection anticipated for production

Implementing a large-scale test of the ORS data elements has resulted in changes and refinements to 

several of the elements as the survey moves into production. These changes were made in consultation 

with the Social Security Administration and reflect the desire to collect data that meet the needs of SSA. 

What is most notable is that relatively few data elements are changing; the pre-production test has 

demonstrated that the definitions and wording of the majority of the data elements are working well in 

data collection.

There was only one change to the physical requirements elements. This involved rewording the 

question on climbing ramps or stairs to ensure that the climbing was work-related and not simply the 

result of someone, say, choosing to walk up stairs to a third floor office.

Among the environmental elements, there were two changes. The first consolidated the presence of 

hazardous contaminants into one data element rather than two. The second added an indicator for the 

presence of personal protective equipment for hazardous contaminants, moving mechanical parts, high 

exposed places, and noise intensity level.

All of the mental and cognitive elements are being revised for production. Highlights are provided 

below:

• The question on task complexity has been removed. A question has been added, “what type of 

decision-making is required to perform the tasks of the occupation?”

• The question on work controls has been removed. A question has been added, “what type of 

supervision does this occupation have?”
Occupational Requirements Survey

Pre-Production Collection Report

6

• A question has been added on pace of the work performed.

• A question has been added regarding what controls the pace of work (the work or the worker).

• The question on work deviating from the norm has been removed. A question has been added, 

“how often do (work/schedule/location) change in this occupation?”

• The questions on contacts have been revised to rescale the options for frequency of contacts.

Future testing plans

Two additional tests of data collection are planned for June-August 2015. The first is the ORS job 

observation pre-test. The job observation pre-test is intended to assess whether the data collected 

through ORS interview collection methods are systematically different than data collected through 

direct observation. This pre-test involves re-contact of a subset of establishments that were interviewed 

as part of the pre-production test. FEs will observe select jobs within the establishment and record data 

on physical and environmental data elements. This pre-test is in response to both Federal Register 

Notice public comments and an external subject matter expert’s recommendations for testing and 

validation of ORS survey design.7 A written report of this test will be available in fall 2015.

The second test is a test of the revised mental and cognitive elements, to understand respondents’ 

perception of the wording of these questions. Data will be collected from establishments who 

participated in ORS feasibility tests. If necessary, the results from this test will be used to further refine 

the mental and cognitive elements after the first year of ORS data is collected.

Lessons learned and overall assessment

At this point, the results from the ORS pre-production test demonstrate that data on occupational 

requirements can be collected using the processes established by BLS, but final conclusions about the 

feasibility of these methods will be made when the analysis of pre-production data is complete. 

Training and procedures are designed for consistent data collection by field economists. The feedback 

channels between FEs and the procedures team ensure that questions are answered quickly, 

information is disseminated to all ORS staff, and any needed modifications to procedures are made and 

communicated effectively. The data review process will evolve as new procedures are implemented and 

the indicators used to identify inconsistencies in collected schedules are refined.

The response rate for ORS is comparable to that for the NCS and item level response rates provide 

preliminary evidence that usable data on job requirements are being collected. The elements that 

address the physical, environmental, and vocational preparation requirements (tested through 

feasibility tests in prior years) appear to be working well in pre-production collection and few changes 

are being made to those elements as BLS moves into production.

There are some aspects to collection that need to be addressed going forward. Training and procedures 

for ORS production should address best practices for identifying the proper respondent as well as 

strategies for dealing with establishments for which FEs are collecting both ORS and NCS data elements. 

Item level response rates indicate that some respondents are having difficulty providing duration data 

for some job requirements and tools and procedures to assist field economists in obtaining measures for 

those data elements are needed. 

 7 The subject matter expert’s report, “Occupational Requirements Reliability and Validity Literature Research,” is 

available here: http://www.bls.gov/ncs/ors/handel_report_feb15.pdf. 
Occupational Requirements Survey

Pre-Production Collection Report

7

The mental and cognitive elements have largely been rewritten for production, which will require 

additional training and procedures that assist FEs in collecting these new elements from respondents. 

The test of mental and cognitive elements scheduled for summer 2015 should help shape this. Minor 

refinements may be made during the first year of production collection to mental and cognitive 

elements as part of the OMB clearance process for production collection. However, major changes

cannot be made until the second year of collection. If the summer test and initial collection indicate 

further major changes to the mental and cognitive elements are needed, tests of the mental and 

cognitive elements will occur in FY2016.

Occupational Requirements Survey • NCSinfo@bls.gov • www.bls.gov/ors • 202.691.6199