1 | Page

FY 2014 Occupational Requirement Survey (ORS) Test Plans

September 30, 2013

This document outlines the testing that the Bureau of Labor Statistics (BLS) will perform during FY 2014

under the joint interagency agreement with the Social Security Administration (SSA).  Section I of this

document provides an overview of the testing that will be done in FY 2014 while Sections II and III

provide more detailed information. Section IV lists research activities that BLS will be conducting in

support of this testing effort.  

Section I – Testing Overview

In FY 2014, the BLS will perform work to meet the following objectives:

 Collect data to continue evaluating whether the National Compensation Survey (NCS) can

produce estimates of sufficient quantity and quality to meets the needs of the SSA;

 Continue to evaluate survey design options and develop the processes, protocols, aids, and

collection procedures to meet SSA data needs; and

 Provide documentation to the SSA summarizing the work performed by the BLS, conclusions

drawn, and recommendations for future data collection, testing, and research.

In order to accomplish these objectives, the BLS will conduct various Feasibility Tests followed by a Pre‐

Production Test, and complete related research.   

In FY 2014, the BLS will test the collection of mental and cognitive demands of work data elements and

resolve issues discovered during FY 2013 testing.  All tests will include procedures and protocol

development, training, data collection, evaluation, and debriefing activities.  Specifically, the FY 2014

Feasibility Tests will include the following activities:

 Design, develop, and conduct a test to determine how best to collect occupational requirements

data elements and NCS data elements from the same establishment;

 Design, develop, and conduct a test to determine how best to collect occupational requirements

data elements from America’s largest firms and State governments;

 Design, develop, and conduct tests for the new mental and cognitive demands of work data

elements for use in SSA’s disability evaluation process, and evaluate the use of occupational task

lists as developed by the Employment and Training Administration’s O*NET program during data

collection.  The initial test will take place within the metropolitan Washington, D.C. area.  The

BLS and SSA will evaluate and modify the initial results prior to starting a second phase of

testing.  The BLS and SSA will evaluate the results of the second phase of testing  to determine if

further testing is needed to capture the mental and cognitive demands of work in a way that

meets SSA program needs;   

 Design, develop, and conduct a test to refine the methods to develop more efficient approaches

for data collection as identified during FY 2013 testing;  
2 | Page

 Design, develop, and conduct a test to determine how best to collect occupational requirements

data elements when a personal visit is not optimal due to respondent resistance, collection

costs, or other factors; and

 Capture and evaluate changes in data coding as a result of observing the work environment, the

sampled occupation, or both.  

At the conclusion of Feasibility Testing, the BLS will conduct a nationwide test to evaluate all aspects of

the survey in a possible production environment.  Data collection and capture for this Pre‐Production

Test are expected to run for two or more quarters and will conclude in FY 2015.  The data capture

portion of this test will be followed by a full evaluation of data, processes, survey design, and other

program elements.   

Section II – Feasibility Testing

BLS will conduct a series of Feasibility Tests through spring 2014.  BLS has identified five independent

Feasibility Tests and one test that will be conducted during all Feasibility Testing:

1. NCS/ORS Joint Collection Test – to determine if and how BLS can collect NCS and ORS data from

the same establishments without diminishing data quality in any product line.

2. Central Office Collection Test – to determine how BLS will collect ORS data from America's

largest firms and State governments while balancing data quality—as measured by overall and

item level response and cost of collection—without negatively impacting NCS.

3. New Data Element Tests – to determine if and how to collect the new mental and cognitive

demands of work data elements and evaluate the use of occupational task lists as developed by

the Employment and Training Administration’s O*NET program during data collection.   

4. ORS Only Efficiency Innovations Test – to identify the most efficient way to collect the ORS data

elements tested in FY 2013 when not also collecting NCS data elements.

5. Alternative Modes Test – to determine how to collect high quality ORS data via phone, e‐mail,

or both so that BLS can balance cost and data quality.

6. Observation Test – to determine if and when data coding is changed as a result of observing the

work environment, the sampled occupation, or both.  These data will be captured during each of

the five independent tests and the Pre‐Production Test.

As BLS conducts each of these Feasibility Tests, BLS will capture the results of the data collection effort

in a database for review and analysis purposes.  All data will be reviewed for accuracy and

completeness.  BLS will use this information to develop data edits and edit ranges.  BLS will not compute

any weighted estimates from the data obtained during these Feasibility Tests, but some unweighted

tabulations may be calculated.

All data elements that were captured during Phase 3 of the FY 2013 tests as well as any additional data

elements planned for production mode or needed to evaluate these tests will be captured as part of

these tests.  Examples of additional data elements needed include establishment and occupation status

codes, establishment mode of collection, and information about observing the jobs (see Observation

Test section below).  As the tests are planned in greater detail, data elements may be added or modified

subject to resource constraints.
3 | Page

Within 60 days of completing the feasibility testing, BLS will provide SSA with a comprehensive report of

all feasibility testing conducted in FY 2014.  The BLS will discuss the results of feasibility testing with the

SSA to determine if further testing is needed to capture the cognitive and mental demands of work in a

way that meets SSA program needs.

1. NCS/ORS Joint Collection Test

The goal of the NCS/ORS Joint Collection Test is to determine if and how BLS can collect NCS and

ORS data from the same establishments without diminishing data quality in any product line.  Two

options of collecting the data jointly are to (1) collect all ORS data during initial NCS data collection

activities, i.e. initiations, which would require the move to a “global” question approach, and to (2)

collect as much ORS data as possible during NCS initiation and then collect any remaining ORS

elements during NCS update periods which occur over a three or more year time period.  For this

test, BLS will evaluate response rates to learn about potential ORS response rates and any potential

impact on NCS response rates due to dual collection of NCS and ORS data elements.  At a minimum,

BLS will look at the response rates for the ORS data elements.  If possible, BLS also will look at the

NCS item level response rates for establishments in this test and for similar establishments not in

this test.

This test will be conducted nationwide, and will include approximately 100 establishments.   These

establishments will be selected primarily from private industry.  The establishments will be a mix

current NCS reporters, both private and public sector, that are providing updated wage and benefit

data on a regular basis and private sector establishments being initiated for the first time.  The

establishments in the test will be picked to ensure a broad coverage of industries and establishment

size classes across all the regions.

Collection of ORS data elements in this test will include the efficiency protocols developed and being

tested during the ORS Only Efficiency Innovations Test described below.  Half of the collections for

this test will occur in the fall of 2013.  The second half of collection will occur in February and March

following a protocols/strategies review in January.  Evaluation of the test will take place in April,

before the start of the Pre‐Production Test in May.  There will be one debriefing session for all BLS

and SSA participants in this test for each month of test collection.

2. Central Office Collection Test

The goal of this test is to determine how BLS can collect high quality ORS data from large firms that

provide data for all establishments from a central office location, such as the corporate

headquarters of the firm.  Since some of these firms could be in the NCS sample, this collection

should be done without negatively impacting NCS products. In this test, BLS is trying to determine

whether the traditional NCS respondent can provide the data for all occupations at all

establishments or locations, whether another central respondent can provide these data, or if BLS

staff must go to the individual establishments to obtain the data.  BLS also wants to determine if and

when the responses for ORS data elements are the same for a given occupation at all establishments

in the firm and when the responses may vary by location.  For this test, BLS will evaluate potential

ORS response rates and any probable impact on NCS response rates due to dual collection of NCS
4 | Page

and ORS data elements.  At a minimum, BLS will look at the response rates for the ORS data

elements.  If possible, BLS also will look at the NCS item level response rates for establishments in

this test and for these establishments/firms in prior quarters if already participating in the NCS

sample.

This nationwide test will be conducted for private industry firms only; consisting of approximately 40

firms from wide range of industries.  For each firm selected, BLS will obtain ORS data for employees

in several establishments, representing the breadth of the firm’s operation to the extent possible.   

Test collections will start in the fall of 2013 and end in March 2014 to allow sufficient time for

collection within ORS workload constraints.  There will be one debriefing session for all BLS and SSA

participants in this test for each month of test collection.

3. New Data Element Tests

The purpose of these tests is to determine how to collect the new cognitive data elements and

evaluate the use of occupational task lists as developed by the Employment and Training

Administration’s O*NET program during data collection.  Any additional ORS data elements or major

revisions to the definitions for any FY 2013 ORS data elements will be included in this test.  The

primary goal will be to develop workable collection protocols for these new data elements.  During

these tests, BLS will only collect the data elements being tested.  BLS will not collect wages, leveling,

environmental conditions, vocational preparation information, or physical demands that were

tested during FY 2013 and are included in other Feasibility Tests.

The first collection part of these tests will be composed of approximately 25 establishments – all

located in the metropolitan Washington, D.C. area.  The second collection part will include

approximately 125 establishments – all located in a single metropolitan area other than

Washington, D.C.  Both parts of the test will include a mix of private and public sector

establishments.  The geographic area for this second part will be selected to closely mirror the

country in employment by major industry strata and have sufficient establishments to support the

test.  Collection goals will be set by industry for each of the test parts to ensure a broad coverage of

ownership and industries.     

Collections for the first collection phase should occur early in Fiscal Year 2014.    Collections for the

second phase should occur early in Calendar Year 2014.  Evaluation of the results and the

determination if any additional testing is needed will occur after the second phase of collection.   

Debriefings for these tests will be in‐person daily sessions for all BLS and SSA participants who

collected or observed a schedule.

4. ORS Only Efficiency Innovations Test

The goal of this test is to identify the most efficient way to collect the ORS data elements tested in

FY 2013 when not also collecting NCS data elements.  This may include the introduction of skip

patterns in the questioning process, new ways to ask the questions, reordering the questions,

and/or other innovations.  This test will evaluate best practices from the FY 2013 Phase 3 efficiency

test, determine needed refinements and/or new ideas for making ORS more efficient, and test those
5 | Page

refinements/new ideas.  This protocol is expected to reduce respondent burden, or the perception

thereof, while continuing to collect high quality data without a reduction in anticipated response

rates.  For this test, BLS will evaluate response rates to learn about potential ORS response rates and

any potential impact on NCS response rates due to dual collection of NCS and ORS data elements.  

At a minimum, BLS will look at the response rates for the ORS data elements.   

This nationwide test will include approximately 90 establishments.  BLS will identify the group of

establishments eligible for inclusion in this test to ensure BLS has a broad coverage of industries and

establishment size classes.   

Test collections will occur in the fall of 2013.  BLS will evaluate the results of the test in early

Calendar Year 2014 so that the protocols for the Pre‐Production Test can be finalized and

implemented in May.  There will be one debriefing session for all BLS & SSA participants in this test

for each month of test collection.

5. Alternative Modes Test   

The goal of the Alternative Modes Test is to determine how to collect high quality ORS data via

phone, e‐mail, and/or fax so that BLS can balance cost and data quality and evaluate the quality of

data obtained via personal visit versus data obtained without a personal visit.  This test is attempting

to determine if data collected via phone or e‐mail/fax matches data collected via personal visits.  As

a result, it is very important to collect one establishment from each mode from every set of

establishments identified for this test.

BLS will collect data from approximately 180 establishments.  The test establishments will be

selected from all parts of the country, with no preference to any specific geographic areas.  

Approximately ninety establishments in this test will be collected with a personal visit and

approximately 90 establishments without a personal visit but by some combination of phone, e‐

mail, and fax.

BLS will select groups of similar establishments for this test and collect at least one establishment

per mode from each group.  Each group will contain businesses located in the same geographic area

in similar industries and the same employment size class.  The group selection will maximize the

types of industries and size classes included in the test.   

Test collection period should run from February 2014 through April 2014.  There will be one

debriefing session for all BLS and SSA participants in this test for each week of test collection.

6. Observation Test

The goal of the Observation Test is to gather data to determine if and when ORS data are different

when the environment, sampled incumbents, or both are observed by a Field Economist during data

collection.  Specifically, BLS wants to know if the coded ORS data are different when the

environment and incumbents are observed as opposed to collecting data exclusively from a

traditional NCS respondent. If differences are found, BLS wants to determine if they vary by

occupation, occupational group, or other characteristic.  Field Economists will code whether an
6 | Page

observation took place, whether the observation caused them to code data differently, and

document each situation.  These data will be captured during each of the five independent tests and

the Pre‐Production Test.  BLS plans to analyze the data collected as part of this test after the

conclusion of the Feasibility Tests and again after the conclusion of the Pre‐Production Test in FY

2015.

Section III – Pre‐Production Test

At the conclusion of Feasibility Testing and upon receipt of OMB approval, the BLS will conduct a

nationwide test to evaluate all aspects of the survey in a possible production environment.  The goal of

the Pre‐Production Test is to identify any issues or problems that BLS would then resolve prior to the full

production start.  This test will be composed of approximately 2,500 establishments selected from a

nationwide sample design.  Approximately fifteen percent of these establishments will be selected from

State and local government establishments and the remainder of the sample will be private industry

businesses.

This test will mirror an ORS production environment as closely as possible.  BLS anticipates using the

sample design, collection procedures, data capture system, training processes, review processes, and

estimation methodology that are as close to those planned for production as possible at the time the

test begins.  As such, collection will likely include units that are in the current NCS sample.

This is a national test and BLS expects that the units will come from all parts of the United States and all

in scope industries.  The units in this test will be collected via personal visit that require travel away from

the field economist's home geographic area, via personal visit within the field economist's home

geographic area, and via personal contact using the phone and/or e‐mail.  The final mode of collection

used for each establishment will be chosen depending on the respondent's availability.

The Pre‐Production Test is expected to begin after the conclusion of the Feasibility Tests and end in FY

2015.  A decision will then be made to determine if BLS can collect and produce the data needed by SSA

using the NCS platform and, if so, when BLS will move into production.

Section IV – Additional Test and Research Activities

In addition to the specific Feasibility Tests discussed in Section II, BLS plans additional tests using a

usability testing service.  This service will allow BLS to test and determine quickly the best way to phrase

questions for specific ORS data elements.  While these tests will not provide data that could be used for

computing SSA requested tabulations or estimates, using this service will allow BLS to conduct a

qualitative evaluation of alternative ways to ask questions that were problematic during FY 2013 ORS

testing.  Specific test questions will be based on input provided from Field Economists and their

experiences in collecting ORS data.  No establishments will be contacted for these tests, rather,

questions will be asked of individuals identified by the usability testing organization.  Each set of test

questions may be asked from a maximum of 300 people, with each person asked up to four ORS

questions and four additional open ended questions about the wording used in the those questions.  
7 | Page

Respondents might be contacted more than once.  These tests will be coordinated and evaluated by the

BLS’s Office of Survey Methods Research.

The BLS will use data collected in testing, input from SSA, and analysis from internal BLS experts, to

research various issues identified in FY 2013.  These research projects in FY 2014 include the following:

 Sample selection, data review, estimation, validation, and other issues related to survey design;

 Development of measures of reliability of estimates;

 Measurement of survey error; and

 Resolution of how best to handle jobs whose duties and responsibilities cross multiple

occupational classifications.  

BLS will prepare a summary report to document the results of all FY 2014 Feasibility Testing and

Research with recommendations for the future. These recommendations will address the possible need

for future testing in FY 2015 beyond the previously mentioned Pre‐Production Test.