1 | P a g e

U.S. DEPARTMENT OF LABOR

BUREAU OF LABOR STATISTICS

Occupational 

Requirements 

Survey

Phase 2 Summary Report
2 | P a g e

Executive Summary 

The National Compensation Survey (NCS) recently completed Phase 2 of the Occupational Requirements Survey (ORS) 

tests conducted in cooperation with the Social Security Administration (SSA). The main objectives of this test were to 

evaluate ORS collection protocols and aids that had been updated since Phase 1 testing (e.g., streamlined collection 

tools; implementation of a probability-based establishment selection method; refined frequency questions; limited 

phone collection), and to assess ORS collection outside the DC metropolitan area using an expanded number of BLS field 

economists. The results of the Phase 2 test demonstrate the effectiveness of the revised materials and procedures and 

the continued viability of BLS collection of data relevant to the SSA’s disability program. Respondents agreed to 

participate in the test, BLS field economists were able to capture the required data from typical NCS respondents, and 

individual data element response rates were very high. 

Field collection for Phase 2 testing was conducted in the Indianapolis - Anderson - Columbus, IN and Portland - 

Vancouver - Hillsboro, OR-WA metropolitan areas from January 28, 2013 through March 21, 2013. Establishments were 

selected from the current NCS sampling frame, excluding establishments currently active in any NCS sample. Probability 

Selection of Occupations (PSO), a method of randomly selecting individual items from a larger population (in this case 

occupations), was used to determine the occupations selected for collection of detailed job requirement information. 

Twenty-four experienced field economists from BLS regional offices collected the data. More than 40 percent of the 

interviews were observed by BLS staff. The majority of interviews were done by personal visit, but a small number were 

collected by phone. 

Upon completion of the ORS collection, respondents were asked questions to gauge their reactions to the survey, and 

BLS field economists completed a post-interview debriefing. Daily debriefing sessions were held with BLS field 

economists, observers, and other staff for the purposes of discussing interviewing experiences, identifying potential 

issues with the materials and procedures, and sharing lessons learned. End-of-city debriefing sessions held in 

Indianapolis and Portland summarized the major findings identified during the test period and allowed for expanded 

discussion of these and other issues between BLS and SSA staffs. 

Background and Test Overview 

In April 2012, the Bureau of Labor Statistics (BLS) signed an interagency agreement with the Social Security 

Administration (SSA) for the purpose of designing, developing, and carrying out a series of tests to assess the feasibility 

of using the National Compensation Survey (NCS) platform as a means to accurately and reliably capture data relevant to 

the SSA’s disability program purposes. The resulting initiative – the Occupational Requirements Survey (ORS) Project –

recently completed the second of three phases of testing planned for fiscal year 2013 as outlined in the Interagency 

Agreement Deliverable sent to SSA on September 28, 2012. That document outlines the work the BLS will perform, key 

objectives, and a detailed test plan: 

In fiscal year 2013, the BLS will perform work to meet the following objectives: 

1. Evaluate survey design options and begin developing the protocols, aids, and final design to meet SSA data 

needs; 

2. Collect data to test and refine the protocols and aids; and 

3. Provide documentation to the SSA summarizing the work performed by the BLS, conclusions drawn, and 

recommendations for future data collection. 

In order to accomplish these objectives, the BLS will conduct a series of field tests with general test goals as described 

below: 
3 | P a g e

Phase 1 – Initial Proof of Concept Testing: The primary goal of this phase of testing will be to ensure that the BLS 

field economists know how to describe the survey and ask for the new data elements. In addition, the BLS will 

create and test and initial set of data collection protocols and a preliminary set of data collection aids. 

Phase 2 – Collection Protocol Testing: The primary goal of this phase of testing will be to test collection of the new 

data elements while performing a selection of occupations from each respondent. In addition, the BLS will refine 

the collection protocols and aids based on an analysis of the test results. 

Phase 3 – Broad Scale Testing: The primary goal of this phase of testing will be to test the BLS’s ability to select a 

sample of occupations within each establishment, collect the new data elements needed by SSA, and collect other 

NCS data elements that are of research interest to SSA such as wages and job leveling information. In addition, the 

BLS will test the feasibility of collecting the data needed by SSA along with all the NCS data elements needed to 

produce the Employment Cost Index, Employer Costs for Employee Compensation, and various benefits products. 

Phase 1 testing was successfully conducted in the Washington, D.C. consolidated metropolitan statistical area (CMSA) 

from November 28, 2012 through December 6, 2012. BLS staff completed interviews with 27 establishments, collecting 

detailed information on SSA data elements for 104 occupations. This report details the methods and findings of the 

Phase 2 test. Phase 3 is scheduled to start in early May. 

Phase 2 Test Objectives 

The primary goals of Phase 2 testing were to expand the number of BLS field economists that understand how to 

describe the ORS and ask respondents for information regarding the data elements, and obtain information on new 

additions to the Phase 1 collection procedures. Some key additions to the tests were: 

1. Probability Selection of Occupations (PSO) – used a disaggregating technique to select occupations 

randomly from a list of employees working for an establishment. 

2. Rarely / “Almost Never” – offered an additional frequency choice in order to better capture the 

existence of elements that occur very infrequently (approaching never). 

3. Phone Interviews – provided guidance for the use of this collection method. 

4. Dictionary of Occupational Titles (DOT) Coding – matched establishment jobs to the occupational list 

within the DOT. 

5. Work Setting – determined the work environment in which the job occurs. 

Another primary goal of the Phase 2 test was to assess the effectiveness of revised ORS collection tools. BLS staff 

created streamlined collection tools based on the results of the Phase 1 test and the Phase 2 test objectives, and 

evaluated a modified set of data collection protocols and data collection aids. The experiences of Phase 2 testing will be 

used to further refine data collection protocols and aids, analyze the collected data, revise the methods for computing 

estimates, and validate future estimates. 

Test Logistics and Methods 

Field collection for Phase 2 testing was conducted in the Indianapolis - Anderson - Columbus, IN metropolitan area 

(January 28, 2013 – February 20, 2013) and the Portland - Vancouver - Hillsboro, OR-WA metropolitan area (February 28, 

2013 - March 21, 2013). Establishments were selected from the current NCS sampling frame, excluding establishments 

currently active in any NCS sample. Probability Selection of Occupations (PSO) was used to determine the occupations 

selected for collection of detailed job requirement information. Twenty-four experienced field economists from BLS 

regional offices collected the data, and more than 40 percent of the interviews were observed by BLS staff. The majority 

of interviews were done by personal visit, but a small number were collected by phone.
4 | P a g e

BLS field economists tested three alternative collection approaches. Upon completion of the ORS collection, 

respondents were asked questions to gauge their reactions to the survey, and BLS field economists completed a postinterview debriefing. Daily debriefing sessions were held with BLS field economists, observers, and other staff for the 

purposes of discussing interviewing experiences, identifying potential issues with the materials and procedures, and 

sharing lessons learned. End-of-city debriefing sessions held in Indianapolis and Portland summarized the major findings 

identified during the test period and allowed for expanded discussion of these and other issues between BLS and SSA 

staffs. 

Key Findings 

The results of the Phase 2 test were very promising overall. 

BLS field economists completed 227 interviews representing a total of 240 establishments, collecting detailed job 

information for 1,094 occupations. Probability selection of occupations was successfully implemented in over 90 

percent of interviews. As was true in Phase 1 testing, a number of issues arose in Phase 2 that will be addressed and 

evaluated in subsequent phases of testing. These include refined guidance and protocols for: collecting job leveling 

information, physical demand strength and other frequency estimates, phone collection, and the use of clarifying 

examples. However, the consensus opinion of BLS staff is that Phase 2 objectives were successfully met, and these 

activities lay a strong foundation for future tests. 

Phase 2 training consisted of self-study (background readings on the SSA disability program and relevant ORS concepts), 

web-based and in-person instructions provided prior to test collection in each city, and on-the-job training (precollection practice interviews, debrief sessions, and informal field economist exchanges). The training approach used in 

Phase 2 worked well and will be repeated in subsequent tests. The use of practice interviews prior to data collection 

and the use of debriefing session where BLS field economists, observers, and other staff could exchange information and 

suggestions about collection issues were particularly successful. 

The materials prepared to aid ORS cooperation were very well received, particularly the two-sided ORS flyer with the 

SSA letter on the back, the Zooming in on Compensation Statistics booklet, the city-specific economic summary sheets, 

and the EA&I contact information. 

BLS field economists expressed a strong preference for the collection approach that asked respondents for information 

about each ORS element for all selected occupations before moving to the next element. Respondents and field 

economists had very positive reactions to the collection aids, but recommended that a corpus of standard examples be 

developed to help clarify ORS concepts for respondents and ensure testing consistency. 

Most data were collected from typical NCS respondents such as human resource directors, small business owners, and 

location managers. 

Cooperating establishments were able to answer the vast majority of questions asked in the survey, resulting in a very 

high item response rate. This was true of both personal visit and phone interviews. However, as in Phase 1, there were 

certain words and concepts that proved challenging to administer effectively. Respondents continued to vary in their 

interpretations of the term required, and to have difficulty with the concept of average performance. Some respondents 

struggled to provide frequency estimates for tasks that were short but repetitive, or which varied from day to day or 

between employees in a given occupation, and for nonstandard works shifts (e.g., 12 hour shifts). The issue of 

accommodations and how to explain why they are excluded in ORS was consistently discussed in the daily debriefs. One 

successful approach used by a field economist was to explain that the focus of the survey is the job and not the 

individual doing the job, and that individual accommodations at an establishment should not be collected. Field 

economists said that it would be helpful to be able to clearly explain why they are being excluded in a manner that is 

acceptable to all parties.
5 | P a g e

Phase 2 Test Methods and Materials 

Field collection for Phase 2 testing was conducted in the Indianapolis - Anderson - Columbus, IN metropolitan area 

(January 28, 2013 – February 20, 2013) and the Portland - Vancouver - Hillsboro, OR-WA metropolitan area (February 28, 

2013 - March 21, 2013). 

Selected Establishments 

Establishments were selected from the current NCS sampling frame for businesses in the two metropolitan areas, 

excluding all establishments currently active in any NCS sample but including some units recently rotated out of NCS 

production. The target number of completed interviews for Phase 2 testing was 250 establishments (125 per test city) 

representing as broad a mix of industries as possible given the size and time constraints of the test. At the conclusion of 

the testing period, BLS completed 120 interviews in each test city (see the Results section for more details about 

participating establishments). 

Field Economists 

Phase 2 data were collected by 24 experienced NCS field economists from BLS regional offices. Eight senior field 

economists who had participated in Phase 1 testing also collected in both Phase 2 test cities, and served as mentors to 

the field economists new to ORS. Interviewer training for Phase 2 collection followed regular BLS practices, and 

consisted of five components: 

 Self study – field economists were provided the SSA background materials, ORS high-level test plans, and 

element definitions. 

 Webinars – In the week prior to the start of test data collection, field economists participated in web-based 

training sessions in which they were in given an overview of the SSA disability process, and information about 

the ORS test objectives, procedures, and materials. There were two webinar training sessions prior to 

Indianapolis collection and one prior to Portland collection. The imbalance in webinar sessions between test 

cities was offset by an additional day of in-person training in Portland). 

 In-person training – This occurred during the field economists’ first days in the test city. In-person training 

consisted of: review of the technical memorandum; practice with the data capture spreadsheet; an overview of 

instrument edits and data analysis objectives; a calibration exercise; mock interviews; and a mock interview 

debrief session. 

 Mentoring - In addition to formal training, each field economist new to ORS was assigned a mentor who 

collected in prior ORS testing. The mentor served as the field economists’ primary resource for technical 

advice. Mentors made all first-week appointments for the mentee. Mentees observed mentors conducting two 

interviews prior to conducting their own interviews, and then were themselves observed by their mentor on 

their first two interviews. Mentors also reviewed the mentees’ write-up of their initial schedules. 

 On-the-job training (OJT) – Throughout the Phase 2 Test fielding period, field economists engaged in a number 

of activities designed to reinforce formal ORS training concepts. During data collection, OJT was provided 

through formal daily debriefing sessions and informal conversations between field economists and project staff 

in which lessons learned, best practices, and challenging issues were identified. 

Refinement Procedures 

Field economists were provided with their collection assignments prior to their arrival in the test city, and immediately 

began the refinement process (e.g., using Internet searches, phone calls) to validate the assigned establishments. Only 

establishments that could provide data locally (i.e., in Indianapolis or Portland areas) were considered in scope. Field 

economists were instructed to proceed to the next establishment if data had to be procured from outside the test area 
6 | P a g e

(e.g., from a corporate office in another city), or when refinement proved problematic after reasonable effort had been 

made. 

To secure appointments, field economists contacted potential respondents and attempted to speak with company 

personnel with the same job titles that NCS traditionally works with (e.g., HR Directors, Personnel Managers, and Hiring 

Officials). If those individuals were not available, the field economist worked with the establishment contact to identify 

the person most familiar with the target data elements to interview. When the appropriate company contact was 

located, the field economist used a standardized script or “spiel” to explain the purpose and importance of the survey, 

and attempted to schedule an appointment. Some of the field economists sent e-mails to establishment contacts to 

provide additional background information about the purpose of and procedures for the interview. All potential 

respondents were informed that the data collection effort was part of a test being done at the request of the Social 

Security Administration and that participation was voluntary. 

When contact could not be made with listed establishments, or the establishment points of contact expressed 

reluctance or indicated that they were unavailable during the test field period, the field economists were instructed to 

forego the normal NCS conversion attempts and simply advance to the next establishment on their assignment list. This 

process continued until each field economist had secured their allotted appointments. 

Occupational Selection for Responding Units 

The Phase 2 tests evaluated the use of NCS Probability Selection of Occupations (PSO) procedures as an occupational 

selection tool. Field economists were instructed to use standard NCS coverage and procedures for PSO as much as 

possible. They attempted to obtain an employee list, refine the list based on NCS inclusion/exclusion criteria, and then 

select the appropriate number of occupations for collection. Field economists selected 8 occupations for the largest 

establishments (i.e., those with more than 250 employees), and fewer occupations for smaller establishments (e.g., 4 

occupations for companies under 50 people). If the respondent was unwilling or unable to do formal PSO, a fall-back 

selection procedure was used. Field economists attempted to identify the most highly populated occupations in the 

establishment and then select between four and eight of those that also spanned different major SOC classifications. 

PSO could be done at the start of the interview or in advance. 

Standard NCS worker characteristics (e.g., full-time/part-time, union/non-union, time/incentive, supervisor/lead worker 

information, etc.) were collected for all selected occupations. These characteristics were used to help identify unique 

occupations. 

Data Collection Tools and Protocols 

Three paper data collection tools were tested in Phase 2. Each version was designed to collect the required data 

elements: Job Details (e.g., title, occupational employment size, SOC/DOT code, worker characteristics, and work 

environment); Specific Vocational Preparation (SVP); Physical Demands (PD); and Environmental Conditions (EC). The 

tools differed in their visual format, question flow, and collection method. 

 A Single Quote Tool was designed to obtain the occupational requirements one occupation at a time. Definitions 

for select terms were provided at the bottom of the page. 

 A Multiple Quote Tool was designed to collect the occupational requirements of all selected jobs 

simultaneously. Field economists were instructed to ask each item about all occupations before proceeding to 

the next item. 

 An Advanced Multiple Quote Tool was developed for small-scale feasibility testing in Portland. This tool 

changed the order in which the elements were asked (e.g., asking Environmental Conditions questions prior to 

Physical Demand items), and was designed to allow field economists to first determine whether each job 

required certain characteristics or not (e.g., does the job require the worker to be exposed to extreme heat or 

not?), and then to go back and administer follow-up questions only for eligible occupations (e.g., how often does 

the job require the worker to be exposed to extreme heat?). 
7 | P a g e

Based on feedback received from field economists and observers during Indianapolis data collection, the collection tools 

were modified prior to the start of Portland collections. 

o An item was added to the SVP element that asked if there was a literacy requirement for the job. 

o The ‘reaching’ items in the PD element were reordered such that the “any reaching” question was asked 

prior to the directional reaching questions. 

o The frequency questions for ‘lifting/carrying’ items in the PD element were reversed (i.e., “constantly” 

was asked first, followed by “frequently” and “occasionally”).

o The order of the crawling, crouching, kneeling, and stooping items were reversed to ask first about 

stooping, then crouching, kneeling, and crawling. 

Use of the Advanced Multiple Quote tool was restricted to limited testing by senior BLS field economists in Portland. For 

all remaining Phase 2 interviews, the decision of whether to use the Single Quote tool or standard Multiple Quote tool 

(or combination thereof) was left up to the individual field economists, though the Multiple Quote tool was 

recommended for schedules in which it appeared likely that many of the selected occupations would have similar 

requirements. Field economists were instructed to read the questions as worded on the tool for testing consistency, but 

were permitted to provide additional explanations and ask follow-up questions as necessary. 

In addition to these collection forms, a number of aids were developed to help clarify the survey task and definitions. 

Respondent visual aids provided frequency definitions and examples for the noise intensity level response categories 

and of the type of frequency and weight information that respondents should consider when answering the 

lifting/carrying items. Field economist aids provided definitions and clarifying examples of ORS elements and answers to 

commonly asked questions. Field economists were encouraged to consult these aids during the interview and to share 

them with respondents as necessary. 

Both in-person and remote data collection were tested in Phase 2. The preferred method of collection was personal 

visit; if the respondent indicated a willingness to provide information by personal visit and another method (e.g., phone, 

email), it was mandatory to select personal visit. In the event that a respondent refused or was unable to schedule an 

in-person appointment, however, field economists were permitted to collect information over the phone or by email. In 

addition, a small group of schedules was assigned to phone collection conducted by senior BLS field economists in each 

Phase 2 test city. Remote collection protocols were very similar to those for in-person collection. Field economists 

were required to use the collection tools, collect all of the ORS Phase 2 elements, ask questions as worded, and probe 

unclear answers or in situations in which respondents’ answers did not match expected job patterns. They also were 

encouraged to frequently refer to the field economist aids to provide more detailed definitions, explanations, and 

examples to the respondent as needed. 

Data Capture and Review 

Data from the Collection Tools and Establishment Information sheet were entered into a data capture spreadsheet on a 

flow basis. The tool was designed to permit easy data entry by field economists, the ability to review the captured data, 

and capacity to tabulate results. Table 1 lists the data elements collected in the capture instrument. 

Review parameters were developed for Phase 2 and were used to evaluate data elements for internal 

consistency. Specifically, three internal consistency spreadsheets (one for each SSA data element area – SVP, PD, EC) 

were used in the data review and analysis. The parameters identified expected relationships between the individual 

ORS data elements, as well as consistencies between ORS and NCS data elements. 
8 | P a g e

Table 1. Information Recorded in the Phase 2 Test Data Capture System 

Element Items Captured

General Information

Schedule number; establishment employment size; 

NAICS code; selected occupations (title, SOC, 

employment, FT/PT, union/nonunion, incentive/time, 

supervisory status, work environment, job description)

Specific Vocational Preparation (SVP) 

Title; SVP; SVP level; education level; training and 

certifications; time to average performance; literacy 

requirement

Physical Demands (PD)

Standing/walking/sitting; reaching; lifting/carrying; 

pushing/pulling; gross manipulation/fine manipulation; 

keyboarding/words per minute; hand arm controls/foot 

leg controls; 

climbing/stooping/kneeling/crawling/crouching;

visual acuity; speaking; noise intensity level; loud 

noise/moderate sound discernment

Environmental Conditions (EC)

Extreme cold/extreme heat; wetness/humidity; 

vibrations; fumes/odors/gases/poor ventilation;

toxic caustic chemicals; high exposed places; moving

mechanical parts

Observers 

To help garner feedback about the interview and data collection processes, an observer accompanied the field 

economists on approximately forty percent of the data collection appointments. The observers represented a mix of 

roles and stakeholders within the ORS Project (e.g., NCS/ORS management; BLS Assistant Regional Commissioners and 

Branch Chiefs; ORS mentors; BLS survey methods and operations staff), with priority given to those individuals serving 

on the ORS Project. An attempt was made to have each observer observe three collection interviews. 

The observer did not participate directly in ORS collection, but was responsible for taking notes about the collection 

process. They noted the duration of the interview (and the time needed to administer sections within the interview), 

and their observations for each item and the interview as a whole. 

Other Debriefing Activities 

In addition to the Observer tool(s), the following debriefing activities were conducted to assess the effectiveness of 

Phase 2 ORS materials and procedures: 

 Respondent and Field Economist Debriefing – at least 10 minutes were set aside at the end of each ORS 

interview to ask respondents about their interview preparation and experiences. This debrief was 

administered either by the field economist or the observer, and consisted of a brief set of questions targeting 

potential comprehension issues, perceptions of burden, reactions to the visual aids, etc. In addition, field 

economists were asked to complete a post-interview questionnaire about their impressions of respondent 

comprehension issues and the effectiveness of the interview process and materials. 
9 | P a g e

 Daily Collection Debriefs – each day during the Phase 2 test in which there was a collection, an in-person 

debriefing session was held to assess the day’s interviews, discuss problems or questions that arose, and share 

ideas for best practice solutions. All available field economists and observers who had participated in that day’s 

collection interviews attended these daily debrief meetings. Others involved in the ORS project attended, as 

available. 

 City Debriefs – city debriefs were held in Indianapolis on February 8, 2013, and in Portland on March 7, 2013. 

The purpose of these sessions was to summarize key findings from the Phase 2 tests in the respective cities 

based on all the information collected to date through the various assessment activities, to solicit additional 

feedback about components of the test, and begin to identify issues and recommendations relevant to future 

ORS test collections. These meetings were attended by all available ORS Project staff and by SSA sponsoring 

officials. 

 End-of-Phase Questionnaire – field economists were asked to complete a brief questionnaire at the end of 

Phase 2 testing. The purpose of the questionnaire was to collect feedback on aspects of the Phase 2 test that 

were not covered in depth by the other debrief activities (e.g., the effectiveness of Phase 2 training, calibration 

exercises, use of the data capture tool, and debriefing sessions). 

Phase 2 Test Results 

This section reports on results compiled throughout the Phase 2 test. They reflect information obtained from each 

debriefing component and feedback collected from observers and field economists during the collection and city 

debriefing sessions. Although the information garnered from each of these components is summarized here and not 

reported individually by source as was true in the Phase 1 summary, each source contributed significantly to the overall 

Phase 2 findings and conclusions. This section begins with a broad assessment of the feasibility of collecting the data 

needed by SSA through the ORS, and then provides an overview of the collection effort along with more detailed 

descriptions of issues that arose during the test. 

Feasibility 

The main objective of the development and evaluative work that will be done for the ORS project in fiscal year 2013 is to 

assess whether it is feasible for BLS to collect data relevant to the SSA’s disability determination program. The results of 

this collection protocol test suggest that the collection of the ORS data elements using a probability selection of 

occupations is viable. In Phase 2, BLS field economists completed interviews with 227 respondents representing a total 

of 240 locations. New in Phase 2 was the use of probability selection of occupations to determine on which jobs to 

collect detailed job requirement information. Probability selection of occupations was completed in approximately 92% 

of the interviews while the remainder used the same process as Phase 1 testing. These processes combined resulted in 

1,094 occupations for Phase 2 testing. As was true in Phase 1 testing, a number of issues arose in Phase 2 (e.g., with 

certain data elements, aspects of the collection and assessment, etc.) that will be addressed and evaluated in 

subsequent phases of testing; many of those are reflected in the sections that follow. However, the consensus opinion 

of BLS staff is that Phase 2 objectives were successfully met, and these activities lay a strong foundation for future tests. 

Participating Establishments 

Table 2 provides additional details about the establishments that participated in Phase 2 and the occupations for which 

ORS data were collected. As can be seen, BLS staff continued to be successful in securing interviews from a variety of 

industry groups and collecting data for a range of occupations in an extremely tight fielding period. 

Data Collection Assignments and Appointments 

Field economists’ pre-collection activities continued to be very similar to NCS and Phase 1 testing. As was true in Phase 

1, efforts to turn around reluctant respondents were not required. Field economists were instructed to move on to the 

next establishment on their list in the event that they encountered reluctance. 
10 | P a g e

 

Indianapolis field economists were given their assignments one week prior to arrival in the test city, and during 

debriefings indicated that this did not allow them enough time to make appointments. Instead of coming into the test 

city with most of the appointments secured, Indianapolis field economists reported that they had to juggle making 

appointments with other week-one test activities such as training and mentoring appointments. In response to this 

concern, Portland field economists were given their assignments two weeks before they arrived in the city. This 

improved their ability to make appointments before formal training and mentoring appointments began, and an effort 

will be made in Phase 3 to provide field economists their assignments as early as possible. 

Field economists thought that having their assignments organized geographically was helpful. Not only was it efficient in 

terms of scheduling appointments in the same general proximity within the metropolitan area, but it also allowed those 

who were unfamiliar with the area to gain familiarity with at least a portion of the metropolitan area. 

When making assignments where some ownership/industry sectors were limited in the number of establishment units, 

field economists often found that sectors closed for collection before they were able to fully work through their assigned 

units. This caused the need for additional switching of units to provide enough opportunities for field economists to 

reach their individual collection goals, and in turn, caused a delay in their ability to schedule appointments. 
11 | P a g e

Table 2. Description of Establishments and Selected Occupations for Phase 2 Testing 

Ownership Industry 

Group

# of Participating Units Employment 

Size

(range)

# of Jobs 

Collected

Jobs Collected

(selected) Overall By City

Private 

Industry

Education 43

Indianapolis = 19

4 – 522 165

 Teachers (All grade levels; Special Ed)

 Tutors

 Self-Enrichment Education Teachers

 Teacher Assistants

 Secretaries & Administrative 

Assistants

 Vocational Education Teachers, 

Postsecondary

Portland = 24

Financial 

Activities 33

Indianapolis = 16

1 – 1200 143

 Insurance Sales Agents

 Insurance Policy Processing Clerks

 Loan Officers

 Computer User Support Specialists

 Tellers

 Advertising Sales Agents

 Customer Service Representatives

Portland = 17

Goods 

Producing 39

Indianapolis = 21

1 – 717 187

 Molding, Coremaking and Casting 

Machine Setters, Operators and 

Tenders, Metal & Plastic

 First-line Supervisors of Production 

and Operating Workers

 Industrial Production Managers

 Inspectors, Testers, Sorters, Samplers 

& Weighers

 Mechanical Engineers

 Customer Service Representatives

 Painters, Construction & 

Maintenance

 Assemblers & Fabricators, All Other

Portland = 18

Healthcare 40

Indianapolis = 23

4 – 1400 189

 Registered Nurses

 Nursing Assistants

 Personal Care Aides

 Medical Secretaries

 Medical Assistants

 Child, Family & School Social Workers

 Dental Assistants

Portland = 17

Service 

Producing 39

Indianapolis = 19

3 – 675 156

 Waiters & Waitresses

 Heavy & Tractor-Trailer Truck Drivers

 Sales Representatives, Wholesale & 

Manufacturing except Technical & 

Scientific Products

 Laborers and Freight, Stock and 

Material Movers, Hand

 Refuse and Recyclable Material 

Collectors

 First-Line Supervisors of Retail Sales 

Workers

 Retail Salespersons

 Office Clerks, General

Portland = 20

Total 194

Indianapolis = 98

1 – 1400 840 Portland = 96
12 | P a g e

Table 2, continued 

Ownership Industry 

Group

# of Participating Units Employment 

Size

(range)

# of Jobs 

Collected

Jobs Collected

(selected) Overall By City

State and 

Local 

Government

Education 9

Indianapolis = 5

92 - 1809 52

 Teacher (All grade levels; Special Ed)

 Teacher Assistant

 Janitor & Cleaner

 Secretary & Administrative Assistant

 Bus Driver

 Dining Room & Cafeteria Attendant

Portland = 4

Financial 

Activities 13

Indianapolis = 7

4 - 570 64

 Librarian

 Library Technician

 Library Assistants, Clerical

 Usher, Lobby Attendant & Ticket 

Taker

 Janitor & Cleaner

Portland = 6

Goods 

Producing 4

Indianapolis = N/A

16-91 17

 Landscaping & Groundskeeping 

Worker

 General & Operations Manager

 Electrician

 Production, Planning & Expediting 

Clerk

 Heating & Air Conditioning 

Mechanics & Installers

Portland = 4

Healthcare 10

Indianapolis = 5

6 - 4063 64

 Regulatory Affairs Specialist

 Office Clerk, General

 Social & Human Service Assistant 

 Registered Nurse

 Probation Officers & Correctional 

Treatment Specialist

Portland = 5

Service 

Producing 10

Indianapolis = 5

38 - 4530 57

 Police Patrol Officer

 Office Clerk, General

 Construction & Building Inspector

 Municipal Firefighter

 Childcare Worker

Portland = 5

Total 46

Indianapolis = 22

4 - 4530 254 Portland = 24

Marketing 

For Phase 2, BLS assembled the following set of marketing items: ORS one-page flyer with a copy of the letter that SSA 

sent to BLS; a one-page ORS flyer with reference to the SSA website; the BLS Customer Service Guide; a Zooming in on 

Compensation Data booklet; Indianapolis and Portland Area Economic Summary sheets; the most recent ECI/ECEC/EBS 

releases; and business cards listing the contact information of the Economic Analysis and Information (EA&I) units for 

the Chicago and San Francisco Regions. BLS also created two sets of letters aimed at respondents in Phase 2 testing - 

one targeting government units and the other targeting private sector units. In addition, field economists used the SSA 

Occupational Information System Project (OIS) and the BLS Occupational Requirements Survey websites to inform 

respondents about the ORS collection efforts. 

The Phase 2 debriefing results demonstrate that field economists especially liked the two-sided ORS flyer with the SSA 

letter on the back. The other materials that were widely used by field economists were the Zooming in on 

Compensation Statistics booklet, the Indianapolis/Portland Economic Summary sheets, and the EA&I contact 

information. There was not widespread use of the ORS regional letters or respondent-specific marketing packets. This 
13 | P a g e

likely was due to the nature of the ORS assignment and the Phase 2 test study; field economists had a short window to 

make appointments and were instructed to move to another unit if the first few attempts were unsuccessful. For 

collection in Phase 3 and beyond, use of the respondent letters and tailored marketing packets may increase as fuller 

recruitment efforts are implemented. 

Interview Duration 

Based on the approved testing plan submitted to the Office of Management and Budget (OMB), field economists were 

allotted one hour to collect the ORS data and conduct the respondent debriefing in each establishment. On average, 

data collection took 53 minutes per schedule, and data was collected on 4 occupations at each establishment. The 

average occupation took 13.4 minutes to complete. PSO took an average of 5 minutes to complete when not done in 

advance. Job details and education and training took an average of 16 minutes to complete, or 4 minutes per 

occupation. Physical demands took the longest to complete of the sections, at an average of 26.8 minutes per schedule 

or approximately 6 minutes per occupation. Environmental conditions took 4.7 minutes to complete on average. The 

post-interview debrief averaged 2.5 minutes. 

Collection Protocols 

The Collection Approaches 

The most popular choice among the three collection approaches tested in Phase 2 was the Multiple Quote method in 

which field economists collected data about each element for all the selected jobs at one time before moving on the 

next element. BLS field economists reported that this approach seemed to provide the best balance between collecting 

quality data and reducing respondent burden, and they liked that the definitions of the various elements were 

incorporated directly into the collection tool. Feedback from the senior BLS field economists who used the Advanced 

Multiple Quote method were similarly positive, with some staff indicating that they would like to use this tool 

exclusively in future collection efforts. The approach of asking all of the ORS data elements in their entirety for a single 

selected occupation before moving on to the next occupation received positive feedback in limited situations, such as 

small companies with one or two selected occupations. However, most field economists felt that the single-quote 

approach did not result in an efficient interview, and produced a higher level of respondent burden when more than one 

or two occupations were selected. When given a choice of which tool to use (as opposed to being required to use each 

tool equally, as in Phase 1), staff almost universally opted not to use the single-quote tool. 

The Spiel 

The spiel consisted of written introductory language designed to aid ORS cooperation, as well as a series of bullet points 

covering topics that field economists were required to address when initially contacting selected establishments. The 

Phase 2 spiel incorporated a number of changes based on recommendations made during Phase 1, and field economists 

said that these changes resulted in significantly improved interactions with respondents and greater chance of securing 

cooperation. 

Respondent Visual Aids 

Results from the field economist and the respondent debriefings revealed nearly universal agreement that the 

respondent visual aid (with concept definitions and frequency scale) was well liked and effective. All of the respondents 

who saw the visual aid reported that it was helpful. The multi-colored weight chart, on the other hand, proved difficult 

for respondents to understand and for field economists to explain. Comments such as “the multi-colored weight chart 

did not work visually” were fairly common in the debriefings.

Field Economist Reference Guide 

This document essentially served as technical manual that field economists could consult during the interview. It 

contained definitions of the factors, as well as some examples pulled from the Revised Handbook for Analyzing Jobs 

(RHAJ). Staff working in Phase 2 found it to be very helpful, and referred to it quite frequently during interviews. The 

main shortcoming that field economists found with this document was that it had insufficient examples of the 

conditions and factors that one encounters in a modern work environment. Most of the examples either were 
14 | P a g e

production related or so specific that they had minimal relevance to the respondent’s actual work situation. Field 

economists recommended that more effort needs to be devoted to finding more common, contemporary, and 

understandable examples for each factor. Other project participants noted that while examples have the power to 

quickly crystallize a vague concept for respondents and to help with recall, they may have a biasing effect if not 

implemented with care and consistency. 

Phase 2 BLS Field Economist and Observer Training 

Phase 2 test collections allowed for further development and refinement of the training procedures for staff new to 

ORS. As the training staff gained greater insight into the ORS data collection process, they were able to create training 

materials that helped get new staff up to speed more efficiently, as reflected in field economists’ responses to items on 

the end-of-phase questionnaire (Tables 5 and 6). 

Table 5. The Field Economist Training Adequately Prepared Me for Phase 2 Collection – Percent Agreement

 Strongly Agree 6.7%

 Agree 80.0%

 Neither Agree nor Disagree 6.7%

 Disagree 6.7%

 Strongly Disagree 0.0%

Table 6. Field Economists’ Overall Rating of Phase 2 Field Economist Training

 Excellent 40.0%

 Good 46.7%

 Fair 13.3%

 Poor 0.0%

Beyond these general reactions, field economists and observers were largely positive about their experiences in a 

number of the individual components of Phase 2 training. They reported that the pre-collection practice (or ‘mock’) 

interviews gave them the opportunity to rehearse using the collection and observer tools, and gave them confidence 

that BLS staff could ask the questions as intended and collect high-quality data pertinent to the SSA elements. The daily 

debriefing sessions continued to be an excellent training tool, as well. The semi-structured setting once again provided 

an exchange of information which allowed for training and clarifications of materials already provided. Finally, field 

economists and observers emphasized the value of other informal, on-the-job learning opportunities. These on-going 

conversations helped to solidify lessons learned, best practices, and to identify areas in need of further development. 

Data Collection 

Data collection for Phase 2 once again proved to be a valuable learning experience for field economists, as it did in Phase 

1. Over the course of collecting data in the Phase 2 cites a number of recurring issues began to emerge. As has been the 

case since the beginning, reaching and weight were the two data elements that posed the most problems, but some 

additional areas of confusion began to surface as well. 

Respondents 

Field economists’ experience administering the Phase 2 ORS items and respondent debriefs provided valuable insight 

into respondents’ understanding of the ORS concepts and their reactions to the survey experience, and it allowed field 

economists to develop better and more consistent probes when answers appeared inconsistent. As in Phase 1, the term 

require (e.g., “...does this job require the worker to...”) continued to be confusing for some respondents in Phase 2. 

Phase 2 introduced language intended to clarify this concept (e.g., how the job is generally performed; core duties of the 

occupation), but respondents varied in their interpretations of these descriptions, as well. For example, one respondent 

said that a core duty of a job was not required if another person in the occupation was able to perform the task. 
15 | P a g e

The concept of average performance also continued to challenge respondents. Field economists continued to provide 

the given definition and description of the term. Some respondents said that for jobs requiring prior experience, 

incumbents were required to be satisfactory on day one, while other respondents wanted to use the probationary time 

as the indicator. 

The issue of accommodations and how to explain why they are excluded in ORS was a frequent topic in the daily 

debriefs. One successful approach used by a field economist was to explain that the focus of the survey is the job and 

not the individual doing the job, and that individual accommodations at an establishment should not be collected. 

Although it proved easy enough to explain that accommodations should be excluded, field economists said that it would 

be helpful to be able to clearly explain why they are being excluded in a manner that is acceptable to all parties. 

Respondent Reaction to ORS 

Respondents generally were very receptive to ORS, and most said that they felt it is a worthy endeavor. The following 

bullets list some examples of typical respondent comments made during the respondent debrief: 

 The survey is appropriate; there is an obvious need for this study. 

 This is a great undertaking. 

 The SSA test is a good idea and I like the idea that SSA is assessing their disability program. 

 It made me think about and review my own jobs and what I should add or change to my job descriptions. 

 I like the idea that SSA is considering a disability assessment. 

In some instances, however, respondents had more pointed observations about the survey: 

 The shorter the survey the better for me, even though I understand you need to get your data. 

 It is hard to average out the work in a job as it varies day by day. 

 It was difficult to know all aspects of the job (e.g., some of the fine details about physical demands aren't so 

well known).

 I found it hard to distinguish between requirements and accommodations. 

In anticipation of Phase 3 testing, Phase 2 respondents were asked questions during their debriefing about providing 

ORS data through alternate methods of collection, and their ability to provide wage information for the selected 

occupations. As the results presented in Table 8 (below) make clear, most respondents felt that some form of remote 

collection of ORS data would be feasible, and that they would have been able to provide wage data about employees in 

the selected occupations. 

Table 8. Respondents’ Responses to Questions about Mode Preference and Wage Data Availability

Do you think we could have collected this information through email or a phone interview?

 Yes 72%

 No 20%

 Maybe 9%

Would we have been able to obtain current wage rates for all of the employees in each of the discussed job 

titles from you?

 Yes 87%

 No 6%

 Other 7%

Field Economists 

Phase 2 field economists in most cases felt that the collection steps for the ORS were very similar to the NCS program, 

and the refinement of the sample units typically followed the same process as in NCS. This was reflected in daily debrief 

discussions as well as field economists’ responses to items asked in the end-of-phase questionnaire (see Table 7). 
16 | P a g e

Table 7. Field Economists’ Responses to Debriefing Questions about Phase 2 Pre-collection Activities 

Were the refinement, contact and collection any different for this establishment than it would have been 

for an NCS schedule?

 Same as NCS 96%

 Different than NCS 1%

 Other 3%

How long did it take you to research the establishment and make the appointment?

 Less than 15 minutes 18%

 15-30 minutes 61%

 30-60 minutes 15%

 More than 60 minutes 6%

Debriefing Activities 

Phase 2 debriefing activities remained unchanged from Phase 1 (e.g., interview observations; completion of observer 

and respondent debrief, and post-interview debrief tools; attendance at the daily and end-of-city debriefing sessions). 

Of the 120 interviews conducted in Indianapolis, 60 had observers; in Portland 42 of 120 interviews were observed. 

The daily debrief sessions were attended by ORS project staff – including all field economists (unless they were on an 

interview), observers, project staff, and managers – and the discussions again were active and productive. The end-ofcity debriefing sessions were attended by all available ORS Project staff and managers, with representatives from the 

SSA attending via phone. 

Each of the Phase 2 debriefing activities remained useful, albeit with some appearance of diminishing returns towards 

the end of collection as trends solidified and less new information emerged. The respondent debriefings and daily 

debriefing sessions were the most crucial assessment elements, offering insights into issues that will need to be 

addressed and further evaluated in Phase 3 or fiscal year 2014 (e.g., the optimal order of ORS elements; the use 

clarifying examples; and the length and repetitiveness of the survey/respondent burden). 

The in-house debriefing sessions were larger than they had been in Phase 1, and often there many more interviews that 

needed to be discussed. In Indianapolis, the sessions were structured around collection appointments, with field 

economists and observers sharing their interview experiences, identifying potentially problematic items or procedures, 

and exchanging approaches that they found helpful in clarifying meaning and securing collection goals. For Portland, the 

decision was made to alter the way in which the meetings were structured, moving from a specific interview-focused 

approach to an industry-focused approach that allowed BLS staff to more efficiently identify survey trends across 

different industries. The end-of-city debriefings provided the opportunity for BLS staff to summarize their Phase 2 

findings, and for SSA to ask questions about the Phase 2 test changes and to get feedback from field economists about 

what was and was not working. As in Phase 1, significant problems were identified as well as smaller issues that need 

adjusting, but there was consensus that the ORS materials and procedures significantly improved between the two test 

phases and throughout Phase 2. 

Phase 2 observers reported that the observer tools remained useful for recording timing information and noting 

problems that occurred during the interview, but the utility and efficiency of those tools could be improved. 

Suggestions included providing additional space for writing notes, requiring less item-by-item notation, and formatting 

the tool as a booklet with major elements on one page (for reference) and a notes page facing it. Most of the field 

economists favored the Multiple Quote collection tool, so the multi-quote observer tool was used most often. For 

Phase 3, BLS plans to explore a redesign of the observer tool to make it easier to keep pace with the interview. 
17 | P a g e

Data Capture System 

The data capture excel spreadsheet was once again used in Phase 2. It was an updated version of the spreadsheet used 

in Phase 1. Some changes from Phase 1 collection included: 

 Edits were added to the system, prompting field economists to review their coding and provide additional 

documentation when coding is unusual for a specific job. 

 Each element had don’t know, no response and never added as an option. 

 The 9-digit DOT code and the 8-digit SOC code were required using the proper numerical formatting. The 

system provided a prompt if there is an error and would not allow the incorrect format to be entered. The 

formatting error is true for all other elements as well. 

 The updated spreadsheet included an area to indicate the supervisory level of the job. 

The use of the spreadsheet proved to be a functional option in Phase 2 while work on the data capture system 

continued, but field economists had somewhat mixed views about the usability of this tool and the perceived value of 

the new Phase 2 edits. 

Table 9. Field Economists’ Responses to Questions about Phase 2 Data Capture Tool

The Phase 2 data capture tool (spreadsheet) was easy to use.

 Strongly Agree 6.7%

 Agree 33.3%

 Neither Agree nor Disagree 33.3%

 Disagree 20.0%

 Strongly Disagree 6.7%

The edits helped identify situations where the data may have been inconsistent or incorrect.

 Strongly Agree 0.0%

 Agree 40.0%

 Neither Agree nor Disagree 46.7%

 Disagree 13.3%

 Strongly Disagree 0.0%

Work continued towards a Phase 3 Data Capture system during Phase 2 testing. Acceptance testing was conducted by 

BLS. At the end of Phase 2, the first version of the production data capture application was deployed and used in a 

parallel test along with the data capture spreadsheet. Staff were to report back any issues or suggestions that they have 

with the system in order to improve it for Phase 3. A tracker was implemented to identify system bugs and issues. The 

new system was shown in several demonstrations throughout Phase 2 to demonstrate progress and request feedback 

from staff. 

Data Analysis 

BLS staff were tasked with preparing edits to determine the validity and accuracy of the data gathered in Phase 2 as well 

as creating calibration activities to help ensure inter-coder reliability in data collection. 

Calibration Activities 

One activity was conducted in each of the two test cities prior to the beginning of collection to help ensure that different 

field economists code the same specific vocational preparations, physical demands and environmental conditions for a 

job the same way during a collection interview. This also served to augment the training provided to the field 

economists by emphasizing best practices developed throughout the previous test phase. 

Each field economist participating in collection for the city was given a dental hygienist job description and a single 

quote data collection tool. Field economists were instructed to code all elements in which they felt there was sufficient 
18 | P a g e

supporting documentation in the job description. They also were asked to make notes in places in which they thought 

there was enough information to code the data element but would be more comfortable if they could ask additional 

questions of the respondent. In such instances, the field economists also were asked to note the specific questions they 

would ask. Additionally, field economists were asked to note instances in which they could determine the existence of a 

required element but would be unable to determine the frequency. 

Once all participants completed their coding based upon the job description, facilitators led the group through a 

discussion that looked at each question on the collection tool as well as the answers arrived at by the field economists 

and why they gave each particular answer. Additional discussion was generated by the facilitators once the targeted 

answers were revealed so as to emphasize core collection concepts such as what is required for the job as opposed to 

just preferred by the employer, the threshold which needs to be reached for certain elements to be coded as present, 

and activities that are considered incidental to the job and should not be coded. An answer key with the target answers, 

the job description point of reference supporting the targeted answer as well as the DOT coding was provided to the 

field economists once discussion had concluded. 

Data Analysis – Phase 2 Report Summary

BLS staff devised tools, procedures, and systems for completing analysis and validation of the micro data collected 

during Phase 2. These included one calibration activity in each test city, the refinement of review parameters from 

Phase 1 for data analysis, and the data review itself. The calibration activities were intended to ensure consistency in 

coding among field economists with regards to SVP, physical demands, and environmental conditions. For the review 

parameters, staff analyzed data collected during Phase 1 to develop expected relationships between each data element 

which were then integrated into the data capture spreadsheets completed by the senior BLS field economists. 

 Excel formulas were developed to identify coded data elements that did not meet these expectations. For data review, 

two approaches were taken. Schedules first were reviewed by BLS staff for consistency using the edits developed as part 

of the review parameters. Review questions concerning any data entries which were flagged as potentially inconsistent 

were sent to the field economist who collected the schedule in question. Their answers indicated whether data had 

been changed, documentation added, or both. Further analysis of schedules then was performed using SAS software. A 

comparison was made by SOC code between the coding of each element done in ORS Phase 2 and the value given by the 

DOT. 

ORS values were higher on the frequency scale than DOT values more often than DOT values were higher than ORS 

values. Gross manipulation was the element with the largest proportion of occupations with higher ORS values than 

DOT values: 40% of occupations had higher DOT values than ORS values. Speaking had the largest proportion of 

differences, with 72% coded higher in ORS than the DOT value. The two elements dealing with the use of the hands, 

Fine Manipulation (fingers) and Gross Manipulation (whole hand) had the lowest proportion of matches with large 

proportions of ORS coding both higher and lower than the DOT value.