U.S. DEPARTMENT OF LABOR

BUREAU OF LABOR STATISTICS

Occupational 

Requirements 

Survey

Phase 1 Summary Report

Fiscal Year 2013 
1 | P a g e
2 | P a g e

Executive Summary 

The National Compensation Survey (NCS) recently completed Phase 1 of the Occupational Requirements 

Survey (ORS) tests conducted in cooperation with the Social Security Administration (SSA). The main 

objective of the three ORS tests in fiscal year 2013 is to assess whether it is feasible for BLS to collect 

data relevant to the SSA’s disability program using the NCS platform. The results of the Phase 1 proofof-concept test suggest that this approach is viable. Respondents agreed to participate in the test, BLS 

field economists were able to capture the required data from traditional NCS respondents, and 

individual data element response rates were very high. 

Field collection for Phase 1 testing was conducted in the greater Washington, D.C. area from November 

28, 2012 through December 6, 2012. Establishments were selected from the current NCS sampling 

frame, excluding establishments currently active in any NCS sample. Nine experienced field economists 

from BLS regional offices collected the data, and each interview was observed by a BLS national office 

staff member or an SSA representative. 

Upon completion of the ORS collection, respondents were asked questions to gauge their reactions to 

the survey, and BLS field economists and observers completed a post-interview debriefing. Daily 

debriefing sessions were held with BLS field economists, observers, and other staff for the purposes of 

discussing interviewing experiences, identifying potential issues with the materials and procedures, and 

sharing lessons learned. A final end-of-phase debriefing session summarized the major findings 

identified during the test period and allowed for expanded discussion of these and other issues between 

BLS and SSA staffs. 

Key Findings 

The results of the Phase 1 test were very promising overall. Test objectives were successfully met and 

ORS has a strong foundation on which to build for future development and testing. 

 BLS field economists completed interviews with 27 establishments, collecting detailed information 

for 104 occupations. It took between 8 and 10 minutes to collect information for each occupation. 

 BLS field economists had minimal difficulty gaining cooperation of sampled establishments. Many 

BLS field economists noted that one of the most effective pieces of information to gain cooperation 

was the high name recognition of the SSA. 

 Most data were collected from traditional NCS respondents such as human resource directors, small 

business owners, and location managers. 

 Cooperating establishments were able to answer the vast majority of questions asked in the survey 

resulting in a very high item response rate. 

 The training approach used in Phase 1 worked very well and will be repeated in subsequent tests. 

 The use of daily debriefings where BLS field economists, observers, and other staff could exchange 

information and suggestions about collection issues was particularly successful. 

 The materials prepared to aid ORS cooperation were very well received by respondents. 

 It is not enough to simply record the respondent’s answer; it is important to verify consistency 

across questions. For example, is it possible to sit 6 hours a day and also climb stairs frequently? 

 Certain words and concepts proved to be unclear or confusing to respondents. These included 

“required,” “accommodations” and prior work experience. 

As a result of our collection efforts during Phase 1, BLS identified several refinements that will be tested 

and evaluated as part of Phase 2. These include refined guidance and protocol for collecting the 

Physical Demand Strength and additional choices for the frequency scale. 
3 | P a g e

Background and Test Overview 

In 2012, the Bureau of Labor Statistics (BLS) signed an interagency agreement with the Social Security 

Administration (SSA) for the purpose of designing, developing, and carrying out a series of tests to 

assess the feasibility of using the National Compensation Survey (NCS) platform as a means to accurately 

and reliably capture data relevant to the SSA's disability program. The resulting initiative–the 

Occupational Requirements Survey (ORS)–recently completed the first of three phases of testing 

planned for fiscal year 2013 as outlined in the Interagency Agreement Deliverable sent to SSA on 

September 28, 2012. That document outlines the work the BLS will perform, key objectives, and a 

detailed test plan: 

In FY 2013, the BLS will perform work to meet the following objectives: 

1) Evaluate survey design options and begin developing the protocols, aids, and final design to 

meet SSA data needs; 

2) Collect data to test and refine the protocols and aids; and 

3) Provide documentation to the SSA summarizing the work performed by the BLS, conclusions 

drawn, and recommendations for future data collection. 

In order to accomplish these objectives, the BLS will conduct a series of field tests with general test goals 

as described below: 

Phase 1- Initial Proof of Concept Testing: The primary goal of this phase of testing will be to 

ensure that the BLS field economists know how to describe the survey and ask for the new data 

elements. In addition, the BLS will create and test an initial set of data collection protocols and a 

preliminary set of data collection aids. 

Phase 2- Collection Protocol Testing: The primary goal of this phase of testing will be to test 

collection of the new data elements while performing a selection of occupations from each 

respondent. In addition, the BLS will refine the collection protocols and aids based on an 

analysis of the test results. 

Phase 3- Broad Scale Testing: The primary goal of this phase of testing will be to test the BLS’s

ability to select a sample of occupations within each establishment, collect the new data 

elements needed by SSA, and collect other NCS data elements that are of research interest to 

SSA such as wages and job leveling information. In addition, the BLS will test the feasibility of 

collecting the data needed by SSA along with all the NCS data elements needed to produce the 

Employment Cost Index, Employer Costs for Employee Compensation, and various benefits 

products. 

This report details the methods and findings of the Phase 1 test. Phase 2 begins in late January and 

Phase 3 is scheduled to start in April. 
4 | P a g e

Phase 1 Test Objectives 

The primary goal of Phase 1 testing was to ensure that the BLS field economists understood how to 

describe the ORS and ask respondents for information regarding the new data elements. The key data 

elements of interest were: 

1. Specific Vocational Preparation (SVP) - an indicator of “time to proficiency,” defined as the 

amount of time required by the typical worker to learn the techniques, acquire the information 

and develop the facility needed for average job performance. 

2. Physical Demand (PD) - characteristics or activities that an occupation requires of a worker, 

measured in such a way to support SSA disability determination needs 

3. Environmental Conditions (EC) of occupations that replicate as closely as possible those that the 

SSA currently uses. 

In addition, BLS staff created, implemented, and evaluated an initial set of data collection protocols and 

data collection aids, and used the experience of Phase 1 testing to begin learning about the methods for 

computing estimates, analyzing the collected data, and validating future estimates. 

Test Logistics and Methods 

Field collection for Phase 1 testing was conducted in the greater Washington, D.C. area from November 

28, 2012 through December 6, 2012. Establishments were selected from the current NCS sampling 

frame, excluding establishments currently active in any NCS sample. The target sample size for Phase 1 

testing was 25 – 30 establishments representing as broad a mix of industries as possible given the size 

and time constraints of the test. Nine experienced NCS field economists (FE) from the BLS regional 

offices served as collected the data, and each interview was observed by a BLS national office staff 

member or an SSA representative. 

BLS field economists tested three alternative collection approaches. Upon completion of the ORS 

collection, respondents were asked to complete a short questionnaire to gauge their reactions to the 

survey. Daily debriefing session were held with field economists, observers, and other staff for the 

purposes of discussing interviewing experiences, identifying potential issues with the materials and 

procedures, and sharing lessons learned. A final end-of-phase debriefing session summarized the major 

findings identified during the test field period and allowed for expanded discussion of these and other 

issues between ORS and SSA staffs. 

Key Findings 

The results of the Phase 1 test were very promising overall. 

BLS field economists completed interviews with 27 establishments, collecting detailed information for 

104 occupations. As with any new collection effort, there were a number of issues that arose in Phase 1 

that will be addressed and evaluated in subsequent phases of testing. However, the consensus opinion 

of BLS staff is that the Phase 1 test objectives were successfully met and that ORS has a strong 

foundation on which to build for future development and testing. 
5 | P a g e

The Phase 1 test collected data on an average of 4 occupations at each establishment, and it took 

between 8 and 10 minutes to collect the ORS information for each occupation, depending on the 

instrument used. 

Phase 1 training consisted of self-study (background readings on the SSA disability program and relevant 

ORS concepts), in-person instruction provided during ORS orientation meetings, and on-the-job training 

(pre-collection practice interviews, debrief sessions, and informal field economist exchanges). The 

training approach used in Phase 1 worked very well and will be repeated in subsequent tests. The use of 

practice interviews prior to data collection and the use of daily debriefings where BLS field economists, 

observers, and other staff could exchange information and suggestions about collection issues were 

particularly successful. 

The materials prepared to aid ORS cooperation, in particular a 1-page flyer explaining the background, 

purpose, and specifics of the ORS test and the prepared script were very well received. In addition, many 

BLS field economists noted that one of the most effective pieces of information at their disposal was the 

name recognition of the SSA. 

BLS field economists expressed a preference for the collection approach which collected information 

one occupation at a time. Respondents had very positive reactions to the respondent visual aid, and 

field economists had some suggestions for improving the aid, such as removing the ‘extreme heat’ and 

‘extreme cold’ scales and adding examples to improve collection of the lifting and carrying elements. 

During the collection of data for Phase 1, it became very clear that it was not enough to simply record 

the respondent’s answer. Answers to questions asked earlier in the interview may very well prove to 

raise flags about answers to the current question. Is it possible to sit 6 hours a day and also climb stairs 

frequently? Can a worker reach forward and to the side constantly? 

The definitions in the Dictionary of Occupational Titles (DOT) do not provide enough information to code 

all required data elements consistently. To supplement the definitions, BLS staff used examples of work 

such as those provided in The Revised Handbook for Analyzing Jobs (RHAJ). These examples, too, had 

limitations. For example, most are drawn from the manufacturing sector which has declined in 

employment. 

There were certain words and concepts that proved to be unclear or confusing to some respondents. 

“Average performance” also was an issue in some interviews. The issue of accommodations and how to 

explain why ORS excludes them came up consistently in the daily debriefs. One successful approach 

used by field economists was to explain that the focus of the survey is the job and not the individual 

doing the job, and that individual accommodations at an establishment should not be collected. The 

matter of prior work experience also proved to require greater explanation for some respondents. There 

are amounts of previous work experience that are preferred and there are amounts that are required. 

Another topic of discussion among the BLS field economists was the measurement of the SVP scale. The 

intervals are very wide between some steps and the credit allowed for undergraduate education 

resulted in some coding abnormalities that yielded some unexpected results, with certain occupations 

having an SVP much higher or lower than expected. 
6 | P a g e

Phase 1 Test Methods and Materials 

Field collection for Phase 1 testing was conducted from November 28, 2012 through December 6, 2012. 

The testing location was the greater Washington, D.C. area. 

Selected Establishments 

Establishments were selected from the current NCS sampling frame for businesses in the greater 

Washington, D.C. area, excluding all establishments currently active in any NCS sample but including 

some units recently rotated out of NCS production. The target number of completed interviews for 

Phase 1 testing was 25 – 30 establishments representing as broad a mix of industries as possible given 

the size and time constraints of the test. At the conclusion of the testing period, 27 interviews were 

completed (see Results section for more details about participating establishments).

Field Economists 

The data collectors for the Phase 1 test were 9 experienced NCS BLS field economists from the BLS 

regional offices. Interviewer training for Phase 1 collection followed regular BLS practices and consisted 

of three components: 

 Self study – BLS field economists were provided the SSA background materials, researched SSA 

website, and reviewed the Dictionary of Occupational Titles (DOT), among other documents. 

 In-person training – This occurred during two meetings held at the BLS National Office. In the 

ORS Orientation meeting (October 23-24, 2012), project managers introduced BLS field 

economists to the ORS program and SSA representatives presented an overview of the disability 

determination process and the required data elements. At the Phase 1 Test Kick-off Meeting 

(November 27, 2012, also at BLS), SSA provided additional background about the disability 

adjudication process (including physical residual functioning capacity assessment), and the BLS 

field economists were provided a document containing detailed instructions for the Phase 1 

test. 

 On-the-job training (OJT) – Leading up to and throughout the Phase 1 Test fielding period, BLS 

field economists engaged in a number of activities designed to reinforce formal ORS training 

concepts. Prior to data collection, each field economist conducted at least two practice 

interviews with BLS staff unaffiliated with the ORS and NCS (e.g., Human Resources personnel, 

program managers), and participated in a calibration exercise in which all BLS field economists 

observed the same practice interview, individually recorded ORS information, and then 

compared and discussed their answers with one another. During data collection, OJT was 

provided through formal daily debriefing sessions and informal conversations between BLS field 

economists and other staff in which ‘lessons learned,’ ‘best practices,’ and challenging issues 

were identified. 

Refinement Procedures 

Field economists were provided with the establishment listings and case assignments upon arrival to the 

National office. Each field economist was given a list of private sector establishments to recruit, and 5 

field economists were additionally asked to collect for one government sector unit. The BLS field 

economists were notified that the numerical order of their assignments represented the collection 

preference, with the lower number being preferred. 

Once the field economists had their assignments, they began to research the establishments using the 

Internet. The collectors made phone calls to contact potential respondents and attempted to speak 
7 | P a g e

with company personnel with the same job titles that NCS traditionally works (e.g., HR Directors, 

Personnel Managers, and Hiring Officials). If those individuals were not available, the field economist 

worked with the establishment contact to identify the person most familiar with the target data 

elements to interview. When the appropriate company contact was located, the field economist used a 

standardized script to explain the purpose and importance of the survey, and attempted to schedule an 

in-person appointment (phone interviews were not permitted as part of Phase 1 testing). Some of the 

collectors sent e-mails to establishment contacts to provide additional background information about 

the purpose of and procedures for the interview. All potential respondents were informed that the data 

collection effort was part of a test being done at the request of the Social Security Administration and 

that participation was voluntary. 

When contact could not be made with listed establishments, or the establishment points of contact 

expressed reluctance or indicated that they were unavailable during the test field period, the field 

economists were instructed to forego the normal NCS cooperation attempts and simply advance to the 

next establishment on their assignment list. This process continued until each field economist had 

secured three viable appointments. 

Occupational Selection for Responding Units 

At the start of the interview, BLS field economists collected or verified information about the 

participating establishment (e.g., total employment, NAICS code), and then selected between 4 and 6 

occupations (as time and cooperation allowed) for the ORS collection. There was no formal sampling 

process for occupational selection, but field economists attempted to select occupations that were (a) 

highly populated in the establishment, (b) representative of a range of major Standard Occupational 

Classification (SOC) codes, and (c) convenient for the respondent to report. Field economists were 

instructed to select the most narrowly defined level of an occupation (i.e., the most detailed, distinct 

establishment description for an individual job), and to avoid selecting combined jobs. Neither 

occupational work schedules nor NCS occupational characteristics (e.g., full-time/part-time, union/nonunion, time/incentive) were collected, except for situations in which it was apparent that the 

occupational requirements varied by these characteristics. 

Data Collection Approaches and Protocols 

Three collection approaches were tested in Phase 1. Each version was designed to collect the required 

data elements (i.e., Specific Vocational Preparation (SVP), Physical Demands (PD), and Environmental 

Conditions (EC)), but they varied in their format, question flow, and collection method. 

 Collection Approach A was designed to obtain the occupational requirements one occupation 

job at a time. 

 Collection Approach B also collected occupational requirements one occupation at a time. The 

main difference was that the response options in Approach B were grouped together under a 

single question stem where appropriate. The purpose of this approach was to obviate the need 

for field economists to repeat the same question for related items 

 Collection Approach C collected the occupational requirements of all selected jobs 

simultaneously. 

In order to test the effectiveness of each collection approach, field economists were asked to use a 

different collection approach during each of their three personal visit interviews. 

A set of aids for respondents and interviewers was also developed. The field economist visual aid listed 

the SSA-provided definition and examples for each data element, as well as definitions of the 
8 | P a g e

measurements of frequency (i.e., never, occasionally, frequently, constantly). BLS field economists were 

encouraged to consult this aid during the interview, and to share it with the respondent as necessary. 

An additional, two-sided respondent visual aid provided a place for BLS field economists to list each of 

the jobs being surveyed, the frequency definitions, and examples for the noise intensity level response 

categories. BLS field economists were told to use the respondent aid during every interview. Finally, 

BLS field economists were provided with a document that contained answers to questions the 

respondent may ask; they familiarized themselves with this document prior to the interview, and could 

refer to it in the event that questions arose during the appointment. 

Data Capture and Review 

Data, including establishment information, were entered into a data capture spreadsheet on a flow 

basis. The spreadsheet was designed to permit easy data entry by BLS field economists, to provide the 

ability to review the captured data, and to ensure the capacity to tabulate results. Table 1 lists the data 

elements collected in the spreadsheet. 

Initial review parameters were developed for Phase 1 and were used to review and analyze data 

elements for internal consistency. Specifically, three internal consistency spreadsheets (one for each 

SSA data element area – SVP, PD, and EC) were used in the data review and analysis. The parameters 

identified expected relationships between the individual ORS data elements, as well as consistencies 

between ORS and NCS data elements. 

Table 1. Information Recorded in the Phase 1 Test Data Capture Spreadsheet 

Element Items Captured

General Information

Schedule number; establishment employment size; 

NAICS code; selected occupations (title, SOC, 

employment, FT/PT, work environment, job 

description)

Specific Vocational Preparation (SVP) Title; SVP; SVP level; education level

Physical Demands (PD)

Standing/Walking/Sitting; Reaching; Lifting/Carrying; 

Pushing/Pulling; Gross Manipulation/Fine Manipulation; 

Keyboarding; Hand Arm Controls/Foot Leg Controls; 

Climbing/Stooping/Kneeling/Crawling/Crouching;

Visual Acuity; Speaking

Environmental Conditions (EC)

Extreme Cold/Extreme Heat; Wetness/Humidity; 

Vibrations; Fume/Odors/Gases/Poor Ventilation;

Toxic Caustic Chemicals; High Exposed Places; Moving 

Mechanical Parts; Noise Intensity Level; Loud 

Noise/Moderate Sound Discernment

Observers 

To help garner feedback about the interview and data collection processes, an observer accompanied 

the field economist on each data collection appointment. The observers represented a mixture of roles 
9 | P a g e

and stakeholders within the ORS Project (e.g., BLS survey methods and operations staff; NCS/ORS 

management; SSA officials); with priority given to those individuals serving on ORS related teams but not 

serving as a collector. Each observer participated in only one interview. 

The observer did not participate in ORS data collection, but was responsible for recording information 

about the interview. They noted the duration of the interview (and the time needed to administer 

sections within the interview), and their observations for each item and the interview as a whole. 

Phase 1 Other Debriefing Activities 

In addition to the observer form, BLS assessed the effectiveness of Phase 1 ORS materials and 

procedures by conducting the following debriefing activities: 

 Respondent Debriefing – At least 10 minutes were set aside at the end of each ORS interview 

to ask the respondent about their interview preparation and experience. This debrief was 

administered either by the field economist or the observer, using a respondent debriefing 

outline with 11 questions targeting potential comprehension issues, perceptions of burden, 

reactions to the visual aids, etc. 

 Post-Interview Debrief – Immediately after leaving each ORS appointment, the field economist 

and observer jointly answered 10 questions that asked whether they observed any data 

element comprehension issues, if the interview process and materials seemed effective, and if 

there were any suggested improvements for ORS collection (either from the respondent or the 

field economist or the observer). 

 Daily Collection Debrief – Each day during the Phase 1 test in which there was a collection, an 

in-person debriefing session was held to assess how the daily interviews went from the 

perspective of both the field economist and the observer. These sessions were moderated by a 

facilitator and had a dedicated note taker to ensure that all aspects of the discussion were 

captured. All 9 BLS field economists (whether they conducted an interview that day or not) and 

the observers who went out on collection interviews that day participated in these daily debrief 

meetings. Others involved in the ORS project attended, as available. 

 End of Phase Debrief – On December 6, 2012, the Debrief Team conducted a final debriefing 

session. The purpose of this session was to summarize key findings from the Phase 1 test 

based on all the information collected in the various assessment activities, to solicit additional 

feedback about components of the test, and begin to identify issues and recommendations 

relevant to Phase 2 preparation. The meeting was attended by BLS staff and by SSA sponsoring 

officials. 

Phase 1 Test Results 

This section reports on results compiled throughout the Phase 1 test. They reflect information obtained 

from each debriefing component and feedback collected from observers and interviewers during the 

collection and end-of-phase debriefing sessions. Although the information garnered from each of these 

components is summarized here and not reported individually by source, each source contributed 

significantly to the overall Phase 1 findings and conclusions. This section begins with a broad 

assessment of the feasibility of collecting the data needed by SSA through the ORS, then provides an 

overview of the collection effort and presents more detailed descriptions of issues that arose during the 

test. 
10 | P a g e

Feasibility 

The main objective of the development and evaluative work that will be done for the ORS Project in FY 

2013 is to assess whether it is feasible for BLS to collect data relevant to the SSA’s disability program. 

The results of this initial proof-of-concept test suggest that this approach is viable. Respondents agreed 

to participate in the test and BLS field economists were able to capture the required data from 

traditional NCS respondents. In Phase 1, BLS field economists completed interviews with 27 

establishments, collecting detailed job requirement information for 104 occupations. As with any new 

collection effort, there were a number of issues that arose in Phase 1 (e.g., with certain data elements, 

or aspects of the collection and assessment) that will be addressed and evaluated in subsequent phases 

of testing; many of those are reflected in the sections that follow. However, the consensus opinion of 

BLS staff is that the Phase 1 objectives were successfully met and these activities lay a strong foundation 

for future tests. 

Participating Establishments 

Table 2 provides additional detail about the establishments that participated in Phase 1, and the 

occupations for which ORS data were collected. BLS field economists were successful in securing 

interviews from a variety of industry groups and collecting data for a range of occupations in an 

extremely tight fielding period. 

Table 2. Description of Establishments and Selected Occupations for Phase 1 Test 

Industry Group Number of 

Participating Units

Employment 

Size

(range)

Number of Jobs 

Collected

Jobs Collected

(selected)

Private Industry, 

Education 4 15 – 200 17

 Teachers

 Management

 Office and Admin. staff

 Installation, Maintenance 

and repair

Private Industry, 

Financial Services 4 4 - 80 14

 Property manager

 Laundry and dry cleaning

 Financial specialists

 Audio/Video equipment 

technicians

Private Industry, 

Goods Producing 4 12 – 114 15

 Mechanics, pipefitters, 

and electricians

 Management

 Installation, 

maintenance, and repair

Private Industry, 

Health Care

4 3 – 85 16

 Healthcare practitioners

 Office and administrative 

support

 Home health aide

Private Industry, 

Services 5 3 – 187 14

 Sales

 Hair stylist

 Store manager

 Housekeeping

State and Local 

Government 6 65 – 775 28

 Faculty

 Registered nurses

 Police officer

 Clerical
11 | P a g e

. 

Data Collection Appointments 

ORS field economists reported that their pre-collection activities included searching for the company’s 

phone number, conducting Internet searches to find establishment websites, gaining cooperation and 

similar activities. For the Phase 1 test, BLS field economists were instructed to move on to the next 

establishment on their list in the event that they encountered reluctance. This allowed BLS field 

economists to focus their time on collecting the needed test data as opposed to spending hours 

attempting to get cooperation from respondents who did not want to cooperate. 

Notwithstanding these procedural differences, field economists reported that it took 30 to 60 minutes 

to secure an appointment. Once BLS field economists were able to speak with the respondents, they had 

a range of experiences securing the appointment: Some set up appointments in as little as 5 minutes, 

while others had to go through several layers of contacts and information sharing (e.g., with an 

establishments’ national corporate office or legal department) before being able to make an 

appointment. Another limiting factor encountered by ORS field economists was the ever-presence of 

voicemail which sometimes made initial contacts difficult. 

The BLS field economists reported that the materials prepared to aid in data collection were very useful 

in addressing any concerns raised by their establishment contacts. Several field economists referred 

respondents to the SSA and BLS websites as well, and found that to be a useful tactic. One of the most 

effective pieces of information at BLS field economists’ disposal was the name recognition 

establishment contacts had for SSA – most field economists reported that this was instrumental in 

securing test participation. 

Interview Duration 

Based on the approved testing plan submitted to the Office of Management and Budget (OMB), BLS 

field economists were allotted one hour to collect the ORS data and conduct the respondent debriefing 

in each establishment. On average, data was collected on 4 occupations at each establishment. 

Collection of the average occupation took between 8 and 10 minutes. 

Collection Protocols 

The Collection Approaches 

Each of the three collection approaches had things that worked and things that did not, but overall there 

were some universal comments and suggestions that applied to all three approaches: 

1. Add questions that address prior work experience and other types of training besides a formal 

college degree. 

2. Group similar questions together. Switching from Yes/No to Occasionally/Frequently/Constantly

often caught the respondent off guard. 

3. The order of the reaching questions could be tweaked to ask overall reaching last. 

4. Ask about moderate sounds before asking about loud sounds. 
12 | P a g e

Other Protocol Outputs 

Introductory Script 

During the Phase 1 collection the field economists reported that they were all using the prepared 

introductory script, although they were making minor changes to it somewhat to reflect their own 

personal style. There seemed to be no specific issues with the introductory script as it was presented in 

Phase 1. Some field economists concluded there should be a greater emphasis on what specific 

elements ORS would be asking about, while others thought confidentiality needed to be addressed at 

greater length. However, neither of these views was universal. Some felt that accommodations needed 

to be addressed up front at first contact, but this was also a minority opinion. 

Visual Aids 

One clear theme can be gleaned from reviewing all of the notes from the various debrief meetings – the 

respondents loved their visual aids. Whether it is actually useful or just helps them feel involved in the 

process, there was universal praise. There were, of course, some suggestions for improving the aid, 

such as removing the extreme heat and cold scales and adding some weight examples. 

Phase 1 BLS Field Economist and Observer Training 

Phase 1 training consisted of self-study (background readings on the SSA disability program and relevant 

ORS concepts), in-person instruction provided during ORS orientation meetings, and on-the-job training 

(pre-collection practice interviews, debrief sessions, and informal field economist exchanges). BLS field 

economists and observers reported that all of these activities were important and helped to ensure that 

they were prepared for the Phase 1 test. The orientation meetings and SSA presentations provided 

crucial information about the ORS Project objectives and elements, and the supporting SSA 

documentation was mined heavily by ORS team when developing Phase 1 materials and procedures. 

The observer information session was attended by all Phase 1 observers, and participants found the 

session extremely useful, and even offered a number of suggestions that resulted in improved materials 

and procedures. 

Everyone was particularly positive about the benefits of doing practice interviews prior to data 

collection. These interviews gave field economists and observers the opportunity to practice using the 

collection approaches and observer methods, and affirmed that BLS field economists could ask the 

questions and collect the data on all the SSA data elements. The daily debriefings also were excellent 

training tools. This semi-structured forum provided an avenue for valuable information exchange about 

collection issues. Because data collection scenarios all are unique, BLS field economists found these

debriefing sessions allowed them to discuss different types of situations and possible solutions to these 

scenarios. Finally, the end-of-phase debriefing gave the BLS staff the opportunity to synthesize and 

report on information gleaned from the spectrum of assessment activities, to raise questions relevant to 

Phase 2 testing, and share Phase 1 experiences with our SSA sponsors. 

BLS field economists and observers also underscored the importance of other informal, on-the-job 

learning opportunities. These on-going conversations helped to solidify lessons learned, best practices, 

and to identify areas in need of further development. For Phase 1 testing there was no formal 

mentor/mentee process, but it did not stop the field economists from discussing the program objectives 

and collection strategies amongst themselves. 
13 | P a g e

Data Collection 

The process of going out and collecting live data from actual respondents proved to be a fruitful and 

valuable endeavor. Once the field visits began, the feedback at the debrief meetings began to flow. Field 

economists became more familiar with the various concepts, procedures and collection approaches in a 

more substantial and concrete way. It also was at this point that a lot of very specific questions 

pertaining to definitions, concepts and fallbacks began to surface. 

Respondents 

Meeting with respondents and actually asking the questions allowed BLS field economists to gather 

some valuable information on how these questions might be perceived from the other side of the table. 

There were certain words or concepts that proved to be unclear or confusing to some respondents. A 

good example of this was our use of the word “required.” Some respondents shied away from this 

word, not wanting to be seen as “requiring” anything specific of their employees that could potentially 

cause a legal issue. Some field economists theorized that the trepidation may also have stemmed from 

fear of appearing to discriminate against those with disabilities. 

 

Another concept that proved to be rather perplexing to respondents was the issue of “average 

performance.” There was much discussion with certain respondents trying to clarify the concept. Many 

respondents said something along the lines of “I wouldn’t hire them if they weren’t ready to work on 

day one.” Some respondents also said it was difficult trying to answer this item when they were asked 

to exclude any orientation period. Consistent interpretation of this concept across the respondent 

population seems unlikely until BLS and SSA can provide additional guidance to field economists and 

respondents. 

The issue of accommodations and how to explain why ORS excludes them came up consistently in the 

daily debriefs. One successful approach used by field economists was to explain that the focus of the 

survey is the job and not the individual doing the job, and that individual accommodations at an 

establishment should not be collected. Although it proved easy enough to explain that accommodations 

should be excluded, it would be helpful to be able to clearly explain why they are being excluded in a 

manner that is acceptable to all parties. 

The matter of prior work experience also proved to require greater explanation for some respondents. 

There are amounts of previous work experience that are preferred and there are amounts that are 

required. Along those same lines there were ranges of prior work experience (5-7 years) collected. 

BLS Field Economists 

The BLS field economists tasked with the collection of the data in Phase 1 proved that they are skilled in 

all aspects of their jobs - locating respondents, explaining complex survey concepts in plain language, 

selling the importance of cooperating in the survey, guiding the respondent in their answers, and further 

acting as a resource during the interview. 

During the collection of data for Phase 1, it became very clear that it was not enough to simply check the 

box corresponding to the respondent’s answer in the forms. Answers to questions asked earlier in the 

interview may very well prove to raise flags about answers to the current question. Is it possible to sit 6 

hours a day and also climb stairs frequently? Can a worker reach forward and to the side constantly? 

The BLS field economists also found themselves having to act as a “human visual aid,” physically 

demonstrating the various movements during the interview. 
14 | P a g e

Another topic of discussion among the BLS field economists was the measurement of the SVP scale. The 

intervals are very wide between some steps and the credit for undergraduate education resulted in 

some coding abnormalities that yielded some unexpected results, with certain occupations having an 

SVP much higher or lower than expected. 

Phase 1 Debriefing Activities 

Participation in the various Phase 1 debriefing activities (e.g., interview observations; completion of 

observer, respondent debrief, and post-interview debrief forms; attendance at the daily and end-ofphase debriefing sessions) was very high. All interviews had observers, and there was a 100 percent 

return rate on the observer forms and post-interview debrief forms. Respondent debriefing questions 

were asked in all but two interviews (where time constraints prevented their administration), and the 

team received completed forms for all available interviews. On average, the five daily debrief sessions 

were attended by 20 BLS staff – including field economists, observers, team representatives, and 

managers – and the discussions were active and productive. The end-of-phase debriefing session was 

attended by all available BLS staff and managers, as well as representatives from the SSA. 

Reactions to the five debriefing components were very positive. In particular, staff singled out the 

respondent debriefings and daily debriefing sessions as crucial to Phase 1 assessment. Most collectors 

and observers indicated that the respondent debriefings gave them insights that they would not have 

had otherwise into respondents’ perceptions of the survey, potential confusions, and suggestions for 

improving the questions and survey process. In addition, respondents seemed to really appreciate being 

asked about their reactions and suggestions. Several observers felt that additional time could be spent 

on this activity in future testing, and a number of people suggested that the questionnaire items should 

be slightly changed for the next round of tests. It was thought by some that a greater emphasis should 

be placed on gathering data pertaining to specific aspects of the collection, the questions asked, and the 

order of questions as opposed to open-ended or general questions. 

The in-house debriefing sessions were lively discussions in which both BLS field economists and 

observers shared their interview experiences, identified potentially problematic items or procedures, 

and exchanged approaches that they found helpful in clarifying meaning and securing collection goals. 

These meetings also provided an opportunity for BLS field economists and observers to share stories 

they heard from respondents about their personal connections to disability issues, and how that helped 

to put a human face to the ORS. The end of phase debriefing provided the opportunity for BLS field 

economists to summarize their Phase 1 experiences, and for SSA to ask questions about the Phase 1 test 

design and to get answers from the people doing the actual collection. 

Phase 1 also provided valuable information about how best to collect information during and 

immediately following the ORS interview. The post-interview debriefing process entailed Field 

economists and observers discussing their impressions of the interview and completing a structured 

questionnaire. Many field economists and observers said that this process served as a good way to 

organize their thoughts, particularly if they had a long commute back from their collection location. 

Others, however, felt that the information coming out of the exercise was redundant and unnecessary, 

and that the same thing would be covered in the daily debriefings. 
15 | P a g e

Data Analysis 

The Data Analysis team was tasked with preparing tools, procedures, and systems to determine the 

validity and accuracy of the data gathered as well as creating calibration activities to help ensure intercoder reliability in data collection. The team conducted two calibration activities, developed initial 

review parameters for data analysis, and completed the data review itself. 

Calibration Activities 

Two activities were conducted to help ensure that different field economists code the same specific 

vocational preparations, physical demands, and environmental conditions for a job the same way during 

a collection interview. The first activity was a "zero-point" calibration which took place during the Phase 

1 collection training session prior to any data collection. The purpose was to get a baseline reading on 

how similar the collectors' coding was for the same information and to help train the collectors on 

where differing interpretations of the same information might arise. As a result of this activity, it was 

apparent that there were some discrepancies between the ways different field economists coded 

certain elements, especially when the information given by the respondent seems incomplete. The 

second activity was conducted at the end of the Phase 1 debriefing session and after the Phase 1 data 

had already been collected and coded. The purpose of this activity was to gather another reading from 

the field economists and to facilitate reflection on individual collector's interview techniques and 

understanding of the questions. Of particular salience during this activity was the extent to which a 

collector was willing to challenge a respondent's answer to an element or ask more probing questions to 

verify if the respondent understood the question correctly and was providing accurate information. This 

activity showed much greater agreement in coding among field economists than the first activity. 

Data Analysis Tool Development and Results of Data Review 

For the initial review parameters, the team reviewed all individual data elements to develop expected 

relationships between each data element which were then integrated into the data capture 

spreadsheets completed by the field economist. A total of 515 unexpected conditions were created and 

Excel formulas were developed to identify data elements coded that did not meet these expectations. 

For data review, two approaches were taken. Schedules were first reviewed by members of the Data 

Analysis and Review staff for consistency using the edits developed as part of the initial review 

parameters. Review questions concerning any data entries which were flagged as potentially 

inconsistent were sent to the field economist and the answers indicated whether data had been 

changed, documentation added, or both. Further analysis of schedules was then performed on the 

data. SVP variability was compared by SOC group, with the result being that the SVP range for several 

groups was lower than expected. Frequency tables of responses were also compiled, indicating that the 

answer “Occasionally” was chosen far more often by respondents in Phase 1 testing than would be 

expected based on comparison to the DOT. In SOC codes which had jobs from multiple establishments 

there was more variation in all the elements – SVP, physical demands, and environmental conditions –

than was seen for shared SOC codes within a single establishment, which might be a sign of differing 

interpretations or confusion with certain concepts. The analysis and review staff also tabulated data on 

each edit to evaluate their effectiveness. Eighty-three total questions were asked of the field 

economists and resulted in 31 changes to data. In 14 of the cases where the field economist did not 

change the data, however, the field economist indicated they would have called back the respondent for 

clarification had policies been different.