1 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Mode Changes in the NLSY

Holly J. Olson

National Longitudinal Surveys

Survey Methods Symposium

July 16, 2019
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Presentation Overview

 Background on NLSY

 Mode Change Considerations in the NLSY97

 Outcomes of the NLSY97 Mode Change 

 Conclusions
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

National Longitudinal Surveys (NLS)

 NLS surveys gather detailed information about labor market 

activity and other experiences in the lives of seven cohorts of 

men and women

 Three active cohorts 

NLSY79 

NLSY79 CYA (1986)

NSLY97
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

NLS Survey Structure

 Contractor Based System

CHRR – OSU, NORC – University of Chicago

 Each cohort is fielded biennially, alternating years

 Each fielding effort is referred to as a ‘Round’

 Round lasts approximately 4 years

 Divided into 4 phases: design, survey preparations, data collection 

(fielding), data processing
5 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

NLSY79 & NLSY97 Response Rates

17, 89%

16, 81%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Response Rate

Round

NLSY79 NLSY97
6 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Interview Modes in the NLS

 Paper and Pencil Interview 

(PAPI)

 Computer Assisted Personal 

Interview (CAPI) 

 Computer Assisted 

Telephone Interview (CATI)

 In-person

 Phone

 Web– not yet

Interview Method Interview Mode
7 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

NLSY79 & NLSY97 Use of Phone by Round

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

PHONE PERCENTAGE

ROUND

% by Phone NLSY79

% by Phone NLSY97
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Differences between NLSY79 & NLSY97

 Age of Respondents/Life Stages

 Proximity of mode change to shift in interview frequency 

(biennial)

 Technology

 Sample Stability

 Questionnaire Structure
9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

NLS Priorities

 Total Response Rates

 Cost

 Sample Retention

 Sample Representation

 Data Quality
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Reasons for mode change in NLSY97

 COST!!! 

 Other options considered:

 Reducing sample size by dropping oversamples 

 Moving to a triennial collection schedule

 Not going after the hardest to reach cases

 Adaptive design/focus on subsample response rates

 Reducing overall interviewer hours
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Design Considerations for Phone

 Respondent setting is unknown

 Respondent visual aids are removed

 Harder for respondent to retain long strings of information 

 Respondent are more subject to primacy/recency effects

 Phone respondents less likely to report sensitive information 

than self-administering respondents, but more likely than inperson interviewer-administration respondents
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Survey Changes to Adapt to Phone 

 FIs instructed not to state private information

 Category questions changed to yes/no questions

 Streamline questionnaire

Shorten question response lists

 Some sensitive questions moved out of the SAQ section

 Additional materials informing respondents of mode change
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

R18 Fielding Strategy

 Early Bird

Decentralized dialing system (D-CATI)

Use of replicates

 Outbound dialing

Expired replicates

 Case Managed

FI’s ‘own’ cases
14 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

NLSY97 R18 Outcomes

 Production

 Phone vs In-person

 Interview Breakoffs

 Timings

 Additional Factors
15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Cumulative Completes by Week, R10- R18
16 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Round 17, Completed Cases x Average HPC

0

5

10

15

20

25

0

100

200

300

400

500

600

700

800

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

HPC

Completed Cases

Week

Cases per week HPC
17 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Round 18, Weekly Completed Cases x Average HPC

0

5

10

15

20

25

0

100

200

300

400

500

600

700

800

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

HPC

Completed Cases

Week

Int per week Hour per IW
18 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Completes by Respondent Types

Round Respondent Type Fielded Completed Percentage

Round 15 Perfect Responder 5290 5164 98%

Out 1 Round 358 171 48%

Round 16* Perfect Responder 5195 5076 98%

Out 1 Round 328 180 55%

Round 17 Perfect Responder 4975 4816 97%

Out 1 Round 596 268 45%

Round 18 Perfect Responder 4804 4542 95%

Out 1 Round 436 166 38%
19 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Comparison of Breakoffs b/w R17 and R18

Breakoff is defined as a time when the respondent needs to stop 

the interview at any point

Round 17 Round 18

Total Number of Breakoffs 232 687

Mean Breakoffs per Case 1.21 1.27

Percent of Cases with at least one 

Breakoff

3.27% 10.53%
20 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Mid-Fielding Changes

 Questionnaire modified to shorten administration time

 Earlier transition to case managed phase

 Additional monies added to FI hours for in-person R outreach

 Earlier implementation of ‘Last Chance’ bonus
21 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Compounding Factors in 97 Mode Change

 Proximity of interview schedule (biennial) and mode change

 Early expenditures and outlays that may not have been cost 

effective 

 Higher than average burn rate

 Difficulty contacting respondents
22 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Lessons Learned

 Don’t try to do too much at once 

 Outreach is key

More in-person outreach is needed

 Different types of outreach is needed

 Case management structure

 Mid-fielding flexibility
23 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Holly Olson

Assistant Director

National Longitudinal Surveys

www.bls.gov/nls

Olson.holly@bls.gov

Contact Information