A Multi-Mode Survey Design 

for the NLSY97 COVID-19 

Supplement

2023 FCSM Conference

October 25, 2023 

Anthony Washburn

Leah Christian

Quentin Brummet
Outline

01 Background and Data Collection Approach

02 Response Rates Compared to Other Rounds

03 Outreach Effort and Role of Interviewers

04 Conclusions and Future Research
Background and Data Collection Approach
BACKGROUND AND DATA COLLECTION APPROACH : USE OF WEB IN LONGITUDINAL SURVEYS 4

Many longitudinal studies are adding web data collection

• The Panel Study of Income Dynamics introduced web data collection in 2021. 

– Web was introduced after several years of development and testing.

– To evaluate potential mode effects, one group was assigned to a web first sequential mixed mode approach 

and the other to a CATI first design.

– About 40% of completed interviews were from the web mode. 

• The Health and Retirement Study introduced web data collection in 2018.

– Started testing a web mode in 2007 with off-year surveys between waves.

– To evaluate the potential impact, a sequential mixed-mode experiment was included in 2018. Note: only some 

sample members were eligible for the web based on access to the internet, prior interview completion and 

other criteria.

• The Understanding Society Study in the UK introduced web data collection mode in 2016.

– Started testing in 2009, leveraging the innovation panel, and developed an approach to target households 

most likely to respond via web.

– In more recent years, sample members are assigned to either a web-first or CATI-first mixed-mode design. 
BACKGROUND AND DATA COLLECTION APPROACH : METHODS

5

NLSY97 COVID-19 

Supplement • A sequential multi

-mode 

web-first design with 

phone follow-up

• Short survey design 

(12 minutes) covering 

most pressing topics

• Initial outreach by email 

and advance letter with 

email and phone prompting 

later in the field period
BACKGROUND AND DATA COLLECTION APPROACH : METHODS 6

The NLSY97 COVID Supplement was fielded halfway between 

Round 19 and Round 20 of NLSY97

October 2019

Round 19 Data 

Collection Start

March 2020

Pandemic 

Onset

July 2020

Round 19 Data 

Collection End

February–May 2021

COVID 

Supplement

September 2021

Round 20 Data 

Collection Start
BACKGROUND AND DATA COLLECTION APPROACH : METHODS 7

NLSY97 COVID Supplement Approach

Outreach

Initial

• Advance letter w/ web link

• Mass email w/ clickable 

web link

Follow-Up

• Interviewer prompting w/ 

phone, email, & text

• Mass email reminders

• Last chance mailing/email

Data Collection

Fielding

• 2/2021 – 5/2021

• 8,490 fielded cases

• Soft launch 200 cases

Questionnaire

• Work status

• Schooling

• Health

Incentives

Tiered Incentive

• $10 base

• $20 base for targeted 

demographics*

• $25 end of round 

incentive*

*Higher/additional incentives 

for sample members with 

historically lower probability 

of response
BACKGROUND AND DATA COLLECTION APPROACH : OUTCOME MEASURES 8

Outcome measures analyzed

Response Rates

Response rates 

overall and by mode

Comparison to other 

NLSY97 rounds

Outreach Effort

Number of attempts 

per complete

Comparison to other 

NLSY97 rounds

Representation & 

Nonresponse Bias

Demographic 

composition 

compared to other 

NLSY97 rounds

Interviewer Impact

Response rates by 

interviewer outreach

Representation by 

interviewer outreach
Findings
FINDINGS : RESPONSE RATES, OVERALL AND BY MODE 10

About two-thirds of fielded sample responded to the 

COVID supplement, lower than other NLSY97 rounds

• COVID supplement response 

rate was lower than recent 

NLSY97 rounds

• Most of the responses came in 

via the web with some phone 

completion

• Although the response rate was 

lower, the field period was also 

significantly shorter than 

regular NLSY97 rounds

NLSY97 COVID Supplement Field Overall Web Phone In-Person

COVID Supplement 4 mos. 64.2% 54.7% 9.5% 0.0%

NLSY97 Comparison Rounds Field Overall Web Phone In-Person

Round 19 (before supp.) 10 mos. 79.4% 0.0% 76.1% 3.3%

Round 20 (after supp.) 12 mos. 77.0% 0.0% 75.2% 1.8%

COMPLETION RATES BY ROUND AND BY MODE

Note: Response rates are based on total non-deceased cases
FINDINGS : OUTREACH EFFORT 11

The COVID supplement required fewer attempts 

per completion on average

• COVID supplement outreach effort 

was significantly lower than recent 

NLSY97 rounds

• The web-first design reduced the 

effort per complete by more than 

50% compared to round 19 (from 

23.7 attempts on average to 9.4)

• Over 70% of completes in the 

COVID supplement required 5 or 

fewer outreach attempts by 

interviewers

• About 22% of completes did not 

require any interviewer outreach

Completes w/

NLSY97 COVID Supplement Attempts/

Complete

< 6 

Attempts

6-10 

Attempts

11+ 

Attempts

COVID Supplement 9.4 71.1% 18.0% 11.0%

NLSY97 Comparison Rounds Attempts/

Complete

< 6 

Attempts

6-10 

Attempts

11+ 

Attempts

Round 19 (before supp.) 23.6 22.8% 26.6% 50.6%

Round 20 (after supp.) 31.5 15.1% 29.1% 55.8%

OUTREACH EFFORT BY ATTEMPTS THRESHOLDS
FINDINGS : REPRESENTATION AND NONRESPONSE BIAS 12

Representation differed 

slightly from other rounds

• COVID supplement was more 

female, non-black/Hispanic, 

educated, and had higher income 

than other recent rounds 

• Lower income, less educated, and 

black respondents were less likely 

to respond

• Demographic differences across 

rounds likely a reflection of 

variation in accessibility for 

completing web survey

• Interviewer outreach helped to 

reduce demographic gaps

Demographics COVID Round 19 Round 20

Sex

Male 45.9% 49.2% 49.0%

Female 54.1% 50.8% 51.0%

Race/Ethnicity

Black 24.9% 27.2% 27.8%

Hispanic 20.8% 21.1% 21.1%

Mixed 0.9% 0.9% 0.8%

Non-Black/Hispanic 53.4% 50.8% 50.3%

Education

Less than HS 15.3% 17.5% 18.0%

High School 21.8% 24.0% 23.6%

Some College 16.8% 16.1% 16.3%

Bachelor’s+ 37.9% 33.9% 33.5%

Unknown 8.2% 8.6% 8.7%

Income

$0 3.1% 4.0% 4.2%

$1–24K 15.2% 17.0% 16.9%

$25–50K 20.4% 20.6% 20.8%

$50K+ 59.1% 55.3% 55.0%

Unknown 2.2% 3.1% 3.1%

DEMOGRAPHICS BY ROUND
FINDINGS : REPRESENTATION AND NONRESPONSE BIAS 13

Nonresponse bias analysis

• Compare respondent demographics of 

completes for each round to 

the corresponding proportion from 

the full non-deceased sample 

• Absolute relative nonresponse bias: 

the absolute ratio of the direct 

nonresponse bias and estimate 

derived from the full sample

• Average relative nonresponse bias: 

average of the absolute relative 

nonresponse bias (ARNB) across 

all demographic comparisons

AAbbbbbbbbbbbbbb RR RR RRRR

NN NNNN NN NN BB

YY�rr,dddddddd − YY�nn,dddddddd

YY�nn,dddddddd

NN NNNN NN NN BB = YY� ,dd dd − YY� ,dd dd

YY�

rr,dddddddd unweighted estimated proportion derived from 

completes for each round 

YY�

nn,dddddddd unweighted estimated proportion derived from 

full non-deceased sample for each round
FINDINGS : REPRESENTATION AND NONRESPONSE BIAS 14

Nonresponse bias similar for 

COVID versus other rounds

• The average of the absolute relative 

nonresponse bias across the 

demographics tested was similar for 

the COVID supplement compared to 

both Round 19 and Round 20 of the 

NLSY97

• Major reason why nonresponse bias 

for web-first survey was similar to 

other rounds was because of 

interviewer impact

AVERAGE ABSOLUTE RELATIVE NONRESPONSE BIAS
FINDINGS : INTERVIEWER IMPACT 15

Response rate and representation depended 

on outreach effort and mode

• Web with prompting reached 

demographic groups less likely to 

respond via web, and phone 

interviewing reached even more of 

those less likely to respond to web

• Interviewer outreach increased 

representation of males, less educated, 

and lower income respondents

• Interviewer outreach and prompting 

increased representation across key 

outcome measures as non-working 

respondents were more likely to 

respond with prompting or by phone

Key Outcomes Web Comp 

no Outreach

Web Comp 

w/ Outreach Phone Comp

N 1224 3561 831

Response Rate 14.0% 40.7% 9.5%

Sex

Male 35.4% 46.3% 60.0%

Female 64.6% 53.7% 40.0%

Education

Less than HS 11.3% 14.4% 25.4%

High School 15.8% 22.0% 29.6%

Some College 17.6% 16.8% 16.0%

Bachelor’s+ 48.3% 38.4% 20.2%

Unknown 7.1% 8.4% 8.8%

Worked Last Week

Yes 77.3% 75.7% 64.4%

No 22.1% 23.5% 35.4%

KEY OUTCOMES BY INTERVIEWER OUTREACH
Conclusions and Future Research
CONCLUSIONS AND FUTURE RESEARCH : CONCLUSIONS 17

A carefully designed multi-mode survey approach with web-first data 

collection can yield high quality information

Response Rates

Web-first mixedmode approach 

achieved a 66% 

response rate when 

supplemented with 

interviewer prompting 

and additional phone 

outreach on a quick 

fielding timeline

Outreach Effort

A web-first survey 

drastically reduced 

outreach effort 

compared to other 

NLSY97 rounds with 

most respondents 

requiring 5 or fewer 

outreach attempts

Representation & 

Nonresponse Bias

Some demographic 

groups were less 

represented but 

nonresponse bias 

did not significantly 

differ from other 

NLSY97 rounds

Interviewer Impact

Interviewer outreach 

was critical to 

achieving a high 

response rate and a 

more representative 

sample
CONCLUSIONS AND FUTURE RESEARCH : FUTURE RESEARCH 18

Future research can help inform how to scale web-first approach 

for longer, more complex longitudinal surveys

• Testing – Testing is critical to introducing web into longitudinal surveys to understand 

potential impacts to response rates, representation, and survey estimates. 

• Mode preferences – Mode preferences and completion history can be helpful in 

targeting web to those most likely to respond via the internet. 

• Contact information – Web-first mixed-mode surveys require robust contact 

information for efficient follow-up (e.g., email, cell phone number).

• Interviewing staff – Interviewing staff are necessary for executing effective follow-up 

outreach strategies to achieve a high response rate.
Thank you. Anthony Washburn

Research Methodologist at NORC

washburn-anthony@norc.org