Instrument Programming 

for Web and Phone 

Multi -Mode Administration

Rosella Gardecki and Amanda Roose

 2023 FCSM Conference

October 25, 2023 

–––

chrr.osu.edu | chrrinfo@chrr.osu.edu | 614-442-7300
National Longitudinal Surveys

• Sponsored by the Bureau of Labor Statistics, with additional funding from NIA, 

NICHD, and other Federal agencies

• A set of surveys designed to gather information at multiple points in time on 

labor market activities and other significant life events

- NLS Youth 1997 (NLSY97): Men and women born in the years 1980-84

- NLS Youth 1979 (NLSY79): Men and women born in the years 1957-64

- NLSY79 Children and Young Adults: Biological children of women in the NLSY79

- NLS Young and Mature Women (NLSW): Young women born in the years 1943-53 

and mature women born in the years 1922-37

- NLS Young and Older Men (NLSM): Young men born in the years 1941-52 and older 

men born in the years 1906-21

2

OVERVIEW

–––
NLSY97 Cohort

• Longitudinal Sample of 8984 respondents 

born 1980 -1984

- Aged 12 -17 during first interview in 1997

- Oversamples of Black and Hispanic individuals

• Annual interview through 2011 then biennial

• Prior rounds conducted by interviewers, 

either in -person or by phone

- Early in-person interviews included a small 

self-administered portion

• Round 21 currently in the field

• Round 20 data (collected 2021 -22) available 

in early December

3

OVERVIEW

––– OVERVIEW–––
COVID-19 Response: Onset March 2020

• NLSY97 Round 19 data collection

- Scheduled field period fall 2019 – summer 2020

- Moved to full phone effort

• NLSY79 Round 29 response

- Scheduled fall 2020 – fall 2021

- Worked with OMB to add COVID questions

- Focused on impacts to work, health, and income

• NLSY97 response

- Developed an interim survey: COVID-19 supplement

- Collect information on the short -run impacts outside of the normal schedule

- Fielded Round 20 as planned (fall 2021-2022)

- Included questions on impacts to work, health, and income

4

OVERVIEW

–––
NLSY97 COVID-19 Supplement

• Fielded February -May 2021

- Between Round 19 (2019 -2020) & Round 20 (2021 -2022)

• 5,616 respondents completed

- Ages 36-41 at the time of the supplement

• Designed to be a maximum of 12 minutes

- Average time across all modes=8.83 minutes

• Both interviewer -administered phone and self -administered 

web modes

• Preload-free due to concerns about confidentiality

- Questions couldn’t build on historical information

- Challenge for validation

5

OVERVIEW

–––
Building the NLSY97 COVID -19 Supplement

Questionnaire programming challenges

Multi-modal administration: mode determination

On-screen presentation: Variations in question text and 

response options by mode

Questionnaire programming

Security and confidentiality issues

Protecting information collected in survey

Email security

Verifying correct respondent

6

OVERVIEW

–––
CHRR Survey Suite Software

• Survey Suite Components: Designer, Survey, and Investigator

• Survey Features

- In-house software created specifically for longitudinal surveys

- Uses a central database to present questions within a web server

- No need for respondents to download a separate app

- Allows for multimodal administration

- Display adjusts depending on device

7

QUESTIONNAIRE

–––
Mode Determination

• Different links provided to respondents and interviewers

- Access the same survey instrument

- Determined appropriate survey presentation

• Automated process determined which link accessed each question

- Seamlessly present each question in correct format depending on mode

- Allows for multiple mode switches during the survey

8

Example | Alternate non-response categories depending on mode

Interviewer version Self-administered version

Don’t know I don’t know

Refused I prefer not to say

QUESTIONNAIRE

–––
On-Screen Presentation

Simplified appearance 

• Streamlined screen to remove information

- Section title

- Question name

• Reduced possibility of respondent error

- Fast forward 

- “Go to question” box

9

Covid survey

Traditional 

survey

QUESTIONNAIRE

–––
Questionnaire programming

• Shorter survey length to prevent breakoffs

- 46 potential questions

• Review of question wording to simplify language

- Respondent versus interviewer non-response wording

• Removal/rewording of interviewer instructions 

• Limited response category lists

• Time-saving matrix questions were split into separate components

• Minimum and maximum values limited response options on 

subsequent questions

10

QUESTIONNAIRE

–––
Survey Mode 

Differences

Survey Length Time to complete

Web -only 8.18 minutes

Phone-only 12.55 minutes

Mixed mode 13.67 minutes

11

Responses Average number

Web -only 32.83 responses

Phone-only 32.39 responses

Mixed mode 32.55 responses

QUESTIONNAIRE

–––
Item Nonresponse by Mode

12

Web -only Phone-only Mixed

At least one 

don’t know 20.7% 9.3% 27.6%

At least one refusal 8.9% 1.5% 6.9%

At least one 

nonresponse answer 26.7% 10.4% 27.6%

• Highest # refusals: whether the respondent had 

on-line banking

• Highest # don’t know answers: whether the respondent 

received an advance letter/email

QUESTIONNAIRE

–––
13

Questionnaire programming challenges

Multi-modal administration: mode determination

On-screen presentation: Variations in question text and 

response options by mode

Questionnaire programming

Security and confidentiality issues

Protecting information collected in survey

Email security

Verifying correct respondent

Building the NLSY97 COVID -19 Supplement

OVERVIEW

–––
Protecting 

Respondent Data

• Preload-free

• Only personally identifying 

information (PII) collected in survey 

was used for incentive payment

• Disabled backing up on the contact 

information questions

Presentation name 14

SECURITY/CONFIDENTIALITY

–––
Email Security

• Respondents sent an individual link to the survey by advance letter 

and email

• Links used non -sequential random IDs 

- Respondents cannot be connected to public-use data

- Non-sequential IDs prevent guessing of multiple unique links

• Some email products were found to have an automated process that 

opened the link when the email was sent

15

SECURITY/CONFIDENTIALITY

–––
Verification Process

• Could not present or collect PII to 

verify that the actual respondent 

was doing the interview 

• Presented three questions about 

known characteristics

• Post-fielding, matched answers 

to previously collected data

Presentation name 16

SECURITY/CONFIDENTIALIT

––– SECURITY/CONFIDENTIALITY

–––
Verification Outcomes

• Fewer than 100 respondents missed one or more of three verification questions

• Incentive payment information compared to contact information in past rounds 

• NORC called cases for verification if no match

• Fewer than 10 cases were dropped due to lack of verification

17

<100 <10

SECURITY/CONFIDENTIALITY

–––
Conclusions

Item nonresponse was significantly higher 

in the internet survey

Interview length significantly shorter 

for self-administered surveys

Content was limited by confidentiality concerns, 

with no preloads or collection of PII

Additional verification techniques could be explored 

to further mitigate confidentiality concerns

18

CONCLUSIONS

–––

1

2

3

4
Thank 

you!

chrr.osu.edu 

chrrinfo@chrr.osu.edu 

614-442-7300