National Center for Science and Engineering Statistics

Social, Behavioral and Economic Sciences

National Science Foundation

Testing Dependent Interviewing on 

a Self-Administered Survey

FCSM 4 Nov 2021

Jennifer Sinibaldi

Director, Research Program

https://ncses.nsf.gov/about/research
• Dependent Interviewing (DI) Experiment 

• Control group + 2 DI treatments

• Respondent Experience questions

• Results: Did it seem to work?

• Time/burden, measurement error, enjoyment 

Overview

2

Sinibaldi
Background
• Explicitly treat survey as longitudinal

• Make better use of prior responses

• More efficient

• More consistent

• Better experience for respondents

Our Goal

4

Sinibaldi
• “Proactive” Dependent Interviewing (DI) for some questions

• Shorter questionnaire

• More consistency across information that hasn’t changed^

• More pleasant respondent experience

• Try out 2 different formats of DI

• 1-step (DI1) and 2-step (DI2)

• Total of 3 versions of the questionnaire

The Proposed Solution

5

Sinibaldi
• Does DI reduce time to administer?

• Does DI appear to affect measurement error?

• Are respondents receptive to DI?

• We need to ask follow-up questions –> Respondent Experience 

Questions

• Overall, is one DI version superior?

Analysis: Research Questions

6

Sinibaldi
• DI on self-administered questionnaire

• Motivation for doing DI is unique

• 2 styles of DI, 1 step approach (DI1) not typically used

• Literature:

• Seam effect (e.g. Moore et al. 2006)

• Web-based, Self-administered study

• Lugtig and Lensvelt-Mulders 2014; Al Baghal 2017

• Comparisons of Proactive vs. Reactive DI (e.g. Lynn & Jackle 2007)

• 1 step approach (DI1) (Hoogendoorn 2004)

What’s Novel about this?

7

Sinibaldi
Examples of the Questions
• DI 1: the 1-step approach

• Shows the question with the answer selected that R reported last 

time. Ask R to change it or indicate no change on the same screen.

• DI 2: the 2-step approach

• Shows the question with the answer selected that R reported last 

time. Ask “Is this information correct?”. 

• If No, then 2nd screen asking standard question.

The Two Forms of DI

9

Sinibaldi
Example Dependent Interviewing Question- DI1

10

Sinibaldi
Example Dependent Interviewing Question- DI2

11

Sinibaldi
Example Question- Control

12

Sinibaldi
Respondent Experience Questions

13

Sinibaldi

• Topics: Perceived speed, enjoyment, question sensitivity

• Ex: How confident are you that NCSES will protect your answers?

• Ex: To what extent did you enjoy completing today’s survey?

• Ex: How sensitive did you think the questions on this survey 

were?

• Some questions worded the same for all 3 groups

To what extent did you enjoy completing today’s survey?

Did not enjoy at all

Enjoyed a little

Enjoyed somewhat

Enjoyed a great deal
Respondent Experience Questions

14

Sinibaldi

• Different question wording for DI vs. Control

• DI: Do you think pre-filling some of your answers from [prior cycle] 

and asking you to confirm or update them made this survey... 

• Control: If this survey had pre-filled your most recent answers and 

asked you to confirm or update them, would that have made this 

survey... • Much more burdensome

• A little more burdensome

• Neither more or less burdensome

• A little less burdensome

• Much less burdensome
Respondent Experience Questions

15

Sinibaldi

• Questions for DI only
Data Collection
• Survey of Doctorate Recipients (SDR)

• Sample 3,900 cases, mostly 2019 Rs

• Stratified, random assignment to DI1, DI2, Control

• 38 SDR questions, not including R verification questions

• 13 DI questions 

• 9 closed-ended (7 single response, 2 forced choice)

• 4 open-ended

• 20 Respondent Experience questions

Data Collection

17

Sinibaldi
• Field period: 13 Oct 2020 – 23 Nov 2020

• Reference date: 1 Sept 2020

• $30 prepaid incentive

• AAPOR (unwtd) RR1: 65.6%

• Across 3 groups: 64.2% - 66.7%

• DI1 n = 832| DI2 n = 863| Control n= 879

Data Collection

18

Sinibaldi
Results
• Actual survey length (unweighted, trimmed)

• DI1 15.0 min| DI2 14.0 min*| Control 15.4 min

• Could be influenced by format and number of Rs without 

changes 

Research Question: Time to Administer

20

Sinibaldi

* 95% confidence level
• Actual survey length: DI2 significantly faster

• Perceived survey length (wtd): DI1 & DI2 significantly “faster”

*P<0.05, **P<0.01, ***P<0.001

Research Question: Time to Administer

Sinibaldi

Response DI1 DI2 Control**

How would you rate the speed at which you completed today’s survey?

Very slow 0.4% 1.4% 0.4%

Somewhat slow 7.6% 7.6% 16.4%

Somewhat fast 62.7% 59.3% 56.6%

Very fast 29.3% 31.7% 26.6%
• Closed-ended questions: Overall Distributional Differences

Research Question: Measurement Error (1 of 6)

Sinibaldi

Question DI1 vs. DI2 DI1 vs. Control DI2 vs. Control

Employer type n.s. n.s. n.s.

Employer size n.s. n.s. n.s.

Educational institution n.s. n.s. n.s.

Type of educational institution n.s. ** *

Faculty rank n.s. n.s. n.s.

Work activity (forced check all) ǂǂ (1/14) n.s. n.s.

Supervised others n.s. n.s. n.s.

*P<0.05, **P<0.01, ***P<0.001 Based on Pearson Design-Based Chi2

ǂP<0.05, ǂǂP<0.01, ǂǂǂP<0.001 Based on Adjusted Wald F-Test (Bivariate variables only)
• Closed-ended questions: Differences by Response Option

Research Question: Measurement Error (2 of 6)

Sinibaldi

Question DI1 vs. DI2 DI1 vs. Control DI2 vs. Control

Employer type * (2/10) * (3/10) * (1/10)

Employer size * (1/8) * (1/8) * (1/8)

Educational institution n.s. n.s. n.s.

Type of educational institution n.s. * (2/6) * (3/6)

Faculty rank n.s. n.s. n.s.

Work activity (forced check all) * (1/14) n.s. n.s.

Supervised others n.s. n.s. n.s.

* 95% confidence level Based on the Adjusted Wald F-Test
• Open-ended questions

Research Question: Measurement Error (3 of 6)

Sinibaldi

*P<0.05, **P<0.01, ***P<0.001, N.S = Not Significant

Question

DI1 vs. 

DI2

DI1 vs. 

Control

DI2 vs. 

Control

DI1 vs. 

DI2

DI1 vs. 

Control

DI2 vs. 

Control

Any Change Meaningful Change

Employer Name N.S *** *** N.S  

Job Title N.S ** * *** N.S *** ***

Job Duties N.S *** *** N.S *** **
of those who made changes to open-ended questions

Research Question: Measurement Error (4 of 6)

Sinibaldi

Question DI1 DI2 Control***

Employer Name 90.4% 88.5% 47.1%

Job Title 89.5% 88.3% 53.6%

Job Duties 31.3%

DI1 vs. DI2*

51.1% 18.5%

• Open-ended questions: Percent meaningful changes,

*P<0.05, **P<0.01, ***P<0.001
• Any questions where answer was accurate enough although 

could have updated it? (among DI conditions only) 

Research Question: Measurement Error (5 of 6)

Sinibaldi

Response DI1 DI2

Yes, For 1 Question 11.4% 13.0%

Yes, for Multiple Questions 21.8% 21.9%

No 66.8% 65.1%

*P<0.05, **P<0.01, ***P<0.001 Based on Adjusted Wald F-Test
• Any questions where the answer displayed was no longer true, 

and left as-is ? (among DI conditions only) 

Research Question: Measurement Error (6 of 6)

Sinibaldi

Response DI1 DI2

Yes, For 1 Question 3.8% 5.2%

Yes, for Multiple Questions 3.3% 1.0%

No 92.9% 93.8%

*P<0.05, **P<0.01, ***P<0.001 Based on Adjusted Wald F-Test
• How sensitive do you think the questions in this survey were?

Research Question: What do Rs think of DI? (1 of 3)

Sinibaldi

Response DI1 DI2 Control

How Sensitive?

Not at all 34.2% 39.0% 35.7%

A Little 36.3% 37.2% 35.6%

Somewhat 24.1% 16.7% 24.1%

Very 5.3% 7.1% 4.7%

*P<0.05, **P<0.01, ***P<0.001
• Do you think pre-filling some of your answers from 201X and asking 

you to confirm or update them (made/would make) this survey...?

Research Question: What do Rs think of DI? (2 of 3)

Sinibaldi

Response DI1 DI2 Control***

Pre-Filling Burdensome?

Much More 0.1% 0.2% 0.9%

A Little More 1.6% 1.0% 6.0%

Neither 10.9% 11.2% 21.7%

A Little Less 24.3% 26.2% 42.2%

Much Less 63.1% 61.5% 29.2%

*P<0.05, **P<0.01, ***P<0.001
• To what extent did you enjoy completing today’s survey? 

Research Question: What do Rs think of DI? (3 of 3)

Sinibaldi

Response DI1 DI2 Control

Extent of Enjoyment?

Not at all 12.4% 13.7% 16.7%

A Little 33.7% 35.5% 35.0%

Somewhat 43.7% 40.6% 41.7%

A Great Deal 10.3% 10.2% 6.6%

*P<0.05, **P<0.01, ***P<0.001
Conclusion
Research Questions

• Does DI reduce time to administer? Yes

• Does DI appear to affect measurement error? Small 

indications of both improvement and increase

• Are respondents receptive to DI? Not a strong positive or 

negative sentiment

• Overall, is one DI version superior? DI2 slightly preferred

Conclusion: Not seeing dramatic results (+/-) which seems to 

indicate a smooth transition to DI.

Sinibaldi 33
Jennifer Sinibaldi

Director, Research Program

National Center for Science and Engineering Statistics

jsinibal@nsf.gov

Sinibaldi

https://ncses.nsf.gov/about/research

https://ncses.nsf.gov/about/funding-opportunities

Thank You!