RTI International is a registered trademark and a trade name of Research Triangle Institute. www.rti.org

"Second Chance" Respondents to the 

National Survey on Drug Use and Health

A Natural Reinterview Reliability Analysis

Peter Frechtel, Lauren Klein Warren, and Adam Lee

RTI International
Are survey respondents telling the truth?

 A popular question among those in the survey research field. This 

topic is often called “reliability” or “measurement error”.

 Also a popular question among lay people. Maybe the most common 

question I get when I talk to people about what I do for a living, 

especially for surveys with sensitive questions.

 This is the main topic of this presentation.

2
Acknowledgements

 The presentation is sponsored by RTI International’s Center for 

Survey Statistics, with the research included in the presentation 

stemming from ongoing methodological work conducted under the 

contract for the National Survey on Drug Use and Health (NSDUH). 

The NSDUH is funded by the Substance Abuse and Mental Health 

Services Administration (SAMHSA), Center for Behavioral Health 

Statistics and Quality under project no. 0215638 (for 2018-2022) .

 The views expressed in this presentation do not necessarily reflect 

the official position or policies of SAMHSA or the U.S. Department of 

Health and Human Services; nor does mention of trade names, 

commercial practices, or organizations imply endorsement by the 

U.S. Government. 

3
Study Background

 The NSDUH provides national, state and substate data on 

substance use and mental health in the civilian, noninstitutionalized 

population age 12 and older. 

 Data are collected on a quarterly basis each year.

 Approximately 67,500 interviews completed annually.

 Conducted by RTI under contract with SAMHSA.

 Face-to-face data collection only until the onset of the COVID-19 

pandemic. Now, mixed mode: web and in-person. Web data 

collection began in Q4 of 2020.

4
“Second Chance” cases

 On NSDUH, web breakoffs are much more common than in-person 

breakoffs

 Many breakoffs occur early, but some don’t

 Some of those breakoffs are recontacted in person, and restart the 

survey

 The gods of survey research have bestowed upon us a rare and 

precious gift.

5

Start web survey End web survey

Start in-person 

survey

End in-person 

survey

BREAKOFF
Who are the Second Chance cases?

 Almost 600 in the 2021 NSDUH out of 69,850 usable cases

 About 350 got far enough to answer key questions in both modes

 There are some demographic differences between them and the 

other usable cases

– The most glaring one: very few are aged 12-17.

– Others: more female, more non-Hispanic Black, less urban

6
What should we do with our gift?

 Three obvious ideas:

1. Use them to try to determine causes of breakoffs

2. Use them to make back-end adjustments for web breakoffs 

that were not successfully recontacted

3. Use them to try to examine consistency of responses

 We did a little on #1, and more than a little on #2, but 

this presentation is about #3.

– How often are people responding inconsistently across 

modes?

– What characteristics are associated with inconsistent 

response?

7
Past NSDUH studies on reliability/validity

 Validity Study (2000-2001)1: compared self-reported drug use 

to drug use based on urine samples

– Good agreement; focused on past month use; questions about 

quality of the tests, bias with respect to urine sample

 Reliability Study (2006)2: compared self-reported drug use 

between two in-person interviews

– Very good agreement

– Random sample of respondents

– Same mode

1Harrison, L. D., Martin, S. S., Enev, T., & Harrington, D. (2007). Comparing drug testing and self-report of drug use among 

youths and young adults in the general population. (DHHS Publication No. SMA 07-4249, Methodology Series M-7). 

Rockville, MD: Substance Abuse and Mental Health Services Administration, Office of Applied Studies.

2Substance Abuse and Mental Health Services Administration. (2010). Reliability of Key Measures in the National Survey on 

Drug Use and Health (Office of Applied Studies, Methodology Series M-8, HHS Publication No. SMA 09-4425). Rockville, 

MD.

8
Reliability Study vs. Second Chance Study

Validity Study (2000-2001)

(past month use) Reliability Study (2006) Second Chance Study (2021)

Cigarettes 15.4 3.4 15.0

Alcohol N/A 4.2 12.0

Marijuana 10.2 3.2 16.6

9

% Inconsistent, Lifetime Use

 Why such a difference between RS and SC?

– SC cases are all web breakoffs; RS cases are a random sample

– “Mode effect”

– 2006 vs. 2021

 Let’s try some data analysis.
Second Chance Study: variables involved

 Dependent variables: consistency indicator for lifetime 

use of cigarettes, alcohol, marijuana, any of the three

 Independent variables: we tried to avoid data mining.

– Primary sociodemographic variables (PSVs): household 

size, age category, gender, race/ethnicity, education level, 

employment status, region, urbanicity

– Hypothesis variables (HVs): days between interviews, 

reluctance to respond, number of years since initiation, 

device used in web interview, past year psychological 

distress, data quality problems

10
Second Chance Study: results

Independent Variable

Statistical Significance

Evidence of Association

χχ2 Models

Days between interviews None None None

Response propensity None None None

Web modules completed CG, AL, ANY AL, ANY Some, but direction is unclear

Years since initiation AL None Very little

Device type CG, AL, MJ, ANY AL Some: phone users seem to be 

more inconsistent

Serious psychological distress None None None

Item nonresponse AL, ANY AL Some

Roster age mismatch Sparse data: no statistical testing None

Time per question None None None

All PSVs Some evidence that those with low education level are more inconsistent

11

AL = inconsistent response for lifetime alcohol; CG = inconsistent response for lifetime cigarettes; MJ = inconsistent response 

for lifetime marijuana; ANY = inconsistent response for any of the three substances.
Second Chance Study: conclusions / next steps

 Data analysis didn’t turn up that much.

– We’ll keep accumulating Second Chance cases, though.

– We can improve our data quality indicator and our measure of reluctance 

(number of contacts required in field, number of days between mailing and 

submission of broken-off web response).

 Recontacting people

– We’d love to ask these folks directly about their responses, but that’s a 

heavy lift, and it might not lead to much insight anyway.

– More likely, we’ll ask them more general questions about their experience 

with the web survey, beginning in 2024 or later.

12
More Information

Peter Frechtel

Senior Research Statistician

202.816.3966

frechtel@rti.org

13