Using Close-Ended Web Probes to Inform the 

Redesign of the National Health Interview Survey 

(NHIS)

Paul Scanlon, PhD

2018 FCSM Research and Policy Conference

March 7th, 2018

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research
Presentation Overview

1. Web Probing and Cognitive Interviews

2. The NCHS Research and Development Survey (RANDS)

3. NHIS Chronic Lung Condition Questions

4. Methodological Considerations for Using Close-Ended Web Probes

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Web Probing and Cognitive Interviewing

▪ The goal of question evaluation studies is to reduce the potential measurement 

error of survey items.

▪ Traditional method is Cognitive Interviewing

• Purposive sampling

• Semi-structured interviews

• Qualitative data and analysis

▪ Newer method is Web Probing

• Wide range of samples—from convenience to representative

• Structured interviews*

• Quantitative and qualitative analysis*

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Web Probing and Cognitive Interviewing

Fヴoマ Edgaヴ aミd “Iaミloミ, ヲヰヱ7 さApples aミd Oヴaミges: What is the ヴight ケuestioミ ┘heミ Ioマpaヴiミg WeH PヴoHiミg aミd Cogミiti┗e Iミteヴ┗iewiミg?ざ AAPOR

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Close-Ended Web Probing

▪ Close-ended web probes provide a way to mix qualitative and quantitative 

findings

• Use findings from Cognitive Interviews to design probes

• Answer categories are patterns of response

• Embedded into questionnaire following question or questions under evaluation

• Findings can be analyzed alongside other quantitative variables

▪ Probes are designed around the patterns of interpretation found during cognitive 

interviews

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Example of a Close-Ended Web Probe...

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations

NHIS Question: Would you say your health in general is excellent, very good, good, 

fair, or poor?

Probe Question: When you answered the previous question about your health, 

what did you think of? 

1. My diet and nutrition

2. My exercise habits

3. My smoking or drinking habits

4. My health problems or conditions

5. The amount of times I seek health care

6. The amount of pain or fatigue I have

7. My conversations with my doctor
NCH“’ ReseaヴIh aミd De┗elopマeミt “uヴ┗ey ふRAND“ぶ

▪ Non-production survey designed to examine the potential uses of web panels

▪ Used Gallup Panel

• さRepヴeseミtati┗eざ, ReIヴuited, “tatistiIally-sampled

▪ Conducted over two rounds in 2015 and 2016

• Second round included close-ended web probes

▪ n=4784 (Round 1=2304; Round 2=2480)

▪ Only web respondents

• Panel itself includes non-web panelists, but they were not included in this survey

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
An Example of this Mixed-Method Question Evaluation Process: 

The NHIS Lung Questions

▪ Currently NHIS asks three separate questions about chronic pulmonary disease

• Emphysema ever

• COPD ever

• Chronic Bronchitis in last 12 months

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Current NHIS Scheme

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations

Have you ever been told by a doctor or other health professional that 

you had emphysema?

1. Yes

2. No

Have you ever been told by a doctor or other health professional that 

you had chronic obstructive pulmonary disease, also called COPD?

1. Yes

2. No

During the past 12 months, have you been told by a doctor or other 

health professional that you had chronic bronchitis?

1. Yes

2. No
An Example of this Mixed-Method Question Evaluation Process: 

The NHIS Lung Questions

▪ Currently NHIS asks three separate questions about chronic pulmonary disease

• Emphysema ever

• COPD ever

• Chronic Bronchitis in last 12 months

▪ Cognitive interviewing revealed large potential for measurement error

• COPD INCLUDES emphysema and chronic bronchitis!

• Counting acute bronchitis in chronic question

▪ A さIoマHiミedざ ケuestioミ ┘as suggested aミd tested oミ RAND“

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
An Example of this Mixed-Method Question Evaluation Process: 

The NHIS Lung Questions

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations

Have you ever been told by a doctor or other medical professional that 

you have Chronic Obstructive Pulmonary Disease or COPD, 

emphysema, or chronic bronchitis?

1. Yes

2. No

[9] Refused
An Example of this Mixed-Method Question Evaluation Process: 

The NHIS Lung Questions

▪ Currently NHIS asks three separate questions about chronic pulmonary disease

• Emphysema ever

• COPD ever

• Chronic Bronchitis in last 12 months

▪ Cognitive interviewing revealed large potential for measurement error

• COPD INCLUDES emphysema and chronic bronchitis!

• Counting acute bronchitis in chronic question

▪ A さIoマHiミedざ ケuestioミ ┘as suggested aミd tested oミ RAND“

• Wanted to find a question that captured the same construct, but with a lower potential for 

measurement error

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
An Example of this Mixed-Method Question Evaluation Process: 

The NHIS Lung Questions

Does the combined question capture the same constructs as the separate questions 

do?

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Findings from Cognitive Interviewing

EPHEV and COPDEV: Ever been told...

Emphysema/COPD

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations

Constructs 

Captured

Emphysema/COPD

General Chronic 

Lung Disease

Heart Disease
Findings from Cognitive Interviewing

CBRCHYR: In the past 12 months...IhヴoミiI HヴoミIitis?

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations

Constructs Captured

Specific Diagnosis of 

Chronic Bronchitis

Any Diagnosis of 

Bronchitis (Including 

Acute Bronchitis)
Findings from Cognitive Interviewing

Combined Question: Ever been told...

COPD, emphysema or chronic bronchitis?

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations

Constructs Captured

COPD/Emphysema/Chronic 

Bronchitis

General Chronic Lung 

Disease

Acute Bronchitis

Heart Disease
Findings from Web Probing

▪ Probes were administered after combined question (NEWLUNG) in Round 2 of 

RANDS

▪ Oミly ヴespoミdeミts ┘ho aミs┘eヴed NEWLUNG さyesざ ヴeIei┗ed pヴoHes

▪ Probes designed to tease out specific condition respondents were thinking about 

(and see if acute issue was present or not)

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations

Which condition were you told you had? [Select All]

1. COPD

2. Emphysema

3. Chronic Bronchitis

4. Bronchitis

5. Something else

Thinking about the most recent time you had symptoms of Chronic 

Obstructive Pulmonary Disease or COPD, emphysema, or chronic 

bronchitis, how long did the symptoms last?

1. Less than one week

2. One week to one month

3. One month to three months

4. Three or more months
Findings from Web Probing

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Findings from Web Probing

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
An Example of this Mixed-Method Question Evaluation Process: 

The NHIS Lung Questions

Does the combined question capture the same constructs as the separate questions 

do?

▪ Cognitive Interviews indicate that combined question 

• Does pick up the intended, in-scope constructs

• Does not appear to capture out-of-sIope さheaヴt diseaseざ oヴ さaIute HヴoミIhitisざ 

constructs

▪ Web probing used to see if this latter is just a function of the cognitive 

interviewing sample

• Only small percent of respondents use a potentially out-of-scope interpretation

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Methodological Considerations for the Use of Web Probes

▪ One of the perceived drawbacks of adding cognitive probes to field tests is the 

idea that they will effect the performance of other survey items.

▪ By evaluating the response between Rounds 1 (without probes) and 2 (with 

probes) of RANDS, we find no evidence of this.

▪ Explored two areas:

1. PヴoHes’ effeIts oミ o┗eヴall ヴespoミse

2. PヴoHes’ effeIts oミ ヴespoミse to otheヴ suヴ┗ey iteマs

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Methodological Considerations for the Use of Web Probes:

Overall Item Response by Round

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Methodological Considerations for the Use of Web Probes:

Completion and Breakoffs by Round

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Methodological Considerations for the Use of Web Probes:

Effects on Response to Other Items

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations
Thank you!

Paul Scanlon

National Center for Health Statistics

pscanlon@cdc.gov

National Center for Health Statistics

Center for Questionnaire Design and Evaluation Research

1. Web Probes and Cognitive Interviews 2. RANDS 3. NHIS Lung Questions 4. Methodological Considerations