# Presentation: 510808

## Q-BANK: Development of a Tested-			Question Database

- Questionnaire Design Research Lab
- Office of Research and Methodology
- National Center for Health Statistics

## Cognitive Interviewing at NCHS

- A Brief Overview

## Questionnaire design problems

- Some possible threats to accuracy derived from questionnaires:
- Questions not understood as intended
- Don’t adequately capture respondent experience
- Pose an overly challenging response task
- Don’t provide a legitimate option for an answer

## Questionnaire design problems (2)

- Errors can be systematic (bias) or random (noise; high variance)
- Problems may not be visible in the actual data, inviting delusions about the quality of self-reports that have been collected

## “In the last year, have you been bothered by pain in the abdomen?”

- Seems to be straightforward

## “In the last year, have you been bothered by pain in the abdomen?”

- Seems to be straightforward
- But suppose we ask:
- What do people think of as their “abdomen”?

## What, to you, is your abdomen?

## “In the last year, have you been bothered by pain in the abdomen?”

- Seems to be straightforward
- But suppose we ask:
- What do people think of as their “abdomen”?
- What sort of threshold do people use for answering “yes”?

## “In the last year, have you been bothered by pain in the abdomen?”

- Seems to be straightforward
- But suppose we ask:
- What do people think of as their “abdomen”?
- What sort of threshold do people use for answering “yes”?
- What period of time are people thinking about while answering?

## “In the last year, have you been bothered by pain in the abdomen?”

- Possible revisions:
- Show shaded picture of abdomen
- Drop “bothered”
- Use “In the past 12 months”
- “In the past 12 months, have you had pain in the abdomen?  By abdomen, we mean the shaded area in this picture.”
- Clear alternatives address these problems, with no apparent drawbacks

## Shaded picture of the abdomen

## What is cognitive interviewing? 

- 1) An administration of a draft questionnaire...
- 2) With additional verbal information collected about the responses...
- 3) And used to either:
- Evaluate the quality of responses
- Or help to determine whether the question is generating the sort of information its author intends.
- Within these general parameters, there are a number of variants:

## Practices of cognitive interviewing

- The “purist” version– encourage participants to think out loud while answering questions; interviewer “prompts” participants without much other activity.

## Practices of cognitive interviewing (2)

- Newer versions rely much more heavily on direct probing.  But even so, there can be much variation:
- Probes scripted or based on interview/participant particulars (more “active” role for the interviewer)
- Probes that are “cognitive” or more narrative in nature
- A relatively realistic administration of the questionnaire (with debriefing at the end) or intensive discussion throughout

## Some general practices

- We generally opt for:
- Designing a _semi-scripted_ interviewing protocol.  It asks some probes in anticipation of potential problems, but allows improvisation.
- Selecting a group of interviewees who would be typical recipients of the questionnaire and collectively cover a range of situations.  Get a demographic/educational mix as much as possible.
- Considering the interviewers to be _investigators_ and not simply data collectors (methodologists with subject matter knowledge).

## What sorts of probes should you ask?

- Consider this simple model of the response process as a starting point:
- _Comprehension_:  Respondent interprets the question
- _Retrieval_:  Respondent searches memory for relevant information
- _Estimation/Judgment_:  Respondent evaluates and/or estimates response
- _Response_:  Respondent provides information in the format requested

## Probes in cognitive interviews	

- Traditionally the majority of attention focuses on comprehension
- _Direct comprehension of terms_: What does 'dental sealant' mean to you?"
- But we also find that the most efficient way to learn about frame of reference and interpretation is to ask for narrative about the basis for the respondent’s answer
- Why did you answer that way?
- You said ___; tell me more about that.
- Or, ask for _paraphrase_ of the question.

## Probes in cognitive interviews (2)

- To get at other aspects of the response process:
- _Recall_: How did you figure your answer to that?  How do you remember that?
- _Confidence_: How certain are you about that? (can be used to identify judgment issues/uncertainty)
- _Response categories_:  How did you pick an answer to that?  Was it difficult to find a response category?
- You can’t go wrong with:
- “Tell me more about that?”
- “How so?”

## The end result

- Qualitative analysis follows one or more rounds of interviews
- Project reports include rich data on
- Participant experiences and how these line up with question wording
- Participant interpretations of the questions
- Problems participants had answering
- Potential ambiguities, trade-offs involved in asking question a certain way
- Reports provided to sponsor, including recommendations for changes when appropriate
- Then what?

## 2001 QUEST conference
Losing knowledge
Repeat testing

- 2001 QUEST conference
- Losing knowledge
- Repeat testing

## Implications:
Personal
Administrative
Survey Research

- Implications:
- Personal
- Administrative
- Survey Research

## Inability to compare studies
Hinders methodological conclusions
Prevents insight 
Stagnant method

Limited use to survey research

- Inability to compare studies
- Hinders methodological conclusions
- Prevents insight
- Stagnant method
- Limited use to survey research

## National Center for Health Statistics original developers
Access to past evaluation projects
Methodological research in question design

- National Center for Health Statistics original developers
- Access to past evaluation projects
- Methodological research in question design

## Individual questions
Test reports
Test findings
Methods used
Question characteristics

- Individual questions
- Test reports
- Test findings
- Methods used
- Question characteristics

## Numerous, open categories 
Accessed by drop-down list
Examples:
Demographic:  Income, Race, Gender, Nationality
Health:  Insurance, Chronic conditions, Smoking
Employment: Status, Occupation, Job change, Salary, Benefits 

- Numerous, open categories
- Accessed by drop-down list
- Examples:
- Demographic:  _Income_, _Race_, _Gender_, _Nationality_
- Health:  _Insurance_, _Chronic conditions_, _Smoking_
- Employment: _Status_, _Occupation_, _Job change_, _Salary_, _Benefits_

## Events, Actions and Behaviors
Objective Characteristics
Subjective Characteristics
Speculations
Attitudes/Opinions
Knowledge Tests
Other

- Events, Actions and Behaviors
- Objective Characteristics
- Subjective Characteristics
- Speculations
- Attitudes/Opinions
- Knowledge Tests
- Other

## Yes/no
Select one textual
Select one numeric
Mark all that apply
Allocation
Open-ended textual:  Delineated & Non-delineated
Open-ended textual: Delineated & Non-delineated
Other 

- Yes/no
- Select one textual
- Select one numeric
- Mark all that apply
- Allocation
- Open-ended textual:  Delineated & Non-delineated
- Open-ended textual: Delineated & Non-delineated
- Other

## Interviewer difficulty
Visual design problem
Missed instructions
Unknown terms
Ambiguous concepts 
Overly complex
Assumption
Double barrel
Questionnaire effects
Recall & estimation error
Inadequate response options
Problematic answer space
None as tested 

- Interviewer difficulty
- Visual design problem
- Missed instructions
- Unknown terms
- Ambiguous concepts
- Overly complex
- Assumption
- Double barrel
- Questionnaire effects
- Recall & estimation error
- Inadequate response options
- Problematic answer space
- None as tested

## Survey Research Tool for:
Questionnaire Designers
Subject Analysts
Survey Methodologists

- Survey Research Tool for:
- Questionnaire Designers
- Subject Analysts
- Survey Methodologists

## Co-Sponsors: 
Office of Behavioral and Social Science Research, 
Bureau of the Census, 
National Science Foundation, 
National Cancer Institute, 
Bureau of Labor Statistics
NCHS provides internet access

- Co-Sponsors:
- Office of Behavioral and Social Science Research,
- Bureau of the Census,
- National Science Foundation,
- National Cancer Institute,
- Bureau of Labor Statistics
- NCHS provides internet access

## On-line to steering committee members
Testing and refinement
Broader access possibly by end of year
Looking for sponsors and partners

- On-line to steering committee members
- Testing and refinement
- Broader access possibly by end of year
- Looking for sponsors and partners

## Past 
Integrating interagency needs
Additional fields for establishment surveys
Additional fields for self-administered surveys
Reworking existing categories: response error typologies, question type typologies, question topic categories
Establishing procedures
Methodological requirements of evaluation reports

- _Past _
- Integrating interagency needs
- Additional fields for establishment surveys
- Additional fields for self-administered surveys
- Reworking existing categories: response error typologies, question type typologies, question topic categories
- Establishing procedures
- Methodological requirements of evaluation reports

## Current
Coding consistency
Populating the database
Staffing limitations
Funding limitations

- _Current_
- Coding consistency
- Populating the database
- Staffing limitations
- Funding limitations

## Future
Maintaining a methodological research agenda
Continual eye toward development

- _Future_
- Maintaining a methodological research agenda
- Continual eye toward development

## Searching in Q-BANK

- Agency Name
- Survey Title
- Response Error
- Advanced Searches

## Q-BANK Search Fields

## Agency Name Search

## Agency Name Search

## Agency Name Search

## Q-BANK Reports

## Searching on Survey Title

## Searching on Response Error

## Advanced Searches

## Advanced Searches

## Cognitive Testing Final Reports

- Don’t get lost in the weeds
- Important information about a tested question is lost without referencing the final report
- Conclusions about specific questions can be distorted without the final report

## Contents of a Typical Report

- Introduction
- Method
- Procedures
- Summary of findings
- Question-by-question analysis

## Why refer to the final report?	

- Search categories may not be straightforward
- Example: “At this time, is **(CHILD) **covered by health insurance that is provided through an employer or union or obtained directly from an insurance company?”
- 	Type of response error = double barreled question

## Why refer to the final report?

- Search categories may not be straightforward
- Example: “At this time, is **(CHILD) **covered by health insurance that is provided through an employer or union or obtained directly from an insurance company?”
- 	Type of response error = double barreled question
- The interpretive process and underlying construct can’t be gleaned from the search category
- Example: “Have you ever taken any herbal supplements for yourself?”
- 	Type of response error = ambiguous concept

## Final Reports

- Q-Bank is a good tool for quickly finding relevant reports of different cognitive testing projects
- Searches not meant to replace final reports

## Questions?

- Kristen Miller
- KSMiller@CDC.GOV