# Presentation: 823847

## 

Kristen Miller
Questionnaire Design Research Lab 
National Center for Health Statistics, USA

- Kristen Miller
- Questionnaire Design Research Lab
- National Center for Health Statistics, USA

- Purpose of cognitive test
- Structure of protocol
- Types of cognitive test data & possible analyses
- Things to keep in mind

## Interviews: Designed to examine the stages of the question response process
Comprehension, retrieval, judgment, response

Analysis: Identifies...
potential response errors:  (e.g. vague concepts, recall problems, inadequate response options)   
patterns of interpretation

- Interviews: Designed to examine the stages of the question response process
- Comprehension, retrieval, judgment, response
- Analysis: Identifies...
- potential response errors:  (e.g. vague concepts, recall problems, inadequate response options)
- patterns of interpretation

## Traditional Cognitive Testing

- Protocol Semi-structured
- Data collected are qualitative
- Qualitative methodologies for analysis
- Small samples:  approx. 10 – 30 participants

## Structured Questionnaire
Designed to illustrate:
Administration ease
Interpretations
Factors considered by respondent
Degree of consistency with physical abilities
Differences between subgroups

- Structured Questionnaire
- Designed to illustrate:
- Administration ease
- Interpretations
- Factors considered by respondent
- Degree of consistency with physical abilities
- Differences between subgroups

## Demographic Section:  Country, language, gender, age, SES

Question Testing Sections
Core Question
Interviewer Coding
Open-ended Follow-up Probe
Cognitive Follow-up Probe
Functioning Follow-up Probe

Health Questions:  subjective health, chronic condition list

Interviewer Debriefing

- Demographic Section:  Country, language, gender, age, SES
- Question Testing Sections
- Core Question
- Interviewer Coding
- Open-ended Follow-up Probe
- Cognitive Follow-up Probe
- Functioning Follow-up Probe
- Health Questions:  subjective health, chronic condition list
- Interviewer Debriefing

## Did the respondent...
need you to repeat any part of the question?  Yes/No

have any difficulty using the response options? Yes/No

ask for clarification or qualify their answer?  Yes/No

- Did the respondent...
- need you to _repeat_ any part of the question?  Yes/No
- have any difficulty using the _response options_? Yes/No
- ask for _clarification_ or qualify their answer?  Yes/No

## Vision			83.89%
Hearing			88.47%
Cognitive			85.72%
Mobility			91.03%
Self-care			93.65%
Communication		86.70%

N=1,287

- Vision			83.89%
- Hearing			88.47%
- Cognitive			85.72%
- Mobility			91.03%
- Self-care			93.65%
- Communication		86.70%
- N=1,287

## Number of response problems by age

|  | 0 | 1 or 2 | 3+ | n |
| --- | --- | --- | --- | --- |
| 18-25  | 58.29% | 12.56% | 29.15% | 199 |
| 26-30 | 58.26 | 18.39 | 22.99 | 174 |
| 31-40 | 69.42 | 14.05 | 16.53 | 242 |
| 41-50 | 61.54 | 19.46 | 19.00 | 221 |
| 51-60 | 64.43 | 17.45 | 18.12 | 149 |
| 61+ | 57.67 | 15.87 | 26.46 | 189 |
|  |  |  |  | 1174 |
| Chi-square=19.74, p<.05 |  |  |  |  |

## 
Can you tell me what you were thinking?  Why did you answer [fill with respondent’s answer to the core question]?

- Can you tell me what you were thinking?  Why did you answer [*fill with respondent’s answer to the core question*]?

## Varied in quality
A lot missing
Very brief
Provides some insight 
E.g. “Because I wear glasses”

- Varied in quality
- A lot missing
- Very brief
- Provides some insight
- E.g. “Because I wear glasses”

## How often they had difficulty
The amount of effort they needed to put into the activity 
The degree of concern or worry they had regarding the difficulty
Other domain-specific questions
Communication: Shyness question
Cognitive: Reason question

- How _often_ they had difficulty
- The _amount of effort_ they needed to put into the activity
- The degree of _concern or worry_ they had regarding the difficulty
- Other domain-specific questions
- Communication: Shyness question
- Cognitive: Reason question

## Do you believe that your memory or concentration difficulties are...
	[Mark all that apply] 
because you have too many things to do?
because you getting older? 
or, because of something else? 

Only busy:   	28.78%
Aging:		25.84%
Something else:	43.07%

- Do you believe that your memory or concentration difficulties are...
- 	[Mark all that apply]
- because you have too many things to do?
- because you getting older?
- or, because of something else?
- Only busy:   	28.78%
- Aging:		25.84%
- Something else:	43.07%

## Reason by degree of problem

|  | Sometimes | A lot | Can’t do
 | n |
| --- | --- | --- | --- | --- |
| Only busy | 89.05% | 10.95% | 0.00% | 137 |
| Aging | 77.24 | 22.76 | 0.00 | 123 |
| Something Else | 73.66 | 24.40 | 1.95 | 205 |

## Presence of a condition
Use of an assistive device
Degree of difficulty doing related activities 

- Presence of a condition
- Use of an assistive device
- Degree of difficulty doing related activities

## Examine the consistency between:
Washington Group Question responses and 
Follow-up functioning questions responses
Goal: explain the discrepancies
Misunderstood word?
Cultural difference?
Translation problem?
Interviewer error?
Error in the follow-up questions?
Other data assist in explanation

- Examine the consistency between:
- Washington Group Question responses and
- Follow-up functioning questions responses
- Goal: explain the discrepancies
- Misunderstood word?
- Cultural difference?
- Translation problem?
- Interviewer error?
- Error in the follow-up questions?
- Other data assist in explanation

## Response patterns:  Vision

| Pattern | Wears Glasses | Functioning Problem | Reports Disability |
| --- | --- | --- | --- |
| A | No | No | No |
| B | Yes | No (corrected) | No |
| C | No | Yes | Yes, disability |
| D | Yes | Yes (not corrected) | Yes, disability |
| E | Yes | No (corrected) | Yes, disability |
| F | No | No | Yes, disability |
| G | Yes | Yes (not corrected) | No |
| H | No | Yes | No |

## Response patterns: Hearing

| Pattern | Aid | Disability | Missed Words | Functioning Problem |
| --- | --- | --- | --- | --- |
| A | No | No | No | No |
| B | No | Yes | Yes | Yes |
| C | No | No | Yes | No |
| D | No | No | No | Yes |
| E | No | Yes | No | Yes |
| F | No | Yes | Yes | No |
| G | No | No | Yes | Yes |
| H | No | Yes | No | No |
| I | Yes | No | Yes | Yes |
| J | Yes | Yes | Yes | Yes |
| K | Yes | No | No | No |
| L | Yes | Yes | No | Yes |
| M | Yes | Yes | Yes | No |
| N | Yes | No | No | Yes |
| O | Yes | No | Yes | No |
| P | Yes | Yes | No | No |