Count me in: challenges of conducting a cognitive interview study with 

adults with an intellectual and developmental disability

Annie MacFadyen, Amanda Wilmot, Amanda Titus

National Center for Health Statistics

Collaborating Center for Questionnaire Design and Evaluation Research

2023 Federal Committee on Statistical Methodology Research and Policy Conference 

October 25, 2023

National Center for Health Statistics
Acknowledgements

 Study conducted on behalf of the Administration for 

Community Living and the National Center for Health 

Statistics 

 NCHS CCQDER Staff

• Kristen Miller (CCQDER Director)

• Amanda Wilmot

• Amanda Titus

• Lauren Creamer

• Stephanie Willson 

 All our research participants
Outline

 Project background

 Methodology

 Project lifecycle

• Set up

• Data collection

• Analysis and write up

 Lessons learned
Purpose

 Share strategies to facilitate effective interviews

• Supported by experience

 Cognitive evaluation study

 People with ID/DD are rarely interviewed in cognitive interview studies

“I’ve had problems throughout my life when I’ve tried to voice my 

opinion, people just shut me out. They don’t want to listen to nothing I 

have to say.”
Project background
What are intellectual and developmental disabilities?

 Diverse group of conditions due to an impairment in physical, learning, 

language, behavior areas

 Begin during development, before age 22

 Affect a person’s day-to-day life

 Intellectual disability is a type of developmental disability
Background of the study

 People with ID/DD can participate in cognitive interviewing 

studies

 “Nothing about us without us”

 Challenges to our typical project model emerge

• With responsiveness and respondent-centered approach it 

is not only possible but also beneficial 

• Challenges are not excuses for exclusion
Methodology
Cognitive interviewing methodology

 Purpose of question evaluation

• Detect flaws

• Understand the question constructs

 Interpretivist cognitive interviewing style

• Focus on respondent’s personal experiences

• Data: the narrative of how the respondent came to their answer

• Emergent, unscripted probing

• Elicit respondent narratives and life experiences
Project methodology 

 Two rounds of one-on-one cognitive interviews

 Interviews with adults with ID/DD and, for some, a proxy 

respondent

• Round 1: 27 interviews, Fall 2019 (in-person)

• Round 2: 29 interviews, Spring 2023 (virtual)

 About one hour each

 $40 renumeration

 16 questions evaluated on functional difficulties 
Project Lifecycle
Typical lifecycle of a CCQDER cognitive interviewing 

study

Ethical 

research

Principal 

Investigator 

assigned to 

project IRB/OMB 

approvals

Questions for 

evaluation

Interviewer 

and recruiter 

briefing

Sample 

design

Respondent 

recruitment

Screening 

and 

scheduling

Informed 

and 

voluntary 

consent 

Interpretive 

Cognitive 

interviewing

Thank you 

letter and 

payment

Summary 

notes 

(Q-Notes)

Data analysis

Report 

writing

Report 

publication 

(Q-Bank)

Set-up

Analysis

Data collection
Working with respondents with ID/DD
Typical lifecycle of a CCQDER cognitive interviewing 

study

Set-up

Analysis

Data collection

• ERB/OMB

• Finalizing questions

• Purposive sample plans

• Preparing team

• Ads and flyers

• Screening and scheduling

Ethical 

research

Principal 

Investigator 

assigned to 

project IRB/OMB 

approvals

Questions for 

evaluation

Interviewer 

and recruiter 

briefing

Sample 

design

Respondent 

recruitment

Screening 

and 

scheduling

Informed 

and 

voluntary 

consent 

Interpretive 

Cognitive 

interviewing

Thank you 

letter and 

payment

Summary 

notes 

(Q-Notes)

Data analysis

Report 

writing

Report 

publication 

(Q-Bank)
Setting up the project: challenges

 Challenges

• Recruitment

• Gate-keeping in the ID/DD community

• Typical methods to recruit hard-to-reach 

populations did not work

• Initial lack of rapport and trust

• Screening and scheduling

• Schedule management

• Increased time
Setting up the project: solutions 

• ‘State to local’ approach

• Sponsor brokered introductions to organizations at the high level

• Foundation formed for local-level relationships

• Consistent person to interface with organizations

• Embracing longer processes

• Repeated explanation of the study

• Multiple calls, as needed

• Relationship building for each respondent and their loved ones
Typical lifecycle of a CCQDER cognitive interviewing 

study

Set-up

Analysis

Data collection

• Continual informed consent

• Interpretive cognitive interview

• Closing interview

• Thank you letter and payment
Data collection: challenges

 Challenges

• Informed consent

• Is consent capacity impaired for those with ID?

• Communication 

• Diverse language and communications abilities

• Use of non-verbal cues

• Varied depth of description

“I have a lot of difficulty remembering a lot of things. If you tell me too much 

information at once, it won’t register as quick in my brain as you’d think it 

would [...] if you break it down and tell me little by little, then I can repeat it 

back to you.”
Data collection: solutions part 1 

• Continual, affirmative consent 

• Repeated explanation of the study before interview started

• Reminders during and after interview

• Additional safeguards in place as needed
Consent capacity

1. Please tell me, in your own words, what you will be asked to do if you take part in 

this study?

2. When I say your taking part is completely voluntary, what does that mean to you?

3. Do you have to answer all of the questions in this study?

4. What will you tell me if you don’t want to answer a question?

5. What can you do if you start the study but don’t want to finish it?

6. If your [aide/caregiver/LAR] wants you to take part, but you do not want to, are you 

still allowed to refuse?
Data collection: solutions part 2 

• Rapport building through consistency

• One recruiter: same person to screen, schedule, and set up interview

• Returning at the end of virtual interviews to say goodbye

• Interpretive interview style as a strength

• Flexibility of approaches, probe styles, and pace

“You have to explain things to me in a different way. So that I can understand.” 

• Utilizing experienced, adaptable interviewers

• Understanding emerges more slowly, or even post-interview
Typical lifecycle of a CCQDER cognitive interviewing 

study

Set-up

Analysis

Data collection

• Iterative review of videos

• Summary notes 

• Data analysis 

• Report writing
Analysis: challenges

 Challenges

• Complexity of analysis

• Diversity within the sample itself

• Non-linear structure of the conversation

• Cognitive delay and order effects

• Non-verbal cues
Analysis: solutions 

• Embracing longer processes

• Expecting to learn additional information through iterative review

• Organization as well as analytic summary

• Video recording was vital

• Debriefing notes for all interviews

• Systematic way to highlight and preserve certain aspects of the interview

• Respondent self-perception

• Interviewer adjustments

• Two principal investigators

• Tracking complex patterns

• Limit potential biases 
Summary
ONE-SIZE-DOES-NOT-FIT-ALL.jpg (640×300) (people-equation.com)
Lessons learned

 Cognitive interviews with adults with ID/DD are rewarding

• Motivated community with strong self-advocacy 

• Sample should be constructed carefully to account for 

heterogeneity within group

• Experienced, adaptable interviewers should be used to make the 

most of each interview

• Expectation of greater time and effort for all stages of project

• Understanding that both informed consent and good rapport are 

vital to cultivate throughout all stages 

 It is the responsibility of the research team to include people with 

IDD directly in any research about them.
Selected references

 Horner-Johnson, W., & Bailey, D. (2013). Assessing Understanding and Obtaining 

Consent from Adults with Intellectual Disabilities for a Health Promotion Study. 

Journal of policy and practice in intellectual disabilities, 10(3), 10.1111/jppi.12048.

 Jeste, D. V., Palmer, B. W., Appelbaum, P. S., Golshan, S., Glorioso, D., Dunn, L. B., ... 

& Kraemer, H. C. (2007). A new brief instrument for assessing decisional capacity 

for clinical research. Archives of general psychiatry, 64(8), 966-974.

 Miller, K. (2014). Introduction. In K. Miller, S. Willson, V. Chepp & J.L. Padilla (Eds.), 

Cognitive interviewing methodology. Hoboken, NJ: Wiley.
For more information, contact CDC

1-800-CDC-INFO (232-4636)

TTY: 1-888-232-6348 www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the 

official position of the Centers for Disease Control and Prevention.

Thank you!

Annie MacFadyen

AMacFadyen@cdc.gov