Understanding the Interviewer in Cognitive Interviewing: Reflexivity 

and Improving Data Quality

Zachary Smith, MA

Collaborating Center for Questionnaire Design and Evaluation Research

Federal Committee on Statistical Methodology Research and Policy Conference

Washington, DC

October 25-27, 2022

National Center for Health Statistics

The findings and conclusions in this presentation are those of the authors and do not necessarily represent the official position of 

the National Center for Health Statistics, Centers for Disease Control and Prevention. 
“We are not hemmed in by the fixed scope, order, and wording of items 

on a survey questionnaire...by being responsive to informants, we can 

evade the restrictions imposed by our a priori thinking about which 

topics are important and what they mean. By being a bit more directive, 

we can pull the conversation back to issues we need to address, even as 

we continue to encourage our interviewees to speak in terms that are their 

own. Throughout the interview, we retain freedom to probe, follow up, 

challenge, double back, abandon a fruitless line of inquiry, ask if we have 

understood correctly, or simply express our fascination and ask the 

interviewee to say more.”

- Soss, Joe, “Talking Our Way to Meaningful Explanations,” Interpretation and Method (New York: Routledge, 2014), p. 169
Outline

 Reflexivity: what is it?

 Applicability to cognitive interviews

 Study details and context

 Findings

 Preliminary recommendations
Reflexivity: what is it?
Reflexivity: definitions

“An awareness of how the habits and experiences that one is bringing to 

bear on a situation shape and construct that situation” (Jackson 2014: 271)

 Critical consciousness of own actions in research process (Crapanzano 2010: 56)

 Counter to objectivity: researcher as instrument/means; need to 

document and analyze researcher role (Schwartz-Shea 2014: 133)

“A keen awareness of, and theorizing about, the role of the self in all 

phases of the research” (Schwartz-Shea 2014:133)
Reflexivity: methodology and 

methods

“Centrally, it demands the explicit 

articulation of how the research was 

actually done, why, and with what 

effects for the resulting interpretation 

that is presented.” (Wilkinson 2014: 402)

Reflexive journal (Lincoln and Guba 1985, Roller 

and Lavrakas 2017)

Writing practices, member checks, 

methodological description, attention to 

researcher role in generation of 

evidence (Schwartz-Shea 2014: 138)

Roller and Lavrakas (2015: 42) ->
Reflexivity: cognitive interviewing

“The goal is not to take interviewers out of the process or otherwise 

minimize their ‘footprint’ on the interaction, but to clearly demarcate and 

understand their role in it.” (Willson and Miller 2014: 33)

 Reduction of bias – undue distortion of research outcomes

 Reduction/interrogation of asymmetrical power relation (Kvale 2006)

 Transparency/credibility in cognitive interviewing report (Chepp and Scanlon 2014)

 Related to, but not identical with, literature on interviewer effects; 

potential for fruitful cross-epistemological dialogue (West and Blom 2017)
Study details and context
Reflexivity study: details

 Multi-mode: notes and in-depth interviews

 Summary notes:

• In what ways did you feel you influenced the direction of and/or data gathered in 

the interview? Consider, for example, your positionality, the way you framed 

probes, whether certain topics were more or less comfortable for you, whether 

you had any difficulties with the questionnaire, or any other aspect of the 

interview you find relevant.

 In-depth interviewers

• Hour to two-hour interviews with each interviewer (n = 6)

 Study context: cognitive interviews evaluating questions on cannabis

• Interviews n = 90; interviewer n = 7
Study findings
Findings: a framework for analyzing reflexivity
Findings: challenges and 

interviewer presence

“The respondent was a young new mother. She had her baby 

with her for much of the interview. It was a little distracting

from time to time but the respondent was making an effort to pay attention and answer the questions, so 

I did my best and proceeded. She gave the baby to her brother about two-thirds of the way through, 

which gave us some completely uninterrupted time. She shared with me early in the interview that her 

mother had passed away a couple of years ago from cancer. It was in this context that she heard about 

CBD products for pain relief. I offered my condolences and noted that this must have been very difficult

for her to go through. Learning about that and also being able to identify with 

the challenges of early motherhood she must be going through made me feel 

some emotional connection with the respondent. I think that this helped me 

to be patient and flexible as the respondent juggled caring for her baby. I felt 

uncomfortable asking her about whether she used marijuana while pregnant 

but she seemed unfazed by the question, answering that she did not smoke 

while pregnant.”
Findings: technical considerations

 Problems with instrument 

 Probing style – concurrent or retrospective?

 From interview to summary notes
Findings: prior knowledge

 Different kinds of knowledge:

• Of respondent directly

• Positive: flow, digging deeper

• Negative: off-limits topics, assumed shared knowledge

• Of subject matter

• Do we know the right questions to ask?

• From prior interviews
Findings: presence and challenges

 Unmediated interviewing process
Findings: presence and challenges

 Mediated interviewing process
Findings: setting and baggage

 Setting: 

• Location of interview, interviewer, and interviewee 

“R was sitting cross-legged in a closet for privacy and was casually dressed in a hoodie. I was also 

at home in a soft armchair, no headset, no official background. I think that our relatively casual 

and informal settings created an informal, almost intimate setting for the interview. This may have 

helped the respondent to open up about how marijuana helps her cope with anxiety and 

depression.”

 Baggage: 

• Assumptions about respondents

• Troubling respondent behaviors 

• Personal views on cannabis 

“I am a proponent of legalizing marijuana so I think it was easy for me to listen to and sympathize with 

her story. I don’t share these views with respondents, but I’m sure they bias me. I may not have 

probed as much as I could have for some questions because of this.”

• Discomfort with topics or instrument
Findings: co-creation of data

“They’re not going to answer a question you don’t ask.”

 It’s not just getting the data, getting the narrative

• “It’s not like I can scoop the info out of the respondent’s head without impact on 

the data at all.”

• The respondent narrative is “already framed by the [survey] questions and how I 

choose to ask about them.”

 Respondents learning how to do a cognitive interview

 Interviewers learning how to probe

• “Just as they’re learning how to do a cognitive interview, you’re learning how to 

probe them, for better or for worse.”
Preliminary recommendations
Preliminary conclusions and recommendations

 More than just minimizing bias

 Understanding constitution of data, broadly defined

 How to assess?

• Use of single question vs. reflexivity journal

 Implications for analysis?

• Tagging/coding respondents based on use of single question

 How to document?

• Separate section of topline analysis on single question

• Integration into question-by-question findings
For more information, contact CDC

1-800-CDC-INFO (232-4636)

TTY: 1-888-232-6348 www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the 

official position of the Centers for Disease Control and Prevention.

Thank you!

For more information contact: Zachary Smith, zsmith@cdc.gov

Q-Bank: providing access to survey question evaluation reports, question design and performance https://wwwn.cdc.gov/qbank/

Q-Notes: designed to facilitate the management and analysis of cognitive 

interviews https://www.cdc.gov/nchs/ccqder/products/qnotes.htm