Supplementing Cognitive Interviewing with Natural Language Processing Approaches from Data Science

Stephanie Eckman, RTI International

Katherine Blackburn, RTI International

Peter Baumgartner, RTI International

David Henderson, RTI International

Y. Patrick Hsieh, RTI International

Patricia Green, RTI International

Kelly Kang, National Center for Science and Engineering Statistics (NCSES)

www.rti.org

RTI International is a trade name of Research Triangle Institute. RTI and the RTI logo are U.S. registered trademarks of Research Triangle Institute.

1

2

COVID Module for SED

Survey of Earned Doctorates (SED)

New questions measuring the impact of COVID pandemic on doctoral students’ graduate experiences and career plans

2021 SED (July 2020-June 2021 graduates): 

Added questions on the impact areas 

Followed by open-ended questions

2022 SED (July 2020-June 2022 graduates): 

Similar questions on the impact areas

Followed by close-ended questions 

Sponsored by the National Science Foundation, the National Institutes of Health, the U.S. Department of Education, and the National Endowment for the Humanities, the Survey of Earned Doctorates or SED is an annual census of all individuals who earn a research doctorate in the US. This survey is conducted to assess the changing characteristics of doctorate recipients over time and initial postdoctoral employment opportunities over time and support informed decision making about how to support and improve of this country’s doctoral education system.

In 2021, the SED fielded new questions to collect data about the impact of the COVID pandemic on doctoral students. Because the situation was novel, open-ended follow-up questions were used to obtain more information, which is rare for a survey with about 55,000 respondents per year

The responses captured from the open-ended follow up questions for the COVID impact areas in 2021 survey were used to develop a set of close-ended follow-up questions for the 2022 survey.  

2

Starting Point: COVID Impact Questions

3

Questions added in 2021

Yes and No response options

Each Yes response followed by open-ended question (with Yes and No response options)

How did the timeline for completing your doctoral degree change?

How was your research disrupted?

etc.

The timeline for completing my doctoral degree changed.

My research was disrupted.

Funding for my doctoral studies was reduced or suspended.

My immediate postgraduate employment or education plans changed. 

My longer term career plans or goals changed (e.g., type of employer, research focus).

My plans about where to live in the year after graduation were affected.

My graduate experience or career plans changed in other ways.

7 COVID impact core questions were added to 2021 SED with Yes or No option.  ‘Yes’ response had a follow-up open-ended question to capture more specific verbatim responses.

Open end follow up questions:

How did the timeline for completing your doctoral degree change?

How was your research disrupted?

How was your funding reduced or suspended?

How did your employment or education plans change?

How did your career plans or goals change?

How did it impact your decision on where to live in the year after graduation?

In what other ways did your graduate experience or your plans change?

These questions went through some cognitive testing to confirm the wording, and then were implemented on the 2021 SED. 

For the 2022 round, we wanted to:

Revise questions to make them more clear

provide closed response options that included the most commonly mentioned themes from the open-ended responses reported in 2021.

3

NLP Cluster Analysis

Initial

Quantitative Results

Question Revision Process

Manual Coding & New Questions

Cognitive 

Interviews

Qualitative Results

4

Open-End 

Verbatim Data

Revised 

Questions

NLP Cluster Analysis

Full

This is an overview of the timeline for the project. 

The open ended responses collected from the 2021 SED respondents were used in two different ways. I will refer to these throughout the presentation as the “qualitative arm” which is the process shown on top in this diagram, and the “quantitative arm” which is the process shown on bottom. 

First, survey staff reviewed the open-ended verbatim responses to develop and manually code response options for the close-ended follow-up question for each impact area. 

The new closed-ended questions were then evaluated with cognitive interviews. 

This resulted in a report based on the qualitative data collected from the cognitive interviews. 

Simultaneously, the “quantitative arm” of the project used natural language processing and cluster analysis of the same open-ended verbatim responses to identify clusters of common themes. 

Both arms contributed to decisions on the final response options for the close-ended questions to be added in the 2022 SED. 

Similar themes emerged from the NLP & cluster analysis and the manual coding & cognitive interviews. 

These results suggest that data science techniques work well to supplement cognitive interviews for questionnaire testing. 

Now we will walk through these steps more in depth. 

4

5

4,021 open-ended responses from 1,305 respondents to 2021 COVID impact questions

Qualitative: Question Testing

Manually coded into themes

Used to create response options for the closed ended follow-up questions for further testing

First, let’s focus on the qualitative arm of the research. 

The dataset used for the qualitative research contained 4,021 responses from the first 1,305 respondents in the 2021 SED. 

These respondents open ended responses were manually coded into themes to develop close-ended responses to test for the 2022 SED. 

5

Revised COVID Impact Questions 

6

Questions in 2022

Same Yes or No response options

Added more context in the questions with examples 

Closed-ended follow-up items (with Yes and No response options) added to capture primary responses provided in 2021

The pandemic changed the timeline for completing my doctoral degree. 

The pandemic led to a reduction or suspension of funding for my doctoral studies. 

The pandemic affected my research (e.g., limited access to resources, changed research plan).

The pandemic changed my immediate postgraduate employment plans (e.g., limited job opportunities, less desirable employment, work visa status).

The pandemic changed my longer term career plans (e.g., pursuit of different type of job or employer).

The pandemic affected my plans about where to live in the year after graduation. 

The pandemic changed my graduate experience or career plans in other ways.

Based on the results of the manual coding, we revised the impact questions to provide more context and specific examples. 

We also developed closed-ended responses. 

On the left are the revised COVID impact questions. In the instrument, these were presented in a grid with yes/no response options. 

Each YES response triggered follow up questions about the impact on the students’ studies or future plans. 

6

7

The pandemic changed the timeline for completing my doctoral degree. 

The pandemic led to a reduction or suspension of funding for my doctoral studies. 

The pandemic affected my research (e.g., limited access to resources, changed research plan).

The pandemic changed my immediate postgraduate employment plans (e.g., limited job opportunities, less desirable employment, work visa status).

The pandemic changed my longer term career plans (e.g., pursuit of different type of job or employer).

The pandemic affected my plans about where to live in the year after graduation. 

The pandemic changed my graduate experience or career plans in other ways.

New Follow-up Items 

(If Yes)  How was your research affected?

                 Select Yes or No for each
| As a result of the pandemic, I have limited or no access to resources I need (e.g., lab, data, hardware, software, human subjects, archives). |
|-----------------------------------------------------------------------------------------------------------------------------------------------|
| I had to make changes to my research plan (e.g., goals, topic, focus, approach, scope) as a result of the pandemic.                           |
| The pandemic disrupted my research in other ways - Specify:                                                                                   |

Here is the follow up question about the COVID impact on the doctoral student’s research. 

7

8
| As a result of the pandemic, there are limited job opportunities in the employment I desire.                                       |
|------------------------------------------------------------------------------------------------------------------------------------|
| I may have to accept a less desirable job in terms of the type of position, employer, and/or location as a result of the pandemic. |
| I encountered difficulty with my visa status or international travel as a result of the pandemic.                                  |
| The pandemic changed my immediate postgraduate employment plans in other ways - Specify:                                           |

New Follow-up Items 

The pandemic changed the timeline for completing my doctoral degree. 

The pandemic led to a reduction or suspension of funding for my doctoral studies. 

The pandemic affected my research (e.g., limited access to resources, changed research plan).

The pandemic changed my immediate postgraduate employment plans (e.g., limited job opportunities, less desirable employment, work visa status).

The pandemic changed my longer term career plans (e.g., pursuit of different type of job or employer).

The pandemic affected my plans about where to live in the year after graduation. 

The pandemic changed my graduate experience or career plans in other ways.

(If Yes)  How did your immediate postgraduate plans change? Select Yes or No for each

And the impact on immediate postgraduate plans. 

8

9
| As a result of the pandemic, I plan to pursue my career in a different type of job or field than I had considered before.                               |
|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| I plan to pursue my career with a different type of employer (e.g., from academe to industry) than I had considered before as a result of the pandemic. |
| The pandemic opened new opportunities in the areas I had not considered before.                                                                         |
| The pandemic changed my longer-term career plans in other ways – Specify:                                                                               |

New Follow-up Items 

The pandemic changed the timeline for completing my doctoral degree. 

The pandemic led to a reduction or suspension of funding for my doctoral studies. 

The pandemic affected my research (e.g., limited access to resources, changed research plan).

The pandemic changed my immediate postgraduate employment plans (e.g., limited job opportunities, less desirable employment, work visa status).

The pandemic changed my longer-term career plans (e.g., pursuit of different type of job or employer).

The pandemic affected my plans about where to live in the year after graduation. 

The pandemic changed my graduate experience or career plans in other ways.

(If Yes)  How did your longer-term career plans change?   Select Yes or No for each

And the impact on long term career plans. 

Note that each set of closed-end responses has an Other-Specify option 

Additionally, the final impact question “The pandemic changed my graduate experience or career plans in other ways” is also followed up with an open-ended question as catch-all. 

9

Cognitive Interviews

One-on-one in-depth interviews with recruited doctoral students over Zoom

Tested new question wording – comprehension, response process, and missing themes

10

Interviewers trained to conduct cognitive interviews

Cognitive interviews completed

Approximate hours for training, completing interviews, writing notes, analyzing results, and producing reports

4

60

500

After we had the revised questions and response options, we tested them via cognitive interviews.

The cognitive interviews consisted of one-on-one in-depth interviews with recruited doctoral students over Zoom. 

Respondents to answered the survey questions, and then answered a series of probes about each question, diving further into their understanding of the question text and their response process. 

A diverse sample across public vs private institutions, sex, citizenship status, and field of study was selected to represent the SED survey population as much as possible. 

10

Qualitative Findings – Cognitive Interviews

Recommended question changes (small tweaks)

Confirmation of previous manually coded themes 

Missing themes

Limited access to collaborators/advisors as a “resource”

Limited networking opportunities

Mental health

Positive impacts

Accelerated doctorate completion timeline, more time to work, opportunities for pandemic-related research

Increased job opportunities because of remote work option

11

Our final qualitative findings suggested question changes, which consisted primarily of small tweaks. 

For example, we found that respondents sometimes forgot the original question stem when answering the follow ups, so we included reminders in follow up questions (e.g., “long-term career plans”). 

We also confirmed that the closed ended questions captured the majority of participants’ experiences during the pandemic. 

We did uncover evidence of some themes that were missing in the new survey questions. 

Limited access to collaborators/advisors as a “resource”

Limited networking opportunities 

Mental health during the pandemic

And we found that respondents sometimes wanted to talk about positive impacts of the pandemic:

	Opportunities increased because of remote work option

	speed up timeline

	more time to work

	involved in pandemic-related research

11

4,021 open-ended verbatim responses from 1,305 respondents

12

Quantitative: Initial Analysis

NLP of text responses

Create semantic embedding for each response, represented by a vector

Step 1

Step 2

Findings:

Cluster analysis supported key findings from qualitative analysis

Identified new themes missing from initial qualitative analysis

NLP = Natural Language Processing

Clustering algorithm

Group similar vectors into clusters

Now let’s talk about the quantitative arm of the analysis. 

We used the same 4,021 responses that were used in the manual coding. 

First, we used natural language processing to an embedding of each text response. The algorithm turns the text response into a vector of numbers. 

Next, a clustering algorithm was used to group similar vectors into clusters. The clustering algorithm assigned a cluster to about 2/3rds of responses (responses not assigned a cluster were not similar enough to other groups of responses)

This initial analysis identified clusters that were similar to those found in the qualitative analysis, but also identified clusters that were not included in the qualitative work 

12

Quantitative: Full Analysis

13

Then we used a larger set of responses collected during the 2021 round and repeated the analysis. The goals for this full analysis step were:

to examine changes in the open-ended responses over time – as you likely remember, during this stage of the pandemic, things were changing quickly

to assess whether additional response options should be added to the close-ended follow-up questions for the 2022 SED.  

We found that the proportion of responses that are uncoded, outliers, or “other” increased in the full predicted dataset. 

If these proportions had remained relatively similar across datasets, we could have justified keeping the existing coding scheme, however, this was not the case.

13

Reran process on all responses received before Feb 17, 2021 (n=49,903)

NLP of text responses

Analysis Goals

Assess changes in responses over time

Clustering algorithm

Uncover possible new response options to include

NLP Cluster Analysis

Initial

Quantitative Results

Question Revision Process

Manual Coding & New Questions

Cognitive 

Interviews

Qualitative Results

14

Open-End 

Verbatim Data

Revised 

Questions

NLP Cluster Analysis

Full

As a reminder, at this point in the project we had two sets of proposed clusters or themes 

One coming out of the cognitive interviews

One coming out of the quantitative analysis on the large data set

So a natural question is whether there is any agreement between the two

14

Qualitative and Quantitative Methods Identified Similar Themes

15

Qualitative: 27 Clusters

Quantitative: 54 Clusters

Research setbacks and delays

Lab closure

Employment difficulties

Change to postdoc plans

Reconsidering academia

Accelerated and delayed graduation

Relocation or travel issues

Here are examples of the clusters that emerged with the largest number of responses. These clusters were captured in both sets

An example is the cluster “lab closure”, which maps onto the response option: “As a result of the pandemic, I have limited or no access to resources I need (e.g., lab, data, hardware, software, human subjects, archives).” 

This topic was captured in both the qualitative and quantitative analyses

15

16

5 clusters: 

transition to remote work

considering remote employment

illness of self or family

personal health/ family/childcare issues

childcare impacting dissertation

2 clusters:

missed opportunities

general uncertainty

Qualitative: 27 Clusters

Quantitative: 54 Clusters

Research setbacks and delays

Lab closure

Employment difficulties

Change to postdoc plans

Reconsidering academia

Accelerated and delayed graduation

Relocation or travel issues

Qualitative and Quantitative Methods Identified Similar Themes

We were surprised and pleased to see how well the themes identified by the qualitative and quantitative approaches agreed.  

Of the 54 quantitatively derived clusters, 49 were related to the topics identified through our manual review. The topics unique to the quantitative analysis were about personal illness, childcare issues, and serious consideration of working remotely in the future. 

Of the qualitatively developed clusters, only 2 were not clearly associated with the NLP clusters. These focused on missed opportunities for collaboration or attending conferences in person and a general uncertainty about the future.  

All clusters on the edges of the Venn diagram had relatively few responses associated with them. 

While the themes overlapped, the relative importance of the themes varied by method. For example, the proportion of responses in quantitatively derived clusters about timeline disruptions (delays or accelerations) was double the proportion found in the initial manual review of responses. This finding made sense as the manual coding was based on the responses provided early in the data collection while the quantitative analysis included many more responses and responses from those who were graduating later in the academic year who had more opportunity for their timelines to change. 

Similarly, while the cognitive interview data showed us that many people spoke about their mental health difficulties, the cluster analysis revealed that responses to the survey questions were much less focused on mental health issues. Participants may have seen the cognitive interview as a space to vent and discuss the struggles they faced in the past year. This environment could have led to a higher report of mental health issues compared with spontaneous report on the open-ended survey questions. Due to the goals of the COVID questions in the SED, and the small percentage of overall responses focused on mental health issues, we decided not to proceed with adding a new question.

16

Qualitative and Quantitative Findings Together– Revised Questions

Cognitive interviews are limited

How much weight do you put on the feedback received during the interviews as you make question changes? 

NLP & Cluster Analysis helped to guide the changes with increased confidence

Example: Mental health came up frequently during the interviews but was a relatively small cluster within the NLP/ML analysis

17

Together, the qualitative and quantitative data helped guide the decision process of what question changes to implement, as well as give weight to the new themes that came up during cognitive interviewing. Because of the overlap between the themes that emerged in the qualitative and quantitative data, we were able to inform the decision about whether to move forward with new topics. 

For example, the topic about mental health. This topic came up during almost half of the cognitive interviews. However, during the cluster analysis, two clusters emerged, one focused on emotional and personal issues (1.8% of responses to COVG) and another about isolation and the lack of social interaction (6.1% of responses to COVG). Together, these two clusters contain 1.7% of responses across all questions, and 7.9% of responses to the original COVG. While the cognitive interview data showed us that many people spoke about their mental health difficulties, the cluster analysis revealed that responses to the survey questions were much less focused on mental health issues. Due to the goals of the COVID questions in the SED, and the small percentage of overall responses focused on mental health issues, we decided not to proceed with adding a new question.

Another possible issue around this topic of mental health is that participants may have seen the interview as a space to vent and discuss the struggles they faced in the past year. This environment could have led to a higher report of mental health issues compared with spontaneous report on the open-ended survey questions. 

A limiting factor of cognitive interviews is how much weight to give the feedback that comes up during the interviews. Using the combination of both the quantitative and qualitative analysis helped us make a decision about how much importance mental health issues hold for doctoral students during the pandemic. 

17

18

Future Applications

Use NLP rather than coding to identify themes in open ended responses

Cost of NLP clustering (n=50K text responses) ≈ Cost of manual coding (n=4K text responses)

Fewer cognitive interviews needed to test and validate coded responses 

Use machine learning to code Other-specify responses

Identify needed changes to existing response options

While it is rare to be able to implement open-ended questions on such a large survey, the technique was helpful to confirm what question changes to move forward with. The quantitative analysis also confirmed that the manual coding and closed ended questions that were created included the majority of respondent’s issues around the pandemic. 

In the future, this technique could help to:

1. Identify new response options for questions with open-ended responses. Given the overlap in the manually developed codes and the quantitatively developed clusters, it would be faster to use NLP to identify the key themes and have methodologists review the unclustered responses to see if there are important themes missed by the NLP coding. 

2. Reduce cognitive interview burden (time and money) by supplementing cognitive interview work with NLP. The cognitive interviews could focus more on wording and understanding of the questions.

3. While it was not the focus of  this presentation, the quant. approach will be most useful in coding the open-ended responses to specified response categories (substantially reducing labor costs). 

18

Thank you

19

Thank you!  

For more information, contact:

Stephanie Eckman (seckman@rti.org) 

19

# image1.png

# image2.png

# image5.png

# image8.png

# image9.svg

# image17.svg

# image18.png

# image19.svg

# image20.png

# image21.svg

# image10.png

# image11.svg

# image12.png

# image13.svg

# image14.png

# image15.svg

# image16.png

# image7.png