Qualitative Code Suggestion: A Human-Centric Approach to Qualitative Coding
Authors: Cesare Spinoso-Di Piano
Samira Abbasgholizadeh Rahimi
Jackie CK Cheung
Venue: EMNLP 2023 Findings (2023)
Keywords: qualitative coding
human-centric
TL;DR: We approach qualitative coding in a human-centric fashion and show that this creates technical challenges which lead us to experiment with systems that can suggest relevant codes for highlighted passages.

Abstract: Qualitative coding is a content analysis method in which researchers read through a text corpus and assign descriptive labels or qualitative codes to passages. It is an arduous and manual process which human-computer interaction (HCI) studies have shown could greatly benefit from NLP techniques to assist qualitative coders. Yet, previous attempts at leveraging language technologies have set up qualitative coding as a fully automatable classification problem. In this work, we take a more assistive approach by defining the task of qualitative code suggestion (QCS) in which a ranked list of previously assigned qualitative codes is suggested from an identified passage. In addition to being user-motivated, QCS integrates previously ignored properties of qualitative coding such as the sequence in which passages are annotated, the importance of rare codes and the differences in annotation styles between coders. We investigate the QCS task by releasing the first publicly available qualitative coding dataset, CVDQuoding, consisting of interviews conducted with women at risk of cardiovascular disease. In addition, we conduct a human evaluation which shows that our systems consistently make relevant code suggestions.

Qualitative Code Suggestion:
A Human-Centric Approach To Qualitative Coding
Cesare Spinoso-Di Piano
Samira Abbasgholizadeh Rahimil2* Jackie Chi Kit Cheungl2
McGill University*
Mila Quebec AI Institute"
camira.rrhimin jackie. cheung@)mcgill......
Abstract
Interviemer: WWht sume the decisions that you lavw macie that have been cifficutt terms of managing and preventing card ovesculer disesses...
Code Suggettions
Qualitative coding is content analysis method in which researchers read through text cor- pus and assign descriptive labels or qualitarive codes passages. It is an arduous and man- ual process which human computer interaction studies have shown could greatly benefit from NLP techniques to assist qualitative coders. Yet, previous attempts at leveraging language technologies have set up qualitative coding as a fully automatable classification problem. In this work, we take a more assistive approach by defining the task of qualitative code sugges- tion (QCS) in which a ranked list of previously assigned qualitative codes is suggested from an identified passage. In addition to being user- motivated, QCS integrates previously igmored properties qualitative coding such as the se- quence in which passages are annotated, the importance rare codes and the differences annotation styles between coders. We in- vestigate the QCS task by releasing the first publicly available qualitative coding dataset, cVDQuoding, consisting of interviews con- ducted with women at risk of cardiovascular disease. addition, we conduct human evalu- ation which shows that systems consistently make relevant code suggestions.
CrD Medications CVD Prevention, Health Measure
Participant: Mell I'mm committed to taking the med conion see car cordhologitt anymore, told me dion't have he and manage wine,
referred my GP:
Management of CVD, Life Mediffcat onn
wright tning
I
The dedsion shape
Resiioon Made, Deciion making by the pubients
weight and improse lifestule it's oenainyy something Lake into account glion car dosascull health.
Figure I: Excerpt from one of our interview transcripts along with qualitative code suggestions made by one of systems (GPT-3.5). The codes that originally assigned to the passages by the qualitative coder are underlined.
coding process, a researcher carefully scans each document in their corpus, identifies passages that are associated with their research question, and assigns these passages descriptive labels or quali- tative codes (Figure 1).
Despite its widespread use, coding remains an arduous and time-consuming process as it requires manually annotating studied corpus line by line. For instance, studies have shown that coding a 1- hour interview transcript will typically take perienced researcher between 4 to 8 hours of anno- tation (Miles al., 2019).
1 Introduction
In qualitative research, qualitative coding is a con- tent analysis method that is used to analyze textual corpora such as cultural and political texts, ques- tionnaires and interview transcripts. First used in social science studies, qualitative coding has be- come a ubiquitous and universal method of anal- ysis geared at providing researchers with an in- depth understanding of their studied corpus (Elliott, 2018). It is now employed in several research dis- ciplines including medicine. education, and psy- chology (Chapman et al., 2015; Gough and Scott, 2000; Smith and McGannon, 2018). During the
To increase coding efficiency and reduce its cognitive load on researchers, NLP practitioners have attempted to insert language technologies into the coding process by formulating it as a text classi- fication task. However, as human-computer interac- tion (HCT) studies have shown (Rietz and Maedche, 2021), qualitative coders prefer receiving sugges- tions for identified passages as they code rather than having yet-to-be-seen passages automatically coded for them. This aversion towards fully au- tomatic coding system may explain why, this day, all available qualitative coding tools are used solely as "electronic filing cabinets" (Fielding and
"Corresponding author

---

Lee, 2002). That is, if automatic coding is not approached in a user-centered manner, despite de- sire for automated assistance, coding tools such as NVivo' and MAXQDA will continue to be used for their bookkeeping features only (Marathe and Toyama, 2018).
coders. This dataset is available by request the corresponding author, due to its sensitive nature.
We experiment with classification, information-reetiivall and zero-shot prompting techniques to model QCS and conduct a human evaluation which shows that our systems consistently make relevant code suggestions."
In this work, we take a more user-centered ap- proach qualitative coding by proposing the task of qualitative code suggestion (QCS) in which ranked list of previously assigned codes sug- gested from passages identified by a qualitative coder. Our framing of qualitative coding reveals technical subtleties which have not been addressed in previous work. For instance, our task definition exposes the necessity to evaluate code suggestions based the sequence in which documents coded as well the importance, and difficulty, of detecting rare codes. In addition, unlike other NLP tasks that use human-sourced annotations, QCS ex- pects differences in anmotation styles and avoids at- tempts to correct or homogenize these differences. Although more challenging, we believe that this human-centric approach is appropriate given that qualitative coding is meant to aid researcher in their personal understanding of corpus.
Related Work
In qualitative research, content analysis is defined as "a family research techniques for making sys- tematic, credible, or valid and replicable inferences from texts and other forms of communications" (Drisko and Maschi, 2016). These techniques rely heavily qualitative coding which has been used extensively throughout different arcas of research, first in the social sciences and, now, across most fields (Macnamara, 2006). There are two types of qualitative coding: deductive and inductive (Sal- daña, 2021). In deductive coding, all codes are predefined and the coding process serves as a way of validating or invalidating a hypothesis. In induc- tive coding, codes are defined ad hoc as documents analyzed and, thus, the coding process is used to discover new phenomena underlying corpus. In terms of assistive technologies for qualitative coding, studies have investigated various computa- tional approaches to automate parts of the deductive qualitative coding process. Efforts by Crowston et al. (2012); Scharkow (2013); McCracken et al. (2014); Liew et al. (2014) have investigated the of rule-based learning and supervised learning to automate the assignment of codes passages, More recently, Grandeit et (2020); Xiao et al. (2023) have explored using BERT-based models and GPT3 respectively to predict a previously as- signed code from a passage. In all cases, deductive coding was cast as fully automatable classifica- tion problem and metrics such as F1 and annotator- system agreement scores were used to measure performance.
To summarize, the research questions we address in our work are: 1) What previously unexplored technical challenges arise from approaching qual- itative coding in a user-centered manner? and 2) How effective are current NLP methods at handling these challenges? To this end, the contributions of our work are as follows:
Grounded in HCI studies on automated cod- ing assistance, we propose the task of qual- itative code suggestion (QCS) in which pre- viously assigned codes are suggested from passages identified by qualitative researchers. This user-centered approach to qualitative cod- ing reveals the importance of considering the sequence in which passages are coded, detect- ing rare codes, and preserving differences in coders' annotation styles.
We release the first publicly available quali- tative coding dataset, named CVDQuoding, consisting of transcripts of interviews with women at risk of cardiovascular diseases along with the annotations of two qualitative
At the same time, there has been extensive work from the HCI community on how qualitative coders would like automation to assist them during the cod- ing process (Marathe and Toyama, 2018; 0018; Feuston and Brubaker, 2021; Chen et al., 2018: Haug et al..
'The code can he found at petps://githhal com/ desare-spinoso/(Ss.
software

---

2021). These studies show that while researchers desire automation, they believe the coding agency should remain their hands. Thus, methods that introduce language technologies in the qualitative coding process should favour assisting coders by highlighting ambiguous passages (Drouhard et al., 2017), by helping them remain consistent in their annotation style (Sporer, 2012), or by interactively refining the definition of codes (Bakharia et al., 2016). These assistive approaches differ from pre- vious human-in- -ooe-- annotation studies (Mon- tani and Honnibal, 2018) since, as Jiang et al. (2021a) point out, "qualitative analysis is a unique case for human-Al collaboration because uncer- tainty is essential to qualitative analysis itself." In other words, qualitative coding is unique because the act of coding data, and the uncertainty that comes with it, is valued by researchers as it culti- vates their understanding of their corpus.
towards complete automation as coders could still reject suggestions.
2. Rare codes. Even though new codes may be rare past the point of data saturation, their oc- currences remain possible and scientifically important to the coders (van Rijnsoever, 2017) That is, the rarity of a code is in and of itself an important scientific finding. Thus, while sug- gesting relevant codes may alleviate some of coding's cognitive load, detecting rare codes may be just as important.
Annotation style. Finally, since coding is meant to cultivate a particular researcher's un- derstanding of a corpus, no two coders will have the same annotation style (Loslever et al., 2019). Thus, previous efforts to homogenize code annotations, for instance by grouping similar codes from different coders together, should be avoided. Although convenient for modeling purposes, this merging is not sup- ported by the user-centered approach we take in this work.
3 Qualitative Code Suggestion
We present properties of qualitative coding which have surfaced from the user-centered perspective we take in this work. We show that these properties warrant a new task definition that centers around suggestions and rare codes.
3.2
Task Definition
Our task definition, which applicable to both deductive and inductive coding, integrates the pre- viously discussed properties into the main QCS task and the novel code detection subtask.
3.1 Properties of Qualitative Coding
In qualitative coding, researchers code their corpus to gain a deeper understanding of a certain phc- nomenon related to their research question. As a result, in deductive and inductive coding, the first few documents that are coded are critical for the researcher's own understanding. Once the re- searchers have read through enough documents, properties of qualitative coding arise which, ac- cording to HCI studies (Rietz and Maedche, 2021; Marathe and Toyama, 2018), should be considered when developing assistive coding techniques. We further develop these properties below:
Qualitative code suggestion (QCS) Consider the setting in which a corpus comprised of documents D {dl, .2......dpp .vr. Each docu- ment, d, is a set of text spans )=m len(d) which have been identified and assigned codes by a qual- itative coder. Now, suppose K E {1....N) N] is a particular coder's point of data saturation. In this case, we collect all the codes assigned to text spans from documents d1 to dx and create a set of codes C C (cI...., Cm.. The task of QCS that we define here consists of ranking, in order of relevance, the set of codes in C with respect every identified text span in the set of documents past the point of data saturation, i.c. {t E d, K + ++...N) NJ.
1.
Data saturation. Coders that carry out qual- itative coding reach a personal point of data saturation after which few new codes will appear (Saunders et al., 2018). It is at this point where suggesting relevant codes for an identified passage may be helpful for qual- itative coder. These suggestions could help the qualitative coder consider relevant code assignments without needing to sift through irrelevant codes. At the same time, suggest- ing several relevant codes minimizes the shift
Novel code detection subtask Furthermore. since documents coded sequentially, some rare codes may only surface in the test portion of the corpus, (d +1,. , d [)
and, thus, might not
*We leave the additional task of text span extraction as future work as this extension to QCS goos beyond the scope of this study.

---

be part of C. As detecting rare codes may be just important as suggesting relevant ones, we add a novel code detection subtask to our task defini- tion which consists of assigning the novel code to a text span if one of its truly assigned codes is not in C. To create training instances for this subtask, we assign the novel code to all text spans in the training set that have been assigned code with training frequency of 1. All of the text spans in the validation and sets with at least one assigned code that does not appear in the training set are assigned the novel code.
codes in the first five interviews, and this trend pers off as they code more transcripts (Figure 2). In addition, coder finds new rare codes in ev- ery transcript whereas coder ceases to create new codes by interview 7. Moreover, both coders clearly exhibit significantly different annotation styles with coder 1 creating 207 codes with an av- erage of 4 text spans per code and coder 2 cre- ating 23 codes with average of 19 text spans per code. This difference can be attributed the specificity of coder-2's codes, which tend to be more abstract (c.g., "Health Mcasures Taken") than coder- I's codes (c.g., "CVD-related examinations, tests or measures"). For additional details about CVDQuoding including a sample list of codes, see Appendix A.1.
4 The CVDQuoding Dataset
To investigate the QCS task, we release the first publicly available qualitative coding dataset which we name CVDQuoding. In this section, we describe the dataset and how it was created. In addition, draw parallels between CVDQuod- i ing and the properties of qualitative coding described in Section 3.1.
Bumbber meay codter trrnddeent
Coeer
CVDQuoding is a dataset consisting of 15 tran- scripts of interviews conducted with women at risk of cardiovascular diseases (CVDs) as well as anno- tations carried out by two qualitative coders. These qualitative coding annotations consist of text spans identified by the qualitative coders, as well as the codes that were assigned to each text span. Ad- ditional details about the dataset as well as com- parisons with previous qualitative coding datasets, which have all been closed-source, can be found in Table 4 in Appendix A.I.
Figure 2: Distribution of the number of new codes in- troduced per interview each coder.
The annotations in CVDQuoding were origi- nally created as part of a larger project inves- tigate the interest of women at risk of CVDs in using AI tools (Bousbiat et al., 2022) and were subsequently adapted for our purposes. During the interviews, women were asked about the health challenges they face and their opinions about us- ing an AI tool to help them manage their risk of CVDs. For this work, extracted the 15 inter- view transcripts and tagged the text spans coded by each coder and the questions asked by the inter- viewer using XML, Readers may refer to Table in Appendix A.2 for the complete list of questions asked during each interview and to Figure 6 in Ap- pendix A. for an example of a tagged transcript excerpt.
Modeling Approaches
We investigate the ability of current NLP meth- ods rank lists of codes by relevance as well detect novel codes from text spans. To this end, we describe our data formatting decisions and the modeling paradigms we explore.
We choose explore three distinct modeling paradigms, classification, information--retrival and zero-shot prompting, based on the following intu- itions. Firstly, the classification paradigm assumes that the confidence scores of a multi-label clas- sifier can be used to rank codes. Secondly, the information-retrieval paradigm assumes that codes can be treated as documents and that text spans are the queries used to rank them. Finally, the zero- shot prompting paradigm assumes that the descrip- tiveness codes is sufficient for a large language model (LLM) to rank them based on the text span.
CVDQuoding exhibits the properties of qual- itative coding presented in Section 3.1. terms of data saturation, both coders create many new

---

5.1 Data Formatting
code.
For the novel code detection subtask, a classifi- cation head is trained on top the vector of scores Vi which consists of the scores computed between each code E C and the input instance (g. t)s- Thus, if there are 10 codes in the training set (ex- cluding novel), then for every instance (q. t); the classifier is passed 10-dimensional vector y1 com- puted from either the bi-encoder or cross-encoder.
For all three modeling paradigms, we associate ev- ery contiguous text span d2 that was coded by one of the qualitative coders with its immediate context c. In the case of CVDQuoding, we define the context as being the question asked right be- fore L. Thus, our case, every model has access to a tuple (g. t) consisting of a question, and a text span, t, highlighted by qualitative coder. We use the previous question as an approximation of the context as most coding decisions in CVDQuoding involve considering text span as the answer to preset question. For example, the code "Interest in AT tools" assigned to "Very interested, yes" because it the answer to "Are you interested in using AI tools for CVD?".
5.4 Zero-shot Prompting Paradigm
In the zero-shot prompting paradigm, we provide an autoregressive LLM M with prompt contain- ing general instructions, the list of codes and a text span t to code along with its previously asked question q. Refer to Appendix A.3 for the prompt template. Upon generation, the suggested codes are extracted via an exact-match search and the order in which the codes are generated is used their predicted rank. Furthermore, the generation "predicts" the novel code either if . generates the string "None of the above" first (which is included in the prompt template) or no exact matches are found in its generation.
5.2 Classification Paradigm
We build |C| + 1 multi-label binary classifier trained on the first K' annotated transcripts to pre- dict the assignment of each code E C as well as the novel code text span. We the scores produced for each code sort the set of codes C for a test instance (q, t) {(g,t) t E d K+1... N). Furthermore, an instance is assigned the novel code only if it is given the highest score among all codes.
Experiments
We investigate approaches to modeling the QCS task in order to determine the ability of cur- rent NLP-based methods to allow for a more user- centered approach to qualitative coding. We dis- cuss the experimental setup to test our modeling paradigms as well as our evaluation methodology.
5.3 Information-retrieval Paradigm
In the information-retrieve paradigm, we treat instance tq as query and use two neural- retrieval architectures to rank the set of codes C with an additional step for the novel code detection subtask.
6.1 Experimental Setup
We build two neural-retrieval architectures origi- nally presented by Reimers and Gurevych (2019): the bi-encoder and the cross-encoder. In the bi- encoder, representation is learned for the text span t, the previous question q and the code c. The representations of the test span, h4, and the previ- ous question, hy. are max pooled together and a score is computed by applying a cosine similarity berween the pooled representation and the code representation, fe. In the cross- cross-encoder, represen- tations are learned for the concatenations of the code with the question, [SEP] c, and with the text span, [SEP] c. The representations, hy [SEP] and he [SEP]= are max pooled together and a classifica- tion head placed on top of the pooled represen- tation to produce scores. both cases, the code scores produced are used to rank all but the novel
To experiment with each of the paradigms pre- sented in the previous section, we sort CVDQuod- ing by annotation-tine group it by coder, and consider it at different possible points of data sat- uration. More specifically, for each coder, we consider training on {di,.... dx} and testing on {dK+1..... d1s} for K = 1... 14 where d, is the 7th annotated transcript in annotation-tima We re- serve 20% of each training set for validation. We summarize our workflow in Figure 3.
We train and test models from each of our three modeling paradigms and for all of the configu- rations discussed above. For the classification paradigm, we use |C] + SVM classifiers as well as a single DistilBERT (Sanh et al., 2020) classi- fier with a multi-label sequence classification head. We use DistilBERT due to the computational costs

---

Interview question
Text
Code(s)
Interview 15
Span
Interview K
q1
C1
Interview .diquestion_
Ininiover:
Train ... K
q2
C14 C22
Pericpprt:
q2
C11
Css Cz2
Gn
Test
q2
C25
CVDQuoding
K+1 ... 15
Figure 3: Visualization of the data formatting we apply to raw annotations from CVDQuoding to run our experiments with our different modeling paradigms.
hyperparameter tuning over 2 14 = 28 in- dividual datascts. For the Information-rctricee paradigm, we use DistilBERT as the encoder for the bi-encoder architecture. For the cross-encoder, we experiment with both DistilBert and ConvBERT (Jiang et al., 2021b). We ConvBERT based on the intuition that text spans often contain phrases code's description.
6.2 Evaluation Methodology
We subject our modeling paradigms to two distinct rounds of evaluations. In Section 6.2.1, we evalu- ate how well our systems provide code suggestions based the original annotations from CVDQuod- ing. In Section 6.2.2, further evaluate our systems suggestions by asking human judges to de- termine their relevance. We present visualization of our two-phase evaluation in Figure 4.
which are lexically similar If this the case, then ConvBERT's span-based ker- nels may be better suited at soft matching code's description in a text span than a fully attention- based masked language model like DistilBERT. Moreover, ConvBERT has computational costs (e.g., GPU memory requirements) in the same order of magnitude as DistilBERT. Finally, for our zero- shot prompting paradigm, we OpenAl's GPT- 3.5 Turbo (Brown et al., 2020) accessible through APP Additional hyperparameter configurations and training details can be found in Appendix A.3.
Interview question
Text Span
Code(s)
Code ranking
C3
C12 Cas Caz Ca
Original Annotations Evaluation
Human Evaluation
Caz
Relevant: Cas Ca2 Cs
In addition to the models discussed above, we use an information-retrievall baseline commonly used in neural-retriever papers. We use the Okapi BM25 retriever (Trotman et al., 2014) to compute scores between each code and instance (q, l). In addition, place logistic regression model top of the vector of scores to make the novel class prediction.
Figure 4: The original annotations evaluation relies the codes assigned to the test text spans in CVDQuod- ing while the human evaluation asks judges deter- mine which of the top-4 suggested codes are relevant (for sample test instances).
6.2.1 Original Annotations Evaluation
QCS For the main QCS task, compute the mean reciprocal rank (MRR) score and a soft normalized discounted cumulative gain at k (sNDCG@k) score. These metrics allow us to

---

where C. is the set of codes originally assigned to text span by one of the two qualitative coders. measure how highly our systems rank the codes that were originally assigned to the text spans in CVDQuoding by the qualitative coders.
alternative and relevant code assignments which the original coders may not have considered.
To compute the MRR, we exclude the rank of the novel code as we are interested in a system's 's ability to suggest previously assigned codes. Ad- ditional details about the computation of the MRR are presented in Appendix A.4.
Evaluation setup To gather human judgements, we hired two human evaluators, both with expe- rience in qualitative research, and asked them to judge the relevance of our systems' code sugges- tions. In particular, we randomly sample 32 in- stances from the annotations of coders and 2 at the same point of data saturation of K 10 and sign human evaluator to each sample. We extract the top-4 suggestions of each model described Section 6.1% and ask the evaluator to judge whether each suggestion is "Relevant" or "Irrelevant" based on the question and the text span. Additional de- tails about the human evaluation can be found in Appendix A.5.
To compute the sNDCG@k metric, we approx- imate suggested code's с relevance score for a text span i, (c), and subsequently carry out the standard NDCG computation. The relevance score, (e)( is computed by calculating the BERTSccoee (Zhang et al., 2019) between suggested and true codes. More specifically, the relevance score rel;(c,, for a code E C and text span i is
rel,(c) = max BERTScoree(, c') d' E C.ee
Metries We use the annotations from the human evaluation to recompute the rank-based metries pre- sented Section 6.2.1. We recompute the MRR and the sNDCG@k of the 32 code suggestions based on the human evaluator's s relevance judge- ments. That is, for a sampled instance i, Cirue becomes the set of codes selected "Relevant" by the evaluator. In addition, we also compute the pre- cision 4 (P@4) of each system using the human evaluator's s annotations. Details about the computa- tion of P@4 are presented in Appendix A.4.
Once all the relevance scores are computed, we use them in the standard NDCG computa- tion. To do so, we sort the list of relevance factors (rel,(c) trru__rankscoofee scores list for a text span i. In addi- tion, the predicted_rp Beleccceeddd_sssseeee rank_scores for a text span i is computed as the list of relevance factors sorted by the model's scores for each code. Both these lists are then used in the standard NDCG formula. Additional details are presented in Appendix A.4. Finally, we add a cutoff l: to sNDCG as is usu- ally done in standard NDCG to account only for the top k results creating sNDCG@K. We choose to evaluate our systems with k 4 as no more than 4 codes are assigned to text spans throughout CvDQuoding.
c E C) to compute the
Results
We present the results of our experiments with re- spect to both the original annotations and the hu- man evaluation annotations.
7.1 Original Annotations
Using the original annotations as the gold stan- dard labels, we present the MRR and sNDCG@4 scores for the main ranking task of QCS as well as the macro and micro F1 for the novel code de- tection subtask. In particular, we show the MRR and ANDCG@4 scores at K = 10 for coders and 2 (Table 1) and the Fi scores for the novel code prediction task at K 5 (Table 2). In addition, we provide plots of the MRR for coders with respect to K (Figure and in Appendix A.6). Additional results for all points of data saturation, presented using both scatter plots and tables, can be found in Appendix A.6.
Novel code detection For the novel code detec- tion subtask, we compute both the macro and micro F1 scores. We report both averages due to the im- balance in the ratio between novel and not-novel classes.
6.2.2 Human Evaluation
In addition to using CVDQuod i ng's original anno- tations to evaluate systems' ability to provide relevant code suggestions, conduct human evaluation where ask human judges ascer- tain the relevance of our systems' code suggestions. This additional evaluation phase is necessary be- cause is common for text spans to have several
"We do not consider the Okapi BM25 baseline suggestions due to their poor quality.

---

MRR
7.2
Human Evaluation
Coder
Coder
Coder
Coder
Using the human evaluation annotations as the gold standard labels, we present the results for the same rank-based metrics, along with the added P@4 4 re- sults, and contrast them with the results in the pre- vious section (Table 3).
Okapi BM25
0.15
0.53
0.70
0.79
SVM
0.51
0.75
0.80
0.85
DistiBBERT
0.55
0.75
0.70
0.79
Bi-Encoder
0.48
0.74
0.80
0.86
Cross-Encoder
0.55
0.64
0.81
0.82
(Di)tilBERT)
Fonsser
Systems consistently make several relevant code suggestions While the human evaluators have recall with respect to the original annotations of 0.94 and 0.95 for coders and respectively, they identify, on average, 3 times more relevant codes than in the original annotations. As a result, we observe a statistically significant jump between the MRR and sNDCG@4 scores computed on the sam- plc of 32 text spans using the original annotations and the human evaluation annotations across all models. On average, the human-evaluatiion-drrive MRR is 66% and 34% higher than the MRR com- puted using the original annotations for coders and 2, respectively. In fact, in the case of coder 2, the recomputed MRR for GPT-3.5 is 1. This jump is less drastic for the sNDCG@4 metric with rela- tive increases of 40% and 19% for coders I and 2 respectively. In addition to making consistently rel- evant code suggestions, our best performing mod- els achieve P@4 scores above 0.5 indicating that, on average, our systems are able to suggest more than one relevant code for a given passage. Alto- gether, these additional results demonstrate that hu- man evaluations are necessary for QCS systems as they provide interpretations for codes which were highly ranked but which the qualitative coders may not have originally considered in their code assign- ments.
0.59
0.77
0.83
0.86
(ConyBERT)
GPT-3.5
0.57
0.73
0.74
0.83
Table I: Results of the models' performance on the QCS main ranking task. The MRR and sNDCG@4 scores are computed based on the original annotations from the CVDQuoding dataset and using K" 10 as the point of data saturation.
Macro FI
Micro Fl
Coder 1
Coder
Coder 1
Coder
Okapi BM25
0.56
0.57
0.59
0.80
SVM
0.32
0.30
0.44
0.30
DistiBBERT
0.55
0.45
0.59
0.80
Bi-Encoder
0.36
0.45
0.57
0.80
Cross-Encoder
0.51
0.46
0.59
0.78
(DistiBEETT)
Cross-Encoder
0.36
0.57
0.57
0.79
(ConvBERT)
GPT-3.5
0.41
0.44
0.53
0.80
Table 2: Novel class detection subtask results. Macro and micro Fi scores are computed using K- = 5.
The refereaa and zero-shot prompt- ing paradigms are the best performing On the main QCS ranking task, we observe that the information-retrieva paradigm, modeled through the ConvBERT cross-encoded and the zero-shot prompting paradigm, modeled through GPT-3.5, achieve the highest MRR and sNDCG@4 scores across both coders for most data saturation points. While GPT-3.5 achieves the highest MRR in most cases for coder (11 out of 14), ConyBERT ob- tains the largest number of maximal MRR scores for coder 2 (7 out of 14). In addition, ConvBERT obtains the largest number of maximal sNDCG@4 scores for both coders.
MRR
<NDCG@4
PQ4
Coder
Codar
Coder I
Coder
Cider
Codkr
SVM
0.93
0.978
0.55
DisUBERT
0.72
0.91
0.969
0.975
0.41
0.65
Hi-Eneader
0.86
0.90
0.79
0.66
Fneoder
0.78
0.93
0.965
0992
0.46
0.68
(DiiiBBER)
Cross Encoder
(ConyBERT)
1.887
0.95
0.973
01987
0.51
0.56
GPT-3.5
0.74
1.00
0.973
0.989
0.32
0.53
Table 3: MRR, sNDCG@4 and P@4 scores computed using the annotations collected during the human evalu- ation.
Detecting novel codes is a challenging subtask For the novel code detection subtask, modeling paradigm is able to outperform the Okapi BM25 baseline. These relatively poor performances in- dicate that detecting novel codes is a challenging subtask.
Analysis
We discuss the implications of our results in rela- tion to the three properties of qualitative coding which we uncovered through our numan-centrie ap-

---

Vodil
Mode
Cris Paoeeer (huuatitt) Cuso-recder
Orannrrode (caviaor)
02
PURE DUAA SaaniaGn
Parns Duta Suta
Figure 5: Plot of MRR across data saturation points K = to 14 for coder (left) and for coder 2 (right)
proach and the ability of our modeling paradigms consider these properties.
recomputed MRR and the additional results from the P@4 show that not only are our systems able to correctly rank relevant codes, but they are also likely to highly rank more than one relevant code, potentially forcing the coder to further reflect on their coding decisions without needing to consider irrelevant codes.
Firstly, across both coders and all models, we observe steady rise in our systems' ability to make code suggestions consistent with the original annotations of CVDQuoding (See scatter plots through annotation-time in Figure 5 as well in Appendix A.6). This poor performance for small values of K is consistent with inductive coding which, for the first documents, is exploratory and unpredictable by nature. In fact, the moment at which this rise in performance plateaus may be helpful signal to identify when coder has reached their personal point of data saturation and, thus, when code suggestions could start to be beneficial. Secondly, across all saturation points and for both coders, we observe relatively poor performance detecting novel codes. This difficulty suggests that the "catch-all" bucket method we use in this work is inappropriate and that more sophisticated representations of novelty need to be learned to properly detect rare codes. Finally, we notice that, average, our systems' MRR and the sNDCG@ @4 calculated using coder 2's annotations are 58% and 7% higher respectively than when calculated using coder I's annotations. This result is natural given coder 2's annotation style of assigning high-level codes to larger text spans is much more amenable to machine learning techniques. Thus, in the future, methods that are able to identify annotation styles and incorporate them in their modeling may be more suitable at solving QCS in user-tailored fashion.
Conclusion
In this work, we approached qualitative coding a novel user-centered manner and showed that this approach gives rise to technical subtlcties related to qualitative coding which have not been previ- ously investigated. These subtleties include the importance of considering coder's personal point of data saturation, the difficulty of detecting rare codes and the necessity to avoid homogenizing dif- ferent annotation styles. We showed that each of these properties is reflected in both the CVDQuod- ing dataset we introduced and the experimental results of our modeling of QCS. Lastly, our human evaluation showed that solely relying automatic metrics computed with respect the original anno- tations made by the qualitative coders is insufficient to describe the performance of QCS systems. The results derived from our human evaluation show that QCS systems can consistently provide sev- eral relevant code suggestions and, thus, that this human-centric approach to qualitative coding may be able to truly assist researchers in their study of textual corpora.
Limitations
In addition, the ranked-based metrics computed using human judgments point to a promising future for the downstream viability of QCS systems. The
This work includes a few limitations which we leave as future work. Firstly, we did not conduct

---

an evaluation of the impact of QCS in an applied setting. This lack of downstream experiments pre- vents us from determining the full impact of auto- matically providing code suggestions to qualitative coders. For instance, an automatic QCS system may remove more agency from the coder than an- ticipated if they begin to blindly trust suggestions. Secondly, despite our best efforts, our work falls short of investigating all possible technical avenues solving QCS. In particular, we do not investigate fine-tuning sequence- sqquence-o-ssqueeme to-sequence models as pre- liminary experiments showed poor performance. In addition, we do not explore the full range of abilities offered to us by LLMs such as GPT-3.5. For instance, using GPT-3.5 to generate synthetic annotations may have helped the performance of our larger neural-based models which are known perform poorly in data-scarce settings.
study.
The authors acknowledge the material support of NVIDIA in the form of computational resources.
References
Aneesha Bakharia, Peter Bruza, Jim Watters, Bhuva Narayan, and Laurianne Sitbon. 2016. Interactive topic modeling for aiding qualitative content analysis. In Proceedings of the 2016 ACM on Conference Human Information Interaction and Retrieval, pages 213-222.
1. C. Bousbiat, G. Roland, Y. Shahram, C. Rodriguez, P. Pluye, J. Légaré, H. Bergman, I. Vedel, G. Bis- leri, MP. Gagnon, and S. Abbasgholizadeh-Rahne 2022. "CVD Prevention among Women in Primary Care Using AI: Work-in Yookk-n-PPuggeed Progress". Family Medicine Forum/Forum en médecine familiale, Toronto, On- tario.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Ethics Statement
We strongly believe in the work done by qualitative researchers and the human understanding that they provide to complex bodies of text. This work is an effort to develop methods that help rather than replace the arduous work conducted by qualitative researchers and coders.
The data collection of this study has been ap- proved by the Faculty of Medicine and Health Sci- ences Institutional Review Board (IRB) of McGill University (IRB Internal Study Number: A03-B22- 21A). All necessary participant consent has been obtained and the appropriate institutional forms have been archived.
AL Chapman, M Hadfield, and CJ Chapman. 2015. Qualitative research in healthcare: an introduction to grounded theory using thematic analysis. Jour- nal of the Royal College of Physicians of Edinburgh, 45(3):201-205.
Man-Chen Chen, Margaret Drouhard, Rafal Kocielnik. Jina Suh, and Cecilia R Aragon. 2018. Using ma- chine learning to support qualitative coding in cial science: Shifting the focus to ambiguity. ACM Transactions on Interactive Intelligent Systems (TiS)) 8(2):1-20.
A Acknowledgements
We would like to thank the anonymous reviewers for their feedback and valuable suggestions. In addition, we would like to thank Andrei Mircea, Aylin Erman, Dan Poenaru, Jad Kabbara, Ian Po- rada, Ines Arous, Sabina Elkins and Yu Lu Liu for their insightful feedback on the multiple iterations of this paper.
K. Crowston, E.E. Allen, and R. Heckman. 2012. Using natural language processing technology for qualita- tive data analysis. International Journal of Social Research Methodology, 15(6))5233-43.
James W Drisko and Tina Maschi. 2016. Content anal- ysis. Pocket Guide to Social Work Re.
This work was supported by the Fonds de Recherche du Québec Nature et Technologies (FRQNT). Jackie Jackie Chi Kit Cheung is supported by the Canada CIFAR AI Chair program. Samira Ab- basgholizadeh Rahimi is a Canada Research Chair (Tier 2) in Advanced Digital Primary Health Care, and received salary support (Research Scholar Ju- nior 1 Career Development Award) from the Fonds de Recherche du Québec-Sante (FRQS) during this
Margaret Drouhard, Nan-Chen Chen, Jina Suh, Rafal Kocielnik, Vanessa Pena-Araya, Keting Cen, Xiangyi Zheng, and Cecilia R Aragon. 2017. Aconium: Vi- sual analytics to support collaborative qualitative cod- ing. In 2017 IEEF Pucific Visualization Symposium (Pacific Vis), pages 220-229. IEEE.
Victoria Elliott. 2018. Thinking about the coding pro- cess qualitative data analysis. The Qualitative Report, 23(1 ():28)0)288881..

---

Jessica Feuston and Jed R Brubaker. 2021. Putting tools in their place: The role of time and perspec- tive in human-ai collaboration for qualitative analy- sis. Proceedings of the ACM on Human- ComputtC Interaction, 5(CSCW2):1-25.
M.B. Miles, A.M. Huberman, and Saldana. 2019. Qualitative Data Analysis: A Methods Sourcebook. SAGE Publications.
Ines Montani and Matthew Honnibal. 2018. Prodigy: A new annotation tool for radically efficient machine teaching. Artificial Intelligence to appear.
Nigel G. Fielding and Raymond M. Lee. 2002. New pat- terns in the adoption and use of qualitative software. Field Methods, 14(2):1772116
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert, Sentence embeddings using siamese nett-ks In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Stephen Gough and William Scott. 2000. Exploring the purposes of qualitative data coding in educational enquiry: Insights from recent research, Educational Studies, 26(3):339-354
T. Rietz and A. Maedche. 2021. Cody: An ai-based sys- tem to semi-automate coding for qalitative research.
Philipp Grandeit, Carolyn Haberkem, Maximiliane Lang, Jens Albrecht, and Robert Lehmann. 2020. Using BERT for Qualitative Content Analysis in Psy- chosocial Online Counseling. In Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pages 11-23, On- line. Association for Computational Linguistics.
Johnny Saldaña. 2021. The coding manual for qualita- tive researchers. The coding manual for qualitative researchers, pages 1-440.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
Saskia Haug. Tim Rietz, and Alexander Maedche. 2021. Accelerating deductive coding of qualitative data: An experimental study on the applicability of crowd- sourcing. In Proceedings of Mensch und Computer 2021, pages 432 443.
Benjamin Saunders, Julius Sim. Tom Kingstone, Shula Baker, Jackie Waterfield, Bernadette Bartlam, Heather Burroughs, and Clare Jinks. 2018. 8. Satura- tion qualitative research: exploring its conceptual- ization and operationalization. Quality & quantity, 12:1883-1990
Jialun Aaron Jangg Kandrea Wade, Casey Fiesler, and Jed R Bruhaker. 202 la. Supporting serendipity: Op- portumities and challenges for human-ai collaboration in qualitative analysis. Proceedings of the ACM on ComaunCommuttr Interaction, S(CSCW1):1-23,
Michael Scharkow, 2013. Thematic content analysis using supervised machine learning: An empirical evaluation using German online news. Quality & Quantity, 7712)776
Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen,
Jiashi Feng, and Shuicheng Yan. 2021b. Convbert: Improving bert with span-based dynamic convolu- tion.
Brett Smith and Kerry R McGannon. 2018. Developing rigor in qualitative research: Problems and oppor- tunities within sport and exereise psychology. In- ternational review of sport and exercise psychology. 1111)101--22
Jasy Suet Yan Liew, Nancy McCracken, Shichun Zhou, and Kevin Crowston. 2014. Optimizing Features in Active Machine Learning for Complex Qualitative Content Analysis. In Proceedings of the ACI. 2014 Workshop Language Technologies and Computa- tional Social Science, pages 44-48, Baltimore, MD, USA. Association for Computational Linguistics.
Siegfried Ludwig Sporer. 2012. Making the subjec- tive objective? computer-assisted quantification of qualitative content cues to deception. In Proceed- ings of the workshop on computational approaches to deception detection, pages 78 18-85.
Pierre Loslever, Taisa Guidini Gonçalves, Káthia Marçal de Oliveira, and Christophe Kolski. 2019. Using fuzzy coding with qualitative data: example with sub- jective data in human-compute interaction. TheoreT- icul issues in ergonomics science, 20(4):559448)
Andrew Trotman, Antti Puurula, and Blake Burgess. 2014. Improvements bm25 and language models examined. In Proceedings of the 19th Australasian Document Computing Symposium, ADCS '14, page 58 65, New York, NY, USA. Association for Com- puting Machinery.
Jim Macnamara. 2006. The fork in the road of media and communication theory and practice. 4th Annual Summit on Measurement, pages 1-
Frank van Rijnsoever. 2017. can't get no) satura- tion: a simulation and guidelines for sample sizes in qualitative research. PloS one, 12(7):e018 689..
M. Marathe and K. Toyama. 2018. Semi-automated cod- ing for Qualitative research: A user-centered inquiry and initial prototypes. volume 2018-April.
Ziang Xiao, Xingdi Yuan, Q Vera Liao, Rania Abdel- ghani, and Pierre- Yves Oudeyer. 2023. Support- ing qualitative analysis with large language models: Combining codebook with gpt-3 for deductive cod- ing. In Companion Proceedings of the 28th Inter- national Conference on Intelligent User Interfaces. pages 75-78.
N. McCracken, J. Suet Yan Yan, and K. Crowston. 2014. Design of an active learning system with human cor- rection for content analysis. Proceedings of the Work- shop on Inteructive Language Learning, Visualiza- tion, and Interfaces, pages 59-62.

---

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. CoRR, abs/1904.09675.

---

A
Appendix

---

A.1
Dataset Characteristics
Corpus size (in number of words)
Source
Number of annotations
Publicly available?
(Crowston et al., 2012)
84870
3011
No
(Grandeit et al., 2020)
N/A
10000
No
(Xiao et al., 2023)
N/A
668
No
CVDQuod i
63927
1175
Yes?
Table 4: Comparative analysis of our dataset with previously closed-source datasets. In this case, the number of words counted using space separation and the number of annotations refer to the number of text spans that have been assigned one or more codes.
Qquestion22Inneevveewe Have you ever been informed about your cardiovascular health, and if so how have you been informed??<question.22
Participant 1: <code coder='2 2' Value="Understanmee
of CVD'>So, I've been in cardiology since '96. used to work at the Montreal Heart Institute before so <code coder='1' value= Sources of information' >I attended cardiac rounds and went to conferences and things like that. also like to read health, media, papers and then conferences. </coce<<//ood/>
Figure 6: Excerpt from Interview 1 of the CVDQuoding dataset with XML tags to identify the question asked and the text spans have been annotated. In the code tag, attribute coder identifies which qualitative coder was responsible for the annotation and the attribute value identifies the code assigned to the text span.
Distribution of the number of text spans per codes
Coder
Coder
Number codes
50
10
5
20
20
25
30
Number text spans
Figure Frequency of code assignment counts.
"Email the corresponding author.

---

Distribution of the length of the text spans (whitespace separation)
Coder Coder
150
Number
13,
15,
19. 16]
[17, 32)
133, 64)
Fe5, 1281
1129, 2561
[257, 512)
[513.)
spans length
Figure 8: Distribution of the length of text spans.
Annotator
List of codes
Coder
Life-style modification, Needs assure data confidentiality and security, Increasing trust in the tool, Lack of knowledge, Additional comments about diet recommenda- tions, CVD misconceptions, Level of interest in tracking information and prompts, Level of interest in educational modules, Additional comments about educational modules, Level of interest transparency and explainability, Enthusiastic about technology, App prompts, Frequency of push-notifications, Level of interest in personalized workouts, Additional comments about explainability
Coder
Technology for CVD care, Cardiovascular Health Information, Accessibility, Deci- sions Made, Additional Features Suggested, Understanding of CVD, Challenges, Trust and Reliability, Transparency, Decision Involvement, Weight Tracking, Educa- tional Models, Technical Support, Data Monitoring, Pedometer
Table 5: List of 15 codes created by coders and 2 sorted by frequency over the entire CVDQuoding dataset.

---

A.2 Interview Questions
Interview Questions
Question
How do you think cardiovascular diseases generally described and understood by the public?
Question 2
Have you ever been informed about your cardiovascular health? How?
Question 3
What are the most frequent and important decisions you face related to your cardio- vascular health? What your usual role in making decisions about your cardiovascular health and preventing CVD?
Question
Question
What do you think are some challenges and needs in preventing and managing cardiovascular diseases (from your perspective as a woman at risk of CVDY? Have you ever considered decision support system to help you in any decisions related your cardiovascular health?
Question 6
Question
What are your thoughts on using digital technology (e.g., mobile apps, Al system- s/robots) to make decisions in relation to your cardiovascular health?
Question 8
How would you like us to design and develop this Xi-Care tool that is useful, helpful and effective for women at risk of CVD (e.g. no risks to users)?
Question
On a scale of to with being not at all interested and 5 being very interested, what is your level of interest in monitoring tools that track health data over time?
Question 10
On a scale of to 5 with being not at all interested and 5 being very interested, what is your level of interest in step-count feature?
Question 11
On a scale of 1 to with being not at all interested and 5 being very interested, what your level of interest in a weight tracking feature?
Question 12
On a scale of 1 to 5 with 1 being not at all interested and 5 being very interested, what is your level of interest in educational modules cardiovascular health?
Question 13
On a scale of 1 to 5 with being not at all interested and 5 being very interested, what your level of interest in guided exercise activities?
Question 14
On a scale of to 5 with being not at all interested and 5 being very interested, what is your level of interest in diet recommendations?
Question 15
Would you like to be able to follow your progress and receive push-notifications through the Xi-Care tool?
Question 16
How difficult/easy do you think it will be for you to integrate the Xi-Care tool into your daily life?
Question 17
To what extent would you trust rely on the Xi-Care tool to make assessments about your cardiovascular health and prevent and manage CVD?
Question 18
Do you foresee any challenges with integrating the Xi-Care tool in your daily life in terms of ethics? Could you please describe these challenges?
Question 19
In terms of transparency, how important is it for you to be able to understand how the Xi-Care tool works?
Question 20 Is there something else you'd like add about ethical aspects in regards the Xi-Care tool that will be empowered by Al (Justice; mon-maleficnnee; Autonomy; Beneficence; ExplicabilityTTrasppeeegg
Table 6: List of questions asked during the interviews of the CVDQuoding dataset. Xi-Care is the name of the app being proposed to participants of the study help them control their risks of cardiovascular diseases (CVDs).

---

A.3 Hyperparameter Configurations We provide additional hyperparameter and archi- tecture details for all our models.
We use component-wise max pooling method max(-,-) aggregate hy and hg together.
Ppool = max hg)
A.3.1 Baseline
Finally, cosine similarity is applied between the pooled representation, Apool and the code represen- tation, hes generate score. A class weighted mean-squared error loss is then computed and back- propagated through the three encoders. For the novel code detection subtask, a linear classifier is placed on top of the vector of encoder scores. For the training of the linear classifier, we use another class weighted cross entropy loss and keep the three encoders frozen.
Okapi-BM25. We use an open-source imple- mentation of the BM25 algorithm². We use the Okapi implementation with its default parameters h 1.5, 6 = 0.75, 0.25 and tokenizer with space separation and lowercasing.
A.3.2 Classification Paradigm
SVM. We use the SVM implementation from sklearn'. We transform all inputs to if-idf features with separate encodings for the question and text span. We use the standard radial basis function ker- nel and tune the regularization parameter C' using the values {0.001, 0.01, 0.1, 10). A class weight parameter is computed for each code as well as for the novel code and is inversely proportional to the class frequencies in the training set.
We tune the learning rate using the values {1 10-5 10-9} with a weight decay of 0.01. We use larger batch size of 32, since training sets now have size of |C| x N where is the number of coded text spans. We train for 25 epochs with early stopping on the validation loss set with a patience of 5 epochs.
DistilBERT. We the DistiBBERT implemen- tation from Hugging Face particular, we use DistilBERT's tokenizer and DistilBertForSe- quenceClassificationn module which accepts a vari- able number of labels, |C| + 1 in this case. We tune the learning rate using values {1 x 10-5, x 10-5} with weight decay of 0.01. We use batch size of 8 and train for 25 epochs with early stopping on the validation loss with a patience of 5 epochs. We also compute a class weight inversely proportional the class frequencies in the training set and apply it to the cross entropy loss. All our experiments consistently show that using class weights benefits performance.
-nnoseE In the cross-encoder, we create representations for the concatenation of codes with the question and the text span. That is, for every code e in the training set, the text span and the previous question q are each independently con- catenated with the code e using the special [SEP] token. Both concatenations are passed to different encoders Eg and Eg to obtain contextual repre- sentation. Truncation is applied to the start of both t and q to comply with the encoder's 's maximum token input length.
heISEP = Ft [SEP) c)
by [SEP] = Eg(g [SEP] e)
A.3.3 Information-retrievll Paradigm
Along with hyperparameter details, we provide additional architecture details about both the bi- encoder and cross-encoder.
The representations are pooled using a component-wise max pooling method max(-.))
Bi-Encoder. In the bi-encoder, create repre- sentations for the code c, the text span and the pre- vious question q with three distinct BERT encoders, Eg, Eg and E.. our case, we use DiST as the encoder for all three inputs. We pass each input whose start is truncated to the model's 512 token input limit to its respective encoder and extract its CLS token representation.
Пooot max(h; [SEP] c: hy [SEP]
We place classification head on top of hpool identical architecture to the DistilBERT classi- fication head. We compute a class weighted cross entropy loss and backpropagate it through both en- coders. For the novel code detection subtask, a lincar classifier is placed top of the vector of encoder scores, which, in this case, are logits. We use a class weighted cross entropy loss and keep the two encoders frozen.
h (t), E2( Ee(c)
htttps///ggtuubbcoom//drabwwwww bm25
In this case, we experiment with DistilBERT and ConyBERT, both cases, we tune the learning
http:s///scikitteaaaoooossttttaa whanss//hugginggtcc..o//

---

using the values {1 10-3,2 ,2 x 10-93 with weight decay of 0.01. We use batch size of 32 and train for 30 epochs with early stopping on validation loss set to have patience of 5 epochs. We train for more epochs because we noticed that the cross-encoders took longer to converge.
A.3.4 Zero-shot Paradigm
We use OpenAI's GPT-3.5 API to generate LLM re- sponses our zero-shot prompt. In particular, use the model checkpoint 1rtt3.5-turbo--0 with temperature of The question and the text span are truncated to allow for 64 token genera- tion. At the time our experiments, this version of GPT-3.5 allowed for 4096 tokens. Figure 9, we show the template that we use to prompt GPT-3.5.
You helpful assistant that suggests qualitative codes for qualitative searcher. The coders you can suggest are: [AVAILABLF] [AVAILABLF_CODESS] and None of the above. Which of the previous codes would you assign to the following ex- cerpt from an interview with a woman at risk of cardiovascular disease (CVD): "Question: [QUESTION] Answer: [AN- SWER]"
Figure 9: Template used prompt generative LLM M for ranking of the codes for passage. [AVAIL- ABLE CODES] is a placeholder for the list of codes from C to rank for instance (g, f),, [QUESTION] is placeholder for the previous question and [ANSWER] is placeholder for the text span highlighted by one of the coders.

---

predicted_rank:(C A.4 Automatic Metries
our system for text span i
We provide complete descriptions for the automatic metrics presented in Section 6.2.1 and 6.2.2.
sDCG,
sNDCG; sIDCG;
true_rankkscores)
MRR To compute the MRR, we consider the set
sIDCG; Σ
log(i)
of true codes, Ci assigned to text span i well C
iE2...1C-11
the predicted rank, predicted_ {1....,[C]] J. assigned each code E C for text span by one of our systems. this computation, we exclude the novel code as we are interested in a system's ability to suggest previously assigned codes. The reciprocal rank, RR,, for a passage i is computed the maximum predicted reciprocal rank across all true codes. That is,
predicted_ Beatted_ran_sscorree
sDCG = Σ 162..11C-1
log(t)
Finally, we add a cutoff A to sNDCG as is usually done in standard NDCG to account only for the top results creating sNDCG@K.
RR,
cc C
The MRR is then computed by averaging across all test instances i with |C 1.
P@k To compute the P@k, let N₂ be the num- ber of relevant codes, as indicated by some gold standard annotation, in the first & suggested codes. Then the P@k for text span i computed as
N k
P@k;
We then average the instance-level P@k over all test instances.
To compute the sNDCG metric, we approximate suggested code's relevance score for text span i, (c), in order to carry out the standard NDCG computation. The relevance score, rel(()), is computed by using the BERTScore (Zhang et al., 2019) the approximation for the affinity between suggested and true codes. More specifically, the relevance score rel, for code c and a text span i is
sNDCG@k
rel(c) = max BERTScore(c,e)) d' Cinup
Once all the relevance scores are computed, we use them in the standard NDCG computa- tion. To do so, we sort the list of relevance factors (rel((c)
C) to compute the true_rank_scone list for text span i. In addi- tion, the srrddicted_rank_soee_ is computed as the list of relevance factors sorted by the model's s scores for each code. We can then com- pute the sNDCG score for the suggestions made by
for text span i

---

A.5 Human Evaluation
We conducted the human evaluation by first run- ning a pilot study. The pilot study was used to assess whether the instructions of the human eval- uation were clear and to estimate the duration of the study for the full 32 samples. During the pi- lot study, recruited researcher familiar with qualitative research and asked them judge the relevance of 8 instances. Half of these instances were sampled from coder I's human evaluation samples, and the other half came from coder 2's. We made sure that at least one code had been as- signed to every passage we showed evaluators as we wanted to be able to compare the rank-based metries computed using the original annotations and the human evaluation annotations. In addition, we always included the codes from the original coding annotations to avoid having the evaluator mark all suggestions as irrelevant. An example of a question and its corresponding highlighted text span from coder I's annotations as well as list of codes (truncated due to space constraints) used to gather human relevance judgements is shown in Figure 10.
orogy
acample, apps, systemo, hcalth or health in generalt
they rolame cardiovaseular
Answer Ithnk It's more efrective can enouch Demmnr, dienrty your you got this yoar pot think helpful
snow doctor.
for heatih protesstonais, slong with helping regular polien
Relevent
Inolovant
Professionall corra itatiem
Prefenenize
supportec
Not being sware of Ulllictton
Sourocs
O
Indess research
0
pationts
Beraritt
Figure 10: An example of question and text span from coder 'ss annotations shown the human evalu- ator. We truncate the list of codes shown due to space constraints.

---

A.6 Additional Results
A.6.1 sNDCG Plots
Model
SVM
Okapi-BM55
DisGiBERT
B/Encoder
Gross-Encoder (DisoIERR))
Soft NDCG@1
0.8
Cross-Encoder (ConvBERT)
OpenAi GPT-3.5
0.7
Point of Data Saturation
Figure 11: Plot of sNDCG@+ @4 for coder I across K to K 14
Model
0.95
Okspi-BM25
DisalBERT
BiEncoder
0.9
Cross: Encader (DistIBERT)
Solt NDCG@84
Cross Encoder (ConvBERT)
OpenAi GPT-2.5
0.85
0.8
1
3
10
11
12
13
14
Point of Data Saturation
Figure 12: Plot of sNDCG@ @4 for coder 2 across K to
14

---

A.6.2
MRR Tables
5
10
12
SVM
0.28
0.29
0.39
0.41
0.44
0.49
0.52
0.5
0.52
0.51
0.51
0.54
0.47
0.42
DistiBBER
0.45
0.39
0.42
0.43
0.52
0.5
0.55
0.55
0.57
0.57
0.57
Okapi-BMaR
0.19
0.19
0.17
0.14
0.18
0.14
0.17
0.17
0.14
0.15
0.14
0.11
0.18
0.17
Bi-Encoder
0.55
0.41
0.44
0.4
0.45
0.48
0.51
0.54
0.48
0.48
0.54
0.54
0.54
0.55
Cross-Encode Casss-nnodder (DistiBBERT)
0.41
0.45
0.39
0.51
0.5
0.55
0.57
0.56
0.55
0.55
0.56
0.61
0.56
0.54
Cross-Encoder (ConvBERT)
0.69
0.59
0.52
0.42
0.63
0.58
0.61
0.56
0.59
0.60
0.62
0.57
0.52
GPT-3.5
0.74
0.69
0.66
0.64
0.66
0.58
0.6
0.63
0.57
0.56
0.63
0.58
Table 7: MRR for coder for = 1 to K
2
3
5
7
8
10
11
12
13
14
SVM
0.63
0.67
0.36
0.66
0.68
0.46
DistiBBETT
0.67
0.6
0.25
0.27
0.35
0.76
0.76
0.75
0.75
0.75
0.74
0.7
0.71
0.63
Okapi-BM25 BM25
0.23
0.28
0.23
0.22
0.23
0.23
0.22
0.24
0.23
0.23
0.23
0.23
0.2
0.19
Bi-EEnouder
0.67
0.5
0.41
0.73
0.76
0.74
0.7
0.71
0.7
0.74
0.76
0.73
0.71
0,68
Cross Encoder (DistiBBRRT)
Cross-Encoder
0.62
0.63
0.67
0.72
0.72
0.68
0.64
0.71
0.69
0.64
0.59
Cross-Encoder Encoder (ConvBERT)
0.62
0.7
0.74
0.67
0.78
0.78
0.76
0.75
0.77
0.77
0.78
0.76
0.72
0.72
GPT-3.5
0.89
0.78
0.76
0.74
0.77
0.77
0.76
0.74
0.73
0.73
0.73
0.71
0.71
0.75
Table 8: MRR for coder 2 for K = 1 to K
14
A.6.3 sNDCG @4 Tables
3
5
8
10
11
12
13
SVM
0.75
0.72
0.75
0.76
0.77
0.78
0.79
0.79
0.79
0.78
0.79
0.8
0.79
0.77
DistiBERT
0.81
0.77
0.78
0.77
0.77
0.77
0.8
0.79
0.8
0.8
0.8
0.8
0.8
0.79
Okapi-BM25
0.74
0.73
0.72
0.71
0.71
0.71
0.71
0.71
0.71
0.71
0.71
0.71
Bi-Encoder
0.81
0.80
0.82
0.8
0.8
0.82
0.8
Cross-Encoder (DistilBERT)
0.8
0.79
0.79
0.79
0.8
0.82
0.81
0.82
0.8
0.81
0.82
0.82
0.79
0.79
Euss-er (ConyBERT)
0.87
0.83
0.82
0.83
0.8
0.83
0.81
0.84
0.81
0.83
0.83
0.83
0.82
0.82
GPT-3.5
0.84
0.76
0.76
0.74
0.77
0.76
0.74
0.77
0.75
0.74
0.72
0.75
0.71
0.74
Table sNDCG@4 for coder for K
5
10
12
SVM
0.93
0.84
0.85
0.72
0.84
0.84
0.74
0.82
0.81
0.83
0.83
0.82
0.76
0.78
DistiBERT
0.74
0.76
0.86
0.85
0.83
Okapi-BMi BM25
0.93
0.82
0.83
0.81
0.82
0.81
0.82
0.8
0.79
0.79
0.79
0.79
0.8
0.79
Bi- Bi-Encoder
0.93
0.82
0.78
0.87
0.88
0.88
0.87
0.85
0.85
0.86
0.85
0.85
0.83
0.83
Cross-Encoder (DistiBBERT)
0.92
0.84
0.85
0.83
0.85
0.86
0.86
0.84
0.83
0.82
0.84
0.83
0.81
0.81
Cross-Fnender (ConvBERT)
0.86
0.88
0.88
0.87
0.85
0.86
0.86
0.84
0.84
GPT-3.5
0.86
0.86
0.86
0.85
0.81
0.82
0.83
0.81
0.81
Table 10: # for coder 2 for K
1 to K 14

---

A.6.4 Macro Fi Tables
2
3
5
7
8
10
11
12
13
14
SVM
0.44
0.37
0.37
0.32
0.35
0.35
0.35
0.32
0.3
DistiBBERT
0.49
0.52
0.58
0.56
0.55
0.62
0.60
0.63
0.61
0.62
0.64
0.65
0.63
Okapi-BM22 BM25
0.48
0.5
0.56
0,56
0,56
0.54
0.57
0.49
0.56
0.55
0.56
0.56
0.53
0.58
BiEncoder
0.36
0.3
0.32
0.38
0.36
0.38
0.39
0.4
0.41
0.42
0.42
0.43
0.44
0.46
Cross-Encoed Encoder (DistilBERT)
0.53
0.35
0.51
0.6
0.45
0.58
0.59
0.49
0.55
Cross-Encoder (ConvBERT)
0.48
0.38
0.36
0.5
0.41
0.56
0.59
0.47
0.55
0.61
0.59
OpenAl GPT-3.5 (Turbo)
0.26
0.37
0.4
0.46
0.41
0.42
0.42
0.44
0.47
0.53
0.44
0.52
0.41
0.44
Table 11: Macro Fl for the novel code for coder 1 for K
to K = 14
3
5
9
10
11
12
13
14
SVM
0.1
0.27
0.29
0.47
0.3
0.31
0.17
DistilBERT
0.49
0.49
0.4
0.43
0.45
0.45
0.45
Okapi-BM25
0.44
0.6
0.56
0.50
0.53
0.5
0.56
BiEncoder
0.52
0.39
0,4
0.47
0,45
0.45
Mooss-Encoder (DistilBERT)
0.47
0.63
0.56
0.46
0,46
0,56
0.59
Cross-Encoder (ConyBERT)
0.47
0.56
0.64
0.47
0,57
0.55
0.57
OpenAl GPT-3.5 (Turbo)
0.13
0.42
0.42
0.49
0.45
0.46
0.48
Table 12: Macro Fl for the novel code prediction subtask for coder 2 for K -t = 14. Cells past = are
marked with s_" since no novel codes occur.
A.6.5 Míero Fi Tables
2
3
5
7
8
10
11
12
13
14
SVM
0.79
0.58
0.56
0.46
0.44
0.4
0.41
0.39
0.38
0.37
0.35
0.34
0.31
0.22
DistiBBRRT
0.51
0.53
0.58
0.60
0.59
0.66
0.65
0.68
0.67
0.71
0.69
0.74
0.75
0.84
Okapi-BM25
0.55
0.53
0.57
0.57
0.59
0.63
0.65
0.6
0.63
0.62
0.63
0.64
0.64
0.76
BiFncoder
0.36
0.42
0.44
0.56
0.57
0.62
0.63
0.68
0.69
0.72
0.73
0.74
0.77
0.86
Cross-Encoder (DistilBERT)
0.56
0.53
0.45
0.54
0.59
0.61
0.64
0.68
0.68
0.64
0.64
0.8
0.79
Cross-Encoder (ConvBERT)
0.74
0.53
0.53
0.57
0.63
0.62
0.56
0.71
0.7
0.66
0.64
0.62
0.61
OpenAl GPT-3.5 (Turbo)
0.28
0.44
0.47
0.58
0.55
0.57
0.6
0.64
0.66
0.71
0.67
0.71
0.69
0.78
Table 13: Micro F1 for the novel code prediction subtask for coder for K = 1 to K 14.
2
3
5
9
10
11
12
13
14
SVM
0.12
0.36
0.36
0.87
0.3
0.32
0.19
DistilBERT
0.62
0.58
0.66
0.56
0.80
0.81
0.83
Okapi-BM25
0.63
0.62
0.58
0.71
0.76
0.68
0.79
BiEncoder
0.77
0.64
0.66
0.87
0.8
0.81
0.83
Cross-Fneoder (DistilBERT)
0.88
0.67
0.68
0.87
0.78
0.74
0.8
Cross-Encoder (ConyBERT)
0.88
0.65
0.70
0.87
0.79
0.76
0.74
OpenAl GPT-3.5 (Turbo)
0.14
0.65
0.65
0.87
0.8
0.79
0.83
Table 14: Micro Fl for the novel code prediction subtask for coder 2 for K = to K 14, Cells past = are marked with "_" since no novel codes occur.


Reviews:
[{"review_id": "RRVxedKXaQ", "reviewer": "Reviewer_ZTy2", "summary": "", "questions": "n/a", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comments and their feedback. We appreciate the fact that they highlighted the comprehensive nature of our work which proposes to take a more assistive approach to qualitative coding and which also includes a human evaluation. We would also like to thank the reviewer for pointing out that our work develops and releases the first qualitative coding dataset* (*Pending publication). Finally, we would like to thank the reviewer for their strong view of this paper’s excitement and soundness. \n\nWe address the reviewer’s main point below:\n\nWe would be happy to add a figure describing the overall workflow of our approach in the camera-ready version of the paper. This figure will present, at a high level, the collection of the interviews and the annotations carried out by the qualitative coders, the preprocessing of the annotations to create the Quoding dataset, the three methods we study to tackle the Qualitative Code Suggestion (QCS) task, and finally the human evaluation. We highly appreciate the suggestion and believe that these changes would improve the presentation and general readability of the paper.\n\nThank you again for your feedback and your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-comprehensive study\n-new contribution and development of resource\n-verification of the proposed method assisted by human evaluation\n", "reasons_to_reject": "-the workflow was difficult to follow in some cases. It would be useful to provide a diagram describing the overall workflow of the method.", "questions_to_authors": "n/a", "paper_topic_and_main_contributions": "This study investigates the problem of qualitative coding: assigning descriptive labels to the text spans of a document. The authors propose a more assistive approach following an automated code suggestion taking into account a ranked list of previously suggested codes. At the same time, the authors define the problem as QCS: qualitative code suggestion and developed a dataset Quoding consisting of interviews from cardiovascular domains. The findings include identifying the effect of personalized annotation style, how taking into account facts like passage sequence, rare-code appearance makes an impact. The evaluation includes both automated scoring and human judgment.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aCXMqjd69N", "reviewer": "Reviewer_WMQc", "summary": "", "questions": "see weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comments and very detailed feedback. We appreciate the reviewer pointing out that the conceptual task model of qualitative coding we propose is well motivated and grounded in existing literature. We would also like to thank the reviewer for highlighting the detailed and reproducible nature of our experimental setup and for pointing out that the analysis of our experiments support our task definition and the properties of qualitative coding we study (i.e., data saturation, rare codes and annotation styles). Finally, we would like to thank the reviewer for their strong view of this paper’s excitement and soundness.\n\nWe first respond to the reviewer’s main points and then address their presentation/organizational suggestions:\n\n*Main points*\n\nFirstly, as it stands, there has been little work on studying automated assistance in inductive qualitative coding. The Chandra et al. (2019) reference the reviewer cites studies the process of inductive coding in general and not an automated assistive technique in particular. Some of the most notable work in automatic inductive coding includes Guetterman et al. (2018) who developed a WordNet-based clustering technique to group semantically similar text spans to give researchers an idea of commonalities in their dataset before they started the coding process. We would be glad to clarify the Related Works section and add additional references to differentiate our work from previous studies which bring automated assistance to inductive coding. \n\nSecondly, we would like to clarify some confusion regarding the human evaluation. The main purpose of the human evaluation was to determine whether the best QCS models were suggesting relevant codes even when the suggestions didn’t agree with the original annotations from the Quoding dataset. This extra validation step was necessary because we noticed that, in many cases, the qualitative coders did not consider all possible code assignments for a text span. Our human evaluation demonstrated that, even though our systems didn’t always rank the original codes highest, they consistently ranked relevant codes at the top. Thus, our human evaluation was not a user study meant to investigate the implications of QCS methods on coding workflow and findings. We would be happy to clarify the purpose of the human evaluation in Section 7.2 and add a more detailed description in the Appendix.\n\nFinally, in this paper, we do not measure the impact of the QCS task and its different modeling paradigms in a practical setting (See previous paragraph). We believe that such an endeavor merits additional follow-up work to be explored in a human-computer interaction venue. The reason for a rigorous user study being beyond the scope of this paper is that we expect findings to be rich and diverse. For instance, in Rietz and Maedche (2021), the authors dedicate the entire publication to study the implications of retrieving text spans related to a given code (the “dual” of QCS). We would be happy to expand this line of thought in the Conclusion as future work. \n\n*Presentation/organization suggestions*\n\nWe thank the reviewer for their thoughtful and detailed suggestions on the paper’s presentation and organization. We would be happy to reorganize the Results section, as suggested by the reviewer, to make the key insights apparent. The key insights include: 1. The information-retrieval and zero-shot paradigms performing better than the commonly-used classification approach and 2. The statistically significant jumps in performance when model outputs are evaluated by humans. We are happy to apply the reviewer’s organizational suggestions to move Table 1 to the Appendix, to remove some redundant information in sections 5.3 and 6.1 and to place the scatterplots in the main paper to support the Analysis section. We are also glad to address the readability issue raised on lines 661-662 and to add the GPT3.5 prompt which we inadvertently deleted and which we include below\n\t\n\"\"\"You are a helpful assistant that suggests qualitative codes for a qualitative researcher. The coders you can suggest are: [AVAILABLE_CODES], and None of the above. Which of the previous codes would you assign to the following excerpt from an interview with a woman at risk of cardiovascular disease (CVD): “Question: [QUESTION] Answer: [ANSWER]” \"\"\"\n\nThanks again to the reviewer for their constructive feedback and useful comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The conceptual task model of qualitative coding is well-motivated and each of the properties identified — data saturation, rare codes, annotation style —, are grounded on existing work.\n\n\n- The Quoding dataset is a valuable contribution which will drive future research on the task of qualitative coding suggestion (QCS).\n\n\n- The authors provided sufficient details of the different modeling approaches for QCS and rationalized their choices of models for each approach. \n\n\n- The experiment setup is detailed and seems reproducible.\n\n\n- The authors conducted detailed experiments presenting and analyzing results of automated experiments and human subject studies. The high-level insights from these studies reinforced the initial task model outlined by the authors.", "reasons_to_reject": "- The paper fails to situate itself with respect to existing work. The related work section should be updated to differentiate the proposed approach with related work.\n\n- The objective of the user study was not clarified. The authors mention evaluating the “effectiveness of the proposed QCS strategy.” However, what specific research questions they plan to address through the study is unclear. The author should clarify whether the evaluation was meant to be a thorough study of user experience with the proposed QCS method or simply meant to be a verification step.", "questions_to_authors": "see weaknesses.", "paper_topic_and_main_contributions": "The paper proposes a user-centric approach to assist qualitative researchers by providing automated agent generated code suggestions. The key contributions of the paper include: a conceptualized task model of qualitative coding, a publicly available qualitative coding dataset, and a detailed evaluation of qualitative coding suggestion models via automated experiments and human subject studies.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 661-662: hard to read\n- Section 5.4 mentions that the zero shot prompt can be found in the appendix. Could not find the prompt in Section A.4.4.\n\n\nPresentation/Organization suggestion:\n- Table 1 is not informative and can be moved to Appendix.\n- Scatterplots from A.7 should be presented in the main paper with sufficient detail. In the current form, the analysis section is not self-contained.\n- To make space, the authors can make the discussions in Section 5.3 and 6.1 more concise. The modeling approaches are mentioned multiple times: at the beginning of Section 5, section 5.3, and section 6.1 and could be reorganized to make the discussion crisp.\n- Section 7.1 and 7.2 are very difficult to read and the key insights are not apparent. The insights should be highlighted."}}, {"review_id": "dNHoyU6gwL", "reviewer": "Reviewer_ULRd", "summary": "", "questions": "- §2 (On deductive vs. inductive coding) --> What would make inductive coding fundamentally different? I would think that inductive coding just adds an extra step of exploratory analysis where codes are defined before moving on to the official annotation. However, this annotation itself would then not be different from deductive coding.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comments and feedback. In particular, we appreciate the reviewer highlighting the potential of our QCS systems to become full-fledged tools which could assist qualitative coders in their annotations. We also appreciate the reviewer pointing out the development of the Quoding dataset as well as the level of detail and the value of the analysis of our results (Section 8).\n\nWe respond to the reviewer’s main points and answer their questions below:\n\nFirstly, the difference between inductive and deductive coding lies in the fact that each coding technique serves a significantly different purpose. Deductive coding is used to validate or invalidate a pre-defined hypothesis about a corpus. This purpose is why, in deductive coding, researchers design a codebook based on a research question about a corpus and then code the corpus looking specifically for text spans supporting those codes. In contrast, inductive coding is used to discover new phenomena underlying a corpus. This purpose is why, in inductive coding, researchers do not have a pre-defined research question, a codebook is not designed prior to coding and prior exploratory data analysis is not recommended. This difference is one of the reasons why existing fully automatic coding systems (which frame coding as text classification) are not suitable for inductive coding and is what motivated the creation of the QCS task.\n\nSecondly, there is a fundamental difference between qualitative coding and other annotation processes which human-in-the-loop papers have studied. Not only is qualitative coding an annotation process, but it is also - and more importantly - a way for researchers to understand and familiarize themselves with their data. As Jiang et al. (2021) point out in their paper on the challenges of human-AI techniques for qualitative coding, “qualitative analysis is a unique case for human-AI collaboration because uncertainty is essential to qualitative analysis itself.” In other words, qualitative coding is unique because the act of coding data, and the uncertainty that comes with it, is valued by researchers as it cultivates their understanding of their corpus. We would be happy to add additional references to the camera-ready version of the paper to explain how the nature of qualitative coding is fundamentally different from that of other annotation methods.\n\nFinally, as discussed in a response to another reviewer, the human evaluation we perform is not meant to be seen as a user study which investigates how the qualitative coders react and work with a QCS method. The primary purpose of the human evaluation is to validate the relevance of the code suggestions made by each QCS method. As we discuss in the Results section (7.2), our human evaluation shows that the most performant QCS systems (ConvBERT and GPT-3.5) are able to consistently make relevant code suggestions. We have left the study of how relevant code suggestions impact the workflow and findings of qualitative coding for future work as we believe this separate endeavour merits its own human-computer interaction publication. We would be happy to clarify the purpose of the human evaluation and its results in the camera-ready version of the paper. In addition, we would be happy to discuss the necessity of a comprehensive user study in the Conclusion as future work.\n\nThanks again to the reviewer for their constructive feedback and useful comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Potentially helpful tool to many researchers that perform qualitative coding in some capacity\n- Introduction of an open-source dataset for this task\n- Extensive analysis of the results (section 8 is quite valuable)", "reasons_to_reject": "While the authors do a good job of reporting and describing their results, I do have some questions related to clarity and methodological choices.\n\n- Questionable to what degree this task is novel (therefore missing relevant baselines) (see \"Missing References\").\n- Evaluation does not seem most relevant for this task. The main purpose of this system is to help annotators do their work more efficiently. Do the suggestions make annotation more efficient (in terms of annotation time), and do they decrease Annotation Errors by the annotators when used, for instance?", "questions_to_authors": "- §2 (On deductive vs. inductive coding) --> What would make inductive coding fundamentally different? I would think that inductive coding just adds an extra step of exploratory analysis where codes are defined before moving on to the official annotation. However, this annotation itself would then not be different from deductive coding.", "paper_topic_and_main_contributions": "This paper describes a novel approach to perform qualitative annotation. Previous research mostly aimed to create a system that does the annotation process fully automatically, while this paper aims to provide \"qualitative code suggestion\" so that the annotation process becomes a collaboration between human and machine. Furthermore, the authors aim for their approach to not only suggest pre-defined codes, but also detect and suggest novel, rare, codes that have not been defined. It should also be mentioned that the authors introduce the Quoding dataset, which is the only open-source dataset for this type of task.\n\nThe results show that their human-centric approach to qualitative coding can be of great assistance to annotators. However, the system is not capable of performing the novel code detection subtask.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "To me it seems like this proposed task largely overlaps to a large degree with human-in-the-loop annotation, which is a well-studied topic. Tools exist, like Prodigy for instance (Montani & Honnibal, 2018). And also previous ACL-research, such as:\n\nYu et al. (2015). LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop\nLiu et al. (2019). Deep reinforcement active learning for human-in-the-loop person re-identification.\nFan et al. (2019). An interactive visual analytics approach for network anomaly detection through smart labeling.\nGesller et al. (2022). Midas Loop: A Prioritized Human-in-the-Loop Annotation for Large Scale Multilayer Data.\nMendes et al. (2023). Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments.\nKlie et al. (2020). From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains.\n\nIt would have been nice if this work had been discussed in this paper to compare previous approaches on this topic to their approach, and maybe also to provide relevant baselines to compare their QCS to."}}]

Author rebuttal:

Meta-review: The paper proposes a user-centric approach to assist qualitative researchers by providing automated agent generated code suggestions.

Reviewers agreed that:
1. The paper presented a comprehensive study
2. The conceptual task of qualitative coding is well-defined
3. The dataset is useful.

Originally reviewers WMQc and ULRd had some concerns about the connection between this and prior work, and the validity of the user study, which the authors sufficiently addressed in rebuttal.

I recommend accepting the paper into the Main track, and encourage the authors to add a figure illustrating the workflow, extending the related work session, and providing clarification to the human eval.
Decision: Accept-Findings