LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
Authors: Shih-Chieh Dai
Aiping Xiong
Lun-Wei Ku
Venue: EMNLP 2023 Findings (2023)
Keywords: thematic analysis
NLP applications
qualitative research
TL;DR: We propose a Human-LLM collaboration framework for thematic analysis.

Abstract: Thematic analysis (TA) has been widely used
for analyzing qualitative data in many disciplines and fields. To ensure reliable analysis,
the same piece of data is typically assigned to at least two human coders. Moreover, to produce
meaningful and useful analysis, human coders
develop and deepen their data interpretation
and coding over multiple iterations, making TA
labor-intensive and time-consuming. Recently
the emerging field of large language models
(LLMs) research has shown that LLMs have
the potential replicate human-like behavior in
various tasks: in particular, LLMs outperform
crowd workers on text-annotation tasks, suggesting an opportunity to leverage LLMs on
TA. We propose a human–LLM collaboration
framework (i.e., LLM-in-the-loop) to conduct
TA with in-context learning (ICL). This framework provides the prompt to frame discussions
with a LLM (e.g., GPT-3.5) to generate the final
codebook for TA. We demonstrate the utility
of this framework using survey datasets on the
aspects of the music listening experience and
the usage of a password manager. Results of
the two case studies show that the proposed
framework yields similar coding quality to that
of human coders but reduces TA’s labor and
time demands.

LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
Shih-Chieh Dai
Aiping Xiong*
Lun-Wei Ku²
University of Texas at Austin 'sjdai@utexas.edd
2 Pennsylvania State University AAcademia Sinica
axx29epsu. edu '1wku@iis. sinica edu. tw
Abstract
Large language models (LLMs) have ad- vanced artificial intelligence (AI) tremendously re- cently (Brown et al., 2020; Chowdhery et al., 2022; Scao et al., 2022; Touvron et al., 2023). Prompt- ing techniques such as in-context learning (ICL), chain-of ofthou-,-t and reasoning and action elicit decent results on various natural language process- ing (NLP) tasks (Wei al., 2022; Kojima et al., 2022: Suzgun et al., 2022: Yao et al., 2023). LLM- based applications continue multiply, including Copilot for programming. LaMDA generation (Thoppilan et al., 2022), and ChatGPT.3 Indeed, several studies show that LLMs outperform human annotator performance on crowdsourcing tasks (Gilardi et al., 2023; Chiang and Lee, 2023; Ziems et al., 2023), which indicates potential op- portunity leverage LLMs for TA. Thus, in this work, we address the research question: can NLP techniques (i.e., L.Mss) enhance the efficiency of TA?
Thematic analysis (TA) has been widely used for analyzing qualitative data many disci- plines and fields. To ensure reliable analysis. the same piece data is typically assigned to at least two human coders. Moreover, to produce meaningful and useful analysis, human coders develop and deepen their data interpretation and coding over multiple iterations, making TA labor-intensio intensive and mome-consuming. Recently the emerging field large language models (LLMs) research has shown that LLMs have the potential replicate human-like behavior in various tasks: in particular, L.MMs outperform crowd workers on text-annotation tasks, sug- gesting an opportunity to leverage L.LMs on TA. We propose human LLM collaboration framework (i.e., L.LM-in-the-loop to conduct TA with in-context learning (ICL). This frame- work provides the prompt to frame discussions with LLM (e.g., GPT--3.5) to generate the final codebook for TA. demonstrate the utility of this framework using survey datasets the aspects of the music listening experience and the usage of a password manager. Results of the two studies show that the proposed framework yields similar coding quality to that of human coders but reduces TA's labor and time demands."
dialogue
We loop ChatGPT to act as a machine coder (MC) and work with a human coder (HC) on TA. Following Braun and Clarke (2006), we leverage ICL to design the prompt for each step. The HC first becomes familiar with the data, after which the MC extracts the initial codes based on the re- search question and free-text responses. Next, the MC groups the similar initial codes into represen- tative codes representing the responses and which answer the research question. We design discus- sion prompt for the MC and HC to use to refine the codes. The discussion ends once the HC and MC come to agreement, after which the codebook generated. Following the TA procedure, the HC and MC utilize the codebook code the responses and calculate the inner-annotator agreement (IAA) to measure the quality of the work. See Section 3 for detailed information.
1 Introduction
Braun and Clarke (2006) propose thematic analysis (TA) to identify themes that represent qualitative data (e.g. free-text response). TA, which relies on at least two human coders with relevant expertise the whole process, is widely used in qualita- tive research (QR). However, TA is labor-intensive and consuminn For instance, for an in-depth understanding of the data, coders require multiple rounds of discussion to resolve ambiguities and achieve consensus.
We study the effectiveness of this human-LLM
'The code repository is publicly available at: http://hhhhh.cccciiaaaaa-l--aaaa
thttps://githhu com comm/features/cpplott htttts:://openai.com/b.cocc

---

Step 3.
Step 4. Theme Identification Evaluation
Familiarization Exemplar Generation
Step
Code Refinement
Initial Codebook Generation
Action:
Delete
the "Reflective and Mood-setting Songs", codes, since they are all covered under "Emotional Relatable Songs'
Reason:
Exemplars
Themes
Data
Codebook
Step 2.
Initial Code Generation
Codes
Annotated
Annotated
Data
Data
Initial Codes
Decision: Agree
Exemplars
Reason:
These codes
merged Into the 'Emotional and Relatable Songs' code as they all emotional responses.
Aoroaatto-Aggremmet
R Human SPT
Figure 1: Workflow of the LLM-in-the-Loon Loop Framework.
collaboration framework by adopting the frame- work on two cases: Music Shuffle (MS), a sur- vey measuring aspects of music listening experi- ences (Sanfilippo et al., 2020) and Password Man- ager (PM), survey on password manager us- age (Mayer et al., 2022). Considering the LLM input size limitations, sponses of PM into two separate pools and applied one pool for codebook generation. Subsequently, we sampled responses from both pools for final cod- ing and IAA calculation. The results suggest that framework yields
into themes based on their sentiment polarity. Guet- terman et al. (2018) cluster phrases by their Wu- Palmer similarity scores (Wu and Palmer, 1994). Models can be trained to learn how humans code (Rietz al., 2020). Human-Al AI collaboration frameworks for TA have been proposed (Gauthier and Wallace, 2022: Gebreegziabher et al., 2023; Jiang et al., 2021).
randomly divided the re-
QR and NLP researchers are now exploring the opportunity to leverage LLMs for TA. Xiao et al. (2023) investigate the use of LLMs to support deductive coding: they combine GPT-3 with expert-drafted codebook. Gao et al. (2023) pro- pose a user-friendly interface for collaborative qual- itative analysis, leveraging LLMs for initial code generation and to help in decision- maaiinn. Paoli (2023) studies the potential usage ChatGPT for TA following the six TA phases proposed by Braun and Clarke (2006). In the theme refinement phase, the author generates three versions of themes by modifying the temperature parameter," after which the themes are reviewed by HC (the author).
the proposed LLLM-nn-the-ll work quality comparable to two HCs.
The contributions of this work are as follows: 1) we propose a human-LLM collaboration frame- work for TA; 2) we design a loop to facilitate dis- cussions between a human coder and a LLM; 3) we propose solution for long-text qualitative data when using LLM for TA.
2 Background and Related Work
Given the substantial amount of free-text responses, it seems obvious that NL. techniques can be poten- tial solutions to facilitate TA. Existing work tends rely on topic modeling extract key topics that represent the data (Leeson al., 2019; Jelodar al., 2020). Another approach is clustering: for instance, Oyebode et al. (2021) extract phrases, cal- culate their sentiment scores, and group the phrases
Coding can be divided into inductive coding and deductive coding. Inductive coding is bottom-up process of developing codebook, and deductive coding is top-down process in which all data
'The temperature parameter determines the output randomness: higher values yield more random output. https://platfor openai.ppen....mmmmmmmmm foo//ocs//pp www//oos/apiireeeeeeeeeeeeeeeeee aepi-refeeence// chat/create

---

Code Refinement and Initial Codebook Genera- tion The model groups the similar initial codes into certain codes which the MC and HC begin to refine. Step 3 in Figure illustrates a cycle of the discussion process. First an HC reviews the codes and edits the codes if necessary. If the HC modifies any code, he/she states the changes and provides the rationales behind them. After evaluating the changes and the rationales, the MC indicates agree- ment or disagreement with explanations. The HC then reviews the MC's responses and revises the code if needed. The HC initiates a new cycle by editing the codes and providing justification. Such refinement continues until the HC and MC are both satisfied with the result. The refined codes are then used to generate the initial codebook.
Response: This song raminds of singing along with my friends on summer vacation.
Action:
Lef's code reeponea!
"This song reminds me singing along with my friends on summer vacation" refers to a time in the past. Therefore, we got a code: Memories
Figure An exemplar for initial code extraction. Green depicts the quoted sentence, blue indicates the definition of a code, and red represents the code.
is coded using the codebook. We complete both inductive and deductive coding via human-Al col- laboration and showcase the feasibility of our ap- proach using two existing open-access datasets.
Theme Identification and Evaluation The MC and HC generate the final codebook by identify- ing the themes of the codes. MC and HC code the whole qualitative data according to the final codebook. Cohen's x agreement (Cohen, 1960) is calculated to evaluate the work quality of the MC and HC.
3 Human-LLM Collaboration Framework
Figure illustrates the framework, which consists of four steps.
Data Familiarization and Exemplar Generation We used in-context learning (ICL) to design the paradigm of prompt for the initial code extrac- tion (Garg et al., 2022). To write the exemplars for initial code extraction, the HC must be familiar with the qualitative data. Each HC produces four to eight exemplars, following Min et al. (2022). Each exemplar includes response for an open-ended question and the associated actions. Given a free- text response may contain multiple codes, we think "step by step" and "code by code" that were in- spired by chain-of thought (CoT) prompting (Wei et al., 2022; Kojima et al., 2022). To code re- sponse, the format of an action is "quote refers to definition of the code. Thus, we got a code: code". Figure shows one instance. The quote is the sentence in the response that related to the code.
Evaluation
4.1 Metrics
The inner-annotator agreement (IAA) is the main method for evaluating the quality of coding (Krip- pendorff, 2011). IAA measures the clarity and interpretability of a codebook among the coders. A higher IAA suggests the codebook is clear and effectively captures the meaning of the data. We use Cohen's to compare the results of the two datasets. In addition, to evaluate whether the codes from our framework meet the research goal of the original work, we treat the codes generated by the authors of the two datasets as the gold label. We calculate the cosine similarity between codes devel- oped by us and those proposed by the original au- em-
thors. We used the text-embedding-adde bedding provided by OpenAl to embed the codes and then calculated their cosine similarity.
Initial Code Generation At this step, codes are extracted that capture the semantic and latent mean- ings of the free-text responses at fine granularity. The core of the prompt is the exemplar generated in the first step. The prompt also contains the task goal and the open-ended question, which provide the MC with clear understanding of the purpose of the study. The prompts we used are shown in Appendix D.
4.2 Setting
MS In our case study, we focused on the first- asked question because the authors mentioned that some participants provided the same or similar an- swers all four questions. The sampled question "What's the first thing that comes into your mind about this track?" Preprocessing yielded 35 re- sponses for our study.

---

Music Shuffle
Password Manager
LLM is suited for thematic analysis. The re- sults of the MS case study suggest that HC+MC achieves the best agreement compared to all other settings, where the x. of HC+MC is 0.87 (almost perfect agreement), and that of Gold is 0.66 (sub- stantial agreement). Further, even though the K decreases in the Coder 3 settings, Coder 3+MC and Coder 3+HC still exhibit similar agreement com- pared to Gold: The of Coder 3+MC and Coder 3+HC are 0.54 (moderate to substantial agreement) and 0.62 (substantial agreement), respectively. This indicates that our framework achieves reasonable performance. In the results of the PM case study, HC+MC (k 0.81: almost perfect agreement) outperforms Gold (x = 0.77: substantial agree- ment). The cosine similarity also shows that the MC-extracted codes capture the semantic meaning of the codes provided by the authors of the datasets. We thus conclude that the human-LLM LLM collabo- ration framework can perform as well as the two human coders on thematic analysis, but with one human coder instead of two, which reduces labor and time demands.
All
Seen
Unseen
All
HC MC
0.87
0.8
0.82
0.81
Coder3 MC
0.54
0.47
0.34
0.47
Coder + HC
0.62
0.48
0.38
0.43
Gold
0.66
0.77
Table I: Cohen's 's of the two cases. The HC, MC, and Coder 3 used the same codebook for final coding. Seen and Unseen indicate the data was sampled from the responses used or not used for developing the codebook, where All means Seen + Unseen
MS
PM
Similarity
0.8864
0.8895
Accuracy
0.84
0.83
Recall
0.72
0.87
Table 2: Cosine similarity, accuracy, and recall between the codes extracted by our framework and the dataset's ss authors.
PM Of the eight open-ended questions, we se- lected the first question-"Pleete "Please describe how you manage your passwords across accounts"-as this question is general, and seems to have elicited bet- ter responses. Due to the GPT input limitations, we could not feed all the data into GPT-3.5. Thus we randomly sampled 100 responses for codebook development. Such leveraging of partial responses for codebook development has been used by San- filippo al. (2020). We used the 100 responses for codebook development and in the evaluation phase we sampled 20 samples from the two data pools for labeling and IAA calculation, respectively.
A discrepancy discussion process might be neces- sary. The relatively poorer results of Coder still indicate nearly fair to moderate agreement. The k of Coder 3+MC and Coder 3+HC are (0.47, 0.48), (0.34, 0.38), and (0.47, 0.43) for the three condi- tions (i.e., Seen, Unseen, and All). respectively. The PM authors mentioned that if the agreement is lower than k =0.7, they discussed the result and started coding round to re-code the data. They indicated average 1.5 rounds of re-coding. Similarly, our results indicate need for such a mechanism. We leave this as future work.
Following the MS and PM workflow, de- veloped codebook using our framework with a HC and the MC for each case. The HC and MC coded all the responses. We invited another coder (Coder 3) to code the responses using both code- books developed by the HC and MC. The detailed information of the datasets is in Appendix C.
Developing the codebook using partial data is feasible approach. In the Unseen condition, the HC+MC agreement is high (k = 0.82: almost perfect agreement). Additionally, the cosine simi- larity between the codes extracted by MC and the gold codes provided by the authors of the dataset is 0.8895, which indicates the codes effectively cap- ture the semantic meaning of the gold codes. These results imply that using partial data for codebook development is a viable way to address LLM input size limitations for TA.
4.3 Results and Discussion
The results are presented in Table 1: HC + MC indicate Cohen's 's i of HC and MC annotated data, and Coder 3 + MC and Coder 3 HC represent Cohen's 's of Coder and MC labeled data and Coder 3 and HC labeled data, respectively. We treat the result of the original studies as the gold standard for reference. Table 2 shows the result of cosine similarity, accuracy, and recall.
4.4 Error Analysis
To investigate why the IAA dropped significantly in the results Coder 3 with MC and HC, we pull

---

MS
PM
missed the "Positive".
Amhiguity
5
Granularity
8
Distinction The other mismatched responses are due to the coders assigning different codes for a given response. This could be a result of their varied familiarity with the responses and the code- book, leading them to interpret the response or code differently. For instance, Coder 3 stated that sometimes it is hard to find a code that fits the sponse. As result, Coder can only identify the most appropriate code for the response.
Distinction
9
Total
22
30
Table 3: The number of mismatched responses in each category in the two different datasets.
(AII)
(Seem)
PM (Uneen)
PM (AIl)
MC+Coder3 0,54/0.56
0.47/0.72
0,34/0.50
0.47/0.62
HC+Coder3
0.62/0.72
0.48/0.99
0.38/0.54
0.43/0.62
Table 4: The number before "/" is the IAA calculated by code, and the number after "f" represents the IAA calculated by theme.
Overall, we identified the three main factors that led the IAA drop significantly for Coder 3 with MC and HC. However, we argue that thematic anal- ysis is inherently subjective and biased since the coders must align to the research questions when making the codebooks (Braun and Clarke, 2006; Vaismoradi al., 2016). They generated the code- book based on their expertise in the topic of the responses and the research goal. Moreover, such subjective nature is not specific to our work. In- stead, it is common for any thematic analysis. It is arguable whether bias is a drawback for thematic analysis since the goal is to provide a way to un- derstand the collected free-text data. For instance, even though the codebooks in our study are biased, they have captured the semantic meaning of the data to the research questions, and the cosine simi- larity between our codebooks and the codebooks from the dataset is high (0.8864 and 0.8895). The iterative nature of codebook generation leads the codebook to be less biased. As we mentioned in the discussion section, discrepancy discussion process might be solution to improve the IAA.
the sample that they coded differently and the code- book. We found three major reasons: Ambiguity, Granularity, and Distinction that led the low IAA of Coder with MC and HC. Table 3 is the analy- sis of the samples that the three coders (MC, HC, Coder 3) coded differently.
Ambiguity Ambiguity indicates that the coders coded the different codes that actually belong to the same theme. For instance, "Digital Notes" and "Notes" are both under the theme "Written Records and Notes." Given a free-text response, "Keep them in a notes tab on my phone", the coders might code this response as "Digital Notes" or "Notes." These two codes belong to the same theme and are correct but would drop the IAA if one coder selects "Digi- tal notes" and the other selects "Notes." We found that among 22 mismatched responses 5 of them in MS dataset were due to the ambiguity of the codes. In PM dataset, 9 out of 30 are categorized as the reason. To further confirm if the ambiguity of the codes influences the IAA, we calculated the IAA based on the theme. Table 4 represents the re- sult. The result suggests that all the IAA improved, which indicates that the ambiguity of the codes is one factor leading to the low TAA.
5 Conclusion
We propose framework for human-LLM collabo- ration (i.e., LLM-ii-the-loop) for thematic analysis (TA). The result of two case studies suggests that our proposed collaboration framework performs well for TA. The human coder and machine coder exhibit almost perfect agreement (w 0.87 in Mu- sic Shuffle and x 0.81 in Password Manager), where the agreement reported by the dataset au- thors are 0.66 for Music Shuffle and = 0.77 for Password Manager. We also address the L.M input size limitation by using partial data for code- book development, with results that suggest this is a feasible approach.
Granularity Granularity denotes the granularity of the coding work. We found that HC and MC coded more finely than the Coder 3 in 8 out of 22 and 7 out of 30 mismatched responses in MS and PM, respectively. For instance, for the response, "Sleeping, calm I saved it because I liked the way it sounded, I thought it would be a good song to fall asleep to. enjoy listening this song because it sounds sweet, like a cute sweet not cool sweet." Both HC and MC coded this response as "Positive", "Relaxing", and iidep-iiduuiin",. where Coder
Limitations
There are four main limitations in this work.

---

Prompts While we designed specific prompts for each step that successfully elicited the desired output to achieve our goals, it important to note that we cannot claim these prompts to be the most optimal or correct ones. As mentioned by Wei et al. (2022), there is still rooms to explore the potential- ity of prompt to elicit better outputs from LLMs. We believe that there might be better versions of prompts that can achieve better outcomes.
users need to confirm with their IRB before they used the proposed method for thematic analysis.
Acknowledgements
We thank Katie Zhang and Zekun Cai for helping us with the thematic coding tasks. We thank Yi- Li Hsu for fruitful discussions. Thanks as well to the anonymous reviewers for their helpful sugges- tions. The works of Aiping Xiong were in part supported by NSF awards #1915801, #1931441 and #212 This work is supported by the Na- tional Science and Technology Council of Taiwan under grants 11 111-2634---000--022-- and 112-2222- E-001-004-MY2.
Application The survey or interview data might not be able to be put into third-party platform like OpenAL. Specifically, certain Institutional review boards (IRB) may have regulations that the data should be stored on the institution's server. This is especially relevant for data that involves sensitive information, such as health-related studies. There- fore, the users have to check the IRB document or any regulations before they use our method for thematic analysis.
References
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay. Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with statc-of-the-art performance.
Model We only studied the cases on GPT-3.5, while many off-shelf LLMs are available 5. There- fore, we can not guarantee using other LLMs can achieve the comparable result. However, we have demonstrated the effectiveness of our pro- posed framework with GPT-3.5. Future work could adopt this framework other Open-sources LLMs, such as Falcon (Almazrouei et al., 2023) and LLaMA (Touvron et al., 2023). However, the computational resources and cost might be the po- tential limitations.
Virginia Braun and Victoria Clarke, 2006. Using the- matic analysis in psychology. Qualitative Research in Psychology, 3(2):77-001.
Virginia Braun and Victoria Clarke. 2012. Thematic analysis. American Psychological Association.
Virginia Braun and Victoria Clarke, 2019. Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4):589-59) 5977
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models few-shot learners. Advances in Neural information Processing Systems, 33:1877-199
Data The dataset, Music Shuffle, was published in 2020, while the training data for GPT-3.5 extends until September 2021. Therefore, GPT might have encountered or been trained on a similar dataset before. However, as most of the data are regulated by the IRB and the privacy concerns, it is challeng- ing find dataset containing the raw responses data and codebook for reuse. Therefore, we still de- cided to select this dataset for our study. The other dataset, Password Manager, was published in 2022, which does not have the concern, and our proposed method also performs well on this datasct.
Cheng- Baang-Ha&&o Chiang and Hung-yi Lee. 2023, Can large language models be alternative to human evalua- tions? arXiv preprint 205.17230..01957.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint wrXiv:2204.023711.
Ethics Statement
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychologicul mea- surement, 20(1):37-46.
The datasets we used in this work are all public for reuse. All the data do not contain privacy-sensitive information. Therefore, we used these two datasets with OpenAl GPT3.5 model. However, the other
Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Per- rault. 2023. Collabcoder: A gpt-powered workflow for collaborative qualitative analysis.

---

Shivam Garg, Dimitris Tsipras. Percy Liang, and Gre- gory Valiant. 2022. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598.
Peter Mayer, Collins W. Munyendo. Michelle L. Mazurek, and Adam J. Aviv. 2022. Why users (don't) 1) use password managers at large educational institution. In 3/st USENIX Security Symposium (USENIX Security 22), pages 1849-1866, Boston, MA. USENIX Association.
Robert Gauthier and James R Wallace. 2022. The computational thematic analysis tooliit. Proceed- ings of the ACM on Human- Comman-C Interaction, 6(GROUP):1155.
Sewon Min, Xinxi Lyu. Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in- in-context learning work? In EMNLP.
Simret Araya Gebreegziabher, Zheng Zhang, Xiaohang Tang, Yihao Meng, Elena L Glassman, and Toby Jia- Jun Li. 2023. Patat: Human-ai collaborative qualita- tive coding with explainable interactive rule synthe- sis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-19.
Oladapo Oyebode, Chinenye Ndulue, Ashfaq Adib, Di- nesh Mulchandami, Banuchitra Suruliraj, Fidelia An- ulika Orji, Christine Chambers, Sandra Meier, and Rita Orji. 2021. Health, psychosocial, and social issues emanating from the covid-19 19 pandemic based on social media comments: Text mining and thematic analysis approach. JMIR Med Inform. 9(4):22233
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms growd-worken for text- annotation tasks.
Stefano De Paoli. 2023. Can large language mod- els emulate an inductive thematic analysis of semi- structured interviews? an exploration and provoca- tion on the limits of the approach and the model.
Timothy C Guetterman, Tammy Chang, Melissa De- Jonckheere, Tanmay Basu, Elizabeth Scruggs, and VG Vinod Vydiswaran. 2018. Augmenting qualita- tive text analysis with natural language processing: methodologieal study. Journal of Medical Internet Research, 20.3).
K Andrew R Richards and Michael A Hemphill. 2018. A practical guide to collaborative qualitative data analysis. Journal of Teaching in Physical education, 37(2):22--211
Tim Rietz. Peyman Toreini, and Alexander Maedche. 2020. Cody: An interactive machine learning sys- tem for qualitative coding. In Adjunct Proceedings of the 33rd Annual ACM Symposium on User Inter- face Software and Technology, UIST '20 Adjunet, page 90-92, New York, NY, USA, Association for Computing Machinery.
Hamed Jelodar, Yongli Wang. Rita Orji, and Shucheng Huang. 2020. Deep sentiment classification and topic discovery on novel coronavirus covid-19 online discussions: Nlp using lstm recurrent neural network approach. IEEE Journal of Biomedical and Health Informatics, 24(10):2733-274)
Jialun Aaron Jiang. Kandrea Wade, Casey Fiesler, and Jed R Brubaker. 2021. Supporting serendipity: Op- portunities and challenges for human-ai collaboration in qualitative analysis. Proceedings of the ACM on Human- Coman-Comppter Interaction, 5(CSCW1):1-23.
Katie Rose M Sanfilippo, Neta Spiro, Miguel Molina- Solama, and Alexandra Lamont. 2020. Do the shuffle: Exploring reasons for music listening through shuf- fied play. PLoS One, 15(2):00228457
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Iliè, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 2tt:::2222111100100
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay. Hyung Won Chung. Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thouu can solve them.
Klaus Krippendorff. 2011. Agreement and information in the reliability of coding. Communication Methods and Measures, 5(2):93-112.
William Leeson, Adam Resnick, Daniel Alexander, and John Rovers. 2019. Natural language processing (nlp) in qualitative public health research: a proof of concept study. International Journal of Qualitative Methods, 18: 16044099 19887021.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li. Hongrae Lec. Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin. James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Marc
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wa, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprini 2211090

---

Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Weier-Hellsterny Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm. Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. Lamda: Language models for dialog appli- cations.
lect free-text responses (Wertz, 2011; Richards and Hemphill, 2018), has been widely used identify patterns of meanings and gain insight from partici- pants via analysis of qualitative data. One common method for analyzing qualitative data is thematic analysis (Braun and Clarke, 2006, 2012, 2019). For reliable coding, at least two human coders must 1) become familiar with the data, 2) generate codebook, and 3) iteratively code the data until acceptable inter-rater agreement level is reached. Thus TA is labor-intensive and time-consuming.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint
B Model and Parameters
We used OpenAI's gpt-3. 3.-tturb- LLM as GPT outperforms other LLMs on various NLP tasks (Liang et al., 2022). Second, although GPT-4 is smarter than and superior to GPT-3.5, and the input context size of GPT-4 32,7668 for ppt-4-32k) is longer than that for GPT-3.5 (16,384 for gpt-3. 5-turbo-16k), there are more con- straints on GPT-4. For instance, the tokens per minute (TPM) limit for GPT-4 is 40,000, whereas that for GPT-3.5 is 180,000 3g 5-turboo--111 6k). Also, GPT-4 is still in beta, not everyone can ac- cess it. Finally, GPT-3.5 remains powerful LLM compared to others, and it is cheaper.
arXiv:2302.1399 13971.
Mojtaba Vaismoradi, Jacqueline Jones, Hannele Tu- runen, and Sherrill Snelgrove. 2016. Theme devel- opment in qualitative content analysis and thematic analysis.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain thought prompt- ing clicits reasoming in large language models. In Advances in Neural Information Processing Systems.
Frederick Joseph Wertz. 2011. Five ways of doing qualitative analysis: Phenomenological psychology. grounded theory, discourse analysis, narrative re- search, and intuitive inquiry. Guilford Press.
We follow OpenAl's default parameter settings except the temperature, which we set to ensure our study is reproducible. indicates low output randomness: hence the model is less creative.
Zhibiao Wu and Martha Palmer. 1994, Verbs semantics and lexical selection. In Proceedings of the 32nd Annual Meeting Association for Computational Linguistics, ACI. '94, page 133-138, USA. Associa- tion for Computational Linguistics.
C Datasets
Ziang Xiao, Xingdi Yuan, Q Vera Liao, Rania Ab- delghani, and Pierre Yves Oudeyer. 2023. Support- ing qualitative analysís with large language models: Combining codebook with GPT-3 for deductive cod- ing. In 28th International Conference Intelligent User Interfaces. ACM.
Music Shuffle (MS) This dataset facilitates re- search on salient aspects when people listen to mu- sic their personal devices. The survey includes four open-ended questions, and comprises 397 par- ticipants. The authors combined all the responses of the four questions for TA.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Tzhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR).
Password Manager (PM) This is record of password manager usage and general password habits at George Washington University (GWU). The survey includes eight open-ended questions, comprising 277 participants from GWU's faculty, staff, and student body. In the original study, one coder coded all the responses and developed the codebook. Later, another coder coded 20% of the responses and calculated Cohen's Our selected survey questions has 280 responses (n 280).
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence?
A Qualitative Research and Thematic Analysis
Qualitative research, which uses techniques such as interviews and open-ended survey questions to col-
"https://platforr opw..i.eenccco.m cttp:lcccccccc//cccca/// dowi//////////////////ooeeesseee rate- www.-iimtt/vverieee

---

D Prompts
Prompt: Initial Codes Generation
Task:
The goal of study)
are examples for how generate codes. For each example, you will see one response with the codes step by step.
These responses are the answer the question: |Survey Question].
Each generated code have the format:
quote refers to
'mentions
definition code Therefore, we got
code:
Code
Exemplars:
{4 8 exemplars)
Prompt 1: The prompt for initial codes generation.
Prompt: Code Grouping
Here is the survey question: (Survey Question]
IIiitial Codes)
Please organize the codes into themes in ISON format. Ensure that each code belongs only one theme. Assign name each theme.
If there any duplicate codes, please merge them into single entry.
The expected output format should follow this structure: <Name of the themes; <List of codes and their definition belonging the theme>
Prompt 2: The prompt for grouping the codes.
Prompt:
Code Ref ieementnt
Here is your version.
[Themes proposed by the Machine Coder
Here is the revised version.
[Revised Themes by the Human Coder [Actions and Reasons by the Human Coder
What do you think?
Please generate the revised themes.
Please list the parts with which agree and disagree and the reason in ISON.
Prompt 3: The prompt for discussion.


Reviews:
[{"review_id": "bjkvoTQ12h", "reviewer": "Reviewer_Gc6Y", "summary": "", "questions": "1. In reference to Weakness #2, it would be helpful if the authors can share if any experiments were performed to validate whether the relatively lower agreeability of Coder 3 with MC and HC (compared to HC+MC) is indeed indicative of problems of biases in the codebook (biases in the codebook could still lead to \"almost perfect\" inter-coder reliability score for HC+MC if they would have come up during the discussion phase and affected the codebook).", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful and constructive review. We are glad that you find our work \"potential of leveraging LLM in a collaborative approach.\"\n\n### Weakness 1, 2 and Question 1\nWe conduct an analysis on the responses that machine coder (MC), human coder (HC) and Coder 3 coded differently.\n\n\n\n|           | Music Shuffle (MS) | Password Manager (PM) |\n| --------  | -------- | -------- |\n| Ambiguity | 5 | 9 |\n| Granularity | 8 | 7 |\n| Distinction | 9 | 14 |\n| **Total** | **22** | **30** |\n\n**Table1.** Analysis of the mismatched responses\n\n\n\n| | MS (All) | PM(Seen) | PM(Unseen) | PM(All) |\n| -------- | -------- | -------- |-| - |\n| MC+Coder 3 | 0.54/0.56 | 0.47/0.72 | 0.34/0.50 | 0.47/0.62 |\n| HC+Coder 3 | 0.62/0.72 | 0.48/0.69 | 0.38/0.54 | 0.43/0.62 |\n\n**Table2.** The former number is the IAA calculated by code (Level 3), and the latter represents the IAA calculated by theme (Level 2).\n\nTo investigate why the inner-annotator agreement (IAA) dropped significantly in the result coder 3 with machine coder (MC) and human coder (HC), we pull the sample that they coded differently to the codebook. We found three major reasons: **Ambiguity**, **Granularity**, and **Distinction** that lead the low IAA of coder 3 with MC and HC. Table 1 is the analysis of the samples that the three coders (MC, HC, Coder3) coded differently.\n\n**Ambiguity:**\nAmbiguity indicates that the coders coded the different codes (Level 3) that actually belong to the same theme (Level 2). For instance, “Digital Notes” and “Notes” are both under the theme “Written Records and Notes”. Given a free-text response, “Keep them in a notes tab on my phone”, the coders might code this response as “Digital Notes” or “Notes”. These two codes belong to the same theme and are correct but would drop the IAA if one coder selects “Digital notes” and the other selects “Notes”. We found that among 22 mismatched responses 5 of them in the Music Shuffle (MS) dataset were due to the ambiguity of the codes. In the Password Manager (PM) dataset, 9 out of 30 are categorized as the reason. To further confirm if the ambiguity of the codes (Level 3) influences the IAA, we calculated the IAA based on the theme (Level 2 codes). Table 2 represents the result. The result suggests that all the IAA improved, which indicates that the ambiguity of the codes is one factor leading to the low IAA.\n\n**Granularity:**\nGranularity denotes the granularity of the coding work. We found that HC and MC coded more finely than the coder 3 in 8 out of 22 and 7 out of 30 mismatched responses in MS and PM, respectively. For instance, for the response, \"Sleeping, calm I saved it because I liked the way it sounded, I thought it would be a good song to fall asleep to. I enjoy listening to this song because it sounds sweet, like a cute sweet not cool sweet.\" Both HC and MC coded this response as “Positive”, “Relaxing”, and “sleep-inducing”, where coder 3 missed the “Positive”.\n\n**Distinction:**\nThe other mismatched responses are due to the coders assigning different codes for a given response. This could be a result of their varied familiarity with the responses and the codebook, leading them to interpret the response or code differently. For instance, coder 3 stated that sometimes it is hard to find a code that fits the response. As a result, coder 3 can only identify the most appropriate code for the response.\n\nOverall, we identified the three main factors that led the IAA drop significantly for coder 3 with MC and HC. However, we argued that thematic analysis is inherently subjective and biased since the coders must align to the research questions when making the codebooks [1,2]. They generated the codebook based on their expertise in the topic of the responses and the research goal. Moreover, **such subjective nature is not specific to our work. Instead, it is common for any thematic analysis.** It is arguable whether bias is a drawback for thematic analysis since the goal is to provide a way to understand the collected free-text data. For instance, even though the codebooks in our study are biased, they capture the semantic meaning of the data to the research questions, and the cosine similarity between our codebooks and the codebooks from the dataset is high (0.8864 and 0.8895). The iterative nature of codebook generation leads the codebook to be less biased.  As we mentioned in the paper, a discrepancy discussion process might be a solution to improve the IAA (lines 269-280).\n\n[1] Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101.  \n[2] Vaismoradi, M., Jones, J., Turunen, H., & Snelgrove, S. (2016). Theme development in qualitative content analysis and thematic analysis. \n\n### Weakness 3\nWe acknowledge the concern of carbon footprint and will add this to the limitation section.\n\nThanks again for pointing out the concerns and suggestions. We will revise the paper based on your comments and add the codebook qualitative analysis and discussion to the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The work **explores the potential of leveraging LLM in a collaborative approach** along with a human coder (HC) to prepare a codebook with mutual consensus that can then be used effectively by the same human coder (HC) or another human (Coder 3) to perform Thematic Analysis (TA) with decent agreeability. In the current landscape of LLM applications, this contributes insights into LLM applications for an NLP task that involves the processing and categorical analysis of free-text survey responses.\n\n2. Empirical data shows that the codebook prepared according to the proposal is **at least effective to the extent that the human coder (HC) is in \"almost perfect\" agreement with the final code generated by the LLM (MC)**.\n\n3. The use of **(a) in-context learning with exemplars in prompts, and (b) partial responses for codebook development in the PM dataset**, would provide another reference of their efficacy in the literature in relation to useful techniques for prompt engineering while using LLMs.", "reasons_to_reject": "1. The work has only been evaluated with one LLM and two public datasets which makes it **difficult to claim that the presented results alone support the overall idea** that LLMs are effective in the Thematic Analysis (TA) task. Furthermore, the **agreeability of Coder 3 with both MC and HC is significantly lower** in the case of one of these datasets (PM), compared to the Gold reference, which further leads to the question of LLM applicability in this task remaining short of substantiated.\n\n2. The \"almost perfect\" agreement between HC and MC is **perhaps indicative of the biases present in the HC to have impacted the codebook** as the LLM (MC) may have been more inclined to accept the rationale provided by the HC during the discussion phase. With two human coders, this might have led to more to-and-fro discussions to resolve differences of judgment but the other human coder is likely to have been more resilient to biases of the peer in impacting the codebook. The drop in Cohen's kappa when Coder 3 is involved also supports this possibility, and no experiments or discussion is included in the manuscript to rule this out.\n\n3. (Relatively minor concern) In terms of a paper involving LLM, where the advantage is mainly in terms of cost (labor + time) reduction, a holistic approach is warranted where the actual costs of performing inference in terms of **carbon footprint** are also taken into account.", "questions_to_authors": "1. In reference to Weakness #2, it would be helpful if the authors can share if any experiments were performed to validate whether the relatively lower agreeability of Coder 3 with MC and HC (compared to HC+MC) is indeed indicative of problems of biases in the codebook (biases in the codebook could still lead to \"almost perfect\" inter-coder reliability score for HC+MC if they would have come up during the discussion phase and affected the codebook).", "paper_topic_and_main_contributions": "The paper explores the potential of using LLM as a collaborative agent along with a singular human agent to perform Thematic Analysis (TA) task on free-text survey responses. Previous approaches typically involve two or more human expert annotators who identify categorical themes from responses and discuss them to reach a consensus. Leveraging LLM has an obvious advantage in terms of reduction in labor and time. In the proposed approach, the human coder (HC) first identifies exemplars from the survey response data. Then, prompts are created by including the survey question, quote of the response, and identified code (along with definition or rationale) in a chain-of-thought natural language text. Guided by these prompts, the LLM or machine coder (MC) generates initial codes for the rest of the data provided to it. This is followed by a discussion phase between the HC and MC and upon agreement, a codebook is generated. This codebook is then used by HC, MC, and another human coder (Coder 3), to perform the final coding. The inter-coder reliability is evaluated using Cohen's kappa.\n\nThe work is evaluated using one LLM, specifically gpt-3.5-turbo-16k, and two public datasets, Music Shuffle (MS) and Password Manager (PM). Empirical results in Table 1, interpreted according to Landis and Koch (1977), depict an \"almost perfect\" (0.81-1) agreement between HC+MC in all cases (ignoring the near-miss of PM-Seen). The agreeability between Coder 3 and HC is \"moderate\" or \"substantial\" for MS and is similar to that in the Gold reference. However, the agreeability of Coder 3 with both HC and MC is categorically lower (\"fair\" or \"moderate\").\n\n---\n\n**[Update after rebuttal, date: 03-Sep-2023]**\nPlease refer to my official comment in reply to the authors' response *(no other change is being made to the text of the original review)*. As mentioned in the comment, I am increasing the excitement score but retaining the soundness score as it is.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "No ethics concern."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n1. Line 222: \"has\" -> \"has been\"\n2. Line 351: \"can used\" --> \"used\" or \"can use\" (both are suitable in the context of the sentence)\n3. Line 354: \"prposed\" -> \"proposed\"\n\n\nPresentation:\n1. The cosine similarity scores and their discussion are easy to miss (lines 261-263 and 284-288). Furthermore, **I could not locate the actual score for cosine similarity for the MS dataset (lines 261-263)**."}}, {"review_id": "DSQ0GixSks", "reviewer": "Reviewer_rocS", "summary": "", "questions": "A. As mentioned above, a qualitative analysis of codebook generation and coding of free-text responses using this approach would strengthen the paper. Is there such an example you can provide? \nB. Have you considered comparing your LLM-based framework to other language model-based methods (fine-tuned BERT for example)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions for our work. We are delighted that you acknowledge the contributions of our work to qualitative research.\n\n### Weakness 1, 3, and Question A\nWe conduct an analysis on the responses that machine coder (MC), human coder (HC) and Coder 3 coded differently.\n\n\n\n|           | Music Shuffle (MS) | Password Manager (PM) |\n| --------  | -------- | -------- |\n| Ambiguity | 5 | 9 |\n| Granularity | 8 | 7 |\n| Distinction | 9 | 14 |\n| **Total** | **22** | **30** |\n\n**Table1.** Analysis of the mismatched responses\n\n\n\n| | MS (All) | PM(Seen) | PM(Unseen) | PM(All) |\n| -------- | -------- | -------- |-| - |\n| MC+Coder 3 | 0.54/0.56 | 0.47/0.72 | 0.34/0.50 | 0.47/0.62 |\n| HC+Coder 3 | 0.62/0.72 | 0.48/0.69 | 0.38/0.54 | 0.43/0.62 |\n\n**Table2.** The former number is the IAA calculated by code (Level 3), and the letter represents the IAA calculated by theme (Level 2).\n\nTo investigate why the inner-annotator agreement (IAA) dropped significantly in the result coder 3 with machine coder (MC) and human coder (HC), we pull the sample that they coded differently and the codebook. We found three major reasons: **Ambiguity**, **Granularity**, and **Distinction** that lead the low IAA of coder 3 with MC and HC. Table 1 is the analysis of the samples that the three coders (MC, HC, Coder3) coded differently.\n\n**Ambiguity:**\nAmbiguity indicates that the coders coded the different codes (Level 3) that actually belong to the same theme (Level 2). For instance, “Digital Notes” and “Notes” are both under the theme “Written Records and Notes”. Given a free-text response, “Keep them in a notes tab on my phone”, the coders might code this response as “Digital Notes” or “Notes”. These two codes belong to the same theme and are correct but would drop the IAA if one coder selects “Digital notes” and the other selects “Notes”. We found that among 22 mismatched responses 5 of them in the Music Shuffle (MS) dataset were due to the ambiguity of the codes. In the Password Manager (PM) dataset, 9 out of 30 are categorized as the reason. To further confirm if the ambiguity of the codes (Level 3) influences the IAA, we calculated the IAA based on the theme (Level 2 codes). Table 2 represents the result. The result suggests that all the IAA improved, which indicates that the ambiguity of the codes is one factor leading to the low IAA.\n\n**Granularity:**\nGranularity denotes the granularity of the coding work. We found that HC and MC coded more finely than the coder 3 in 8 out of 22 and 7 out of 30 mismatched responses in MS and PM, respectively. For instance, for the response, \"Sleeping, calm I saved it because I liked the way it sounded, I thought it would be a good song to fall asleep to. I enjoy listening to this song because it sounds sweet, like a cute sweet not cool sweet.\" Both HC and MC coded this response as “Positive”, “Relaxing”, and “sleep-inducing”, where coder 3 missed the “Positive”.\n\n**Distinction:**\nThe other mismatched responses are due to the coders assigning different codes for a given response. This could be a result of their varied familiarity with the responses and the codebook, leading them to interpret the response or code differently. For instance, coder 3 stated that sometimes it is hard to find a code that fits the response. As a result, coder 3 can only identify the most appropriate code for the response.\n\nOverall, we identified the three main factors that led the IAA drop significantly for coder 3 with MC and HC. However, we argued that thematic analysis is inherently subjective and biased since the coders must align to the research questions when making the codebooks [1,2]. They generated the codebook based on their expertise in the topic of the responses and the research goal. Moreover, **such subjective nature is not specific to our work. Instead, it is common for any thematic analysis.** It is arguable whether bias is a drawback for thematic analysis since the goal is to provide a way to understand the collected free-text data. For instance, even though the codebooks in our study are biased, they capture the semantic meaning of the data to the research questions, and the cosine similarity between our codebooks and the codebooks from the dataset is high (0.8864 and 0.8895). The iterative nature of codebook generation leads the codebook to be less biased.  As we mentioned in the paper, a discrepancy discussion process might be a solution to improve the IAA (lines 269-280).\n\n[1] Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101.  \n[2] Vaismoradi, M., Jones, J., Turunen, H., & Snelgrove, S. (2016). Theme development in qualitative content analysis and thematic analysis. \n\n\n### Weakness 2\nIn this work, our focus was on exploring the potential way to integrate humans and LLMs for conducting thematic analysis. As a result, the novelty of this study lies primarily in the framework itself.\n\n### Question B\nWe have considered adopting the same workflow on other language models, such as Llama2. We leave this as future work.\n\nThanks again for your helpful suggestions. We will revise the paper based on your comments and add the codebook qualitative analysis and discussion to the revised version. For the presentation issues, such as the acronym QR, we will address them in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Strong problem motivation and problem formulation in the context of recent state-of-the-art literature.\n2. Clear and concise presentation of the problem and solution\n3. Work is novel, and the application of this framework has significant downstream potential for qualitative research applications.\n4. Again, the idea and domain are very novel, the results and the analysis provide a good addition to the LLM theme.\n5. Result is reproducible easily, and seems to be good.", "reasons_to_reject": "1. Minor - I think a much more comprehensive and data-intensive analysis would improve this paper significantly but since it is a short paper this isn't a strong negative against what has been done by the authors. \n2. I am unsure about the technical novelty of the approach - the paper appears to be simply doing prompt engineering on ChatGPT to meet an end goal. I like the domain and the framework but the experimental results, while positive, show the strength of ChatGPT (LLMs) in performing qualitative analysis but not the author's claim of reducing human burdens of performing TA.\n3. Minor - Need to have at least one qualitative analysis of how the machine coder and human coder are selecting similar or different \"codes\" needed for completeness.", "questions_to_authors": "A. As mentioned above, a qualitative analysis of codebook generation and coding of free-text responses using this approach would strengthen the paper. Is there such an example you can provide? \nB. Have you considered comparing your LLM-based framework to other language model-based methods (fine-tuned BERT for example)", "paper_topic_and_main_contributions": "This paper performs an exploratory study on the effectiveness of LLMs in performing thematic analysis on qualitative data. They aim to alleviate the labor-intensive and time-consuming nature of qualitative analysis by introducing their LLM-based framework to work in conjunction with humans to extract a meaningful understanding of qualitative data.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Minor - Presentation of Table 1 should happen on the same page it is first talked about. Same with Figure 1 (because we see Figure 2 before the author even gets to describing Figure 1).\n2. Acronym QR is not defined in the paper."}}, {"review_id": "ytQ1DjcdiH", "reviewer": "Reviewer_Mnmq", "summary": "", "questions": "1) What happens if both coders are machines?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive review. In the following, your concerns are addressed point-by-point.\n\n### Weakness 2\nWe evaluated the effectiveness of the idea by studying two different datasets. The result, Cohen's Kappa, for the two datasets, 0.87 and 0.81, respectively, can prove the idea is actionable. Moreover, we believe that the results from the two different datasets can demonstrate the effectiveness of the framework better than results from one dataset but many similar questions. That is, it is nice to have more experiments. However, all eight questions in PM are of the same type, i.e., qualitative responses to open-ended questions. Further, we selected that specific question to study for two main reasons. First, as we mentioned in the paper, that question is more general (*\"Please describe how you manage your passwords across accounts \"*) compared to other questions, such as *\"Why did you use that creation strategy for your most recent GW account password?\"* Second, the dataset's author stated that the response quality of the first question is the best, and the question we studied is the first question of the dataset. As a result, the responses to the question we selected to study in PM have the best quality compared to other questions. Due to limited resources and pages, we only selected one out of eight questions in PM. \n\n### Weakness 3\n\n|  | Music Shuffle | Password Manager |\n| -------- | -------- | -------- |\n| Accuracy    | 0.84 | 0.83 |\n| Recall | 0.72 | 0.87 |\n\nThe table indicates the accuracy and recall of the codes extracted by our framework and the codes extracted by the dataset’s authors. We will add them to the revised version.\n\n### Weakness 4\nWe conduct an analysis on the responses that machine coder (MC), human coder (HC) and Coder 3 coded differently.\n\n\n\n|           | Music Shuffle (MS) | Password Manager (PM) |\n| --------  | -------- | -------- |\n| Ambiguity | 5 | 9 |\n| Granularity | 8 | 7 |\n| Distinction | 9 | 14 |\n| **Total** | **22** | **30** |\n\n**Table1.** Analysis of the mismatched responses\n\n\n\n| | MS (All) | PM(Seen) | PM(Unseen) | PM(All) |\n| -------- | -------- | -------- |-| - |\n| MC+Coder 3 | 0.54/0.56 | 0.47/0.72 | 0.34/0.50 | 0.47/0.62 |\n| HC+Coder 3 | 0.62/0.72 | 0.48/0.69 | 0.38/0.54 | 0.43/0.62 |\n\n**Table2.** The former number is the IAA calculated by code (Level 3), and the letter represents the IAA calculated by theme (Level 2).\n\nTo investigate why the inner-annotator agreement (IAA) dropped significantly in the result coder 3 with machine coder (MC) and human coder (HC), we pull the sample that they coded differently and the codebook. We found three major reasons: **Ambiguity**, **Granularity**, and **Distinction** that lead the low IAA of coder 3 with MC and HC. Table 1 is the analysis of the samples that the three coders (MC, HC, Coder3) coded differently.\n\n**Ambiguity:**\nAmbiguity indicates that the coders coded the different codes (Level 3) that actually belong to the same theme (Level 2). For instance, “Digital Notes” and “Notes” are both under the theme “Written Records and Notes”. Given a free-text response, “Keep them in a notes tab on my phone”, the coders might code this response as “Digital Notes” or “Notes”. These two codes belong to the same theme and are correct but would drop the IAA if one coder selects “Digital notes” and the other selects “Notes”. We found that among 22 mismatched responses 5 of them in the Music Shuffle (MS) dataset were due to the ambiguity of the codes. In the Password Manager (PM) dataset, 9 out of 30 are categorized as the reason. To further confirm if the ambiguity of the codes (Level 3) influences the IAA, we calculated the IAA based on the theme (Level 2 codes). Table 2 represents the result. The result suggests that all the IAA improved, which indicates that the ambiguity of the codes is one factor leading to the low IAA.\n\n**Granularity:**\nGranularity denotes the granularity of the coding work. We found that HC and MC coded more finely than the coder 3 in 8 out of 22 and 7 out of 30 mismatched responses in MS and PM, respectively. For instance, for the response, \"Sleeping, calm I saved it because I liked the way it sounded, I thought it would be a good song to fall asleep to. I enjoy listening to this song because it sounds sweet, like a cute sweet not cool sweet.\" Both HC and MC coded this response as “Positive”, “Relaxing”, and “sleep-inducing”, where coder 3 missed the “Positive”.\n\n**Distinction:**\nThe other mismatched responses are due to the coders assigning different codes for a given response. This could be a result of their varied familiarity with the responses and the codebook, leading them to interpret the response or code differently. For instance, coder 3 stated that sometimes it is hard to find a code that fits the response. As a result, coder 3 can only identify the most appropriate code for the response.\n\nOverall, we identified the three main factors that led the IAA drop significantly for coder 3 with MC and HC. However, we argued that thematic analysis is inherently subjective and biased since the coders must align to the research questions when making the codebooks [1,2]. They generated the codebook based on their expertise in the topic of the responses and the research goal. Moreover, **such subjective nature is not specific to our work. Instead, it is common for any thematic analysis.** It is arguable whether bias is a drawback for thematic analysis since the goal is to provide a way to understand the collected free-text data. For instance, even though the codebooks in our study are biased, they capture the semantic meaning of the data to the research questions, and the cosine similarity between our codebooks and the codebooks from the dataset is high (0.8864 and 0.8895). The iterative nature of codebook generation leads the codebook to be less biased.  As we mentioned in the paper, a discrepancy discussion process might be a solution to improve the IAA (lines 269-280).\n\n[1] Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101.  \n[2] Vaismoradi, M., Jones, J., Turunen, H., & Snelgrove, S. (2016). Theme development in qualitative content analysis and thematic analysis. \n\n\n### Question 1\nIn this work, we focus on the idea of a human-LLM collaboration framework for thematic analysis. We also believe that leveraging two MC, i.e., two different LLMs, is a valuable direction to explore. We leave this as the future direction.\n\n### Typo, Presentation, and Weakness 1\nWe will address the presentation, such as QR, code, and codebook, in the revised version.\n\nThanks again for your helpful suggestions. We will revise the paper based on your comments and add the codebook qualitative analysis and discussion to the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The human coder + machine coder has higher IAA than the gold standard.\n2) The framework facilitates discussesion between a human and LLM, which can reduce errors caused by hallucination.", "reasons_to_reject": "1) Some concepts like 'QR', 'code' are not clearly explained to the reader. \n2) only one out of the 8 questions is studied on the password management data. \n3)  cosine similarity between generated codes and gold standard codes is reported. However, it is not a sufficient metric. Other metrics like accuracy, recall should be reported.\n4) The analysis of the extracted codes and the missed codes is not provided.", "questions_to_authors": "1) What happens if both coders are machines?", "paper_topic_and_main_contributions": "The paper proposes A human-in-the-loop framework for thematic Analysis (TA). Traditionally, TA has been done by 2 humans. The paper proposes replacing one of the human's with ChatGPT. The framework consists of 4 steps : Data familiarization, initial code generation, refinement and final coding. The human coder + machine coder team gets higher Inter-annotator-agreement than the gold standard on 2 datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Full form of QR should be provided. \nmeaning of 'code' and 'codebook' should be made clear early in the paper. \nCaptions of figures should be strengthened"}, "extra_text": {}}, {"review_id": "a1gZFiiE9d", "reviewer": "Reviewer_WzFN", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive review. In the following, your concerns are addressed point-by-point.\n\n### Weakness 1,2 NLP Venue\nThis work is an interdisciplinary research, which involves qualitative research and human-centered NLP. We proposed a human-LLM collaborative framework for thematic analysis (TA). TA involves NLP techniques, such as extracting codes from the responses, grouping the codes into themes, and labeling the responses with codes. Further, LLM recently attracted peak attention in the NLP community. For instance, the theme track of EMNLP this year is Large Language Models and the Future of NLP. As a result, we believe that it is appropriate to submit this work to EMNLP.\n\n### Weakness 3 Evaluation\nWe evaluated the effectiveness of the idea by studying two different datasets with three coders: MC, HC, and coder 3. The good agreement can prove the idea is actionable. It is for sure nice to have more coders. However, we only evaluated with three coders due to limited resources and pages. We believe that the results from the two different datasets can demonstrate the effectiveness of the framework.\n\n### Weakness 4 Dataset Description\nWe don't have enough space to put detailed information about the dataset due to the page limit. We will add more descriptions about the dataset to address this concern in our next version.\n\n### Missing Reference\nWe conducted initial code generation because we followed the thematic analysis procedure, as we mentioned in the paper. Further, we aim to extract the codes that can answer the research question of the survey, and thus, we did not simply use an unsupervised topic modeling method to categorize the codes. Even though an unsupervised topic modeling approach can extract topic themes that represent the responses, these responses may not be able to answer the research question. We can expand our related work to include this aspect when space allows.\n\n### Presentation- Gold Label\nAs we mentioned in Sec. 4.1 (Line 197-200), the \"Gold\" indicated the IAA reported by the authors of the dataset. The authors of the two datasets coded the responses based on their developed codebook. On the other hand, we coded the responses based on the codebook developed by MC and HC. As a result, we cannot calculate the agreement between the gold standard (coded by the authors of the dataset) and ours (HC, MC, and coder 3). \n\n### Presentation- Cohen's Kappa\nThe authors of the two datasets reported Cohen's Kappa. To compare the result, we calculated Cohen's Kappa as the Inner-Annotator-Agreement.\n\n### Presentation- More than one code\nFollowing the same procedure as the authors of the dataset, one response can have more than one code.\n\nThanks again for your helpful suggestions. We will revise the paper based on your comments and add the codebook qualitative analysis and discussion to the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Paper provides a framework for using prompts to teach LLMs to code free-text answers where the labels used are not already known or pre-defined.\n\nPaper tested on two datasets and found good agreement in coding between the human who prompted the LLM to perform the coding.", "reasons_to_reject": "The paper may be of more interest in a venue where many participants require coding of responses to semi-structured interviews.\n\nThe paper topic is not about empirical methods for NLP. Rather, it describes an application of how prompting an LLM can be used for a task that is common in disciplines where structured interviews need to be analyzed and the set of labels to be used for labeling the data needs to be determined. \n\nFor evaluating generalization, I would like to see several more coders in the evaluation, more than the current one who worked with the LLM and one outside coder. Cohen’s kappa for Coder3 was much lower when matched to the LLM or human. One interpretation is that the LLM did a good job learning the codes from the human that trained it, but those codes have discrepancies with Coder3’s interpretation. Perhaps a more appropriate framework would be for humans to define the set of codes to use (possibly using techniques that have been proposed for helping coders to define the codes) and then training a model that included an LLM to perform the coding. That is, have the “discrepancy discussion” (lines 269-280) before using an LLM.\n\nThe evaluation section needs to better describe the dataset characteristics to help understand the difficulty of the task. Although Section 4.2 Setting states the one question used from each dataset that is coded, a description of each dataset is needed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper describes how a human can use prompts to train an LLM to  generate “codes” for response text to questions (e.g., such as from a semi-structured interview as used in Social Science), use prompts to create themes from the identified codes, and then prompt the LLM to code the dataset using the identified codes. Thus, the LLM serves as a “machine coder” that is taught by a “human coder”. The research question was whether the coding quality of the LLM is comparable to a human. The paper compared agreement between the LLM and human coder in their generated codes and found good agreement. However, the agreement with a third coder was not as good.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Why conduct Initial Code Generation instead of using unsupervised topic categorization to identify the themes? \n\nLennon et al., 2021 generates unsupervised topic categories (without LDA, which was mentioned in the submission) to perform thematic analysis:\nLennon, Robert P., et al. \"Developing and testing an automated qualitative assistant (AQUA) to support qualitative analysis.\" Family Medicine and Community Health 9.Suppl 1 (2021).", "Typos_Grammar_Style_and_Presentation_Improvements": "The section 4. Evaluation seems to be incomplete and is hard to read. It may have been cut down from a longer version, since many items are referred to before they are defined. For example:\nLine 200: “the datasets”, no datasets were mentioned earlier, although in lines 133 and 198  they state “our approach using two existing open-access datasets.” \nAt the end of Section 4.2, it’s stated that the “detailed information of the datasets is in Appendix A.4”. A brief description of the of the datasets should be given in the paper itself. How many responses were there to the two selected questions? How many codes did the authors of the dataset define? Was that number known to the coders or the LLM? If the number of codes was unknown, how was the comparison to the “gold standard” performed?\n\nTable 1 results would be clearer if the “Gold” label was replaced by “MC + Gold” to indicate what was compared. I’d also like to see results for “HC + Gold” and “Coder3 + Gold” to see the agreement between the Gold codes against each of the human codes, in addition to against the LLM codes.\n\nIn lines 191-195, Krippendorff is mentioned in the context of inter-annotator agreement, but then it’s stated that Cohen’s kappa was used to compare the results of two coders. Why not use Krippendorff’s alpha?\n\nHow are text passages handled when more than one code should be assigned to the passage?"}}]

Author rebuttal:

Meta-review: Paper Summary:
This paper proposes using large language models (LLMs) as collaborators in a human-in-the-loop framework for thematic analysis of qualitative data. A human coder provides exemplars to teach the LLM to generate codes, followed by joint codebook creation. Experiments show high agreement between human coder and LLM, but lower agreement with a separate human coder.

I will now provide a snapshot of the reviews. Four reviewers were engaged and unfortunately, I'm unable to remove one of the reviewers. I'm leaving this aspect for the SAC to evaluate and decide which one to expunge.  
Summary of Reviews:

Reviewer 1 found the collaborative LLM approach novel. But only one LLM and two datasets make claims preliminary. Lower coder 3 agreement is concerning and prompts bias questions.

Reviewer 2 thought the problem motivation and presentation were strong, and called the application domain very novel. However, more comprehensive experiments are needed. Unsure of true technical novelty vs. prompt engineering.

Reviewer 3 liked the higher agreement than the gold standard, and the discussion reducing hallucination risks. But felt that some concepts were not clearly explained. The reviewer thinks that metrics like accuracy can be more evaluated in the paper and that missed code analysis is lacking.

R4 noted framework enables teaching LLMs new codes, and the human-LLM agreement was good. However, the topic may be better suited for other venues. More coders are needed in the evaluation. The reviewer also believes that the dataset currently lacks details.

In general, I find incorporating LLMs into the qualitative analysis workflow quite interesting and relevant given the application area. I think it's an eye-opener towards using LLMs to beat down annotation costs. Moreover, all four reviewers praise the novelty. However, concerns exist around evaluation methodology and broader impact. The agreement between LLM and untrained human coders is notably lower, raising questions about bias and generalization. More importantly, the majority of the reviewers recommend authors' using more datasets, and coders, and probe the transparency around disagreements. I note that the authors' rebuttal to the reviewers has shed more light on this aspect. The authors provided a more detailed analysis of the discrepancies between the LLM-trained coder and the untrained coder 3. They identify sources like code ambiguity and granularity differences that account for many of the disagreements. By calculating agreement at the theme level, they showed improved results to address reviewers' concerns. Particularly, I note that the newly adopted theme-level calculation justifies the lower coder 3 agreement as initially reported and alleviates bias concerns. The additional accuracy metrics are useful as well. Personally, I agree with the reviewers that the potential is apparent but current validation is limited and not fully answered by the reviewers and I'm hoping that the authors can improve these areas of the paper in the final draft after extending their experiments.
Decision: Accept-Findings