Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding
Authors: Angelina Parfenova, Jürgen Pfeffer
Accepted at: acl25

Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding

Angelina Parfenova1,2 and Jürgen Pfeffer2

1Lucerne University of Applied Sciences and Arts
2Technical University of Munich

Abstract

Inductive coding traditionally relies on labor-intensive human efforts, who are prone to inconsistencies and individual biases. Although large language models (LLMs) offer promising automation capabilities, their standalone use often results in inconsistent outputs, limiting their reliability. In this work, we propose a framework that combines ensemble methods with code refinement methodology to address these challenges. Our approach integrates multiple smaller LLMs, fine-tuned via Low-Rank Adaptation (LoRA), and employs a moderator-based mechanism to simulate human consensus. To address the limitations of metrics like ROUGE and BERTScore, we introduce a composite evaluation metric that combines code conciseness and contextual similarity. The validity of this metric is confirmed through correlation analysis with human expert ratings. Results demonstrate that smaller ensemble models with refined outputs consistently outperform other ensembles, individual models, and even large-scale LLMs like GPT-4. Our evidence suggests that smaller ensemble models significantly outperform larger standalone language models, pointing out the risk of relying solely on a single large model for qualitative analysis.

1 Introduction

Inductive coding is a fundamental method in social science research, enabling the identification, organization, and analysis of patterns and themes within textual data (Creswell, 2016; Saldana, 2016; Braun and Clarke, 2021). At its core, this method extracts key ideas from text, transforming unstructured documents into a list of codes that capture the underlying themes. Traditionally, this process has relied on human coders who, despite their expertise, are prone to inconsistencies, cognitive biases, and the tendency to overcomplicate labels (MacQueen and Guest, 2008; Bumbuc and Hrybyk, 2016; Bernard, 2016; Morse, 2017). Large language models (LLMs) offer an opportunity to automate this process, potentially improving efficiency and standardization (Bommasani et al., 2021). However, ensuring the reliability and interpretability of LLM-generated codes remains a challenge.

Our study addresses two critical gaps in automated inductive coding: (1) the need for structured refinement processes to improve consistency across codes and (2) the limitations of standard evaluation metrics, which fail to capture the nuanced criteria used by human experts, such as label granularity, alignment, and contextual relevance (Lin, 2004; Zhang et al., 2019; Sellam et al., 2020; Fabri et al., 2021a; Parfenova et al., 2024; Mizrahi

![Overview of the proposed ensemble pipeline for inductive coding. The input x is processed by multiple LoRA fine-tuned models (LLM_1, LLM_2, ..., LLM_N), generating candidate outputs (O_1, O_2, ..., O_N). Moderators compare and evaluate these candidates, selecting y_1, y_2, ..., y_N as refined candidates. Refinement of codes consists of comparing the current code with previous ones and assigning the same code based on the chosen similarity threshold.](page_1012_355_388_388.png)

Figure 1: Overview of the proposed ensemble pipeline for inductive coding. The input x is processed by multiple LoRA fine-tuned models (LLM_1, LLM_2, ..., LLM_N), generating candidate outputs (O_1, O_2, ..., O_N). Moderators compare and evaluate these candidates, selecting y_1, y_2, ..., y_N as refined candidates. Refinement of codes consists of comparing the current code with previous ones and assigning the same code based on the chosen similarity threshold.
et al., 2024). We introduce a novel ensemble-based framework that integrates refinement processes inspired by Retrieval-Augmented Generation (RAG) (Lewis et al., 2020), reusing previously generated codes to eliminate their redundancy iteratively.

This study makes three key contributions: (1) we develop a pipeline that integrates multiple instruction-finetuned LLMs with an iterative refinement process to improve coding consistency; (2) we introduce a composite evaluation metric that captures both semantic and structural quality of generated codes; and (3) we demonstrate that smaller LLM ensembles, refined by merging similar codes, outperform standalone large models, achieving closer alignment with human coded benchmarks.

The results indicate that the proposed ensemble approach improves the quality of the generated codes, while the refinement process further enhances their consistency, resulting in outputs that are more closely aligned with the human-coded data. Additionally, we show that ensemble-based methods reduce variability in code assignments, addressing the subjectivity present in individual LLM outputs. These findings provide a strong basis for improving automated qualitative data analysis while maintaining compatibility with human coding standards.

2 Background

Qualitative data analysis (QDA) plays a critical role in social science research (Miller et al., 1990; Creswell, 2016), enabling researchers to identify, categorize and interpret patterns within textual data. Central to this process is the concept of coding, where meaningful segments of text are assigned concise labels, or codes, that capture their core essence. According to Saldana (2016), a code is "often a word or short phrase that symbolically assigns a summative, salient, essence-capturing and/or evocative attribute to a portion of language-based or visual data." In thematic analysis, one of the most widely used methods in QDA, these codes are further grouped into broader categories to reveal hierarchical relationships and underlying themes within the data (Braun and Clarke, 2021). The source material often includes interviews, documents, or other qualitative data formats, and the process involves summarizing the key ideas in each segment and grouping them into overarching themes.

Recent advances in natural language processing have introduced the use of LLMs for automating qualitative coding tasks (Tornberg, 2023; Parfenova et al., 2024; Fischer and Biemann, 2024). However, we noticed 2 critical points that need to be addressed in this domain. (1) To evaluate codes, some papers noted that traditional summarization metrics like BERT and ROUGE are not good enough for this task (Parfenova et al., 2024; Chen et al., 2024), one research specifically created unsupervised metric to assess codes (Chen et al., 2024). (2) The second critical point in this domain is that although individual LLMs demonstrate remarkable performance, their output often varies due to differences in training data, architectures, and model parameters (Bubeck et al., 2023; Touvron et al., 2023b). This variability introduces subjectivity, making a single LLM's coding decisions as inconsistent as those of individual human annotators. To address these challenges, ensemble methods, techniques that combine multiple models, have been increasingly explored for their ability to take advantage of the complementary strengths of different models and improve performance (Sagi and Rokach, 2018; Jiang et al., 2023b).

Ensemble learning is a widely used strategy to improve model performance by combining the strengths of multiple models, often referred to as "weaker models"(Sagi and Rokach, 2018; Aniol et al., 2019). There are two common approaches to ensemble learning: one involves weighting individual models based on their performance, while the other focuses on aggregating diverse outputs to produce a unified result. One example is the Mix-of-Experts (MoE) framework (Cai et al., 2024), which employs specialized sub-models to make predictions and merges their outputs for improved accuracy. Similarly, LLM-Blender (Jiang et al., 2023a) demonstrates the potential of ensembling by combining ranked outputs from multiple models to achieve superior performance in complex natural language generation tasks.

This study builds upon the concept of ensemble methods but diverges from existing approaches by adopting a moderator-based framework. Unlike fusion techniques that combine outputs probabilistically, our approach incorporates a final decision-making model tasked with selecting the best candidate or proposing a novel output. This design reflects the dynamics of human collaboration, where consensus is sometimes driven by a leader or a final arbiter, rather than by averaging or blending
opinions (Engle et al., 2014). Using this moderator model, we aim to mimic the style of the hierarchical decision-making process in human qualitative coding and demonstrate its effectiveness.

3 Dataset

Our experiments used a dataset of 1,000 code-quote pairs compiled from two main sources: social science research studies and the SemEval-2014 Task 4 dataset (Pontiki et al., 2014) for augmentation. The social science data includes 600 examples from studies across three universities, featuring topics such as technology interaction, social values, and cultural experiences. The SemEval dataset contributes an additional 400 examples, consisting of reviews manually coded by qualitative researchers.

Table 1 summarizes the data sources. Each quote was labeled by 3–5 coders who independently annotated it before reaching a consensus to establish the golden standard. This ensured high-quality labels and reduced variability due to individual coder biases. The test set size was set to 100 examples (see Table 2). The dataset was split into training and testing sets without a separate validation set. Hyperparameters were selected based on the training results and evaluated on the test set.

<table>
  <tr>
    <th>N Quotes</th>
    <th>Description</th>
  </tr>
  <tr>
    <td colspan="2"><b>Social Science Studies Data: 600 quotes</b></td>
  </tr>
  <tr><td>78</td><td>Study about interaction with self-tracking devices (interviews)</td></tr>
  <tr><td>22</td><td>Study about life transitions and mobility (interviews)</td></tr>
  <tr><td>82</td><td>Study about interaction with voice assistants (interviews)</td></tr>
  <tr><td>28</td><td>Study about museums and cultural experiences (interviews)</td></tr>
  <tr><td>25</td><td>Study on doctors’ experiences with pregnant women (interviews)</td></tr>
  <tr><td>110</td><td>Study on technology interactions and user feedback (interviews)</td></tr>
  <tr><td>24</td><td>Study on procrastination and budget planning (interviews)</td></tr>
  <tr><td>56</td><td>Study on technology interactions and user feedback (reviews)</td></tr>
  <tr><td>175</td><td>Study about social expectations (interviews)</td></tr>
  <tr>
    <td colspan="2"><b>SemEval 2014; Task 4: 400 quotes</b></td>
  </tr>
  <tr><td>211</td><td>Restaurant reviews</td></tr>
  <tr><td>189</td><td>Laptop reviews</td></tr>
</table>

Table 1: Summary of Data Sources with descriptions.

<table>
  <tr>
    <th>Statistic</th>
    <th>Overall</th>
    <th>Train</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>Total Quotes</td>
    <td>1000</td>
    <td>900</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Social Science Data</td>
    <td>600</td>
    <td>550</td>
    <td>50</td>
  </tr>
  <tr>
    <td>SemEval Data</td>
    <td>400</td>
    <td>350</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Num of Data Sources</td>
    <td>11</td>
    <td>11</td>
    <td>11</td>
  </tr>
  <tr>
    <td>Unique Codes</td>
    <td>680</td>
    <td>624</td>
    <td>54</td>
  </tr>
  <tr>
    <td>Avg. Quote Length</td>
    <td>254.75±274.28</td>
    <td>280.82±80.89</td>
    <td>234.80±101.61</td>
  </tr>
  <tr>
    <td>Avg. Code Length</td>
    <td>19.95±10.43</td>
    <td>20.01±10.70</td>
    <td>19.27±10.53</td>
  </tr>
</table>

Table 2: Summary statistics of the dataset and train/test splits. Subscript refers to standard deviation where applicable.

To evaluate the ensemble models, we used a second distinct test data set comprising 100 user reviews of ChatGPT sourced from Kaggle (Jikadara, 2023). This data set, characterized by user-generated unstructured feedback, differs from the structured data used for training, providing an opportunity to test model performance on unseen, real-world content. Each review was manually annotated by human coders to create a golden standard reference for evaluation. This dataset was chosen specifically for its diversity in style of language, sentiment, and varied quote lengths to verify the ensemble’s ability to handle varied text types.

4 Pipeline

Our proposed pipeline consists of two key stages, as illustrated in Figure 1. First, an input \( x \) is processed independently by three smaller LLMs (7B and 8B parameter sizes). The outputs from these models are then evaluated by a set of \( N \) Moderators (Moderator_1, Moderator_2, ..., Moderator_N), which refine and consolidate the results. Finally, code merging is performed to ensure consistency across similar inputs, producing the optimal output.

4.1 Phase 1: LoRA Finetuning

In the first phase of the study, we evaluated several open-source models: Llama3 (Touvron et al., 2023a), Falcon (Pineda et al., 2023), Mistral (Team, 2023), Vicuna (Li et al., 2023), Gemma (Team, 2024), and TinyLlama (Jiang et al., 2023c), on an open coding task. We employed various approaches, such as zero-shot, few-shot (providing 1 to 5 examples, see Appendix A), and parameter-efficient fine-tuning (Han et al., 2024) using Low-Rank Adaptation (LoRA) (Hu et al., 2021).

Each model (LLM_i) was fine-tuned on the training dataset and generated an output (\( o_i \)) for a given input (\( x \)). The outputs were evaluated based on their semantic similarity and token overlap using BERTScore (Zhang et al., 2019) and ROUGE (Lin, 2004). From this evaluation, the three models with the highest performance were selected for the moderation phase (see Appendix B).

4.2 Phase 2: Moderation and Refinement

The top three outputs \( \{o_1, o_2, o_3\} \) were passed to Moderators using the prompt template shown in (Appendix C), which incorporated previous model suggestions. The Moderators produced modified outputs \( \{y_1, y_2, \ldots, y_N\} \).

To maintain coherence across similar data points, a refinement stage was applied to all generated
<table>
  <tr>
    <th>Model</th>
    <th>Parameters</th>
    <th>Adaptation</th>
    <th>Prompt</th>
    <th colspan="2">BERTScore</th>
    <th colspan="4">ROUGE</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>P</th>
    <th>R</th>
    <th>F1</th>
    <th>L</th>
    <th>1</th>
    <th>2</th>
    <th>L</th>
  </tr>
  <tr>
    <td>Llama3 (instruct)</td>
    <td>8B</td>
    <td>Finetuning</td>
    <td>Summarize the main idea of a sentence.<br>From the perspective of a social scientist.</td>
    <td>0.72</td>
    <td>0.79</td>
    <td>0.75</td>
    <td>0.18</td>
    <td>0.06</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>Falcon (instruct)</td>
    <td>7B</td>
    <td>Finetuning</td>
    <td>summarize the following sentence as you would in thematic coding.</td>
    <td>0.75</td>
    <td>0.79</td>
    <td>0.77</td>
    <td>0.21</td>
    <td>0.09</td>
    <td>0.21</td>
  </tr>
  <tr>
    <td>Mistral (instruct)</td>
    <td>7B</td>
    <td>Finetuning</td>
    <td>Can you tell me what the main idea of this sentence is in just a few words?</td>
    <td>0.74</td>
    <td>0.79</td>
    <td>0.77</td>
    <td>0.25</td>
    <td>0.11</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>Vicuna (instruct)</td>
    <td>7B</td>
    <td>Finetuning</td>
    <td>Summarize the main idea of a sentence.</td>
    <td>0.73</td>
    <td>0.78</td>
    <td>0.76</td>
    <td>0.19</td>
    <td>0.07</td>
    <td>0.18</td>
  </tr>
  <tr>
    <td>Gemma (instruct)</td>
    <td>7B</td>
    <td>Finetuning</td>
    <td>If you were a social scientist doing thematic analysis, what code would you give to this citation?</td>
    <td>0.72</td>
    <td>0.78</td>
    <td>0.75</td>
    <td>0.17</td>
    <td>0.06</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>TinyLlama (chat)</td>
    <td>1.1B</td>
    <td>Few-shot (5 examples)</td>
    <td>Summarize the main idea of a sentence:<br>Here are examples:</td>
    <td>0.77</td>
    <td>0.74</td>
    <td>0.75</td>
    <td>0.18</td>
    <td>0.03</td>
    <td>0.18</td>
  </tr>
</table>

Table 3: Performance of various open-source LLMs on open coding task across different adaptation methods and prompts. This table presents the BERTScore and ROUGE scores for each model, indicating precision (P), recall (R), and F1 scores for BERTScore, along with ROUGE scores (1, 2, L). Models were evaluated under different scenarios, including finetuning and few-shot approaches, with prompts designed to align with thematic analysis. Detailed fine-tuning results are demonstrated in Appendix B.

codes. This process involved code merging, implemented through embedding similarity analysis. Given a new input \( x \), its embedding \( \phi(x) \) was computed using sBERT (Reimers and Gurevych, 2019) and compared against previously assigned code embeddings \( \phi(p_i) \) using cosine similarity. If the similarity score met or exceeded a predefined threshold (\( \operatorname{sim}(\phi(x), \phi(p_i)) \geq \tau \)), the existing code was retained; otherwise, a new code was assigned. After generation, each candidate code was scored post hoc using the composite evaluation metric \( C(y_i) \) described in Section 5. This score was used only during benchmarking to compare outputs across methods. It was not used during model training, generation, or selection in real-world applications. For inference without gold labels, the refinement mechanism alone is applied.

5 Metrics

To evaluate the performance of individual models that will serve as input to the moderator, two metrics were employed to capture both lexical and semantic similarity: ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2019). BERTScore is a metric that computes the similarity between BERT token embeddings of two codes, which helps assess the meaning in the generated output compared to the reference. ROUGE is a lexical similarity measure that calculates the overlap of n-grams (1-unigram overlap, 2-bigram overlap, L-longest common subsequence) between the generated text and the reference text. ROUGE is particularly effective in summarization task (Fabbri et al., 2021b), which is valuable when the exact wording of the output needs to match the reference.

5.1 Composite Score

To systematically assess the performance of ensemble models in inductive coding tasks, we introduce a composite score that integrates four evaluation scores: (1) cosine similarity, which quantifies the semantic alignment between model-generated codes and human-coded references; (2) METEOR score, which captures lexical similarity while accounting for synonyms and stemming variations (Banerjee and Lavie, 2005); (3) code length penalty, which discourages excessively verbose codes; and (4) Jensen-Shannon divergence, computed between token distributions of reference (golden standard) codes and predicted ones by models. Each component is independently normalized to ensure comparability, and the final composite score is computed as the mean of the four metrics, assigning equal weight to each.

Higher cosine similarity and METEOR scores contribute positively to the composite score, reflecting stronger semantic and lexical agreement with human coders. Conversely, longer codes and greater distributional divergence lower the score, penalizing outputs that deviate from the brevity and consistency typically expected in qualitative coding.

5.2 Metric Validation

To assess the validity of the composite score, we conducted a human evaluation study where experts rated the quality of codes generated by anonymized models (Appendix F). Each expert was presented with ten sentences and their corresponding codes, assigning a rating from 1 to 5 based on clarity, conciseness, and relevance to the text. We then computed Spearman’s correlation between the composite score and the average of all expert ratings,
Figure 2: Performance comparison of different models on inductive coding tasks, measured by Composite Scores (bars) and ROUGE-1 Mean (line). Ensemble models with refinement outperform individual models and non-refinement ensembles.

alongside other evaluation metrics.

The analysis reveals a strong correlation between the composite score and human ratings (Spearman: 0.73, p = 0.039), confirming its alignment with expert judgments. ROUGE-1 and ROUGE-L exhibit moderate positive correlations with the composite score (Spearman: 0.51, p = 0.20 and Spearman: 0.44, p = 0.27, respectively), but their statistical insignificance suggests that they may not fully capture the criteria used by human coders. In contrast, BERTScore F1 shows a negligible correlation with the composite score (Spearman: 0.06, p = 0.90), implying that semantic similarity alone is insufficient for assessing inductive coding quality. Additionally, the correlation between the composite score and code length is near zero (Spearman: -0.02, p = 0.96).

Further analysis reveals that BERTScore F1 exhibits a negative correlation with code length, which contradicts the expectation that qualitative codes should be concise (see Table 4). This misalignment suggests that BERTScore may not be an ideal metric for evaluating inductive coding tasks, as it does not adequately penalize redundancy.

6 Results

The ensemble pipeline with codes merging approach yield significant performance improvements, as detailed in Table 6. Ensemble models, particularly refined variants, consistently outperform individual models across all metrics except BERTScore, which remains stable (F1=0.83-0.86) due to its semantic focus. Among standalone models, GPT-4 achieves the highest composite score (0.44), while Llama3.3 70B leads in ROUGE-1 performance (0.12).

Ensembles perform better, with the Mixtral8x7B ensemble achieving a significant composite score improvement (0.33→0.91) over its individual counterpart. Llama3.3 70B shows a similar trend, improving from 0.38 to 0.50 through ensemble integration. Merging of codes provides additional benefits, elevating Mixtral8x7B to 0.99 (+9%) and Llama3.3 70B to 0.74 (+48% from individual baseline). These improvements demonstrate both the value of model diversity and postprocessing effectiveness in enhancing coding consistency.

Figure 3: Comparison of LLM performance across three evaluation metrics: Coverage (measuring how well a model captures the meaning of the reference codes), Novelty (assessing the degree of new information introduced by the model), and Divergence (quantifying distributional shifts from the reference). Ensemble models with code merging tend to balance coverage and novelty better than individual models.

improvement (0.33→0.91) over its individual counterpart. Llama3.3 70B shows a similar trend, improving from 0.38 to 0.50 through ensemble integration. Merging of codes provides additional benefits, elevating Mixtral8x7B to 0.99 (+9%) and Llama3.3 70B to 0.74 (+48% from individual baseline). These improvements demonstrate both the value of model diversity and postprocessing effectiveness in enhancing coding consistency.

6.1 Composite Score Analysis

The composite score proved to be effective in capturing semantic, lexical, and structural alignment. Refined ensembles consistently outperformed both individual models and ensembles. Although ROUGE-1 primarily captures lexical similarity, it aligned well with the composite score in the ranking model performance. Also, standalone large models like GPT-4 exhibited lower ROUGE-1 scores compared to ensembles, reflecting challenges in lexical precision.

6.2 Coverage, Novelty, and Divergence Analysis

Our evaluation also incorporates the metrics proposed by Chen et al. (2024), which offer greater sensitivity compared to traditional similarity-based measures like BERTScore (see Figure 3). These metrics reveal nuanced distinctions in model performance, though the observed differences were less sensitive than anticipated, except for Jensen-Shannon divergence—a measure we integrate into our Composite Score. Notably, the results suggest that in certain cases, LLM-generated codes may achieve performance levels comparable to or ex-
<table>
  <tr>
    <th>Metric</th>
    <th>Composite Score</th>
    <th>BERTScore F1</th>
    <th>ROUGE-1</th>
    <th>ROUGE-L</th>
    <th>Code Length</th>
    <th>Human Rating</th>
  </tr>
  <tr>
    <td>Composite Score</td>
    <td>1.00</td>
    <td>-0.06</td>
    <td>0.51</td>
    <td>0.44</td>
    <td>-0.02</td>
    <td><span style="background-color:green">0.73*</span></td>
  </tr>
  <tr>
    <td>BERTScore F1</td>
    <td>-0.06</td>
    <td>1.00</td>
    <td>-0.09</td>
    <td>-0.11</td>
    <td><span style="background-color:green">-0.65*</span></td>
    <td>-0.14</td>
  </tr>
  <tr>
    <td>ROUGE-1</td>
    <td>0.51</td>
    <td>-0.09</td>
    <td>1.00</td>
    <td><span style="background-color:green">0.99*</span></td>
    <td>-0.35</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>0.44</td>
    <td>-0.11</td>
    <td><span style="background-color:green">0.99*</span></td>
    <td>1.00</td>
    <td>-0.32</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Code Length</td>
    <td>-0.02</td>
    <td><span style="background-color:green">-0.65*</span></td>
    <td>-0.35</td>
    <td>-0.32</td>
    <td>1.00</td>
    <td>-0.41</td>
  </tr>
  <tr>
    <td>Human Rating</td>
    <td><span style="background-color:green">0.73*</span></td>
    <td>-0.14</td>
    <td>0.49</td>
    <td>0.44</td>
    <td>-0.41</td>
    <td>1.00</td>
  </tr>
</table>

Table 4: Spearman correlation matrix between evaluation metrics. Statistically significant correlations (p < 0.1) are highlighted in green. Asterisks (*) indicate significant values.

ceeding human annotations, raising important questions about the reliability of human-labeled benchmarks as definitive gold standards (Chen et al., 2024). However, we refrain from making definitive claims about this observation and instead treat human-generated codes as the established golden standard in our analysis.

6.3 LLMs Alignment

To assess the alignment between different LLMs in qualitative coding, we separately computed the cosine similarity (Figure 5) and Jensen-Shannon divergence (Figure 6) between the codes generated by models. These similarity metrics demonstrate how consistently different models assign qualitative codes to the same textual input and whether ensemble models enhance convergence toward a unified interpretation. The results clearly indicate that ensemble models, particularly GPT-4 Ensemble, Mixtral 8x7B Ensemble, and Llama3.3 70B Ensemble, exhibit higher semantic similarity and lower divergence compared to individual models.

6.4 Qualitative Assessment and Refinement

Table 5 presents a comparative analysis of codes generated by ensemble models against the human-coded gold standard. The GPT-4 Ensemble demonstrates a tendency toward abstract and generalized codes (e.g., “User Satisfaction and App Effectiveness”), which capture broad themes but often lack specificity. In contrast, the Llama3.3 70B and Mixtral 8x7B Ensembles produce codes that align more closely with the gold standard, offering precise labels. This divergence suggests that GPT-4’s coding strategy prioritizes thematic abstraction, while smaller ensembles have a more effective balance between abstraction and specificity.

The integration of the postprocessing step into our ensemble framework reduces redundancy in generated codes by aligning new outputs with previously assigned codes. As demonstrated in Table 6, refined ensembles produce more concise outputs, achieving an average code length reduction from 6.83 to 4.00 tokens—a 41.5% improvement over standalone models.

Further analysis highlights the impact of post-processing on code diversity. While the human gold standard comprises 47 unique codes with an average length of 2.79 tokens, non-refined models output an excessive number of codes, often generating unique codes for each input. In contrast, code merge integration significantly reduces this redundancy, with Llama3.3 70B Ensemble+ref and Mixtral 8x7B Ensemble+ref producing 53 and 71 unique codes, respectively. This brings the models closer to a human-like coding pattern, as illustrated in Figure 4.

![Comparison of LLMs based on the number of unique codes and average code length](page_1012_682_377_246.png)

Figure 4: Comparison of LLMs based on the number of unique codes and average code length

7 Discussion

In this section, we discuss the implications of our findings regarding the application of ensemble LLMs in automated inductive coding. Our results highlight key advantages of ensemble-based approaches over individual models, particularly in terms of conciseness and the ability to balance abstraction and specificity in qualitative coding.
<table>
  <tr>
    <th>Quote</th>
    <th>Golden Standard</th>
    <th>Llama3.3 70B Ensemble</th>
    <th>GPT4o Ensemble</th>
    <th>GPT4 Ensemble</th>
    <th>Mistral 8x7B Ensemble</th>
    <th>Code Assigned</th>
  </tr>
  <tr>
    <td>"I am not very happy with this because ChatGPT answers the questions I ask but it stops mid-sentence. Then even after continuous efforts it does not give complete answer. This is not very correct. Please fix it so that there is no more trouble. This is a very wrong thing, after some use you are asked to upgrade which is very expensive."</td>
    <td>Technical problems while using</td>
    <td>Unsatisfied with ChatGPT</td>
    <td>User dissatisfaction with incomplete answers</td>
    <td>Unsatisfied with ChatGPT’s incomplete responses</td>
    <td>User: Dissatisfied with expensive, incomplete ChatGPT answers</td>
    <td></td>
  </tr>
  <tr>
    <td>"This is a great app, the bot is so accurate to anything, it gives me tips in gaming, studies, and more. This is a great app! I don’t have any problems with things! Also, there are some problems about the server glitch, but don’t worry, some glitches are just seconds, overall a great app. Note: Mistral 8x7B is more powerful, and scarier than the regular version."</td>
    <td>Helpful app</td>
    <td>Accurate and helpful app</td>
    <td>Helpful and accurate app experience</td>
    <td>User Satisfaction and App Efficiency</td>
    <td>Accurate, helpful tip-providing app</td>
    <td></td>
  </tr>
  <tr>
    <td>"The best so far"</td>
    <td>Best app</td>
    <td>Best so far</td>
    <td>The best experience yet</td>
    <td>User Satisfaction and App Effectiveness</td>
    <td>Top User Rating</td>
    <td></td>
  </tr>
  <tr>
    <td>"Excellent app for learning"</td>
    <td>Excellent for learning</td>
    <td>Excellent learning app</td>
    <td>Excellent learning tool</td>
    <td>User Satisfaction and App Efficiency</td>
    <td>Excellent learning app</td>
    <td></td>
  </tr>
  <tr>
    <td>"Thanks for making my life easier"</td>
    <td>Thankfulness</td>
    <td>Life made easier</td>
    <td>Gratitude for increased ease</td>
    <td>User Satisfaction Feedback</td>
    <td>Gratitude for Ease</td>
    <td></td>
  </tr>
</table>

Table 5: Comparison of qualitative codes generated by ensemble models. The golden standard represents human-coded references.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Merge threshold</th>
    <th colspan="3">BERTScore</th>
    <th colspan="3">ROUGE</th>
    <th rowspan="2">Composite Score</th>
    <th rowspan="2">Code length</th>
  </tr>
  <tr>
    <th>P</th>
    <th>R</th>
    <th>F1</th>
    <th>l</th>
    <th>2</th>
    <th>L</th>
  </tr>
  <tr>
    <td>Mistral8x7B</td>
    <td>-</td>
    <td>0.83</td>
    <td>0.84</td>
    <td>0.83</td>
    <td>0.08</td>
    <td>0.01</td>
    <td>0.08</td>
    <td>0.33</td>
    <td>6.83</td>
  </tr>
  <tr>
    <td>Mistral8x7B Ensemble</td>
    <td>-</td>
    <td>0.83</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.01</td>
    <td>0.08</td>
    <td>0.91</td>
    <td>4.02</td>
    <td></td>
  </tr>
  <tr>
    <td>Mistral8x7B Ensemble + ref</td>
    <td>0.7</td>
    <td>0.84</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.12</td>
    <td>0.01</td>
    <td>0.11</td>
    <td>0.99</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Llama3.3 70B</td>
    <td>-</td>
    <td>0.84</td>
    <td>0.86</td>
    <td>0.85</td>
    <td>0.12</td>
    <td>0.03</td>
    <td>0.12</td>
    <td>0.38</td>
    <td>3.5</td>
  </tr>
  <tr>
    <td>Llama3.3 70B Ensemble</td>
    <td>-</td>
    <td>0.85</td>
    <td>0.86</td>
    <td>0.85</td>
    <td>0.15</td>
    <td>0.02</td>
    <td>0.15</td>
    <td>0.50</td>
    <td>3.57</td>
  </tr>
  <tr>
    <td>Llama3.3 70B Ensemble + ref</td>
    <td>0.5</td>
    <td>0.85</td>
    <td>0.88</td>
    <td>0.86</td>
    <td>0.15</td>
    <td>0.03</td>
    <td>0.15</td>
    <td>0.74</td>
    <td>3.49</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>-</td>
    <td>0.83</td>
    <td>0.84</td>
    <td>0.84</td>
    <td>0.02</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.44</td>
    <td>4.39</td>
  </tr>
  <tr>
    <td>GPT-4 Ensemble</td>
    <td>-</td>
    <td>0.83</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.11</td>
    <td>0.02</td>
    <td>0.10</td>
    <td>0.74</td>
    <td>5.01</td>
  </tr>
  <tr>
    <td>GPT-4 Ensemble + ref</td>
    <td>0.8</td>
    <td>0.83</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.12</td>
    <td>0.02</td>
    <td>0.10</td>
    <td>0.54</td>
    <td>4.62</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>-</td>
    <td>0.85</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.10</td>
    <td>0.04</td>
    <td>0.10</td>
    <td>0.57</td>
    <td>3.8</td>
  </tr>
  <tr>
    <td>GPT-4o Ensemble</td>
    <td>-</td>
    <td>0.85</td>
    <td>0.87</td>
    <td>0.86</td>
    <td>0.14</td>
    <td>0.02</td>
    <td>0.14</td>
    <td>0.74</td>
    <td>4.26</td>
  </tr>
  <tr>
    <td>GPT-4o Ensemble + ref</td>
    <td>0.7</td>
    <td>0.85</td>
    <td>0.87</td>
    <td>0.86</td>
    <td>0.11</td>
    <td>0.00</td>
    <td>0.11</td>
    <td>0.54</td>
    <td>4.42</td>
  </tr>
  <tr>
    <td>Llama3 8B Instruct</td>
    <td>-</td>
    <td>0.83</td>
    <td>0.86</td>
    <td>0.84</td>
    <td>0.12</td>
    <td>0.02</td>
    <td>0.11</td>
    <td>0.61</td>
    <td>8.45</td>
  </tr>
  <tr>
    <td>Falcon 7B Instruct</td>
    <td>-</td>
    <td>0.83</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.09</td>
    <td>0.01</td>
    <td>0.09</td>
    <td>0.47</td>
    <td>12.76</td>
  </tr>
  <tr>
    <td>Mistral 7B Instruct</td>
    <td>-</td>
    <td>0.83</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.07</td>
    <td>0.01</td>
    <td>0.07</td>
    <td>0.67</td>
    <td>11.77</td>
  </tr>
</table>

Table 6: Performance comparison of individual models, standard ensembles, and refined ensembles across key metrics. The merge threshold (refinement) column indicates the similarity threshold between the generated code and codes before it (can look only in the past). Models are evaluated using BERTScore (Precision, Recall, and F1), ROUGE (1, 2, and L), Composite Score, and Average Code Length.

7.1 Ensembles Improve Coding Consistency

A major finding of our study is that ensemble models consistently outperform individual models in inductive coding tasks, as shown in Table 6. The improved performance is particularly evident in Jensen-Shannon divergence (Figure 6), where ensembles exhibit lower divergence, indicating a greater degree of alignment. This suggests that aggregating multiple model outputs helps reduce inconsistencies, reflecting the consensus-building process used by human coders in thematic analysis.

The increased consistency observed in ensemble-generated codes aligns with findings from prior research on LLM evaluation, which suggest that individual models often introduce unwanted variability in their outputs due to differences in training data and architectural biases (Bubeck et al., 2023; Jiang et al., 2023b). In contrast, ensemble methods mitigate this variability by integrating diverse inputs, thereby improving robustness. Our results indicate that this effect holds even for smaller models.

7.2 Postprocessing Enhances Code Stability

The integration of code merging significantly improves code stability, as demonstrated by higher composite and ROUGE scores in refined ensembles (Table 6). By referencing previously assigned codes, refinement reduces redundancy for similar inputs. This is particularly evident in the reduction of unique code counts (e.g., 53 for Llama3.3
Figure 5: Mean cosine similarity heatmap of model-generated codes. Darker green cells indicate higher semantic similarity between models. Ensemble models (GPT-4 Ensemble, Mixtral 8x7B Ensemble, Llama3.3 70B Ensemble) exhibit the highest similarity, demonstrating greater convergence toward a single interpretation of qualitative codes.

Figure 6: Mean Jensen-Shannon divergence heatmap of model-generated codes. Darker green cells indicate lower divergence, meaning greater agreement in assigned codes. Ensemble models exhibit lower JSD, reinforcing their role in reducing variance and improving convergence in qualitative coding tasks.

70B+ref vs. 100 for non-refined models) and code length (41.5% reduction), bringing model outputs closer to human-like efficiency.

Figure 5 reveals that all ensemble models exhibit the highest similarity scores. This suggests that aggregating inputs from multiple smaller models leads to greater convergence in assigned qualitative codes. These ensembles tend to agree more with each other, reducing variance in code assignments compared to individual models.

7.3 Balancing Abstraction and Specificity in Generated Codes

A key distinction among models is their tendency toward abstraction versus specificity. As shown in Table 5, GPT-4 ensembles often generate broad thematic labels, which, while useful for high-level analysis, may lack the granularity required for domain-specific research. In contrast, smaller ensembles produce more precise codes that closely align with human annotations.

This finding reflects a fundamental trade-off in LLM-based coding: while abstraction improves generalizability, excessive abstraction can obscure critical nuances. Prior work has noted that LLMs trained on diverse corpora tend to favor generalized patterns over domain-specific details (Bubeck et al., 2023; Tornberg, 2023). Results suggest that ensemble approaches can mitigate this issue by combining diverse inputs, thus producing more balanced and contextually grounded outputs.

7.4 Future Directions

Building on our findings, we identify two promising directions for future research. First, hierarchical coding strategies could enable models to generate both broad thematic categories and specific subcategories, better reflecting human coding practices (Saldana, 2016). Second, adaptive RAG mechanisms could dynamically adjust retrieval thresholds based on input complexity, further enhancing consistency and efficiency. These methods could significantly improve the interpretability and quality of automated coding systems.

8 Conclusion

In this work, we explored the use of ensemble-based LLMs with refined outputs for automated inductive coding, demonstrating that ensemble approaches consistently outperform individual models in qualitative coding tasks. Our results highlight three key findings: (1) Ensemble methods improve coding consistency, reducing variance and increasing alignment with human-coded benchmarks; (2) Code merging postprocessing enhances coding stability, ensuring greater coherence in label assignments across similar inputs; and (3) Smaller ensemble models outperform standalone high-capacity models, offering a more efficient and scalable alternative to traditional LLM-based approaches. These findings highlight the potential of ensemble approaches and refinement to advance automated qualitative analysis while maintaining alignment with human coding patterns.
Limitations

While our ensemble-based LLMs demonstrate significant improvements in inductive coding, several limitations warrant consideration.

Bias and Fairness

Although ensembling reduces inconsistencies across individual models, it does not fully address biases inherent in the training data of base models. These biases may manifest in the form of skewed code distributions or overgeneralized labels. Future work should investigate bias mitigation techniques, such as adversarial training or dataset balancing, to further improve model fairness and reliability.

Computational Costs

While our ensemble models are smaller and more efficient than large standalone models like GPT-4, they still require higher computational costs compared to single smaller models. For real-world applications, further optimization strategies—such as dynamic ensemble selection—could help balance accuracy with computational efficiency.

Evaluation Metrics

While our composite metric captures multiple dimensions of coding performance, no single metric fully replicates the nuanced judgment of human coders. Future research should explore more sophisticated evaluation frameworks, incorporating human preference modeling, interactive evaluation setups, or multi-criteria decision analysis to better align automated coding with qualitative research standards.

Metric Accessibility and Fairness

While the composite metric demonstrates strong alignment with human judgments, it requires access to gold-standard codes, which limits its use in real-world deployments. This raises a methodological concern when comparing models that are evaluated using this metric against those that do not rely on gold labels. However, we emphasize that the metric is used only after inference and does not inform code generation or refinement. In future work, we aim to explore unsupervised metrics to enable fully gold-free evaluation.

Language Generalizability

Our study focuses exclusively on English-language datasets, which restricts the generalizability of our findings to other languages and cultural contexts. Extending this approach to multilingual datasets could reveal additional challenges, such as language-specific coding conventions or cultural biases, necessitating further adaptations.

Ethics Statement

The use of LLMs in qualitative research introduces ethical considerations, particularly regarding bias, transparency, and the potential for automated systems to replicate or amplify human biases. While our ensemble-based approach mitigates some of these risks by combining diverse model outputs and employing code refinement for consistency, we acknowledge that biases inherent in training data may still influence results. Human oversight remains critical, as demonstrated by our reliance on expert evaluations to establish a golden standard and validate model outputs. We emphasize that LLMs should serve as assistive tools rather than replacements for human expertise, and we advocate for the development of ethical guidelines to ensure responsible use in sensitive research domains. The datasets used in this study were anonymized and handled in compliance with ethical research standards, minimizing risks to participants.

References

Anna Aniol, Marcin Pietron, and Jerzy Duda. 2019. Ensemble approach for natural language question answering problem. In 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW), pages 180–183. IEEE.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.

H. Russell Bernard. 2016. Research Methods in Anthropology: Qualitative and Quantitative Approaches. Rowman & Littlefield.

Rishi Bommasani et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.

Virginia Braun and Victoria Clarke. 2021. Thematic Analysis: A Practical Guide. SAGE Publications.

Sebastien Bubeck, Varun Chandak, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
Simon Bumbuc and Sam Hrybyk. 2016. Subjectivity in qualitative coding: An issue revisited. Qualitative Research, 16(3):253–263.

Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A survey on mixture of experts.

John Chen, Alexandros Lotsos, Lexie Zhao, Caiyi Wang, Jessica Hullman, Bruce Sherin, Uri Wilensky, and Michael Horn. 2024. A computational method for measuring "open codes" in qualitative analysis.

John W. Creswell. 2016. 30 Essential Skills for the Qualitative Researcher. SAGE Publications.

Randi A Engle, Jennifer M Langer-Osuna, and Maxine McKinney de Royston. 2014. Toward a model of influence in persuasive discussions: Negotiating quality, authority, privilege, and access within a student-led argument. Journal of the Learning Sciences, 23(2):245–268.

Alexander Fabbri et al. 2021a. Summac: Re-visiting nli-based models for consistency evaluation. Proceedings of EMNLP 2021.

Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021b. Summeval: Re-evaluating summarization evaluation.

Tim Fischer and Chris Biemann. 2024. Exploring large language models for qualitative data analysis. In Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, pages 423–437, Miami, USA. Association for Computational Linguistics.

Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685.

Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023a. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.

Dongfu Jiang et al. 2023b. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. Proceedings of ACL 2023.

Zhe Jiang et al. 2023c. Tinyllama: Distilling large language models for efficiency. arXiv preprint arXiv:2310.05637.

Bhavik Jikadara. 2023. Chatgpt user feedback dataset. https://www.kaggle.com/datasets/bhavikjikadara/chatgpt-user-feedback. Dataset containing user reviews and ratings for the ChatGPT Android App, updated daily. Includes attributes such as review text, numerical ratings, thumbs up counts, and timestamps. Licensed under MIT.

Patrick Lewis et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Proceedings of NeurIPS 2020.

Chenghao Li, Fangchen Xie, et al. 2023. Vicuna: An open-source chatbot. FastChat: Open Assistant. Available at https://vicuna.lmsys.org/.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries.

Kathleen M. MacQueen and Greg Guest. 2008. Handbook for Team-Based Qualitative Research. AltaMira Press.

George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J Miller. 1990. Introduction to wordnet: An on-line lexical database. International journal of lexicography, 3(4):235–244.

Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949.

Janice M. Morse. 2017. Considering theory derived from qualitative research. Qualitative Health Research, 27(10):1372–1380.

Diana Parfenova et al. 2024. Automating qualitative analysis with llms. Proceedings of ACL 2024.

Felipe Pineda, Raphael Milliere, Andreas Vlachos, Myle Ott, Richard Yates, Amelia Glaese, et al. 2023. The falcon series of language models. arXiv preprint arXiv:2306.01116.

Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35, Dublin, Ireland. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.

Omer Sag and Lior Rokach. 2018. Ensemble learning: A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery, 8(4):e1249.

Johnny Saldana. 2016. The Coding Manual for Qualitative Researchers. SAGE Publications.

Thibault Sellam et al. 2020. Bleurt: Learning robust metrics for text generation. Proceedings of ACL 2020.

Gemma AI Research Team. 2024. Gemma: An instructable, open-source large language model.

Mistral AI Team. 2023. Mistral: Efficient pretraining of transformer language models.
Petter Tornberg. 2023. Using large language models for automated qualitative coding in the social sciences. Nature Machine Intelligence, 5:576–586.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Thibaut Lavril, et al. 2023b. Llama: Open and efficient foundation language models. In Proceedings of the 2023 Annual Conference on Machine Learning (ICML), pages 123–134.

Tianyi Zhang et al. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

A Prompts used for finetuning

In the conducted experiments, several prompts were designed to guide the models in performing open coding tasks. These prompts varied in their level of explicitness, the perspective they asked the model to adopt, and the amount of detail they requested. Additionally, the experiments compared the effect of using a line break versus a period (dot) at the end of each prompt to assess how subtle changes in prompt formatting might influence the model’s performance (see Appendix B). Below is a brief description of each prompt:

• Explicit Instruction (Prompt 1): Summarize the main idea of a sentence. This prompt provides a direct and clear instruction to the model, asking it to summarize the core idea of a given sentence. The expectation is for the model to extract the primary message or theme conveyed in the sentence with no additional context or framing. This prompt is designed to test the model’s ability to perform a straightforward task without needing implicit knowledge.

• Informal Request (Prompt 2): Can you tell me what the main idea of this sentence is in just a few words? This prompt is phrased as a casual, conversational question, asking the model to summarize the sentence in “just a few words.” The informal tone encourages a more concise and simplified response, aiming to capture how well the model can extract the essence of the sentence in a more natural, everyday context.

• Expert Angle (Prompt 3): From the perspective of a social scientist, summarize the following sentence as you would in thematic coding. This prompt takes a more specialized approach, asking the model to assume the perspective of a social scientist performing thematic coding. The expectation here is for the model to not only summarize the sentence but to apply a more analytical and structured lens, possibly introducing higher-level categorizations that would be typical in qualitative data analysis.

• Impersonalization (Prompt 4): If you were a social scientist doing thematic analysis, what code would you give to this citation? In this prompt, the model is asked to act as a social scientist and assign a code, which is a brief label representing the central idea of the sentence. It emphasizes the objectivity of thematic analysis, expecting the model to de-personalize the task and focus on generating an appropriate label that accurately reflects the content.

• Detailed Explanation (Prompt 5): Explain in a couple of words the primary thought expressed in the following text. This prompt asks the model to provide a more detailed, thorough explanation of the primary thought behind the text. It is designed to encourage the model to go beyond a simple summary and delve into the deeper meaning or implications of the sentence.

• Simplified Task (Prompt 6): What is the gist of this sentence? This prompt simplifies the task by asking for the "gist" of the sentence. It challenges the model to provide a very brief and straightforward summary, focusing on distilling the essential meaning of the sentence.

B Detailed fine-tuning results

These results (see Table 7) demonstrate the performance of various models when fine-tuned on the task of open coding using different prompts. BERTScore and ROUGE are reported.

C Moderator prompt template

D Links to models on Hugging Face

• Llama3 8B: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">BERTScore</th>
    <th colspan="4">ROUGE</th>
  </tr>
  <tr>
    <th>P<sub>std</sub></th>
    <th>R<sub>std</sub></th>
    <th>F1<sub>std</sub></th>
    <th>1</th>
    <th>2</th>
    <th>L</th>
  </tr>
  <tr>
    <td colspan="7"><b>Summarize the main idea of a sentence\n</b></td>
  </tr>
  <tr>
    <td>Llama3</td><td>0.7130.060</td><td>0.7580.040</td><td>0.7340.062</td><td>0.141</td><td>0.033</td><td>0.153</td>
  </tr>
  <tr>
    <td>Falcon</td><td>0.7460.073</td><td>0.7820.097</td><td>0.7640.095</td><td>0.176</td><td>0.047</td><td>0.189</td>
  </tr>
  <tr>
    <td>Mistral</td><td>0.7290.076</td><td>0.7870.093</td><td>0.7560.078</td><td>0.178</td><td>0.047</td><td>0.195</td>
  </tr>
  <tr>
    <td>Vicuna</td><td>0.7170.063</td><td>0.7710.095</td><td>0.7360.079</td><td>0.163</td><td>0.028</td><td>0.182</td>
  </tr>
  <tr>
    <td>Gemma</td><td>0.7120.064</td><td>0.7380.078</td><td>0.7450.089</td><td>0.163</td><td>0.030</td><td>0.168</td>
  </tr>
  <tr>
    <td>TinyLlama</td><td>0.7180.072</td><td>0.7750.090</td><td>0.7570.087</td><td>0.164</td><td>0.052</td><td>0.158</td>
  </tr>
  <tr>
    <td colspan="7"><b>Summarize the main idea of a sentence.</b></td>
  </tr>
  <tr>
    <td>Llama3</td><td>0.7180.072</td><td>0.7880.089</td><td>0.7500.072</td><td>0.181</td><td>0.059</td><td>0.166</td>
  </tr>
  <tr>
    <td>Falcon</td><td>0.7380.099</td><td>0.7870.103</td><td>0.7610.096</td><td>0.213</td><td>0.077</td><td>0.210</td>
  </tr>
  <tr>
    <td>Mistral</td><td>0.7190.072</td><td>0.7680.086</td><td>0.7420.075</td><td>0.157</td><td>0.055</td><td>0.148</td>
  </tr>
  <tr>
    <td>Vicuna</td><td>0.7430.079</td><td>0.7780.095</td><td>0.7580.081</td><td>0.193</td><td>0.068</td><td>0.185</td>
  </tr>
  <tr>
    <td>Gemma</td><td>0.7190.071</td><td>0.7790.089</td><td>0.7460.072</td><td>0.172</td><td>0.049</td><td>0.166</td>
  </tr>
  <tr>
    <td>TinyLlama</td><td>0.7360.083</td><td>0.7880.092</td><td>0.7600.081</td><td>0.207</td><td>0.074</td><td>0.199</td>
  </tr>
  <tr>
    <td colspan="7"><b>Can you tell me what the main idea of this sentence is in just a few words?</b></td>
  </tr>
  <tr>
    <td>Llama3</td><td>0.6880.055</td><td>0.7780.084</td><td>0.7290.061</td><td>0.116</td><td>0.034</td><td>0.110</td>
  </tr>
  <tr>
    <td>Falcon</td><td>0.7530.105</td><td>0.7870.108</td><td>0.7680.102</td><td>0.236</td><td>0.104</td><td>0.239</td>
  </tr>
  <tr>
    <td>Mistral</td><td>0.7420.106</td><td>0.7950.106</td><td>0.7660.101</td><td>0.246</td><td>0.106</td><td>0.235</td>
  </tr>
  <tr>
    <td>Vicuna</td><td>0.7410.080</td><td>0.7860.087</td><td>0.7520.063</td><td>0.168</td><td>0.047</td><td>0.161</td>
  </tr>
  <tr>
    <td>Gemma</td><td>0.7110.075</td><td>0.7860.093</td><td>0.7460.079</td><td>0.171</td><td>0.057</td><td>0.168</td>
  </tr>
  <tr>
    <td>TinyLlama</td><td>0.7250.083</td><td>0.7890.090</td><td>0.7540.079</td><td>0.178</td><td>0.067</td><td>0.177</td>
  </tr>
  <tr>
    <td colspan="7"><b>From the perspective of a social scientist, summarize the following sentence as you would in thematic coding.\n</b></td>
  </tr>
  <tr>
    <td>Llama3</td><td>0.6980.059</td><td>0.7840.083</td><td>0.7380.062</td><td>0.130</td><td>0.033</td><td>0.119</td>
  </tr>
  <tr>
    <td>Falcon</td><td>0.7450.109</td><td>0.7920.105</td><td>0.7660.102</td><td>0.210</td><td>0.089</td><td>0.211</td>
  </tr>
  <tr>
    <td>Mistral</td><td>0.6880.060</td><td>0.7850.086</td><td>0.7320.064</td><td>0.139</td><td>0.041</td><td>0.131</td>
  </tr>
  <tr>
    <td>Vicuna</td><td>0.7130.080</td><td>0.7780.094</td><td>0.7430.080</td><td>0.169</td><td>0.061</td><td>0.166</td>
  </tr>
  <tr>
    <td>Gemma</td><td>0.7210.085</td><td>0.7840.093</td><td>0.7490.082</td><td>0.180</td><td>0.070</td><td>0.177</td>
  </tr>
  <tr>
    <td>TinyLlama</td><td>0.7180.073</td><td>0.7760.083</td><td>0.7450.072</td><td>0.165</td><td>0.053</td><td>0.158</td>
  </tr>
  <tr>
    <td colspan="7"><b>From the perspective of a social scientist, summarize the following sentence as you would in thematic coding.</b></td>
  </tr>
  <tr>
    <td>Llama3</td><td>0.6850.082</td><td>0.7810.064</td><td>0.7330.081</td><td>0.136</td><td>0.025</td><td>0.154</td>
  </tr>
  <tr>
    <td>Falcon</td><td>0.7540.066</td><td>0.7780.091</td><td>0.7590.088</td><td>0.181</td><td>0.048</td><td>0.190</td>
  </tr>
  <tr>
    <td>Mistral</td><td>0.7400.067</td><td>0.7800.088</td><td>0.7560.071</td><td>0.172</td><td>0.043</td><td>0.187</td>
  </tr>
  <tr>
    <td>Vicuna</td><td>0.7200.071</td><td>0.7810.094</td><td>0.7530.075</td><td>0.180</td><td>0.038</td><td>0.185</td>
  </tr>
  <tr>
    <td>Gemma</td><td>0.7000.072</td><td>0.7800.085</td><td>0.7460.089</td><td>0.180</td><td>0.046</td><td>0.187</td>
  </tr>
  <tr>
    <td>TinyLlama</td><td>0.7290.076</td><td>0.7780.089</td><td>0.7540.080</td><td>0.169</td><td>0.046</td><td>0.183</td>
  </tr>
  <tr>
    <td colspan="7"><b>If you were a social scientist doing thematic analysis, what code would you give to this citation?\n</b></td>
  </tr>
  <tr>
    <td>Llama3</td><td>0.6920.060</td><td>0.7850.083</td><td>0.7350.064</td><td>0.064</td><td>0.043</td><td>0.126</td>
  </tr>
  <tr>
    <td>Falcon</td><td>0.7360.093</td><td>0.7800.101</td><td>0.7590.092</td><td>0.206</td><td>0.076</td><td>0.200</td>
  </tr>
  <tr>
    <td>Mistral</td><td>0.6860.057</td><td>0.7850.082</td><td>0.7310.061</td><td>0.132</td><td>0.041</td><td>0.123</td>
  </tr>
  <tr>
    <td>Vicuna</td><td>0.7200.070</td><td>0.7810.089</td><td>0.7510.075</td><td>0.183</td><td>0.068</td><td>0.168</td>
  </tr>
  <tr>
    <td>Gemma</td><td>0.7240.085</td><td>0.7840.091</td><td>0.7510.082</td><td>0.170</td><td>0.066</td><td>0.168</td>
  </tr>
  <tr>
    <td>TinyLlama</td><td>0.7200.071</td><td>0.7780.083</td><td>0.7470.072</td><td>0.186</td><td>0.053</td><td>0.182</td>
  </tr>
  <tr>
    <td colspan="7"><b>What is the gist of this sentence?</b></td>
  </tr>
  <tr>
    <td>Llama3</td><td>0.6880.064</td><td>0.7800.086</td><td>0.7250.060</td><td>0.129</td><td>0.042</td><td>0.121</td>
  </tr>
  <tr>
    <td>Falcon</td><td>0.7310.091</td><td>0.7800.090</td><td>0.7540.089</td><td>0.182</td><td>0.080</td><td>0.179</td>
  </tr>
  <tr>
    <td>Mistral</td><td>0.7260.079</td><td>0.7850.095</td><td>0.7530.079</td><td>0.165</td><td>0.057</td><td>0.160</td>
  </tr>
  <tr>
    <td>Vicuna</td><td>0.7200.070</td><td>0.7810.089</td><td>0.7480.072</td><td>0.172</td><td>0.055</td><td>0.162</td>
  </tr>
  <tr>
    <td>Gemma</td><td>0.7070.077</td><td>0.7730.091</td><td>0.7370.076</td><td>0.152</td><td>0.059</td><td>0.146</td>
  </tr>
  <tr>
    <td>TinyLlama</td><td>0.7180.077</td><td>0.7760.079</td><td>0.7410.061</td><td>0.133</td><td>0.032</td><td>0.139</td>
  </tr>
  <tr>
    <td colspan="7"><b>Explain in a couple of words the primary thought expressed in the following text.\n</b></td>
  </tr>
  <tr>
    <td>Llama3</td><td>0.6910.062</td><td>0.7830.085</td><td>0.7330.066</td><td>0.120</td><td>0.038</td><td>0.110</td>
  </tr>
  <tr>
    <td>Falcon</td><td>0.7340.078</td><td>0.7780.090</td><td>0.7540.078</td><td>0.171</td><td>0.049</td><td>0.165</td>
  </tr>
  <tr>
    <td>Mistral</td><td>0.6980.067</td><td>0.7800.088</td><td>0.7330.070</td><td>0.141</td><td>0.038</td><td>0.131</td>
  </tr>
  <tr>
    <td>Vicuna</td><td>0.7200.072</td><td>0.7810.088</td><td>0.7520.072</td><td>0.155</td><td>0.048</td><td>0.148</td>
  </tr>
  <tr>
    <td>Gemma</td><td>0.7060.064</td><td>0.7860.086</td><td>0.7420.066</td><td>0.177</td><td>0.053</td><td>0.170</td>
  </tr>
  <tr>
    <td>TinyLlama</td><td>0.7200.077</td><td>0.7840.091</td><td>0.7500.078</td><td>0.168</td><td>0.071</td><td>0.163</td>
  </tr>
  <tr>
    <td colspan="7"><b>Explain in a couple of words the primary thought expressed in the following text.</b></td>
  </tr>
  <tr>
    <td>Llama3</td><td>0.7080.068</td><td>0.7840.055</td><td>0.7470.063</td><td>0.142</td><td>0.025</td><td>0.152</td>
  </tr>
  <tr>
    <td>Falcon</td><td>0.7520.088</td><td>0.7790.091</td><td>0.7600.086</td><td>0.183</td><td>0.042</td><td>0.193</td>
  </tr>
  <tr>
    <td>Mistral</td><td>0.7380.070</td><td>0.7900.090</td><td>0.7590.073</td><td>0.173</td><td>0.047</td><td>0.183</td>
  </tr>
  <tr>
    <td>Vicuna</td><td>0.7170.066</td><td>0.7800.094</td><td>0.7520.099</td><td>0.161</td><td>0.025</td><td>0.182</td>
  </tr>
  <tr>
    <td>Gemma</td><td>0.7080.068</td><td>0.7780.079</td><td>0.7460.088</td><td>0.172</td><td>0.039</td><td>0.186</td>
  </tr>
  <tr>
    <td>TinyLlama</td><td>0.7280.073</td><td>0.7790.091</td><td>0.7530.099</td><td>0.168</td><td>0.053</td><td>0.168</td>
  </tr>
</table>

Table 7: Detailed Fine-tuning Results. The following table presents the detailed results from fine-tuning experiments, including precision (P), recall (R), F1 score, and ROUGE across different models and prompts.
You will be given a paragraph from the text, which is: {textdescription}.

Definition of the code: A word or short phrase that symbolically assigns a summative, salient, essence-capturing, and/or evocative attribute for a portion of language-based or visual data.

Here is the excerpt to code:
{row["Paragraph"]}

Here are three coding suggestions from previous models:
1. {row["Llama3_Code"]}
2. {row["Falcon_Code"]}
3. {row["Mistral_Code"]}

Please choose the best code or suggest a new code taking into account all these answers.
The output should be the code with no longer than 5 words.

Listing 1: Moderator Prompt Template with Model Suggestions

• Falcon 7B: https://huggingface.co/tiiuae/falcon-7b-instruct
• Mistral 7B: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
• Vicuna 7B: https://huggingface.co/lmsys/vicuna-7b-v1.5
• Gemma 7B: https://huggingface.co/google/gemma-7b-it
• TinyLlama 1.1B: https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Llama 3.3.70B: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct
• Mixtral 7x8B: https://huggingface.co/mistralai/Mixtral-8x7B-v0.1

E LoRA configuration

This section provides the LoRA (Low-Rank Adaptation) configuration used for fine-tuning the models in this study. Below is the code snippet used for configuring LoRA:

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, config)
print_trainable_parameters(model)

generation_config = model.generation_config
generation_config.max_new_tokens = 15
generation_config.temperature = 0.7
generation_config.top_p = 0.9
generation_config.do_sample_sequences = 1
generation_config.pad_token_id = tokenizer.eos_token_id
generation_config.eos_token_id = tokenizer.eos_token_id

F Human evaluation instructions

To validate the quality of automatically generated qualitative codes, we conducted a human evaluation study in which domain experts rated model-generated codes for a given set of quotes. The evaluation form was structured as follows:

F.1 Code Rating Guidelines

Introduction:
Thank you for participating in this evaluation study. Your expertise is crucial in assessing the quality of automatically generated qualitative codes. In this task, you will be presented with a sentence (quote) and a corresponding code assigned by a model. You will rate the quality of the code based on its accuracy, relevance, and conciseness.

Definition of a Code:
According to Saldana (2016), a code is "often a word or short phrase that symbolically assigns a summative, salient, essence-capturing, and/or evocative attribute for a portion of language-based or visual data."

F.2 Instructions for Participants

1. Read the given quote carefully.
2. Review the assigned code.
3. Rate the quality of the code on a scale from 1 to 5 based on the following criteria:

• 1 - Very Poor: The code does not reflect the main idea of the quote at all.

• 2 - Poor: The code partially captures the quote but lacks clarity or relevance.

• 3 - Acceptable: The code is somewhat relevant but could be improved for clarity or conciseness.

• 4 - Good: The code accurately represents the key idea of the quote with minimal ambiguity.

• 5 - Excellent: The code perfectly captures the essence of the quote in a concise and meaningful way.
These ratings were collected for all models across a diverse set of ten quotes. The results were analyzed to compute correlations between human ratings and various automated evaluation metrics, contributing to the validation of the composite score.

Reviewer paper summaries: This paper proposes a pipeline to generate inductive codes using an ensemble of fine-tuned LLMs. The authors evaluate this pipeline against a composite evaluation metric that compares generated codes against a human-generated gold codebook, taking into account semantic and lexical similarity as well as code length and JS divergence from the gold labels. This metric is validated against expert ratings, with a strong Spearman's correlation between expert ratings and metric scores. The porposed pipeline performs well against this evaluation metric.
Summary of strengths: 1. The composite metric is validated against expert judgments, and shows a relatively high (0.73), significant Spearman correlation that outperforms other commonly-used evaluation metrics such as ROUGE and BERTScore.

2. The authors demonstrate that their proposed pipeline generates shorter codes that align more closely with human-generated ones according to the composite metric.
Summary of weaknesses: 1. The details about the pipeline and metric are underspecified. In particular, I found some of the notation that was introduced to be more confusing than helpful. Including some examples of inputs and outputs at each step of the pipeline might improve clarity. I'll include more specifics below. I'd feel comfortable raising my score if most of these points are addressed.

2. On line 236, I understand $x$ to be a code because its embedding $\phi(x)$ is compared against other code embeddings, but $x$ was previously defined as an input sentence on line 220. In addition, would it be more accurate on 241 to say you are calculating $\text{sim}(\phi(x), \phi(p_i))$? And as far as I could tell, $p_i$ was not defined anywhere, but I assume that $p_i$ refers to the code assigned to a previously seen sentence based on lines 238-239.

3. Line 243-244 says the final optimal code $y_{\text{best}}$ was selected using the composite evaluation metric, but doesn't this metric rely on the gold labels? **If so, isn't this a major methodological issue, especially given that you are then evaluating this method against other methods that don't have access to this metric or the gold labels?**

4. What distributions are being compared when calculating the JS divergence in section 5.1? Token distributions? If so, wouldn't this be highly correlated with the METEOR score? 

4. I would have liked to see some IAA numbers for the metric validation in 5.2 in addition to correlation between the proposed metric and the human annotators. This will give a sense of what we might expect from a well-aligned metric.

5. Line 361 references metrics proposed by Chen et al. -- it will be useful for readers to include some background information that describes these metrics and how to interpret them.

6. Line 368 says that "the results suggest that in certain cases, LLM-generated codes may achieve performance levels comparable to or exceeding human annotations", but it's not clear to me what results this sentence is referring to. Is it the results in Figure 3? I'm not sure I get that from the figure.

7. The code length reduction reported in lines 411-412 are actually for improvement of refined ensembles over non-refined, non-ensemble models, right? I am basing this off of the results in Table 6. The decrease in code length from ensembles to refined ensembles is much smaller (and, in the case of GPT-4o, actually reversed.)
Comments, suggestions and typos: 1. Line 116: the phrasing "one research" is awkward --- I might suggest something like "one such work"
2. Line 123: The intro problematizes "subjectivity" here, pointing to variability between individual LLMs. But it's unclear to me that "objective" coding is an attainable goal? How should we think about objectivity / subjectivity in this inherently interpretive process?
3. I am not sure how useful table 4 is to understanding the paper. I wonder if it can be placed in the appendix, and the space better used to add some more details about the pipeline and metric.
Overall assessment scores: 2.5