Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classification
Authors: Gyutae Park, Ingeol Baek, ByeongJeong Kim, Joongbo Shin, Hwanhee Lee
Accepted at: acl25

Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classification

Gyutae Park1, Ingeol Baek1, ByeongJeong Kim1, Joongbo Shin2, Hwanhee Lee1*
1Department of Artificial Intelligence, Chung-Ang University, 2LG AI Research
{pkt0401, ingeolbaek, michael197k, hwanheelee}@cau.ac.kr, jb.shin@lgresearch.ai

Abstract
Dialogue intent classification aims to identify the underlying purpose or intent of a user’s input in a conversation. Current intent classification systems encounter considerable challenges, primarily due to the vast number of possible intents and the significant semantic overlap among similar intent classes. In this paper, we propose a novel approach to few-shot dialogue intent classification through in-context learning, incorporating dynamic label refinement to address these challenges. Our method retrieves relevant examples for a test input from the training set and leverages a large language model to dynamically refine intent labels based on semantic understanding, ensuring that intents are clearly distinguishable from one another. Experimental results demonstrate that our approach effectively resolves confusion between semantically similar intents, resulting in significantly enhanced performance across multiple datasets compared to baselines. We also show that our method generates more interpretable intent labels, and has a better semantic coherence in capturing underlying user intents compared to baselines. We release the code at https://github.com/lilabgyutae/Dyanmic.

1 Introduction
Dialogue intent classification identifies the underlying intent or purpose of a user’s input in a conversation. It is a key component of task-oriented dialogue systems (Degand and Muller, 2020), enabling accurate understanding of user utterances and generation of appropriate responses. However, current intent classification systems face challenges, particularly in managing a large number of intent classes and resolving semantic ambiguity between similar intents (Sung et al., 2023; Cho et al., 2024; Lu et al., 2024). Recent work explores few-shot learning approaches, including retrieval-augmented methods (Milios et al., 2023; Gao et al., 2024; Abdullahi et al., 2024) and prompt-based techniques (Loukas et al., 2023; Parikh et al., 2023; Zhang et al., 2024; Rodriguez et al., 2024), which enable models to learn from limited examples per intent. While retrieval-augmented methods effectively narrow down candidate intents by retrieving examples similar to the input query, these methods also introduce a critical challenge: the retrieved examples often show significant semantic overlap across different intent categories.

As shown in Figure 1, even with just three similar intents (‘Verify PAN’, ‘Bank verification details’, and ‘Account not verified’), the model struggles to make accurate predictions due to their semantic similarity, as indicated by the cosine similarity score of 0.91 in the embedding space. We observe that this high semantic similarity between intent labels makes it challenging for models to

![An example illustrating how ambiguous and similar label names can confuse the model, while refined label names enable clearer decision-making.](page_682_355_682_377.png)

Figure 1: An example illustrating how ambiguous and similar label names can confuse the model, while refined label names enable clearer decision-making.
Figure 2: Overall flow of the proposed dynamic label name refinement method for intent classification.

distinguish between different intents accurately.

We find that these issues can be mitigated by refining the label names to forms that more distinctly differentiate them from other labels. As illustrated in Figure 1, by mapping the label ‘Verify PAN’ to a more descriptive form, such as ‘Verify PAN card details’, it becomes easier to differentiate it from general bank verification intents. This refinement establishes clearer semantic boundaries between intent categories, resulting in more accurate classifications.

In this paper, we present a novel approach that combines dynamic label refinement with similarity-based example selection. Our method involves retrieving semantically similar examples and dynamically refining their intent labels to create more meaningful distinctions between related intents. Through extensive experiments across various model scales and diverse datasets, we achieve significant improvements. Our analysis shows that the improvements are particularly pronounced in datasets with high semantic overlap between intents, with accuracy gains ranging from 2.07% to 7.51% across different model scales.

2 Method

Following recent work in dialogue intent classification (Milios et al., 2023; Chen et al., 2024), our approach leverages retrieval-based in-context learning (ICL) with large language models (LLMs), which has demonstrated effectiveness in tasks involving large label spaces. This approach allows models to dynamically leverage relevant few-shot examples for prediction from the training set.

We introduce a retrieval ICL method for intent classification with dynamic label re-naming, which comprises three steps: (1) retrieving semantically similar examples, (2) refining intent labels using an LLM to generate more descriptive labels, and (3) conducting the final classification with these refined examples.

2.1 Retrieving In-Context Examples

We start by retrieving relevant examples in the dataset for each input query. These retrieved examples are grouped by their original intents to provide comprehensive context for the subsequent steps.

2.2 Dynamic Label Re-naming

Our proposed dynamic label refinement process combines retrieval-based example selection with label generation guided by an LLM, as shown in Figure 2. For each of the specified intent groups, we design tailored instructions for the LLM to analyze these groups and refine the labels accordingly. (see Appendix E for full prompt.) Specifically, we design intent label refinement as a process where the model evaluates whether to retain the original label or propose an enhanced version while preserving the domain-specific semantics. During this process, the model assesses the semantic relationship between the label and its associated examples, deciding either to maintain the original intent name or to generate a more descriptive alternative. For instance, as shown in Figure 2, when analyzing examples like “Verify my PAN card” and “Pan card verification”, the model recognizes that the original label ‘verify_pan’ could be more descriptive and refines it to ‘verify_pan_card_details’ to better capture the specific verification intent.

2.3 Dynamic vs Static Label Refinement

While a static label refinement approach might seem computationally simpler, where intent labels are refined once using all training examples collectively, our experimental validation demonstrates the superiority of dynamic refinement. Table 1 compares both approaches across representative datasets and models.

Dynamic refinement generally outperforms static refinement, though with some exceptions (e.g., static shows advantages on CUREKART and CLINC150 for smaller models). Static refinement shows two key limitations: (1) it cannot adapt to context – for example, "getting_spare_card" needs
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="7">In-Context Learning Methods</th>
  </tr>
  <tr>
    <th>HWU64</th>
    <th>BANKING77</th>
    <th>CLINC150</th>
    <th>CUREKART</th>
    <th>POWERPLAY11</th>
    <th>SOFMATTRESS</th>
  </tr>
  <tr>
    <td><b>Llama3-8b</b></td>
    <td>88.10</td>
    <td>85.88</td>
    <td>95.03</td>
    <td>89.76</td>
    <td>70.87</td>
    <td>82.61</td>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>87.17 (+0.93)</td>
    <td>85.48 (+0.40)</td>
    <td>95.13 (+0.10)</td>
    <td>89.76 (+0.00)</td>
    <td>67.00 (+3.87)</td>
    <td>81.10 (+1.51)</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>84.10 (+4.00)</td>
    <td>81.65 (+4.23)</td>
    <td>87.60 (+7.43)</td>
    <td>85.47 (+4.29)</td>
    <td>68.61 (+1.26)</td>
    <td>83.00 (+0.39)</td>
  </tr>
  <tr>
    <td>Static</td>
    <td>89.03 (+0.93)</td>
    <td>87.95 (+2.07)</td>
    <td>95.51 (+0.48)</td>
    <td>91.94 (+2.18)</td>
    <td>76.10 (+5.23)</td>
    <td>84.60 (+1.99)</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>89.03 (+0.93)</td>
    <td>87.95 (+2.07)</td>
    <td>95.51 (+0.48)</td>
    <td>91.94 (+2.18)</td>
    <td>76.10 (+5.23)</td>
    <td>84.60 (+1.99)</td>
  </tr>
  <tr>
    <td><b>Qwen2.5-7b</b></td>
    <td>87.08</td>
    <td>85.68</td>
    <td>95.36</td>
    <td>90.02</td>
    <td>71.20</td>
    <td>83.79</td>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>86.71 (+0.37)</td>
    <td>87.05 (+1.37)</td>
    <td>95.42 (+0.06)</td>
    <td>89.98 (+0.04)</td>
    <td>70.06 (+1.14)</td>
    <td>84.46 (+0.67)</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>88.38 (+1.30)</td>
    <td>87.30 (+1.62)</td>
    <td>95.58 (+0.22)</td>
    <td>90.40 (+0.38)</td>
    <td>74.76 (+3.56)</td>
    <td>87.40 (+3.61)</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>88.38 (+1.30)</td>
    <td>87.30 (+1.62)</td>
    <td>95.58 (+0.22)</td>
    <td>90.40 (+0.38)</td>
    <td>74.76 (+3.56)</td>
    <td>87.40 (+3.61)</td>
  </tr>
  <tr>
    <td><b>Qwen2.5-1.5b</b></td>
    <td>78.90</td>
    <td>73.31</td>
    <td>82.29</td>
    <td>82.78</td>
    <td>60.84</td>
    <td>73.12</td>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>78.72 (+0.18)</td>
    <td>72.63 (+0.68)</td>
    <td>81.90 (+0.39)</td>
    <td>82.35 (+0.43)</td>
    <td>58.83 (+2.01)</td>
    <td>73.70 (+0.58)</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>74.90 (+4.00)</td>
    <td>69.18 (+4.13)</td>
    <td>83.42 (+1.13)</td>
    <td>86.27 (+3.49)</td>
    <td>55.66 (+3.18)</td>
    <td>73.91 (+0.79)</td>
  </tr>
  <tr>
    <td>Static</td>
    <td>80.76 (+1.86)</td>
    <td>77.34 (+4.03)</td>
    <td>84.02 (+1.73)</td>
    <td>84.10 (+1.32)</td>
    <td>63.10 (+2.26)</td>
    <td>80.63 (+7.51)</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>80.76 (+1.86)</td>
    <td>77.34 (+4.03)</td>
    <td>84.02 (+1.73)</td>
    <td>84.10 (+1.32)</td>
    <td>63.10 (+2.26)</td>
    <td>80.63 (+7.51)</td>
  </tr>
  <tr>
    <th colspan="7">Supervised Fine-Tuning Methods</th>
  </tr>
  <tr>
    <td>CPFT<sup>†</sup></td>
    <td>87.13</td>
    <td>87.20</td>
    <td>94.18</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ICDA<sup>†</sup></td>
    <td>87.41</td>
    <td><b>89.79</b></td>
    <td>94.84</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>QAID<sup>†</sup></td>
    <td><b>90.42</b></td>
    <td>89.41</td>
    <td><b>95.71</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DF<sup>†</sup></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>75.00</td>
    <td>59.60</td>
    <td>73.10</td>
  </tr>
  <tr>
    <td>SBERT-M<sup>†</sup></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>87.38</td>
    <td><b>64.00</b></td>
    <td><b>78.78</b></td>
  </tr>
  <tr>
    <td>SBERT-P<sup>†</sup></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td><b>88.05</b></td>
    <td>62.18</td>
    <td>75.32</td>
  </tr>
</table>

Table 1: Performance comparison of our dynamic label refinement approach across different models. <b>Baseline</b> represents the baseline performance using original intent labels, <b>CoT</b> shows results with chain-of-thought prompting, <b>Static</b> shows results with static label refinement, and <b>Ours</b> shows the results after applying our dynamic label refinement method. Results marked with † are from prior work. Bold indicates the best performance within each method category. Changes are computed relative to baseline.

different refinements ("additional_card_request" vs "duplicate_card_issuance") depending on context, and (2) it disrupts original semantic relationships between intents – while "cancel" naturally relates to "cancel_reservation", static refinements may obscure this connection. In contrast, dynamic refinement adapts to specific query context by refining labels based on retrieved similar examples, allowing the model to better capture semantic relationships specific to the current query context and adjust refinement strategy based on semantic similarity patterns, rather than using fixed refined labels. We also explore an alternative approach using generic identifiers instead of preserving original label names during refinement (detailed analysis in Appendix D).

2.4 ICL for Intent Classification

After obtaining the new labels for each sample, we leverage ICL with the refined labels and examples for final classification. This two-step process where the same LLM both refines the labels and makes the final classification decision helps ensure consistency between the refined semantic understanding and the ultimate intent prediction. Consequently, the overall process involves constructing a prompt that includes: 1) The retrieved examples with their refined intent labels 2) The test query requiring classification 3) Clear instructions for the model to select the most appropriate intent.

3 Experiment

3.1 Experimental Setup

Datasets We evaluate our method on two groups of datasets: DialoGLUE benchmark datasets (Mehri et al., 2020) (BANKING77, HWU64, CLINC150) and HINT3 datasets (Arora et al., 2020) (CUREKART, POWERPLAY11, SOFMATTRESS). For DialoGLUE datasets, we follow the standard 10-shot setting where each intent has only 10 examples for training. For HINT3 datasets, we exclude out-of-scope queries (NO_NODES_DETECTED) from evaluation as they are irrelevant to our intent classification task. Further details about datasets are provided in Appendix A.
Models  We conduct experiments with three different sizes of LLMs to evaluate the effectiveness of our approach. We employ *Llama3-8b-inst.* (Dubey et al., 2024), *Qwen2.5-7b-inst.* (Yang et al., 2024), and *Qwen2.5-1.5b-inst.* as our backbone models.

Baselines  We compare our approach with baseline methods across two categories: **In-context Learning Methods**: We compare against **Raw** (standard retrieval-based classification without refinement) and **Chain-of-Thought (CoT)** (Wei et al., 2023) (structured reasoning approach). Our method (**Ours**) applies dynamic label refinement to the retrieved examples before classification. **Supervised Fine-Tuning Methods**: We compare with state-of-the-art SFT-based approaches including **CPFT** (Zhang et al., 2021), **ICDA** (Lin et al., 2023), **QAID** (Yehudai et al., 2023) for DialogGLUE datasets, and **DF**, **SBERT-M**, **SBERT-P** (Arora et al., 2020) for HINT3 datasets.

Implementation Details  For retrieving semantically similar examples, we use a pre-trained SentenceTransformer model (Reimers, 2019). For each input query, we retrieve the top-20 most similar examples. Following (Milios et al., 2023), we order examples from least to most (Zhou et al., 2022) similar in the prompt, which demonstrated higher accuracy across our datasets. We use this retrieval-based in-context learning setup as our baseline, where the LLM directly performs classification using the original intent labels. While a static label refinement might seem simpler, we opt for dynamic refinement as it enables context-specific label adjustments based on each test query and its retrieved examples. We confirm this choice through experimental validation in Section 2.3.

3.2 Main Results

Table 1 presents a comprehensive analysis of our dynamic label refinement approach across different experimental setups. We structure the investigation around several critical dimensions to clearly demonstrate the effectiveness and implications of our method.

Semantic Disambiguation through Label Refinement  We first confirm that while both Raw and CoT approaches show reasonable performance, they often struggle with semantically ambiguous intents. For example, CoT misclassifies “*Please keep delivery service to the pin code 7021*” as ‘modify_address’ instead of ‘check_pincode’, and “*Hey I didn’t receive the ordered product it’s incomplete*” as ‘refunds_returns_replacements’ instead of ‘delay_in_parcel’. These examples show how structured reasoning often struggles when similar intents have overlapping semantics. Our refinement process tackles this issue by analyzing the semantic relationships between labels and examples, leading to consistent performance improvements over both Raw and CoT baselines across all datasets as shown in Table 1. The improvements are particularly notable in datasets with complex domain-specific terminology, such as BANKING77, where the model effectively leverages existing semantic information in the labels.

Performance Across Model Scales  Our experiments with different model sizes reveal several interesting patterns about the scalability of our approach. The larger models (Llama3-8b-inst. and Qwen2.5-7b-inst.) demonstrate robust baseline performance with moderate improvements across all datasets. Notably, even the smaller Qwen2.5-1.5b-inst. Model achieves significant improvements, particularly on domain-specific datasets (SOFMAT-TRESS, BANKING77), suggesting that our approach effectively enhances performance regardless of model scale. These results demonstrate that our proposed method effectively enhances intent classification performance across various datasets and model architectures. Additional experiments with larger models showing similar trends can be found in Appendix I.

3.3 Analysis

Semantic Similarity Analysis  To validate our hypothesis about semantic relationships between intent labels, we conduct an embedding-based similarity analysis comparing original and refined intent labels across all datasets. Specifically, we leverage the last hidden layer representations from our LLMs to capture the semantic characteristics of each intent label. For each intent, we extract the final hidden state representation and computed pairwise cosine similarities between these representations within each label set.

We observe that refined labels consistently achieve lower average pairwise similarities than original labels across all datasets and model scales, as shown in Table 2. With Llama3-8b-inst., the average similarity drops from 0.86 for original intent labels to 0.74 for refined labels. Qwen2.5-7b-
<table>
  <tr>
    <th>Model</th>
    <th>Original</th>
    <th>Refined</th>
  </tr>
  <tr>
    <td>Llama3-8b-inst.</td>
    <td>0.86</td>
    <td><b>0.74</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7b-inst.</td>
    <td>0.83</td>
    <td><b>0.80</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5b-inst.</td>
    <td>0.95</td>
    <td><b>0.91</b></td>
  </tr>
</table>

Table 2: Average pairwise semantic similarity between original and refined intent labels across various model scales.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">Q2.5-7B</th>
    <th colspan="2">Q2.5-1.5B</th>
  </tr>
  <tr>
    <th>Q2.5-7B</th>
    <th>Q2.5-1.5B</th>
    <th>Q2.5-1.5B</th>
    <th>Q2.5-7B</th>
  </tr>
  <tr>
    <td>HWU64</td>
    <td><b>88.38</b></td>
    <td>81.22</td>
    <td>80.76</td>
    <td><b>88.01</b></td>
  </tr>
  <tr>
    <td>BANKING77</td>
    <td><b>87.30</b></td>
    <td>80.81</td>
    <td>77.34</td>
    <td><b>85.95</b></td>
  </tr>
  <tr>
    <td>CLINC150</td>
    <td><b>95.58</b></td>
    <td>87.51</td>
    <td>84.02</td>
    <td><b>95.22</b></td>
  </tr>
  <tr>
    <td>CUREKART</td>
    <td><b>90.40</b></td>
    <td>82.78</td>
    <td>84.10</td>
    <td><b>88.90</b></td>
  </tr>
  <tr>
    <td>POWERPLAY11</td>
    <td><b>74.76</b></td>
    <td>64.10</td>
    <td>63.10</td>
    <td><b>74.43</b></td>
  </tr>
  <tr>
    <td>SOFMATTRESS</td>
    <td><b>87.40</b></td>
    <td>78.30</td>
    <td>80.63</td>
    <td><b>85.08</b></td>
  </tr>
</table>

Table 3: Performance comparison across different model combinations. The top row means the model used for re-naming, and the second row denotes the models used for classification. Models used: Qwen2.5-7B-inst. (Q2.5-7B) and Qwen2.5-1.5B-inst. (Q2.5-1.5B).

inst. and Qwen2.5-1.5b-inst. follow a similar trend. This decrease in semantic overlap directly correlates with improved classification performance. For instance, on the BANKING77 dataset, Llama3-8b-inst. achieves a 2.07% accuracy improvement along with a 0.1 reduction in label similarity. The performance gains become especially noticeable when the model creates more semantically distinct labels, indicating that reducing label overlap enables clearer distinctions between different intents.

Model Combination To validate the effectiveness of labeling refinement itself of each model, we employ two separate LLMs for intent refinement and classification tasks. We experiment with various combinations of Qwen2.5-7B-inst. and Qwen2.5-1.5B-inst. As in Table 3, using a larger model for refinement followed by a smaller model for classification yields the best performance. For example, using Qwen2.5-7B-inst. for refinement and Qwen2.5-1.5B-inst. for answering achieves notable improvements: +3.31% on CLINC150 and +4.10% on POWERPLAY11 compared to single-model baselines.

Interestingly, even reverse combinations (small model refinement + large model classification) show improvements over the non-refinement baseline, though to a lesser extent. For instance, using Qwen2.5-1.5B-inst. for refinement and Qwen2.5-7B-inst. for answering still achieves improvements on the datasets. This suggests that our label refinement approach is robust across different model scales and configurations. A detailed analysis of performance across different model combinations is provided in Appendix H.

4 Conclusion

We propose a dynamic label refinement method for few-shot dialogue intent classification that mitigates the issues of significant semantic overlap between intent labels. Using the retrieved examples, we refine labels via LLMs to create more semantically distinct intent categories. Experimental results demonstrate that our method consistently improves performance across multiple datasets for various models. We also confirm that our method reduces semantic similarities between intent labels, creating more distinct and interpretable categories.

Limitations

While our approach demonstrates significant improvements in intent classification performance, it requires additional computational overhead compared to traditional methods. The need to run the model once for label refinement and once for classification - increases the computational cost per query. However, we believe this trade-off is justified by the substantial improvements in classification accuracy, particularly for semantically ambiguous intents.

Acknowledgement

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [RS-2021ID211341, Artificial Intelligence Graduate School Program (Chung-Ang University)] and the Chung-Ang University Graduate Research Scholarship in 2023 and was improved by the helpful input and collaboration of researchers from LG AI Research.

References

Tassallah Abdallah, Ritambhara Singh, and Carsten Eickhoff. 2024. Retrieval augmented zero-shot text classification. In Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR ’24, page 195–203. ACM.

Gaurav Arora, Chirag Jain, Manas Chaturvedi, and Krunal Modi. 2020. HINT3: Raising the bar for intent detection in the wild. In Proceedings of the First Workshop on Insights from Negative Results in NLP,
pages 100–105. Online. Association for Computational Linguistics.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Preprint, arXiv:2005.14165.

Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45, Online. Association for Computational Linguistics.

Huiyao Chen, Yu Zhao, Zulong Chen, Mengjia Wang, Liangyue Li, Meishan Zhang, and Min Zhang. 2024. Retrieval-style in-context learning for few-shot hierarchical text classification. Preprint, arXiv:2406.17534.

Qian Chen, Zhu Zhuo, and Wen Wang. 2019. Bert for joint intent classification and slot filling. Preprint, arXiv:1902.10909.

Jaejin Cho, Rakshith Sharma Srinivasa, Ching-Hua Lee, Yashas Malur Saidutta, Chouchang Yang, Yilin Shen, and Hongxia Jin. 2024. Zero-shot intent classification using a semantic similarity aware contrastive loss and large language model. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10776–10780. IEEE.

Liesbeth Degand and Philippe Muller. 2020. Introduction to the special issue on dialogue and dialogue systems. Traitement Automatique des Langues, 61(3):7–15.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Lingyu Gao, Aditi Chaudhary, Krishna Srinivasan, Kazuma Hashimoto, Karthik Raman, and Michael Bendersky. 2024. Ambiguity-aware in-context learning with large language models. Preprint, arXiv:2309.07900.

Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.

Yen-Ting Lin, Alexandros Papangelis, Seokhwan Kim, Sungjin Lee, Devamanyu Hazarika, Mahdi Namazifar, Di Jin, Yang Liu, and Dilek Hakkani-Tur. 2023. Selective in-context data augmentation for intent detection using pointwise V-information. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1463–1476, Dubrovnik, Croatia. Association for Computational Linguistics.

Lefteris Loukas, Ilias Stogiannidis, Prodromos Malakasiotis, and Stavros Vassos. 2023. Breaking the bank with ChatGPT: Few-shot text classification for finance. In Proceedings of the Fifth Workshop on Financial Technology and Natural Language Processing and the Second Multimodal AI For Financial Forecasting, pages 74–80, Macao. -

Zhenyi Lu, Jie Tian, Wei Wei, Xiaoye Qu, Yu Cheng, Wenfeng xie, and Dangyang Chen. 2024. Mitigating boundary ambiguity and inherent bias for text classification in the era of large language models. Preprint, arXiv:2406.07001.

Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020. Dialoglue: A natural language understanding benchmark for task-oriented dialogue. Preprint, arXiv:2009.13570.

Aristides Milios, Siva Reddy, and Dzmitry Bahdanau. 2023. In-context learning for text classification with many labels. Preprint, arXiv:2309.10954.

Soham Parikh, Quaizar Vohra, Prashil Tumbade, and Mitul Tiwari. 2023. Exploring zero and few-shot techniques for intent classification. Preprint, arXiv:2305.07157.

N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.

Juan A. Rodriguez, Nicholas Botzer, David Vazquez, Christopher Pal, Marco Pedersoli, and Issam Laradji. 2024. Intentgpt: Few-shot intent discovery with large language models. Preprint, arXiv:2411.10670.

Mujeen Sung, James Gung, Elman Mansimov, Nikolaos Pappas, Raphael Shu, Salvatore Romeo, Yi Zhang, and Vittorio Castelli. 2023. Pre-training intent-aware encoders for zero- and few-shot intent classification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10433–10442, Singapore. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Asaf Yehudai, Matan Vetzler, Yosi Mass, Koren Lazar, Doron Cohen, and Boaz Carmeli. 2023. Qaid: Question answering inspired few-shot intent detection. Preprint, arXiv:2303.01593.

Haode Zhang, Haowen Liang, Liming Zhan, Albert Y. S. Lam, and Xiao-Ming Wu. 2024. Revisit few-shot intent classification with plms: Direct fine-tuning vs. continual pre-training. Preprint, arXiv:2306.05278.

Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021. Few-shot intent detection via contrastive pre-training and fine-tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1906–1912, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625.
A  Dataset Details

We evaluate our approach using two different groups of datasets. The first group consists of DialoGLUE benchmark datasets (BANKING77: 77 intents focused on banking domain (Casanueva et al., 2020), HWU64: 64 intents spanning across 21 domains (Casanueva et al., 2020), CLINC150: 150 intents covering 10 domains (Larson et al., 2019)), which are widely used for evaluating task-oriented dialogue systems. For these datasets, we follow the standard 10-shot setting where each intent class has only 10 examples for training, reflecting real-world scenarios where collecting large amounts of labeled data is challenging.

The second group includes HINT3 datasets (Arora et al., 2020) (CUREKART: 28 intents in fitness supplements retail domain, POWERPLAY11: 59 intents in online gaming domain, SOFMATTRESS: 21 intents in mattress products retail domain), which contain real user queries from live chatbots. We exclude the NO_NODES_DETECTED label from the test set as it represents out-of-scope queries irrelevant to our task.

B  Impact of the Number of Retrieved Examples

We conduct additional experiments to analyze the impact of the number of retrieved examples on model performance. Figure 3 shows the performance comparison between different model configurations across varying numbers of retrieved examples (10, 20, 30, and 40) on BANKING77 and POWERPLAY11 datasets.

From these results, we observe several key patterns:
• The performance gap between the baseline and refined versions tends to be more pronounced with larger numbers of examples, particularly in BANKING77.
• The larger model (Llama3-8b-inst.) shows more stable performance across different example counts, while the smaller model (Qwen2.5-1.5b-inst.) shows greater variance in performance.
• POWERPLAY11 shows relatively consistent improvement patterns across different example counts, suggesting that the benefits of label refinement are robust across different dataset characteristics.

These findings suggest that our label refinement approach is effective across different numbers of retrieved examples.

![Performance comparison with different numbers of retrieved examples. Solid lines represent performance with label refinement, while dashed lines represent baseline performance without refinement.](page_1012_120_377_377.png)

Figure 3: Performance comparison with different numbers of retrieved examples. Solid lines represent performance with label refinement, while dashed lines represent baseline performance without refinement.

C  Dataset-wise Similarity Analysis

To provide a detailed view of semantic similarity patterns, we analyze the pairwise similarities between intent labels for each dataset. Table 5 shows the average pairwise similarity scores for both original and refined intent labels across different datasets and models.

As shown in Table 5, the reduction in semantic similarity is consistent across all datasets for both models, though the magnitude of reduction varies. For Llama3-8b-inst., POWERPLAY11 shows the largest reduction in similarity (0.156), while CUREKART shows the smallest (0.079). Similarly, for Qwen2.5-1.5b-inst., CLINC150 shows a notable reduction (0.045) while BANKING77 shows a relatively smaller
<table>
  <tr>
    <th>Dataset</th>
    <th>Recall@20</th>
    <th>Avg. Intents</th>
  </tr>
  <tr>
    <td>HWU64</td>
    <td>97.77</td>
    <td>7.54</td>
  </tr>
  <tr>
    <td>BANKING77</td>
    <td>98.93</td>
    <td>7.04</td>
  </tr>
  <tr>
    <td>CLINC150</td>
    <td>99.33</td>
    <td>6.31</td>
  </tr>
  <tr>
    <td>CUREKART</td>
    <td>98.91</td>
    <td>4.31</td>
  </tr>
  <tr>
    <td>SOFMATTRESS</td>
    <td>98.81</td>
    <td>5.86</td>
  </tr>
  <tr>
    <td>POWERPLAY11</td>
    <td>96.44</td>
    <td>6.93</td>
  </tr>
</table>

Table 4: Retrieval analysis with 20 examples

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">Llama3-8b-inst.</th>
    <th colspan="2">Qwen2.5-7b-inst.</th>
    <th colspan="2">Qwen2.5-1.5b-inst.</th>
  </tr>
  <tr>
    <th>Original</th>
    <th>Refined</th>
    <th>Original</th>
    <th>Refined</th>
    <th>Original</th>
    <th>Refined</th>
  </tr>
  <tr>
    <td>HWU64</td>
    <td>0.835</td>
    <td>0.721</td>
    <td>0.772</td>
    <td>0.760</td>
    <td>0.948</td>
    <td>0.910</td>
  </tr>
  <tr>
    <td>BANKING77</td>
    <td>0.889</td>
    <td>0.786</td>
    <td>0.852</td>
    <td>0.838</td>
    <td>0.955</td>
    <td>0.938</td>
  </tr>
  <tr>
    <td>CLINC150</td>
    <td>0.834</td>
    <td>0.705</td>
    <td>0.780</td>
    <td>0.764</td>
    <td>0.946</td>
    <td>0.901</td>
  </tr>
  <tr>
    <td>CUREKART</td>
    <td>0.881</td>
    <td>0.802</td>
    <td>0.800</td>
    <td>0.833</td>
    <td>0.956</td>
    <td>0.925</td>
  </tr>
  <tr>
    <td>SOFMATTRESS</td>
    <td>0.849</td>
    <td>0.761</td>
    <td>0.752</td>
    <td>0.748</td>
    <td>0.943</td>
    <td>0.911</td>
  </tr>
  <tr>
    <td>POWERPLAY11</td>
    <td>0.844</td>
    <td>0.688</td>
    <td>0.838</td>
    <td>0.812</td>
    <td>0.945</td>
    <td>0.903</td>
  </tr>
  <tr>
    <td>Mean</td>
    <td>0.856</td>
    <td>0.744</td>
    <td>0.799</td>
    <td>0.793</td>
    <td>0.949</td>
    <td>0.915</td>
  </tr>
</table>

Table 5: Detailed semantic similarity analysis across datasets and models. Values represent the average pairwise cosine similarity between all intent labels within each dataset. Lower values indicate more semantically distinct intent categories.

change (0.017). These variations might reflect differences in the initial intent label structures across datasets and the models’ ability to refine them effectively.

D Semantic vs Generic Label Refinement

While our main experiments focus on semantic-aware refinement that preserves domain context, we also explore an alternative approach using generic identifiers (w/o ori). In this setting, all original intent labels are first replaced with generic identifiers (e.g., Intent_1, Intent_2) before refinement. During refinement, the model generates new labels without any influence from the original intent names.

The effectiveness of each approach varies with model size. Larger models (Llama3-8b-inst. and Qwen2.5-7b-inst.) generally perform better with original name preservation (w ori), particularly in BANKING77 where domain-specific terminology provides valuable semantic context. However, smaller models (Qwen2.5-1.5b-inst.) often show improved performance with generic identifiers (w/o ori), suggesting that they may benefit from the simplified label space. This performance pattern indicates that the choice between preserving or abstracting intent names should consider the model’s capacity to leverage domain-specific terminology.

Complete results comparing both approaches across all datasets and models are shown in Table 6.

E Full Prompt Template

E.1 Re-naming Prompt Template

The re-naming prompt takes groups of examples organized by their original intent labels and analyzes their semantic meaning. Based on this analysis, it either keeps the original label if it accurately represents the examples, or generates a new more descriptive label while maintaining consistent formatting rules. The prompt enforces lowercase letters and underscores in label names to ensure standardization.

E.2 Classification Prompt Template

The classification prompt presents example pairs of text queries and their corresponding refined intents to establish the task context. It instructs the model to determine the most likely intent for a new query based solely on these examples. The prompt explicitly requires only the intent name as output, without any additional explanation or text, to ensure consistent and clean predictions.

E.3 Chain-of-Thought Prompt Template

The Chain-of-Thought prompt extends the basic classification prompt by requiring the model to explain its reasoning process. For each query, the model must analyze the key elements, provide reasoning, and then determine the intent. This structured approach aims to help the model make more informed decisions by breaking down the classification process into steps.

F Related Work

F.1 Dialogue Intent Classification

Dialogue intent classification aims to identify users’ intentions from natural language utterances. Traditional approaches relied on supervised learning with large labeled datasets (Chen et al., 2019; Larson et al., 2019). The advent of large language models (LLMs) transforms this landscape, enabling effective few-shot learning approaches where only limited labeled data is available (Brown et al., 2020). This shift has been particularly significant as LLMs demonstrate strong few-shot capabilities through in-context learning, reducing the need for extensive labeled datasets (Loukas et al., 2023; Parikh et al., 2023; Chen et al., 2024).
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="3">Llama3-8b-inst</th>
    <th colspan="3">Qwen2.5-7b-inst</th>
    <th colspan="3">Qwen2.5 -1.5b-inst</th>
  </tr>
  <tr>
    <th>Raw</th>
    <th>Refined w/o ori</th>
    <th>Refined w ori</th>
    <th>Raw</th>
    <th>Refined w/o ori</th>
    <th>Refined w ori</th>
    <th>Raw</th>
    <th>Refined w/o ori</th>
    <th>Refined w ori</th>
  </tr>
  <tr>
    <td>HWU64</td>
    <td>88.10</td>
    <td>87.17 (+0.93)</td>
    <td><b>89.03 (+0.93)</b></td>
    <td>87.08</td>
    <td>85.97 (-1.11)</td>
    <td><b>88.38 (+1.30)</b></td>
    <td>78.90</td>
    <td>79.27 (+0.37)</td>
    <td><b>80.76 (+1.86)</b></td>
  </tr>
  <tr>
    <td>BANKING77</td>
    <td>85.88</td>
    <td>87.27 (+1.39)</td>
    <td><b>87.95 (+2.07)</b></td>
    <td>85.68</td>
    <td>86.33 (+0.65)</td>
    <td><b>87.30 (+1.62)</b></td>
    <td>73.31</td>
    <td><b>79.12 (+5.81)</b></td>
    <td>77.34 (+4.03)</td>
  </tr>
  <tr>
    <td>CLINC150</td>
    <td>95.03</td>
    <td>93.73 (-1.30)</td>
    <td><b>95.51 (+0.48)</b></td>
    <td>95.36</td>
    <td>93.73 (-1.63)</td>
    <td><b>95.58 (+0.22)</b></td>
    <td>82.29</td>
    <td><b>86.38 (+4.09)</b></td>
    <td>84.02 (+1.73)</td>
  </tr>
  <tr>
    <td>CUREKART</td>
    <td>89.76</td>
    <td><b>91.94 (+2.18)</b></td>
    <td><b>91.94 (+2.18)</b></td>
    <td>90.02</td>
    <td><b>90.40 (+0.38)</b></td>
    <td><b>90.40 (+0.38)</b></td>
    <td>82.78</td>
    <td><b>86.27 (+3.49)</b></td>
    <td>84.10 (+1.32)</td>
  </tr>
  <tr>
    <td>POWERPLAY11</td>
    <td>70.87</td>
    <td>76.05 (+5.18)</td>
    <td><b>76.10 (+5.23)</b></td>
    <td>71.20</td>
    <td>74.43 (+3.23)</td>
    <td><b>74.76 (+3.56)</b></td>
    <td>60.84</td>
    <td><b>66.99 (+6.15)</b></td>
    <td>63.10 (+2.26)</td>
  </tr>
  <tr>
    <td>SOFMATTRESS</td>
    <td>82.61</td>
    <td>83.40 (+0.79)</td>
    <td><b>85.40 (+2.79)</b></td>
    <td>83.79</td>
    <td>82.60 (-1.19)</td>
    <td><b>87.40 (+3.61)</b></td>
    <td>73.12</td>
    <td>79.44 (+6.32)</td>
    <td><b>80.63 (+7.51)</b></td>
  </tr>
</table>

Table 6: Complete performance comparison including refinement without original name preservation (Refined w/o ori). Results show that while both refinement approaches generally improve over the baseline (Raw), preserving original names during refinement (Refined w ori) tends to yield better or comparable results.

F.2 Retrieval-based In-Context Learning

Retrieval-based approaches have emerged as a powerful paradigm for improving few-shot learning performance. Key developments in this area include: (Milios et al., 2023) propose effective retrieval strategies for in-context learning with many labels, demonstrating significant performance improvements through careful example selection. However, while this approach successfully retrieves semantically similar examples for classification, it introduces new challenges when dealing with intent labels that have high semantic overlap. (Lu et al., 2024) This ambiguity between similar intents creates unnecessary complexity in the classification task, particularly when multiple intents share similar contextual meanings but require different downstream processing. Our work addresses this challenge by introducing a dynamic label refinement approach that helps distinguish between semantically similar intents while maintaining the benefits of retrieval-based example selection.

G Case Study

G.1 Label Refinement Pattern Analysis

Our analysis revealed interesting patterns in how the model refines intent labels, particularly highlighting some suboptimal refinement behaviors:

G.1.1 Verbatim Query-to-Intent Conversion

We observed cases where the model simply converted user queries directly into intent labels:

• Original_text: “I want to return my mattress”

• Refined_intent:
  i_want_to_return_my_mattress

This pattern indicates a potential limitation in our refinement approach where the model sometimes fails to abstract the core intent, instead creating overly specific labels that mirror the input text.

G.1.2 Overly Descriptive Intent Labels

Another pattern emerged where the model generated unnecessarily verbose intent labels:

• Original_intent: size_customization

• Refined_intent:
  how_can_i_order_a_custom_sized_mattress

These findings highlight the need for more sophisticated label refinement strategies that maintain a balance between descriptiveness and practical utility.

H Performance comparison across different model combinations.

Based on Table 3, we conduct additional experiments employing two separate large language models (LLMs) for intent segmentation and classification tasks in order to verify the effectiveness of label segmentation for each model. In particular, for the two models used in the experiment, we report the accuracy of both the model that generates the intents and the model that generates the responses. Consistent with the results shown in Table 3, we find that using a larger model for response generation is effective. Furthermore, we observe that label re-naming, even when performed using labels derived from a smaller model, still yields strong performance.

I Additional Experiments with Large Models

To further validate the effectiveness of our approach across different model scales, we conducted
Examples by intent:
Intent 1: account_not_verified
- How to Verify my Account?
- Account Verification
- Need to Verify my account

Intent 2: delete_pan_card
- Pan card remove
- Delete PAN card
- I want to delete my pan card

Intent 3: bank_verification_details
...
Intent 4: pan_verification_failed
...
Rules for intent mapping:
1. If the current intent name accurately represents its examples, map it to itself
2. If the intent name needs improvement, create a new descriptive name that better represents the examples
3. For new names:
- Use lowercase letters only
- Use underscores between words

INTENT MAPPINGS:
account_not_verified ->
delete_pan_card ->
...
pan_verification_failed ->

Table 7: Re-naming prompt template.

additional experiments with larger models including ChatGPT3.5-turbo-0125 and Llama3-70b-inst.

Results on six benchmark datasets are shown in Table 11 demonstrate that our label refinement approach consistently improves performance even with larger models. Key observations include:

• Both models show consistent improvements across all datasets, with gains ranging from +0.20% to +2.33%

• ChatGPT3.5-turbo-0125 shows particularly strong improvements on CUREKART (+2.33%) and POWERPLAY11 (+1.13%)

• Llama3-70b-inst. achieves notable gains on CUREKART (+1.99%) and SOFMATTRESS (+1.61%)

These results further validate that our label refinement approach is effective across different model scales.

You are an AI assistant specialized in intent classification. Your task is to determine the single most likely intent of a given query based on the examples provided.
Provide only the name of the most probable intent, without any additional text or explanation.

...

Text: "Details for Bank Account Verification"
Intent: bank_verification_details

Text: "Getting error while verifying PAN Card"
Intent: pan_verification_failure

Text: "My PAN card needs to be verified"
Intent: verify_pan_card_details

Query: "My PAN card bank account verification please"
The top 1 most likely intent is:

Table 8: Classification prompt template.

You are an AI assistant specialized in intent classification. Your task is to determine the single most likely intent of a given query based on the examples provided.
For each query:
1. Analyze the key elements and meaning
2. Provide an explanation of your reasoning
3. Extract the most likely intent

Text: "Details for Bank Account Verification"
Intent: bank_verification_details

Text: "Getting error while verifying PAN Card"
Intent: pan_verification_failure

Text: "My PAN card needs to be verified"
Intent: verify_pan_card_details

Query: "My PAN card bank account verification please"
Provide your explanation and intent:

Table 9: Chain-of-Thought prompt template.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">L3-8B</th>
    <th colspan="2">Q2.5-1.5B</th>
  </tr>
  <tr>
    <th>L3-8B</th>
    <th>Q2.5-1.5B</th>
    <th>Q2.5-1.5B</th>
    <th>L3-8B</th>
  </tr>
  <tr>
    <td>HWU64</td>
    <td><b>89.03</b></td>
    <td>80.95</td>
    <td>80.76</td>
    <td><b>88.48</b></td>
  </tr>
  <tr>
    <td>BANKING77</td>
    <td><b>87.95</b></td>
    <td>80.09</td>
    <td>77.34</td>
    <td><b>88.47</b></td>
  </tr>
  <tr>
    <td>CLINC150</td>
    <td><b>95.51</b></td>
    <td>86.82</td>
    <td>84.02</td>
    <td><b>94.79</b></td>
  </tr>
  <tr>
    <td>CUREKART</td>
    <td><b>91.94</b></td>
    <td>84.96</td>
    <td>84.10</td>
    <td><b>91.07</b></td>
  </tr>
  <tr>
    <td>POWERPLAY11</td>
    <td><b>76.10</b></td>
    <td>65.69</td>
    <td>63.10</td>
    <td><b>74.11</b></td>
  </tr>
  <tr>
    <td>SOFMATTRESS</td>
    <td><b>85.40</b></td>
    <td>79.44</td>
    <td>80.63</td>
    <td><b>85.38</b></td>
  </tr>
</table>

Table 10: Performance comparison across different model combinations. The top row shows the model used for re-naming, while the models listed in the second row are used for answering (classification).

<table>
  <tr>
    <th colspan="3">ChatGPT3.5-turbo-0125</th>
  </tr>
  <tr>
    <th>Dataset</th>
    <th>Baseline</th>
    <th>Re-naming</th>
  </tr>
  <tr>
    <td>HWU64</td>
    <td>89.48</td>
    <td>89.70 (<span style="color:red">+0.22</span>)</td>
  </tr>
  <tr>
    <td>BANKING77</td>
    <td>88.78</td>
    <td>89.34 (<span style="color:red">+0.56</span>)</td>
  </tr>
  <tr>
    <td>CLINC150</td>
    <td>96.35</td>
    <td>96.67 (<span style="color:red">+0.32</span>)</td>
  </tr>
  <tr>
    <td>CUREKART</td>
    <td>90.04</td>
    <td>92.37 (<span style="color:red">+2.33</span>)</td>
  </tr>
  <tr>
    <td>POWERPLAY11</td>
    <td>75.08</td>
    <td>76.21 (<span style="color:red">+1.13</span>)</td>
  </tr>
  <tr>
    <td>SOFMATTRESS</td>
    <td>84.18</td>
    <td>85.27 (<span style="color:red">+1.09</span>)</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">Llama3-70b-inst.</th>
  </tr>
  <tr>
    <th>Dataset</th>
    <th>Baseline</th>
    <th>Re-naming</th>
  </tr>
  <tr>
    <td>HWU64</td>
    <td>89.31</td>
    <td>89.56 (<span style="color:red">+0.25</span>)</td>
  </tr>
  <tr>
    <td>BANKING77</td>
    <td>88.31</td>
    <td>88.62 (<span style="color:red">+0.31</span>)</td>
  </tr>
  <tr>
    <td>CLINC150</td>
    <td>96.32</td>
    <td>96.52 (<span style="color:red">+0.20</span>)</td>
  </tr>
  <tr>
    <td>CUREKART</td>
    <td>90.04</td>
    <td>92.03 (<span style="color:red">+1.99</span>)</td>
  </tr>
  <tr>
    <td>POWERPLAY11</td>
    <td>73.46</td>
    <td>74.80 (<span style="color:red">+1.34</span>)</td>
  </tr>
  <tr>
    <td>SOFMATTRESS</td>
    <td>83.79</td>
    <td>85.40 (<span style="color:red">+1.61</span>)</td>
  </tr>
</table>

Table 11: Performance comparison with large language models. Numbers in parentheses show absolute improvement over baseline.

Reviewer paper summaries: The paper proposes a novel approach for few-shot dialogue intent classification. It introduces Dynamic Label Refinement (DLR) method that utilizes retrieval-based in-context learning (ICL) to identify semantically similar examples and dynamically refine intent labels using large language models (LLMs). The refined labels aim to resolve semantic ambiguities between similar intent categories, leading to improved classification accuracy. The experimental results showcase the method’s superior performance across several benchmarks, with up to 7.51% accuracy improvement on datasets characterized by high semantic overlap between intent labels.

This study introduces a novel approach to refining ambiguous intent labels through a two-stage process for the few-shot intent classification task. First, it retrieves the top 20 semantically similar intents from the training set. Then, these intents are refined using large language models (LLMs) to produce clearer intent labels. Experimental results demonstrate that the refined labels effectively reduce ambiguity in the original intent labels, enabling LLMs to classify intents with improved accuracy.
Summary of strengths: 1. The paper addresses a critical problem in intent classification—semantic overlap between similar intents—by proposing a dynamic label refinement mechanism. The solution is shown to effectively mitigate ambiguity.

2. The approach is rigorously evaluated across multiple datasets, including DialoGLUE and HINT3.

3. The refined labels are shown to be more descriptive and semantically coherent. 

4. The paper includes comprehensive analyses, such as the impact of semantic similarity reduction, performance across different model configurations, and comparisons with static label refinement approaches.

- The paper is well-written with straightforward examples and diagrams that effectively illustrate the proposed method.
- The authors demonstrates robust and consistent improvements over baselines across a wide variety of datasets.
Summary of weaknesses: 1. The paper does not discuss how the performance compares to zero-shot/few-shot instruction-tuned LLMs such as chatGPT or LLaMa-instruct-70b.

2. The paper briefly mentions, does not thoroughly explore cases where the method performs suboptimally, such as overly verbose or overly specific label refinements; potential causes or strategies to fix them.

- The authors only compares against LLM-based baselines, lacking comparisons with encoder-based baselines like SentenceBERT or IntentBERT, which could be more effeficent.
Comments, suggestions and typos: See weaknesses.

- Figure 1. cleary => clearly
Overall assessment scores: 3

4