Venue: NAACL 2025

# Learning LLM Preference over Intra-Dialogue Pairs: A Framework for

Utterance-level Understandings

 Xuanqing Liu , Luyang Kong1, Wei Niu, Afshin Khashei, Belinda Zeng,

Steve Johnson, Jon Jay, Davor Golac, Matt Pope

Amazon.com Inc.

Footnote 1: First two authors contributed equally. Corresponding author email: xuanqing@amazon.com

###### Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in handling complex dialogue tasks without requiring use case-specific fine-tuning. However, analyzing live dialogues in real-time necessitates low-latency processing systems, making it impractical to deploy models with billions of parameters due to latency constraints. As a result, practitioners often prefer smaller models with millions of parameters, trained on high-quality, human-annotated datasets. Yet, curating such datasets is both time-consuming and costly. Consequently, there is a growing need to combine the scalability of LLM-generated labels with the precision of human annotations, enabling finetuned smaller models to achieve both higher speed and accuracy comparable to larger models. In this paper, we introduce a simple yet effective framework to address this challenge. Our approach is specifically designed for per-utterance classification problems, which encompass tasks such as intent detection, dialogue state tracking, and more. To mitigate the impact of labeling errors from LLMs - the primary source of inaccuracies in student models - we propose a noise-reduced preference learning loss. Experimental results demonstrate that our method significantly improves accuracy across utterance-level dialogue tasks, including sentiment detection (over \(2\%\)), dialogue act classification (over \(1.5\%\)), etc.

## 1 Introduction

Maintaining high annotation quality, scaling the size of labeled datasets, and managing annotation budgets are three critical yet often conflicting objectives in deploying real-world ML applications. A widely adopted paradigm involves a two-stage process: unsupervised pretraining followed by supervised fine-tuning [e.g., 6, 7, 8]. This approach effectively reduces the size of the labeled dataset required because, during the pretraining phase, models learn to generate universal embeddings across various modalities. Consequently, such pretrained models are often straightforward to adapt to downstream tasks.

In dialogue understanding, moving beyond BERT-like models is essential, as dialogues possess unique characteristics compared to the BERT pretraining corpus (which primarily consists of books and web pages). These differences arise from several factors: First, dialogues involve spoken language exchanges between two or more individuals and are often structured differently, with one line per speaker. This format reduces the effectiveness of tasks such as masked token prediction and next-sentence prediction. Second, the vocabulary in daily dialogues tends to be informal. Finally, dialogues are frequently transcribed from voice recordings, introducing ASR errors and background noise. These distinctive properties have inspired research into developing specialized unsupervised pretraining algorithms for dialogue data [39, 40, 41, 34, 35]. Benchmark evaluations on common dialogue tasks - such as intent detection, next-utterance prediction, summarization, dialogue act classification, and dialogue state tracking - demonstrate the advantages of dialogue-optimized models. These models generally adhere to the classical BERT framework, pretraining on large-scale unsupervised dialogue datasets with dialogue-specific loss functions, including random mask filling, utterance swapping, and contrastive learning. However, it remains unclear whether such pretrained embedding models generalize effectively to specific downstream tasks.

To address this challenge, we require direct supervision signals that are closely aligned with downstream tasks. This motivates the use of in

struction fine-tuned LLMs as phase-2 supervision signals, while retaining traditional unsupervised pretraining as phase-1. However, simply employing LLMs as data labelers and fine-tuning a student model using traditional cross-entropy loss proves suboptimal. The accuracy of LLM-generated labels can be unpredictable, influenced by factors such as the quality of the LLM, the prompting strategy, and the inherent difficulty of the dialogue task. Consequently, the knowledge transferred from the LLM to the student model often deviates from the intended objective. This paper proposes an alternative approach based on preference learning, where pairs of chunks sampled from the same dialogue session (_intra-session pairs_) are labeled by ensembled LLMs. Under reasonable assumption on LLM labeling errors, our method outperforms traditional training algorithms in both data efficiency and generalizability.

## 2 Related work

### Task-oriented dialogue (TOD) system

Task-oriented dialogue understanding lies in the core of building AI assistants to be deployed in domain specific scenarios such as restaurant booking, self-service product troubleshooting, and so on. The objective is to help users achieve their goals in limited turns by understanding users' needs, tracking dialogue states and figure out next best action. Unique to TOD system, intent detection, dialogue act classification, and dialogue state tracking are three critical components of the system. Traditional approaches mostly rely on supervised learning on embedding models LiuLane (2016), by encoding dialogue contexts and employing deep neural networks such as RNN/LSTM or Transformers to infer utterance labels or slot values Barriere et al. (2022); Duran (2021); Chen et al. (2020). In the LLM age, there is a shift from finetuning TOD model for a specific domain Lei et al. (2018) to open domain in-context learning Hu et al. (2022); Arora et al. (2024). Unfortunately, both solutions ignored latency and cost constraints in real-time, commercial products.

### Synthetic label prompting strategies and transfer learning

These two techniques are the foundation of our solution. We discuss the main idea and prior works. **Prompting strategies**. It is often non-trivial prompting LLMs to achieve quality high data labeling. For example, prior work Anagnostidis and Bulian (2024); WorkLu et al. (2021) noticed that few-shot prompting is surprisingly sensitive to factors including the number of example, order of examples, positive / negative sample ratio, or how similar those examples are to the actual input query. In this regard, fine-tuning embedding models on human curated labels are still preferred in production-ready applications. To strengthen the robustness of ICL, a promising solution is through diversified prompting Li et al. (2023); Song et al. (2024), either by starting with a few seeding prompts, and augment more versions using automated pipeline Wang et al. (2022), or repetitvely refine the prompt from diverse perspectives Li et al. (2023).

**Transfer learning**. For better instruction following ability, a popular approach is fine-tuning on synthetic datasets produced by larger LLMs Taori et al. (2023); Chiang et al. (2023); Xu et al. (2023). To foster LLM's reasoning ability, another line of work finetune with synthetic rationales collected from stronger LLMs Wang et al. (2022); Shridhar et al. (2023); Liu et al. (2023); Kang et al. (2024). Similar approach work for task-specific applications too, examples like dialogue generation Xu et al. (2023), information extraction Josifossi et al. (2023); Jeronymo et al. (2023) and code generation Chaudhary (2023); Roziere et al. (2023). Our work focus on per-utterance multi-class classification in TOD system, assuming that even the most capable LLMs can't generate highly accurate labels, so a brand new transfer learning approach is required.

## 3 Proposed framework

### Problem scope

We limit our scope to per-utterance classification, including sentiment detection, dialogue state tracking, dialogue act classification (Fig. 1).

**Intent detection**. Each utterance is mapped to a binary label has_intent (\(y=1\)) or no_intent (\(y=0\)). Positive label means utterance deemed a valid intent (e.g. a question, issue, or complaint). Take customer support for example, we could apply intent detection model to monitor customer speech in real time and figure out whether a customer is seeking for help rather than chit-chatting.

**Dialogue act classification**. We could regard this as an extension of intent detection from binary intent labels to multi-class acts. The objective of 

dialogue act classification is finding out the functions that utterances serve in dialogues - such as commitments, questions, requests, replies, etc. In contact centers, for example, classifying dialogue acts can be valuable at providing appropriate and thoughtful responses to clients adhering to the dialogue acts.

**Dialogue state tracking (DST)**. The objective of DST is extracting and picking up new information into dialogue state as the conversation evolves. This task has great potential in customer service as it not only provides intent types (e.g. _hotel-booking_ in Fig. 0(c)), but also identifies relevant semantic concepts throughout the slot filling process (e.g. _location = San Francisco_).

**Challenge.** When delivering real world applications driven by per-utterance classifiers, the challenges often rooted from obtaining high quality labels. For example, MultiWOZ Budzianowski et al. (2018) is commonly used for benchmarking DST algorithms. Yet the original dataset contains numerous labeling errors, and it took \(4\) future versions Eric et al. (2019); Zang et al. (2020); Han et al. (2021); Ye et al. (2021) (MultiWOZ 2.1-2.4) to correct them. More importantly, we learned that a clean dataset not only ensures us precisely tracking the progress on good valid/test set, but also reduces the reliance on robust model training algorithms Ye et al. (2022). The challenge of labeling leads us to focus on following question -

_Can we design a general solution for performance classification problems, by jointly utilizing small amount of clean, human verified labels and almost unlimited amount of lower quality LLM annotations?_

We share a positive answer in the remainder of this work. Our work is not a simple extension of weakly supervised learning or noise-robust supervised learning, as we utilize characteristics that are unique to per-utterance classifications.

### Workflow

Our workflow involves four stages. Goal of stage 1 is to construct a _prompt bank_ containing diversified prompts that performs well on data annotation work following prompt tuning strategies outlined in Schulhoff et al. (2024); Brown et al. (2020); Wei et al. (2022); Yao et al. (2023); Liu et al. (2021). Predictions led by various prompts are slightly different, we ensemble the outputs together for better results Khalifa et al. (2023); Jiang et al. (2021). Next, we further strengthen the ensemble effect at stage 2 using top-\(K\)/top-\(P\) sampling. After repeated sampling \(N\) times using LLM labeler, we compute \(L\)-dimensional score vector \(S\in[0,1]^{L}\) for dialogue \(\mathcal{D}\) containing \(L\) utterances. Each element \(0\leq S_{i}\leq 1\) is the ratio of positive LLM labels divided by \(N\) (e.g. if \(3\) in \(10\) ensembles labeled \(i\)-th utterance as positive, \(S_{i}=0.3\)). For \(C\)-class classification problem, we transform it into \(C\) one-versus-rest binary classification problems so the same framework still apply.

After we collect LLM labeling scores \(S\), we split a dialogue into multiple segments using a sliding window of stride \(1\). We denote \(x_{i}\) as the \(i\)-th seg

Figure 1: Illustrative examples of intent detection, dialogue act classification, and dialogue state tracking problems.

 

ment covering \(u_{1}\) to \(u_{i}\). Finally in stage 4, we randomly sample two _intra-session_ segments \(x_{i}\) and \(x_{j}\) from the same dialogue and train a student model \(f\) minimizing pair-wise ranking loss:

\[\ell(x_{i},x_{j})=\text{KL}\big{(}\mathbb{I}_{y_{i}\blacktriangleright y_{j}}\parallel \Pr(x_{i}\blacktriangleright x_{j})\big{)}, \tag{1}\]

where \(\mathbb{I}_{y_{i}\blacktriangleright y_{j}}=1\) iff. \(y_{i}=1\) and \(y_{j}=0\) for binary labels; \(\Pr(x_{i}\blacktriangleright x_{j})\) is the probability of \(x_{i}\) being more positive than \(x_{j}\), modeled by network \(f\) under an adaptive margin:

\[\Pr(x_{i}\blacktriangleright x_{j})=\sigma\big{(}\Delta_{i,j}f-\alpha\cdot \Delta_{i,j}S\big{)}, \tag{2}\]

where \(\sigma\) is the Sigmoid function, \(\Delta_{i,j}f=f(x_{i})-f(x_{j})\) is the difference of model predicted scores and \(\Delta_{i,j}S=S_{i}-S_{j}\) is the difference of LLM predicted scores between segment \(i\) and \(j\); \(\alpha\in[0,1]\) is a tunable hyper-parameter controlling margin. We train a student network \(f\) over intra-session pairs to ensure: for any positive+negative pair labeled by LLM (positive \(x_{i}\)_vs._ negative \(x_{j}\)), the student network \(f\) has the same preference as teacher LLM under margin \(\alpha\cdot\Delta_{i,j}S\). This idea made two hidden assumptions: First assuming the LLM score \(S\) is a good estimator of ground-truth correctness probability (_aka._ confidence calibrated Guo et al. (2017)); secondly, single LLM labeler may be biased and high variance, their difference within same dialogue session \(S_{i}-S_{j}\) carries dramatically lower bias and variance due to the differentiation. Therefore estimation error of \(S_{i}-S_{j}\) is more precise than \(S_{i}\) or \(S_{j}\) alone. We discuss and verify two assumptions in the following sections.

### Stage 1-2: How well are LLM scores calibrated to accuracy?

A desirable property of LLM teacher is confidence scores \(S\) calibrated to labeling accuracy, i.e. we expect higher true-positive rate if LLM score \(S_{i}\) closes to one; and near zero true-positive rate if \(S_{i}\) is closer to zero:

\[\Pr(y_{i}=1|S_{i})=S_{i}. \tag{3}\]

If Eq. (3) is true, we could replace ground truth label \(y_{i}\) with soft label \(S_{i}\) without incurring additional gradient bias and variance (see Appendix F for a proof). In addition, Eq. (3) implies monotonicity relationship:

\[S_{i}>S_{j}\Longrightarrow\Pr(y_{i}=1)>\Pr(y_{j}=1). \tag{4}\]

Guo et al. (2017) showed that DNNs are uncalibrated, in that their accuracy falls behind confidence score (DNNs are over-confident). Same findings are reported in LLM world Kapoor et al. (2024); Huang et al. (2024). Among various post-training solutions to calibrate DNNs (e.g. Zadrozny and Elkan (2001); Mozafari et al. (2018)), one simple and effective technique is ensemble different models Lakshminarayanan et al. (2017) which integrates well with our workflow. Remaining question to be answered in this work is -

_Does the same ensemble technique work for LLM predictions? If so, how many ensemble predictions we need to calibrate the scores?_

We design following experiment to answer this question: We sample an intent detection dataset containing around \(600\) transcripts and binary

Figure 2: Overview of our framework to train a small student model using noisy LLM supervision.

 

has_intent/no_intent per-utterance labels. A labeling prompt optimized for Claude3-sonnet1 for this task is provided in Appendix E. We apply the same prompt to ensemble sizes \(n\) between \(1\) and \(30\). In each setting, we run LLM labeling on each input pair \(\langle x_{i},x_{j}\rangle\) for \(n\) times and obtain scores \(S_{i}\) and \(S_{j}\) by averaging LLM predictions. Lastly, we partition the data by value \(S_{i}\) into five buckets: \(S_{i}\in(0.0,0.2]\), \((0.2,0.4]\), \((0.4,0.6]\), \((0.6,0.8]\), \((0.8,1.0]\). Within backet, we compute the percentage of positive ground-truth labels. We apply ECE loss, the standard metric to measure DNN calibration error Guo et al. (2017):

Footnote 1: Available at Anthropic and AWS Bedrock.

\[\text{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{N}\Big{|}\text{acc}(B_{m})-\text{ conf}(B_{m})\Big{|} \tag{5}\]

where \(B_{m}\) is the \(m\)-th bucket partitioned by \(S_{i}\). \(\text{acc}(B_{m})=\text{Pr}(y_{i}=1|s_{i}\in B_{m})\) is the accuracy of \(B_{m}\); and \(\text{conf}(B_{m})\) is the overall confidence score in \(B_{m}\). Due to Eq. (3) lower ECE metric means better calibration.

Despite some random fluctuations, we could observe in Fig. 3 a decline in ECE loss (\(0.22~{}\diagdowndown 0.17\)) as ensemble size increases.

The ensemble technique in Stage 1-2 effectively calibrates LLM scores \(S_{i}\) by introducing fewer gradient biases and variances. Therefore LLM teacher supervisions are good surrogate for ground-truth labels.

### Stage 3-4: Overcoming distribution shifts by intra-session comparison

We generate ranking pairs in a novel way: we sample two chunks for ranking from the same conversation (_intra-session pairs_), instead of different conversations. We make two hypothesis (\(H_{1}\) and \(H_{2}\)) explaining why intra-session pairs are more powerful.

\(H_{1}\)**: Intra-session pairs are harder**. Two chunks sampled from same dialogue are similar in the context (sharing the same topic with overlapping context). As a result, it is harder to tell which chunk is positive label against the other. Once training a student model on top of hard pairs, it forces the model to learn more discriminative textual features from text input, rather than just replying on some keywords. Those intra-session pairs lead to better generalization.

\(H_{2}\)**: LLM labeling errors are canceled by the differentiator**. This hypothesis is more conceptually involved: LLM labeling errors are not uniformly random across all data, instead they cluster on certain type of transcripts. For example, some scenarios are not mentioned in the labeling prompt so LLM has to guess, resulting in more errors in such cases. Fortunately, this type of error typically condensed to certain dialogues, equivalent to a "shifting" effect to the label distribution. By sampling a pair (\(x_{i}\) and \(x_{j}\)) from the same dialogue, their corresponding LLM scores (\(S_{i}\) and \(S_{j}\)) are drifted to roughly the same extent. In the end, the margin of the loss function (1) \(\Delta_{ij}S=S_{i}-S_{j}\) still accurately tracking ground-truth label difference \(y_{i}-y_{j}\).

We design an experiment to validate \(H_{2}\) on two groups: the control group consists of pairs sampled from different dialogues; experimental group consists of pairs sampled from same dialogue. The goal is checking correlation between \(\Delta_{ij}S=S_{i}-S_{j}\) with the probability of \(y_{i}=1\) and

Figure 4: Comparing the correlations between LLM score difference (also the margin of training loss) _w.r.t._ the probability of one label is more positive than the other. We also include linear fittings to both groups.

Figure 3: Visualizing the downward trend of ECE loss as ensemble size increases from 1 to 30.

 

\(y_{j}=0\) (\(y_{i}>y_{j}\) in binary case). We follow the same bucketizing method as previous experiment (5 buckets). We count the percent of \(y_{i}>y_{j}\) cases in each bucket and each group. Result in Fig. 4 shows the ground-truth probability of \(y_{i}>y_{j}\) more sensitive to \(\Delta_{ij}S\) in experimental group than control group. Meaning that our intra-session pairs are indeed less noisy, and a better approximation of golden supervision signal \(y_{i}-y_{j}\).

## 4 Experiments

**Datasets.** We benchmark our method on three important tasks in task-oriented dialogues (TOD): intent/sentiment-detection, dialogue act classification, and dialogue state tracking. We benchmark intent/sentiment detection on MELD Poria et al. (2019) and SILICONE Busso et al. (2017); benchmark dialogue act classification on daily-dialog Li et al. (2017), MRDA Shriberg et al. (2004), BT-OASIS Duran (2021) and dyda_da Chapuis et al. (2020); benchmark dialogue state tracking on SGD Rastogi et al. (2020) and MultiWOZ-2.2 Zang et al. (2020). We put statistics and other details of datasets in Appendix A.

**Baselines.** We want to see how the accuracy change after plugging our workflow into some strong models. We select following baselines accordingly:

* _Claude3-Sonnet_: We pick this model as a strong baseline for measuring LLM annotator performance.
* _FnCTOD_Li et al. (2024): A recent prompting strategy achieving strong results on dialogue state tracking task.
* _ToD-BERT_Wu et al. (2020): A strong baseline for dialogue pretrained small embedding model. This is also the backbone model of our method.
* _FLAN-TS_Chung et al. (2024): T5-XXL fine-tuned on large-scale instructions data including multiWOZ. We include this model as a natural baseline for fine-tuned LLM on TOD datasets.

We summarize features of all baselines with our method in Table 6 of Appendix B.

### Comparing pairwise preference learning vs. pointwise knowledge transfer

To evaluate the transition from pointwise model distillation to pairwise preference learning, we compare the intent detection accuracy of the ToD-BERT model fine-tuned using three approaches: 1) fine-tuning directly on human-labeled data; 2) supervised pretraining with pointwise LLM-generated labels followed by fine-tuning on human-labeled data; and 3) supervised pretraining with pairwise LLM-generated LDA-generated labels followed by fine-tuning on human-labeled data. To assess the impact of data scaling, we vary the sampling ratios during evaluation. Table 1 consistently shows that models leveraging pairwise supervised pretraining outperform the alternatives, particularly in low-data regimes.

### Sentiment detection

Next we benchmark our method with baselines on two sentiment detection datasets. We report classification accuracy over all sentiments defined in each datasets. The results are shown in Table 2. Comparing with ToD-BERT (finetuned directly on human labeled data) and FnCTOD (finetuned on LLM synthetic data), our approach (supervised pretrained on LLM synthetic data using pairwise loss then finetuned on human labeled data) performs better than baselines by around \(2\%\) to \(8\%\).

### Dialogue act classification

Similarly, we benchmark our method against baselines on dialogue act classification problem. Note we adopted the same backbone model as ToD-BERT, and ToD-BERT is still the strongest baseline in this task. Our model out-performed ToD-BERT by around \(1.5\%\) to \(10\%\).

\begin{table}
\begin{tabular}{l|c c c c c} \hline Approach\(\frac{\%}{}\)gold labels & \(0\%\) & \(1\%\) & \(5\%\) & \(10\%\) & \(25\%\) \\ \hline Finetune-only & - & \(27.3\) & \(29.5\) & \(34.7\) & \(69.6\) \\ \hline \multicolumn{6}{c}{_Supervised pretrain \(\rightarrow\) Finetune_} \\ Pointwise pretrain & - & \(31.8\) & \(33.4\) & \(47.2\) & \(77.3\) \\ Pairwise pretrain & - & \(38.4\) & \(45.8\) & \(52.1\) & \(78.4\) \\ \hline \end{tabular}
\end{table}
Table 1: Effective of our approach under various amount of labeled data.

\begin{table}
\begin{tabular}{l|r r r r r} \hline Datasets & Claude & FnCTOD & ToD-BERT & FLAN-T5 & **Ours** \\ \hline MELD & 74.25 & 68.84 & 80.30 & 75.72 & 88.09 \\ IEMOCAP & 76.39 & 61.30 & 87.88 & 82.62 & 90.31 \\ \hline \end{tabular}
\end{table}
Table 2: Benchmarking intent/sentiment detection task.

\begin{table}
\begin{tabular}{l|c c c c c} \hline Datasets & Claude & FrCTOD & ToD-BERT & FLAN-T5 & **Ours** \\ \hline DailyDiaglo & 70.39 & 66.03 & 72.40 & 68.08 & 76.50 \\ MRDA & 62.82 & 81.93 & 88.4 & 60.47 & 89.95 \\ dyda\_da & 71.25 & 74.82 & 79.14 & 68.66 & 85.11 \\ BT-Oasis & 32.85 & 52.76 & 59.24 & 17.13 & 69.62 \\ \hline \end{tabular}
\end{table}
Table 3: Benchmarking dialogue act classification task.

 



### Dialogue state tracking

Finally, we benchmark on two dialogue state tracking (DST) datasets, SGD and MultiWOZ-2.1. In this experiment we benchmark the accuracy of joint prediction of slot/domain/values (aka. **Joint-Acc**). The results are shown in Figure 4.

## 5 Discussion and future work

This paper presents a novel approach to minimizing human effort in labeling high-quality data for a class of per-utterance classification problems. Our method moves beyond traditional LLM labeling and knowledge transfer to student models by leveraging a preference learning and pairwise ranking framework. This framework has been demonstrated to be both theoretically and empirically robust against LLM labeling errors. An intriguing future direction would be to extend this approach to reward model training in reinforcement learning with human feedback (RLHF), another critical domain characterized by noisy labels and the need for robust discriminative model training.

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline Datasets & Claude & FuCTOD & ToD-BERT & FLAN-T5 & **Ours** \\ \hline SGD & 60.7 & 63.9 & 42.5 & – & 47.3 \\ MutiWOE & 27.0 & 37.9 & 16.4 & – & 25.5 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Benchmarking dialogue state tracking task.





## References

* A. Anagnostidis and J. Bulian (2024)How susceptible are llms to influence in prompts?. arXiv preprint arXiv:2408.11865. Cited by: SS3, SS4.1.
* G. Arora, S. Jain, and S. Merugu (2024)Intent detection in the age of lms.. arXiv preprint arXiv:2410.01627. Cited by: SS1.
* V. Barriere, S. Essid, and C. Clavel (2022)Opinions in interactions : new annotations of the SEMAINE database. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France, pp. 7049-7055. External Links: Link Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. Cited by: SS1.
* P. Budzianowski, T. Wen, B. Tseng, I. Casanueva, S. Ultes, O. Ramadan, and M. Gasic (2018)MultiwoZ-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278. Cited by: SS1.
* C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. Chang, S. Lee, and S. Narayanan IEMOCAP. Interactive emotional dyadic motion capture databse. (English). Note: DOI: https://doi. org/10.1007/s10579-008-9076-6, pp. 335-359 External Links: Link Cited by: SS1.
* E. Chapuis, P. Colombo, M. Manica, M. Labeau, and C. Clavel (2020)Hierarchical pre-training for sequence labelling in spoken dialog. In Findings of Association for Computational Linguistics: EMNLP 2020, Online, pp. 2636-2648. External Links: Link Cited by: SS1.
* S. Chaudhary (2023)Code alpaca: an instruction-following llama model for code generation. Code adapacn: an instruction-following llama model for code generation. Cited by: SS1.
* T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020)A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. Cited by: SS1.
* W. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. (2023)Vicuna: an open-source chatbot impressing gpt-4 with 90%* chatgtp quality. See https://vicuna. Imsys. org (accessed 14 April 2023), pp. 6. Cited by: SS1.
* H. Won Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. (2024)Scaling instruction-finetuned language models. Journal of Machine Learning Research25 (70), pp. 1-53. Cited by: SS1.
* J. Devlin (2018)Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Cited by: SS1.
* N. Duran (2021)Bt-oasis corpus. Cited by: SS1.
* M. Eric, R. Goel, S. Paul, A. Kumar, A. Sethi, P. Ku, A. K. Goyal, S. Agarwal, S. Gao, and D. Hakkani-Tur (2019)MultiWOZ 2.1: a consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669. Cited by: SS1.
* C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017)On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70, pp. 1321-1330. Cited by: SS1.
* T. Han, X. Liu, R. Takanabu, Y. Lian, C. Huang, D. Wan, W. Peng, and M. Huang (2021)MultiWOZ 2.3: a multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation. In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13-17, 2021, Proceedings, Part II 10, pp. 206-218. Cited by: SS1.
* K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick (2020)Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738. Cited by: SS1.
* Y. Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and M. Ostendorf (2022)In-context learning for few-shot dialogue state tracking. arXiv preprint arXiv:2203.08568. Cited by: SS1.
* Y. Huang, Y. Liu, R. Thirukovalluru, A. Cohan, and B. Dhingra (2024)Calibrating long-form generations from large language models. arXiv preprint arXiv:2402.06544. Cited by: SS1.
* V. Jeronymo, L. Bonifacio, H. Abonizio, M. Fadaee, R. Lottofoy, J. Zavrel, and R. Nogueira (2023)Impars-v2: large language models as efficient dataset generators for information retrieval. arXiv preprint arXiv:2301.01820. Cited by: SS1.
* Z. Jiang, J. Araki, H. Ding, and G. Neubig (2021)How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics9, pp. 962-977. Cited by: SS1.



Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. 2024. Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks. _Advances in Neural Information Processing Systems_, 36.
* Kapoor et al. (2024) Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. 2024. Calibration-tuning: Teaching large language models to know what they don't know. In _Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertainNLP 2024)_, pages 1-14.
* Khalifa et al. (2023) Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023. Exploring demonstration ensembling for in-context learning. _Preprint_, arXiv:2308.08780.
* Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. _Advances in neural information processing systems_, 30.
* Lei et al. (2018) Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1437-1447.
* Li et al. (2023a) Dawei Li, Yaxuan Li, Dheeraj Mekala, Shuyao Li, Xueqi Wang, William Hogan, Jingbo Shang, et al. 2023a. Daily: Data augmentation for in-context learning via self-paraphrase. _arXiv preprint arXiv:2311.03319_.
* Li et al. (2023b) Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2023b. Self-alignment with instruction backtranslation. In _The Twelfth International Conference on Learning Representations_.
* Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. _arXiv preprint arXiv:1710.03957_.
* Li et al. (2024) Zekun Li, Zhiyu Zoey Chen, Mike Ross, Patrick Huber, Seungwhan Moon, Zhaojiang Lin, Xin Luna Dong, Adithya Sagar, Xifeng Yan, and Paul A Crook. 2024. Large language models as zero-shot dialogue state tracker through function calling. _arXiv preprint arXiv:2402.10466_.
* Liu and Lane (2016) Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. _arXiv preprint arXiv:1609.01454_.
* Liu et al. (2022) Che Liu, Rui Wang, Junfeng Jiang, Yongbin Li, and Fei Huang. 2022. Dial2vec: Self-guided contrastive learning of unsupervised dialogue embeddings. _arXiv preprint arXiv:2210.15332_.
* Liu et al. (2023) Hanming Liu, Zhiyang Teng, Leyang Cui, Chaoli Zhang, Qiji Zhou, and Yue Zhang. 2023. Logicot: Logical chain-of-thought instruction tuning. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
* Liu et al. (2021) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-\(3\)? _arXiv preprint arXiv:2101.06804_.
* Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. _arXiv preprint arXiv:2104.08786_.
* Mehri et al. (2019)Shikib Mehri, Evgenia Razumovskaaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. _arXiv preprint arXiv:1906.00414_.
* Mozafari et al. (2018) Azadeh Sadat Mozafari, Hugo Siqueira Gomes, Wilson Leao, Steeven Janny, and Christian Gagne. 2018. Attended temperature scaling: a practical approach for calibrating deep neural networks. _arXiv preprint arXiv:1810.11586_.
* Poria et al. (2019) Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 527-536, Florence, Italy. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of machine learning research_, 21(140):1-67.
* Ratogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghay Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 8689-8696.
* Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, et al. 2023. Code llama: Open foundation models for code. _arXiv preprint arXiv:2308.12950_.
* Schulhoff et al. (2024)Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yiheng Li, Aayush Gupta, HyJoung Han, Sevien Schulhoff, et al. 2024. The prompt report: A systematic survey of prompting techniques. _arXiv preprint arXiv:2406.06608_.
* Shriberg et al. (2004) Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The icsi meeting 

recorder dialog act (mrda) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pp. 97-100. Cited by: SS3.
* K. Shridhar, A. Stolfo, and M. Sachan (2023)Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 7059-7073. External Links: Link, Document Cited by: SS1, SS2.
* F. Song, B. Yu, H. Lang, H. Yu, F. Huang, H. Wang, and Y. Li (2024)Scaling data diversity for fine-tuning language models in human alignment. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRECCOLING 2024), pp. 14358-14369. External Links: Link, Document Cited by: SS1, SS2.
* F. Song, B. Yu, M. Li, H. Yu, F. Huang, Y. Li, and H. Wang (2024)Reference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, pp. 18990-18998. External Links: Link, Document Cited by: SS1, SS2.
* R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto (2023)Stanford alpaca: an instruction-following llama model. External Links: Link Cited by: SS1, SS2.
* P. Wang, A. Chan, F. Ilievski, M. Chen, and X. Ren (2022)Pinto: faithful language reasoning using prompt-generated rationales. In The Eleventh International Conference on Learning Representations, Cited by: SS1, SS2.
* X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou (2022)Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, Cited by: SS1, SS2.
* J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. (2022)Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, pp. 24824-24837. External Links: Link, Document Cited by: SS1, SS2.
* W. Makes InContext Learning Work, Rethinking the role of demonstrations: What makes in-context learning work?
* Chien-Sheng Wu, S. Hoi, R. Socher, and C. Xiong (2020)Tod-bert: pre-trained natural language understanding for task-oriented dialogue. arXiv preprint arXiv:2004.06871. External Links: Link, 2004.06871 Cited by: SS1, SS2.
* C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang (2023)Wizardlm: empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244. External Links: Link, 2304.12244 Cited by: SS1, SS2.
* C. Xu, D. Guo, N. Duan, and J. McAuley (2023)Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6268-6278. External Links: Link, 2004.06871 Cited by: SS1, SS2.
* S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan (2023)Tree of thoughts: deliberate problem solving with large language models. PreprintarXiv:2305.10601. External Links: Link, 2305.10601 Cited by: SS1, SS2.
* F. Ye, Y. Feng, and E. Yilmaz (2022)Assist: towards label noise-robust dialogue state tracking. arXiv preprint arXiv:2202.13024. External Links: Link, 2202.13024 Cited by: SS1, SS2.
* F. Ye, J. Manotumruksa, and E. Yilmaz (2021)Multiwoz 2.4: a multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. arXiv preprint arXiv:2104.00773. External Links: Link, 2104.00773 Cited by: SS1, SS2.
* B. Zadrozny and C. Elkan (2001)Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, Vol. 1, pp. 609-616. External Links: Link, Document Cited by: SS1, SS2.
* X. Zang, A. Rastogi, S. Sunkara, R. Gupta, J. Zhang, and J. Chen (2020)Muliwoz 2.2: a dialogue dataset with additional annotation corrections and state tracking baselines. arXiv preprint arXiv:2007.12720. External Links: Link, 2007.12720 Cited by: SS1, SS2.
* M. Zhong, Y. Liu, Y. Xu, C. Zhu, and M. Zeng (2022)DialogLM: pre-trained model for long dialogue understanding and summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, pp. 11765-11773. External Links: Link, 2007.12720 Cited by: SS1, SS2.
* Zhi. Zhou, D. Zhang, W. Xiao, N. Dingwall, X. Ma, A. O. Arnold, and B. Xiang (2022)Learning dialogue representations from consecutive utterances. arXiv preprint arXiv:2205.13568. External Links: Link, 2205.13568 Cited by: SS1, SS2.

 



## Appendix A Summary statistics of experiment datasets

## Appendix B Comparing features of baseline models and our method

## Appendix C Sample prompts for Claude

Prompt for daily-dialogue:

```
Dialogue:{dialogue} Lastutterance:{last_utterance} What'sthebestdialogueactofthelastutterance? Choosefrombelowwithoutfurtherexplain: Options: A.Inform B.Question C.Directive D.Commissive E.Noneofabove
```

D,orE Donotoutputanythingelse. Prompt for MRDA:

```
Dialogue:{dialogue} Lastutterance:{last_utterance} What'sthebestdialogueactofthelastutterance?Choosefrombelowwithoutfurtherexplain: Options: A.Statementorsubjectivestatement B.Declarativequestion C.BackChannel D.Follow-me E.Question Avalidoutpathouldbeconeof:A,B,C,D,orE Donotoutputanythingelse. Prompt for MELD:
```
#TaskDescription Inthistaskyowillreceivesabort dialogue.Yourgoalsistocreadthewholedialogue,understandthemestintendeachutterances,andpickouttheutteranceswithpositiveensentiment.
###Outputformat Youneedtcopyeachpositivesentiment utterancestoanjsonarraytogether withtheinitiallinenumber.
##Example Input:
1[Phoebe]OhmyGod,he'slostit.He'stotallylostit.2[Monica]What?
3[Ross]Or!Or,wecouldgotothebank,closeouraccountsandcutthemoffatthesource.
4[Chandler]You'reagenius!5[Joyey]Aww,man,nowewwemon'tbedankbodies!6[Chandler]Now,there'stworeasons.7[Phoebe]Hey.8[All]Hy!9[Phoebe]Ohhthy,youguys,rememberthatcuteclientItoldyouabout?Ibithim.10[Rachel]Where?!11[Phoebe]Onethetouchy. Correctoutput:"'json (positive_utterances":[  "4[Chandler]You'reagenius",  "8[All]hy!") ]

\begin{table}
\begin{tabular}{p{56.9pt} p{56.9pt} p{56.9pt} p{56.9pt}} \hline \hline Methods & TOD finetuned? & LLM distilled & Small size \\ \hline Claude & (unknown) & ✗ & ✗ \\ FnCTOD & ✗ & ✗ & ✗ \\ ToD-BERT & ✓ & ✗ & ✗ \\ FLAN-T5 & ✓ & ✗ & ✗ \\ Ours & ✗ & ✗ & ✗ \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparing baselines and our method along three dimension: TOD finetuned means whether the model is finetuned for TOD tasks; LLM distilled indicates the model is distilled from (imperfect) LLM synthetic labels; Small size means whether the actual inference model is small footprint.

\begin{table}
\begin{tabular}{p{56.9pt}|p{56.9pt} p{56.9pt} p{56.9pt}} \hline \hline Data & \#Classes & \#Plalogues & \#Ultereances \\ \hline _Intent/Sentiment detection_ & & & \\ MELD & \(3\) & \(1,400\) & \(13,000\) \\ IEMOCAP & \(6\) & \(151\) & \(10,039\) \\ \hline _Diadogue act classification_ & & & \\ DailyDialog & \(5\) & \(13,118\) & \(103,630\) \\ MRDA & \(5\) & \(75\) & \(108,202\) \\ dyda\_da & \(4\) & \(87170\) & \(102,000\) \\ BT-Oasis & \(42\) & \(636\) & \(15,067\) \\ \hline _Diadogue state tracking_ & & & \\ SGD & \(53\) (slots) & \(16,142\) & \(329,964\) \\ MultiWOZ-2.1 & \(24\) (slots) & \(8,438\) & \(42,190\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Datasets for each evaluation task and some statistics.





## Appendix D Sample prompts for FLAN-T5

Prompt for daily-dialogue:

```
Dialogue:{dialogue}
``` Lastutterance:{last_utterance} ``` What'sthebestdialogueactofthelastutterance? Options: A.Inform B.Question C.Directive D.Commissive E.Noneofabove PromptforMRDA: Dialogue: {dialogue} Lastutterance: {last_utterance} What'sthebestdialogueactofthelastutterance?Choosefrombelowwithoutfurtherexplain: Options: A.StatementorsubjectivestatementB.Declarativequestion C.BackchannelD.Follow-meE.Question Answer: Prompt forMELD: Dialogue: {dialogue} Lastutterance: {last_utterance} Isthelastutteranceinpositive sentiment?Choose"Yes"or"No".

## Appendix E Intent detection labeling prompt

#Taskdescription Youaregivenaconsursonationbetweenuserandassistant.Typically,theuserhassomequestions/issues/complaints.Yourgoalisttofindouttheutterancecontainingtheuserintent.
#Data description Eachlineoftheconversationcorrespondstoanutterance.Youcanseethespeakerfromaccordingtothesbeginningofaechline.foreexample: As AssistantHii,mynameis[PII],thankyoufordcalling[COMPANY]. [User]Hi,I'mcallingbecauseshippmentarriveddamagedandIneedareplacement. [Assistant]Ise,I'smorytobearyourbadexperienceaboutshippment. """ Heretheuserintentisis"Hi,I'mcallingbecausethemisppmentarriveddamagedandIneedareplacement. Nowitisyourtun,readtheconversationthoroughlyandfindoutallintent

Finally, we plug Eq. (10) into Eq. (8):

\[\begin{split}&\mathbb{E}\left[\frac{\partial}{\partial\theta}L( \theta;(x_{i},\hat{y}_{i}))\right]\\ &=\big{(}f_{\theta}(x_{i})-\mathbb{E}[\hat{y}_{i}]\big{)}x_{i}\\ &=\big{(}f_{\theta}(x_{i})-\mathbb{E}[y_{i}]\big{)}x_{i}\\ &\mathbb{E}\left[\frac{\partial}{\partial\theta}L(\theta;(x_{i}, y_{i}))\right].\end{split} \tag{11}\]

Therefore we have proved that well-calibrated training dataset \(\{x_{i},\bar{y}_{i}\}\) is unbiased training of the model.