Venue: NAACL 2024

# Effective and Efficient Conversation Retrieval for

Dialogue State Tracking with Implicit Text Summaries

Seanie Lee\({}^{\dagger+}\) Jianpeng Cheng\({}^{\ddagger}\) Joris Driesen\({}^{\ddagger}\) Alexandru Coca\({}^{\clubsuit\) Anders Johannsen\({}^{\ddagger}\)

\({}^{\dagger}\)KAIST \({}^{\ddagger}\)Apple \({}^{\clubsuit}\)University of Cambridge

\({}^{\dagger}\)lsnffamily02@kaist.ac.kr

\({}^{\ddagger}\){jianpeng.cheng, joris_driesen, ajohannsen}@apple.com ac2123@cam.ac.uk

###### Abstract

Few-shot dialogue state tracking (DST) with Large Language Models (LLM) relies on an effective and efficient conversation retriever to find similar in-context examples for prompt learning. Previous works use raw dialogue context as search keys and queries, and a retrieval is fine-tuned with annotated dialogues to achieve superior performance. However, the approach is less suited for scaling to new domains or new annotation languages, where fine-tuning data is unavailable. To address this problem, we handle the task of conversation retrieval based on text summaries of the conversations. A LLM-based conversation summarizer is adopted for query and key generation, which enables effective maximum inner product search. To avoid the extra inference cost brought by LLM-based conversation summarization, we further distill a light-weight conversation encoder which produces query embeddings without decoding summaries for test conversations. We validate our retrieval approach on MultiWOZ datasets with GPT-Neo-2.7B and LLaMA-7B/3OB. The experimental results show a significant improvement over relevant baselines in few-shot DST settings.

+
Footnote †: Work done during an internship at Apple.

_Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 96–111 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Dialogue state tracking (DST) is one of the most crucial components in task-oriented dialogue systems. The goal of DST is to track users' intents, slots and values at every turn of a dialogue based on a predefined schema Budzianowski et al. (2018). The challenge of training a supervised DST model lies in the cost of dialogue state annotations, which is not scalable to new schemas, domains or annotation languages. To address these challenges, recent works Hu et al. (2022); Chen et al. (2023) adopt in-context learning with pre-trained large language models (LLM) for few-shot DST. In the few-shot setting, similar dialogue exemplars are retrieved based on the test sample and then these exemplars are added to the LLM prompt for target generation. This approach is attractive since no domain-specific fine-tuning is required for the LLM but it can still generalize to unseen domains.

One challenge in few-shot DST is how to retrieve salient conversation exemplars (e.g., in a set of 3 to 5) from the support set, which serves as demonstrations for the LLM. Ideally, a retrieved exemplar should carry both the same dialogue history and state change as the test sample. However, in a practical few-shot setting (e.g., with at most 100 annotated support examples), it is likely that no exemplar in the support set satisfies the above requirement. Consider a test example with two user turns:

user:book a flight to London Heathrow system: where are you departing from user: Amsterdam It is possible that the closest exemplar we can get from the support set is:

user: I'm leaving Manchester by air system: where are you flying to user: To Paris which neither matches the test dialogue state nor the state change. Nevertheless, we hope that LLM can generalize by learning from such exemplars with an identical user intent. The retrieval task gets harder when the conversation becomes lengthy with only partial history related to the current user input. For example, in another test dialog:

user: what's the weather in London system: sunny user: book a flight to London Heathrow system: where are you departuring from user: Amsterdam the user's current intent is identical to the earlier test sample, but it involves unrelated history. Still we want to match the test sample to a similar exemplar, which reflects the user's intent up to the current point of conversation. This retrieval cannot be easily accomplished with pre-trained dense

retrieval models based on word or sentence similarity. To optimize retrieval performance, previous works (Hu et al., 2022; Chen et al., 2023) fine-tune a dense retriever with "structurally similar" dialogue examples identified from dialogue state annotations with heuristics. Hu et al. (2022) additionally report that including dialogue state information in the retrieval key is helpful. However, the approach is not scalable to a practical few-shot setting (with fewer than 100 annotated support examples), as fine-tuning easily leads to overfitting and catastrophic forgetting McCloskey and Cohen (1989); Lee et al. (2022). It is also impractical to expect every domain owner to create their own fine-tuning data with well-engineered rules.

In this work, we propose a new solution for conversation retrieval starting with the introduction of a LLM-based conversation summarizer. For each exemplar to be indexed and also each test dialog, the summarizer produces a text summarizing _what the user wants at this point of the conversation_. In Section 2, we provide a discussion of this specific summarization choice and how it compares to dialogue state. The summaries are then used as condensed search keys and queries applicable to pre-trained dense retrievers with standard nearest neighbor search. We empirically show that in the few-shot setting, using summaries as retrieval keys and queries is more effective than using raw dialogues.

Notably the conversation summarization task described above can be easily handled by state-of-the-art LLMs via prompt learning, as we will show in an ablation study. However, the deployment of such a retrieval system also introduces extra model parameters and inference cost. Unlike search keys, which can be pre-built offline, a search query needs to be auto-regressively decoded for each test dialogue right during inference. To improve the efficiency of this conversation retriever, our second contribution in this work focuses on distilling a light-weight conversation encoder which embeds a raw dialogue directly into a vector space similar to the embedding of its summary. The light-weight conversation encoder enables efficient conversation search over a vector database without explicit query generation. When evaluated on the MultiWOZ dataset with GPT-Neo-2.7B (Black et al., 2022), LLaMA-7B, and LLaMA-30B (Touvron et al., 2023) for few-shot DST, we find that the distilled conversation encoder is not only more efficient, but also more effective than a cascaded conversation retriever with explicit query generation. Our approach also significantly outperforms relevant baselines, which use annotated dialogues for retriever fine-tuning.

## 2 Conversation Retrieval with Summaries for LLM-based DST

In the context of task-oriented dialogues with multiple turns of interactions between a user and a system, the objective of DST is to predict the accumulated intents, slots and values at each user turn. In a LLM-based approach, the generation of a dialogue state is conditioned on a task-specific prompt. The prompt includes at least the test conversation and a set of \(k\) demonstration examples, from which we expect the LLM to learn to generalize. Considering the size limit of the prompt, \(k\) is expected to be small (3-5 examples). Each of the retrieved examples is an annotated conversation sharing similar features as the test conversation. We expect to retrieve these exemplars with a dense retriever from a "support set" (e.g., 100 annotations) that can be constructed with minimum effort for domain scaling.

There are two major challenges of conversation retrieval for LLM-based DST described above. First, a good representation of search keys and queries need to be found. As we analyze in Section 1, the similarity of two dialogues is not directly quantifiable by semantic distance, but rather requires more sophisticated structural matching mechanism or a higher-order similarity function. This requirement leads to the second challenge as to how to train an effective conversation retriever that can scale across domains. Previous works (Hu et al., 2022; Chen et al., 2023) mainly fine-tune pre-trained dense retrievers with annotated dialogues obtained from the support set. However, fine-tuning is not realistic in a few-shot setting and for every domain.

As shown in Figure 1-(a), our work introduces a query/key generation step in the LLM-based DST. The generation is performed with another LLM which transforms the raw dialogue context into a text summary whose similarity can be evaluated more easily with pretrained retrievers. Specifically, the text summary represents the user's intent up to the current point of the conversation. It grounds the latest user input onto the dialogue history, keeping only information related to the 

current user intent. Note that the summary is a contextual rewriting of the current user intent that is possibly expressed in multiple turns, with applied ellipsis recovery (Hardt, 1997) and co-reference resolution Pradhan et al. (2012). Examples of the conversation summary are in Table 6. The summary can also be viewed as a text description of an updated dialogue state which is to be predicted by the LLM. Unlike the dialogue state, the summary does not maintain all conversation history but only includes information relevant to the current user input.

With the introduction of an explicit query/key generation step, we expect that the conversation retrieval becomes easier and the search index can be built more efficiently. To construct the search index, an offline process can be triggered to generate text summaries for every example in the support set. Note that search key generation does not add any inference cost. However, the query generation step comes at an extra cost since the generation needs to happen in an online process. In the next section, we describe how to make the conversation retrieval more efficient by stepping away from explicit query generation.

## 3 Conversation Encoder Distillation

Note that in the proposed conversation retriever, the LLM-based conversation summarizer needs to be invoked for every test sample to generate the search query as shown in Figure 1-(a). To eliminate the extra inference cost, we propose to distill a light-weight conversation encoder which directly embeds a dialogue into a vector space similar to its summary, by maximizing their embedding similarity. The encoder is trained with large-scale dialogue-summary pairs generated by the conversation summarize in an offline process. After training the model, as shown in Figure 1-(b), we can directly encode each dialogue into a query embedding for maximum inner product search. We call our conversation encoder CONVERSE, standing for **CON**versation embeddings for **VE**rsatile **R**etrieval with implicit **S**ummari**E**s. Next we explain the structure and training objective of CONVERSE.

### Model

PreliminariesIn our problem setup, we are given a set of unlabeled conversations between a user and system, denoted as \({\mathcal{D}^{u}=\{\mathbf{x}_{i}\}_{i=1}^{n}}\) where each conversation \(\mathbf{x}_{i}\) consists of \(l_{i}\) utterances \((\mathbf{u}_{i,j},\dots,\mathbf{u}_{i,l_{i}})\) and each utterance \(\mathbf{u}_{i,j}\) is a sequence of \(T_{i,j}\) tokens \((x_{i,j,1},\dots,x_{i,j,T_{i,j}})\). As shown in Figure 2-(b), the training data of CONVERSE is prepared by invoking the conversation summarizer to generate a summary for each conversation \(\mathbf{x}_{i}\), denoted as \(\mathbf{z}_{i}\), which consists of \(T_{i}^{\prime}\) tokens with \(T_{i}^{\prime}\ll\sum_{j=1}^{l_{i}}T_{i,j}\). We denote the dataset augmented with summaries as \({\mathcal{D}^{a}=\{(\mathbf{x}_{i},\mathbf{z}_{i})\}_{i=1}^{n}}\). For brevity, we omit the first subscript \(i\) if there is no ambiguity. Given the set of conversational-summary pairs, the goal is to train an encoder \(f_{\theta}:\mathcal{V}^{T}\rightarrow\mathbb{R}^{T\times d}\) such that the similarity between a conversation and its summary is maximized, where \(\mathcal{V}\) denotes a set of predefined tokens.

Conversation and Summary EmbeddingTo match a conversation against a summary, we leverage the commonly used architecture in dense retrieval known as the dual encoder Yih et al. (2011); Lee et al. (2019); Karpukhin et al. (2020), where a conversation and a summary are encoded jointly for similarity comparison. State-of-the-art dual encoders Khattab and Zaharia (2020) represent each encoding as multiple vectors, typically the contextualized token vectors, to represent the text. These models largely improve the model expressiveness, and exhibit much stronger performance and robustness compared to their single-vector counterparts Thakur et al. (2021). Based on it, we represent both the conversation embedding \(f_{\theta}(\mathbf{x})\) and the summary embedding \(f_{\theta}(\mathbf{z})\) as a matrix. While the summary encoder in the dual architecture can be directly integrated into off-the-shelf sentence encoders, our conversation encoder (CONVERSE) is designed to reflect the inductive bias of the summarization task.

ConverseRemember that the task of the conversation summarizer is to summarize the current user intent by grounding it to the conversation history. Hence the latest user input (the state delta) is most important and any past utterances irrelevant to the latest input should be dropped out.

Figure 1: Comparison between **(a)** off-the-shelf retriever with query generation and **(b)** CONVERSE w/o query generation.

 

To reflect the nature of the summarization task, we explicitly model the grounding step between the latest user input \({\bf u}_{l}\) and past utterances \({\bf u}_{1},\ldots,{\bf u}_{l-1}\) as a structural bias in CONVERSE. This is achieved with the introduction of a soft retrieval structure that softly retrieves past utterances or tokens which are relevant to the latest user input. Specifically, the soft retrieval is simulated with another neural network \(g_{\phi}:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow[0,1]\), which outputs the relevance score of each token in the utterances \({\bf u}_{1},\ldots,{\bf u}_{l-1}\) conditioned on the latest user utterance \({\bf u}_{l}\). Then, the relevance scores are used to downweight irrelevant token representations of the conversation \({\bf x}\):

\[\hat{f}_{\theta,\phi}({\bf x})=\begin{bmatrix}w_{1,1}f_{\theta}({ \bf x})_{1,1}^{\top}\\ \vdots\\ w_{l-1,T_{l-1}}f_{\theta}({\bf x})_{l-1,T_{l-1}}^{\top}\\ f_{\theta}({\bf x})_{l,1}^{\top}\\ \vdots\\ f_{\theta}({\bf x})_{l,T_{l}}^{\top}\end{bmatrix}\in\mathbb{R}^{T\times d}\] \[w_{j,t}=g_{\phi}(f_{\theta}({\bf x})_{j,t},s_{l}({\bf x}))\in[0 ,1] \tag{1}\] \[s_{l}({\bf x})=\frac{1}{T_{l}}\sum_{t=1}^{T_{l}}f_{\theta}({\bf x })_{l,t}\in\mathbb{R}^{d},\]

where \(t\in\{1,\dots,T_{j}\}\) for each \(j\in\{1,\dots,l-1\}\) and \(f_{\theta}({\bf x})_{j,t}\) is a contextual representation of \(t\)-th token in \({\bf u}_{j}\). Intuitively, an irrelevant token in the conversation history receives a small weight, reducing its contribution to the final similarity scoring against the summary. Conversely, a token in the latest user input always carries the highest weight 1 and contributes more to the similarity computation.

### Training Objective

Given the conversation encoder \(\hat{f}_{\theta,\phi}\) and summary encoder \(f_{\theta}\) with a set of conversation and summary pairs \(\mathcal{D}^{a}=\{({\bf x}_{i},{\bf z}_{i})\}_{i=1}^{n}\), as illustrated in Figure 2-(b), we train the dual encoder to maximize the similarity between a dialogue and its summary with the contrastive loss Henderson et al. (2017):

\[L(\theta,\phi;\mathcal{D}^{a})=-\frac{1}{n}\sum_{({\bf x},{\bf z })\in\mathcal{D}^{a}}\log p_{\theta,\phi}({\bf z}|{\bf x}) \tag{2}\] \[p_{\theta,\phi}({\bf z}|{\bf x})=\frac{\exp(\texttt{sim}(\hat{ f}_{\theta,\phi}({\bf x}),f_{\theta}({\bf z})))}{\underset{({\bf x}^{\prime},{\bf z }^{\prime})\in\mathcal{D}^{a}}{\exp(\sum\texttt{sim}(\hat{f}_{\theta,\phi}({ \bf x}^{\prime}),f_{\theta}({\bf z}^{\prime})))}},\]

where sim is the multi-vector similarity function Khattab and Zaharia (2020), which computes the similarity between the conversation and its summary, denoted as \(\texttt{sim}(\hat{f}_{\theta,\phi}({\bf x}),f_{\theta}({\bf z}))\), by averaging maximum dot product between summary tokens and each conversation token as:

\[\frac{1}{T}\sum_{j=1}^{l}\sum_{t=1}^{T_{j}}\max_{t^{\prime}\in\{1,\dots,T^{ \prime}\}}\hat{f}_{\theta,\phi}({\bf x})_{j,t}^{\top}f_{\theta}({\bf z})_{t^{ \prime}}. \tag{3}\]

In practice, due to computational costs, we sample a mini-batch \(\mathcal{B}\subset\mathcal{D}^{a}\) for computing the denominator of the contrastive loss in equation 2.

### Inference

In LLM-based DST, we are given a small support set of labeled dialogues \(\mathcal{D}^{s}_{1}=\{({\bf x}^{s}_{i},{\bf y}^{s}_{i})\}_{i=1}^{m}\). The search keys can be pre-built offline by calling the conversation summarizer to generate a summary for each dialogue \({\bf x}^{s}_{i}\) from the support set \(\mathcal{D}^{s}_{1}\), resulting in a set of (conversation, label, and summary) triplets denoted as \(\mathcal{D}^{s}_{2}=\{({\bf x}^{s}_{i},{\bf y}^{s}_{i},{\bf z}^{s}_{i})\}_{i=1}^ {m}\). The search index is then built with the summary as the

Figure 2: **Concept.****(a) Generating a summary of a dialogue with language model (LM). (b) Training the retriever to maximize a similarity between the dialogue and generated summary. (c) Given a test dialogue as a query, we retrieve the dialogue (value) of which summary (key) obtains the best similarity score with the query.**



key, and a labeled conversation as the value. The summaries are encoded with the fine-tuned summary encoder described in Section 3.2.

During inference, for each conversation \({\bf x}_{i}^{q}\) from the test set \(\mathcal{D}^{q}=\left\{{\mathbf{x}_{i}^{q}}\right\}_{i=1}^{Q}\), we embed the conversation with the CONVERSE encoder and compute its similarity with every search key using the similarity function in equation 3, i.e., \(\mathtt{sim}(\hat{f}_{\theta,\phi}({\bf x}_{i}^{q}),f_{\theta}({\bf z}_{j}^{s}))\) for \(j=1,\ldots,m\). As shown in Figure 2-(c), the retriever ranks examples \(({\bf x}_{j}^{s},{\bf y}_{j}^{s})\) based on the similarity score and chooses the top-\(k\) exemplars. Finally, the retrieved exemplars are added to the prompt of the downstream LLM for dialog state generation.

## 4 Experiments

### Experimental Setup

CommonWe evaluate LLM-based DST with the proposed conversation retriever on MultiWOZ 2.1 Eric et al. (2020) and 2.4 Ye et al. (2022).

To simulate few-shot scenario, we consider a support set of 100 labeled conversations as the default setting in our comparison. For each experimental run, we randomly sample 100 labeled conversations from the training data of MultiWOZ 2.1/2.4. The analysis of other support set sizes is deferred to an ablation study. During inference, we retrieve the top 5 examples from the support set. The examples along with a test conversation are inserted into the prompt, following Hu et al. (2022). This setting is applied to all comparisons. We use both GPT-Neo Black et al. (2022) and LLaMA-7B/3OB Touvron et al. (2023) as the LLM for DST generation. For evaluation, we report average and standard deviation of Joint Goal Accuracy (JGA) and F1 score Henderson et al. (2014) on all 7,368 test dialogues from MultiWOZ with three runs.

BaselinesWe compare the proposed conversation retriever with the following baselines.

1. IC-DST Hu et al. (2022): It utilizes dialogue labels to construct positive and negative pairs for fine-tuning a pretrained SBERT Reimers and Gurevych (2019) or LinkBERT Yasunaga et al. (2022) as a retriever. The retrieval key is a dialogue context, and the best dialogue context is reported to be previous dialogue state + current user input (which is better than a full dialogue).
2. SM2 Chen et al. (2023): Similar to IC-DST, it fine-tunes SBERT on labeled dialogue data with contrastive loss, where conversations with partial matching slots or values are considered as positive samples. The retrieval key is a dialogue context similar to IC-DST.
3. GTR-T5-Large Ni et al. (2022): It uses a T5 encoder, which is pretrained on large scale corpora for sentence representation, to compute the similarity between conversations for retrieving examples. The retrieval key is the full dialogue.
4. Jina-Large Gunther et al. (2023): Similar to GTR-T5, the pretrained sentence encoder Jina is used to compute similarity between conversations. The retrieval key is also the full dialogue.

OursWe use gpt-3.5-turbo OpenAI (2022) as the conversation summarizer, since it provides reliable summaries that satisfy the task requirement in the prompt (see human evaluation in 4.4 and the prompt specified in Appendix A). First, we evaluate the effectiveness of summary-based search key and query generation, using off-the-shelf retrievers GTR-T5-Large and Jina-Large, which are directly comparable with the baseline.

Second, we evaluate the distilled conversation encoder (CONVERSE). To train CONVERSE, we use the same conversation summarizer to generate a summary for every turn of every conversation from the full MultiWOZ training set, resulting in a total of 56,776 conversation-summary pairs. The parameters \(\theta\) of the dual encoder \(f_{\theta}\) and \(\hat{f}_{\theta,\phi}\) are shared and initialized with LinkBERT Yasunaga et al. (2022), and trained on the conversation-summary pairs for 20 epochs with the objective in equation 2. LinkBERT Yasunaga et al. (2022) is chosen since we empirically find that it offers the best general-purpose weight of initialization. We use the AdamW optimizer Loshchilov and Hutter (2018) with learning rate \(5\cdot 10^{-5}\) and batch size 200. We use eight A100 GPUs for training the model.

### Quantitative Results

Main ResultsThe DST results are shown in Table 1 and Table 2. The first set of comparisons is between conversation retrieval with and without explicit query/key generation. We observe that using the summary as search keys/queries significantly improves the end-to-end (E2E) results, when evaluated with the same off-the-shelf retriever (GTR-T5 or Jina). The result is slightly behind IC-DST which fine-tunes the retriever with dialogue state information in the key. However, after introducing the distilled CONVERSE model, we achieve much 

better E2E results than all baselines. Although our motivation of distilling a conversation encoder is to reduce the inference cost, it turns out that the light-weight model is also helpful in E2E performance. We hypothesize that the improved performance brought by CONVERSE is attributed to two factors. The first and foremost is that we leverage a dual encoder architecture to optimize the matching between conversation-summary pairs. This suggests that the retrieval component is optimized for the task-specific keys and values. A secondary explanation of the performance gain is that the conversation encoder avoids error propagation in explicit summary decoding and re-embedding. It should be noted that the above findings are consistent across different datasets (MultiWOZ 2.1/2.4) and language models (GPT-Neo and LLaMA-7B/30B).

Comparison Against Few-Shot FinetuningRecently, Mosbach et al. (2023) have shown that few-shot fine-tuning outperforms in-context learning in some settings, which makes people wonder how few-shot fine-tuning behaves with 100 labeled dialogues for dialogue state tracking tasks. To answer this question, we compare our method, CONVERSE, against one of the strongest few-shot finentuning methods, DS2 Shin et al. (2022), using BART Lewis et al. (2020) and T5 Raffel et al. (2020) language models. As shown in Table 3, our method, CONVERSE, outperforms the few-shot fine-tuning method, DST. Note that the BART-based DS2 severely overfits to the small labeled dataset and the T5-based model performs worse than the in-context learning method, even though T5 model is pretrained on an additional large-scale labeled dialogue summarization dataset, SAMSum Gliwa et al. (2019).

Out-of Domain GeneralizationTo verify our hypothesis that our unsupervised retriever CONVERSE generalizes better to unseen domain than supervised methods, we hold out the hotel domain from the MultiWOZ dataset and train the retrievers, IC-DST and CONVERSE on the remaining four domains: train, restaurant, taxi, and attraction. Then we evaluate the performance of the few-shot in-context learning with the retrieivers on test examples from the unseen domain, hotel. As shown in Table 4, our model CONVERSE outperforms IC-DST by a large margin, which empirically validates that our unsupervised retriever generalizes better to unseen domain than the supervised one.

### Ablation Study

Size of Support SetWe empirically study the size of the support set (labeled dialogues) in the conversation retrieval task. Notably, a smaller support set requires less annotation effort from the domain owner, placing more emphasis on general

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**MultiWOZ 2.1**} & \multicolumn{2}{c}{**MultiWOZ 2.4**} \\ \cline{2-5} Model & JGA & F1 & JGA & F1 \\ \hline \multicolumn{5}{c}{ILMaM-30B [Touvron et al., 2023]} \\ \hline IC-DST (SBERT) & \(25.41_{+1.83}\) & \(77.82_{+2.16}\) & \(26.01_{+2.27}\) & \(79.01_{+2.50}\) \\ SM2 & \(22.86_{+3.13}\) & \(74.73_{+3.19}\) & \(23.46_{+1.80}\) & \(75.78_{+2.41}\) \\ GTR-TS & \(25.10_{+0.31}\) & \(68.42_{+1.31}\) & \(19.94_{+2.40}\) & \(68.90_{+2.22}\) \\ Jia & \(22.51_{+0.92}\) & \(72.31_{+1.61}\) & \(22.42_{+1.58}\) & \(72.95_{+0.83}\) \\ \hline Sum. + GFR-T5 & \(26.06_{+0.47}\) & \(78.55_{+0.54}\) & \(26.75_{+0.83}\) & \(78.55_{+0.54}\) \\ Sum. + Jina & \(25.10_{+0.17}\) & \(78.07_{+0.54}\) & \(25.81_{+1.45}\) & \(78.98_{+0.86}\) \\ \hline
**CONVERSE** & **27.35**\({}^{+0.27}\) & **79.75**\({}^{+0.55}_{+0.55}\) & **28.23**\({}^{+1.54}\) & **80.45**\({}^{+0.55}_{+0.55}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: JGA and F1 of LLaMA-30B with 100 labeled conversations.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline  & \multicolumn{2}{c}{**MultiWOZ 2.1**} & \multicolumn{2}{c}{**MultiWOZ 2.4**} \\ \cline{2-5} Model & JGA & F1 & JGA & F1 \\ \hline \multicolumn{5}{c}{GFt-Neeo 2.7B [Black et al., 2022]} \\ \hline IC-DST (SBERT) & \(6.76_{+0.57}\) & \(42.91_{+1.27}\) & \(6.81_{+1.15}\) & \(43.42_{+2.13}\) \\ IC-DST (Lin=BERT) & \(6.39_{+1.27}\) & \(40.11_{+1.56}\) & \(6.35_{+1.54}\) & \(40.78_{+2.54}\) \\ SMB & \(4.54_{+0.27}\) & \(35.15_{+0.53}\) & \(5.33_{+0.56}\) & \(35.03_{+0.52}\) \\ GTR-TS & \(4.77_{+0.56}\) & \(28.58_{+0.79}\) & \(4.66_{+0.37}\) & \(32.80_{+0.54}\) \\ Jina & \(5.11_{+0.18}\) & \(30.93_{+1.29}\) & \(5.16_{+0.40}\) & \(30.84_{+1.33}\) \\ \hline Sum. + GFR-T5 & \(6.16_{+0.54}\) & \(40.60_{+1.23}\) & \(6.01_{+0.40}\) & \(40.40_{+2.34}\) \\ Sum. + Jina & \(6.09_{+0.51}\) & \(40.48_{+2.62}\) & \(6.13_{+0.77}\) & \(40.84_{+2.53}\) \\ \hline
**CONVERSE** & **8.07**\({}^{+0.03}\) & **44.11**\({}^{+1.2}\) & **7.85**\({}^{+0.03}_{+0.03}\) & **49.2**\({}^{+1.6}\) \\ \hline \hline \multicolumn{5}{c}{LLaMA-7B [Touvron et al., 2023]} \\ \hline IC-DST (SBERT) & \(18.30_{+0.03}\) & \(69.51_{+1.38}\) & \(18.57_{+2.75}\) & \(70.37_{+2.53}\) \\ IC-DST (LinkBERT) & \(18.09_{+0.03}\) & \(69.41_{+1.58}\) & \(18.97_{+0.53}\) & \(70.29_{+0.53}\) \\ SM2 & \(15.23_{+1.18}\) & \(64.36_{+2.32}\) & \(15.01_{+1.17}\) & \(65.12_{+1.28}\) \\ GT-T5 & \(13.64_{+1.45}\) & \(57.95_{+0.55}\) & \(13.61_{+0.50}\) & \(58.26_{+0.54}\) \\ Jina & \(15.58_{+0.58}\) & \(68.09_{+0.51}\) & \(15.01_{+1.63}\) & \(61.48_{+0.54}\) \\ \hline Sum. + GFR-T5 & \(17.54_{+0.54}\) & \(68.36_{+0.54}\) & \(17.74_{+0.56}\) & \(69.14_{+0.57}\) \\ Sum. + Jina & \(17.85_{+0.54}\) & \(68.70_{+0.54}\) & \(18.37_{+0.53}\) & \(69.65_{+0.54}\) \\ \hline
**CONVERSE** & **19.33**\({}^{+0.03}\) & **71.48**\({}^{+1.03}_{+1.03}\) & **20.35**\({}^{+1.03}_{+1.03}\) & **72.45**\({}^{+1.02}_{+1.52}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: JGA and F1 using labeled 100 conversations with GPT-Neo-2.7B and LLaMA-7B.

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & \multicolumn{2}{c}{**JQA**} \\ \hline Model & MWZ-2.1 & MWZ-2.4 \\ \hline DS2 + BART-Large & \(7.60_{+2.17}\) & \(5.86_{+4.52}\) \\ DS2 + T5-Large & \(17.71_{+1.84}\) & \(19.08_{+1.23}\) \\ \hline
**CONVERSE** + LLaMA-7B & \(19.33_{+0.91}\) & \(20.35_{+1.03}\) \\
**CONVERSE** + LLaMA-30B & **27.35**\({}^{+0.77}\) & **28.23**\({}^{+1.58}_{+1.58}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison against few-shot finentuning methods.

 

ization to unseen dialogue structures. In contrast, a larger support set contradicts the fundamental motivation behind few-shot learning, but it is likely to improve the E2E accuracy, as more test dialogue structures are observable from the exemplars. In Figure 3, we plot the JGA of LLaMA-7B with CONVERSE on varying sizes of the support set constructed from MultiWOZ. As we expect, the JGA increases as the number of labeled conversation increases, even though we do not fine-tune the retriever with any labeled conversations.

Summary vs. state deltaThe conversation summary we adopt in this work concludes the user's current intent when the dialogue takes place. A limitation is that the summary does not directly highlight the state delta carried by the latest user input. As a remedy, we consider a multi-key and query retrieval setup, where we use both the summary and the latest user input as search keys and queries. More specifically, we first retrieve 20 dialogues with CONVERSE and re-rank the 20 dialogues based on the similarity of the latest utterance between the test sample and the support examples, using the pre-trained GTR-T5-Large. As shown in Table 5, re-ranking with the latest user utterance yields marginal performance gains. In future work, we aim to explore a better way of summarizing the conversation structure that reflects both the joint intent and the latest user input.

### Qualitative Results

Visualization of history groundingAs described in equation 1, CONVERSE softly retrieves conversation history based on the latest user utterance. Specifically, the network \(g_{\phi}\) outputs a relevance score between 0 and 1 for each token of the conversation history. In Figure 4, we visualize this relevance score of each token in the history. The tokens with darker blue color indicates a higher weight, which are considered to be more relevant to the latest input.

The examples in Figure 4 shows that the model successfully focuses on relevant part of history. For the first example in Figure 3(a), the user wants to search for a Chinese restaurant in the center with moderate price range. The model assigns large weights to the tokens related to "Chinese", "center", and "price". Similarly, the tokens relevant to booking a taxi gets larger weights in Figure 3(b). For the last example in Figure 3(c), the model pays attention to the tokens related to a museum and ignores many irrelevant ones.

Human Evaluation on Conversation SummarizerThe success of CONVERSE is highly dependent on the output quality of the conversation summarizer, which are used as labels for encoder distillation. We conduct human evaluation of 135 summaries generated by the conversation summarizer, namely gpt-3.5-turbo. Specifically, three human judges are asked to assess whether the generated summaries are consistent with the instructions in the prompt in Table 9. The results indicate that 90.3% of the 135 summaries are deemed consistent with the given prompt.

Examples of the generated summaries are shown in Table 6 and 7. For the first example, the model generates the summary about booking a taxi. It is

Figure 3: JGA of LLaMA-7B with CONVERSE as a function of the number of labeled data.

\begin{table}
\begin{tabular}{l l} \hline \hline  & **JGA** \\ \hline Model & **MultiWOZ 2.1** & **MultiWOZ 2.4** \\ \hline
**CONVERSE + Rerank** & \(19.86\pm 1.22\) & \(20.65\pm 1.28\) \\
**CONVERSE** & \(19.33\pm 0.91\) & \(20.35\pm 1.03\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablations of different ranking with LLaMA-7B.

 

noteworthy that the model focuses on the latest user utterance while disregarding previous user requests for hotel and restaurant reservation. For the second example, the model misses out on the arrival time for generating the summary. Identification and correction of such errors are topics we will explore in future work. We include more examples in Appendix B.

Retrieved ExemplarsIn Table 8, we show the top three most similar examples retrieved by CONVERSE. In this example, the user asks to find an expensive Indian restaurant and a retriever needs to retrieve conversations about a restaurant. Indeed, our CONVERSE retriever assigns high similarity scores to pairs of the target conversation and summaries about finding a restaurant. Note that the language model (LLaMA-7B) with in-context learning successfully generalizes to decode test slot values from the exemplars, though the retrieved exemplars consist of values for food or price range, which are different from the target conversation.

## 5 Related Work

Dialog State TrackingMost of existing works on DST train a supervised model with large-scale labeled datasets Wu et al. (2019); Zhang et al. (2020); Peng et al. (2021); Lin et al. (2020); Lee et al. (2021); Zhao et al. (2022); Kim et al. (2020); Heck et al. (2020); Hosseini-Asl et al. (2020); Ham et al. (2020); Cheng et al. (2020); Platanios et al. (2021). However, a supervised model does not scale well to new domains or annotation schemas. To address the problem, several recent works explore few-shot DST Wu et al. (2020); Li et al. (2021); Gao et al. (2020); Lin et al. (2021); Campagna et al. (2020); Su et al. (2022). Most related are the works of Hu et al. (2022); Chen et al. (2023), who adopt in-context learning with LLM for dialog state generation. The work demonstrated the few-shot generalization ability of LLM applied to DST without parameter updates, but the dialog retriever is still fine-tuned with in-domain data.

Another work related to ours is Shin et al. (2022), which formulates DST as a summarization task. The authors train a T5 language model to decode text summaries, which are then transformed into dialog states with heuristic rules. Different from their work, we do not aim to alter the target of DST as summaries but rather our goal is to enable effective conversation retrieval.

RetrievalOur work mainly focuses on retrieving relevant conversations for in-context learning Liu et al. (2022). There is a vast number of papers Karpukhin et al. (2020b); Khattab and Za

\begin{table}
\begin{tabular}{c c} \hline \hline  & Conversation \\ \hline \multicolumn{2}{c}{:} \\
**SYSTEM**: Booking was successful. \\ The table will be reserved for 15 minutes. ... \\
**USER**: Great. 1 more thing. Can you book a taxi between the 2 places? I would like to arrive at the restaurant in time for my reservation \\ \hline
**Summary**: The user wants to book a taxi to travel between two specific locations. \\ \hline \hline \end{tabular}
\end{table}
Table 7: A failure case of summarization with the LLM.

Figure 4: Visualization of importance scores. Tokens with darker blue gets larger weights based on the latest user utterance.

 

haria, 2020; Izacard et al., 2022; Santhanam et al., 2022) proposing neural network based retrievers which encode queries and keys into low dimensional vectors and compute similarities between them. Hu et al. (2022); Chen et al. (2023) propose to utilize slots and values to represent a long history of conversation for retrieval. However, in order to train the retriever, their approaches require labeled dialogue data to construct positive and negative conversations for each query conversation. Recently, Ravfogel et al. (2023) retrieve texts based on abstract descriptions generated by a LLM.

## 6 Conclusion

The contribution of this work is twofold. First, we proposed an effective way of retrieving conversations in LLM-based DST with conversation summaries as search keys and queries. We then improved the efficiency of the retrieval system by distilling a conversation encoder capable of embedding a conversation into a vector space similar to its summary. This eliminates the cost of decoding an actual summary for each test sample during inference. We validated our CONVERSE encoder for LLM-based DST in a real few-shot setting with 100 conversations in the support set. Results showed that CONVERSE consistently improved both the efficiency and the performance of few-shot DST when using different LLMs, outperforming previous LLM-based DST baselines that rely on annotated dialogues for retrieved fine-tuning.

AcknowledgementWe would like to express our sincere gratitude to Dominik Wagner for the numerous helpful pieces of feedbacks.

## References

* Black et al. (2022) Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. GPT-NeoX-20B: An open-source autoregressive language model. In _Proceedings of BigScience Episode# 5-Workshop on Challenges & Perspectives in Creating Large Language Models_, pages 95-136.
* Budzianowski et al. (2018) Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iinigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. MultiWOZ-A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 5016-5026.
* Campagna et al. (2020) Giovanni Campagna, Agata Forcyciarz, Mehrad Moradshahi, and Monica Lam. 2020. Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 122-132.
* Chen et al. (2023) Derek Chen, Kun Qian, and Zhou Yu. 2023. Stabilized in-context learning with pre-trained language models for few shot dialogue state tracking. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 1506-1519.
* Cheng et al. (2024) Jianpeng Cheng, Devang Agrawal, Hector Martinez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Dain Kaplan, Dimitri Kartsaklis, Lin Li,

\begin{table}
\begin{tabular}{l|l} \hline Target & \\ Conversation & **SYSTEM**: There are 9 Indian restaurants in centre what price range do you want? \\  & **USER**: 1 am looking for expensive Indian food. \\ \hline Gold Label & restaurant-food: indian, restaurant-piceragence: expensive \\ \hline Prediction & restaurant-food: indian, restaurant-piceragence: expensive \\ \hline Exemplar \#1 & **USE**: I am also looking to eat somewhere expensive, in the south area of town. \\  & **SYSTEM**: There are 2 Chinese, 1 Indian , 1 Italian, and 1 Mexican restaurants. Which of those would you like? \\  & **USER**: I would like the Italian place please. \\  & **Summary**: The user wants to find a restaurant in a specific area with a certain price range and cuisine. \\  & **Label**: restaurant-food: ialian. \\ \hline Exemplar \#2 & **USER**: Actually, lalso need a moderate priced restaurant in the same area. \\  & **SYSTEM**: I can find a few that meet that criteria, would you like Indian, or Italian food? \\  & **USER**: Well, everyone said it’s my choice, so I think I would like Italian. \\  & **Summary**: The user wants to find a moderate priced restaurant in a specific area with a specific cuisine. \\  & **Label**: restaurant-food: ialian. \\ \hline Exemplar \#3 & **SYSTEM**: Good news 1 was able to get this for you. Reference idkddjl. Can I help you find other things to do in the area as well ? \\  & **USER**: I am also looking to eat somewhere expensive, in the south area of town. \\  & **SUMMARY**: The user wants to find a restaurant with an expensive price range in a specific area of town. \\  & **LabE**: restaurant-pricerange: expensive, restaurant-area: south \\ \hline \end{tabular}
\end{table}
Table 8: Given the target conversation, we show the top 3 most similar examples retrieved by our model CONVERSE.

 

Dhiyya Piraviperumal, et al. 2020. Conversational semantic parsing for dialog state tracking. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8107-8117.
* Eric et al. [2020] Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 422-428.
* Gao et al. [2020] Shuyang Gao, Sanchit Agarwal, Di Jin, Tagyoung Chung, and Dilek Hakkani-Tur. 2020. From machine reading comprehension to dialogue state tracking: Bridging the gap. In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_, pages 79-89.
* Gliwa et al. [2019] Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_, pages 70-79.
* Gunther et al. [2023] Michael Gunther, Louis Milliken, Jonathan Geuter, Georgios Mastrapas, Bo Wang, and Han Xiao. 2023. Jina embeddings: A novel set of high-performance sentence embedding models. _arXiv preprint arXiv:2307.11224_.
* Ham et al. [2020] Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2. In _Proceedings of the 58th annual meeting of the association for computational linguistics_, pages 583-592.
* Hardt [1997] Daniel Hardt. 1997. An empirical approach to vp ellipsis. _Computational Linguistics_, 23(4):525-541.
* Heck et al. [2020] Michael Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. 2020. Trippy: A triple copy strategy for value independent neural dialog state tracking. In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 35-44.
* Henderson et al. [2017] Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjit Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. _arXiv preprint arXiv:1705.00652_.
* Henderson et al. [2014] Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In _Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL)_, pages 263-272.
* Hosseini-Asl et al. [2020] Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. _Advances in Neural Information Processing Systems_, 33:20179-20191.
* Hu et al. [2022] Yusli Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022. In-context learning for few-shot dialogue state tracking. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 2627-2643.
* Izacard et al. [2022] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. _Trans. Mach. Learn. Res._
* Karpukhin et al. [2020a] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020a. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781.
* Karpukhin et al. [2020b] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020b. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781.
* Khattab and Zaharia [2020] Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over bert. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_, pages 39-48.
* Kim et al. [2020] Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sangwoo Lee. 2020. Efficient dialogue state tracking by selectively overwriting memory. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 567-582.
* Lee et al. [2021] Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with a language model using schema-driven prompting. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 4937-4949.
* Lee et al. [2019] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6086-6096.
* Lee et al. [2022] Seanie Lee, Heae Beom Lee, Juho Lee, and Sung Ju Hwang. 2022. Sequential reptile: Inter-task gradient alignment for multilingual learning. In _International Conference on Learning Representations_.
* Lewis et al. [2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ 

Shuyang Li, Jin Cao, Mukund Sridhar, Henghui Zhu, Shang-Wen Li, Wael Hamza, and Julian McAuley. 2021. Zero-shot generalization in dialog state tracking through generative question answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1063-1074.
* Lin et al. (2021) Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul A Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, and Rajen Subba. 2021. Leveraging slot descriptions for zero-shot cross-domain dialogue statetracking. In _Proceeding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5640-5648.
* Lin et al. (2020) Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020. Mintl: Minimalist transfer learning for task-oriented dialogue systems. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3391-3405.
* Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 100-114.
* Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In _Psychology of learning and motivation_, volume 24, pages 109-165. Elsevier.
* Mosbach et al. (2023) Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. 2023. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation. In _Findings of the Association for Computational Linguistics: ACL_.
* Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al. 2022. Large dual encoders are generalizable retrievers. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9844-9855.
* OpenAI (2022) OpenAI. 2022. Introducing ChatGPT. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt).
* Peng et al. (2021) Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2021. Soloist: Building task bots at scale with transfer learning and machine teaching. _Transactions of the Association for Computational Linguistics_, 9:807-824.
* Platanios et al. (2021) Emmanouil Antonios Platanios, Adam Pauls, Subbro Roy, Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, et al. 2021. Value-agnostic conversational semantic parsing. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3666-3681.
* Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In _Joint conference on EMNLP and CoNLL-shared task_, pages 1-40.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21:140:1-140:67.
* Rayfogel et al. (2023) Shauli Rayfogel, Valentina Pyatkin, Amir DN Cohen, Avshalam Manevich, and Yoav Goldberg. 2023. Retrieving texts based on abstract descriptions. _arXiv preprint arXiv:2305.12517_.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamesebert networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992.
* Santhanam et al. (2022) Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3715-3734.
* Shin et al. (2022) Jamin Shin, Hangyeol Yu, Hyeongdon Moon, Andrea Madotto, and Juneyoung Park. 2022. Dialogue summaries as dialogue states (ds2), template-guided summarization for few-shot dialogue state tracking. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 3824-3846.
* Su et al. (2022) Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2022. Multi-task pre-training for plug-and-play task-oriented dialogue system. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4661-4676.
* Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ 

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Wu et al. (2020) Chien-Sheng Wu, Steven CH Hoi, and Caiming Xiong. 2020. Improving limited labeled dialogue state tracking with self-supervision. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4462-4472.
* Wu et al. (2019) Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 808-819.
* Yasunaga et al. (2022) Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. LinkBERT: Pretraining language models with document links. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8003-8016.
* Ye et al. (2022) Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. 2022. MultiWOZ 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 351-360.
* Yih et al. (2011) Wen-tau Yih, Kristina Toutanova, John C Platt, and Christopher Meek. 2011. Learning discriminative projections for text similarity measures. In _Proceedings of the fifteenth conference on computational natural language learning_, pages 247-256.
* Zhang et al. (2020) Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. Task-oriented dialog systems that consider multiple appropriate responses under the same context. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 9604-9611.
* Zhao et al. (2022) Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu, Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022. Description-driven task-oriented dialog modeling. _arXiv preprint arXiv:2201.08904_.

 



## Appendix A Prompt for Summarization

In Table 9, we provide an instruction and two examples that specify how to summarize a conversation based on the latest user utterance. The first example demonstrate how to ignore irrelevant history -- request for setting an alarm and focus on sending a message. The second example is to generate "The user wants nothing more" instead of summarizing the conversation.

## Appendix B Instruction for Human Evaluation

We conduct human evaluation of 135 summaries generated by the conversation summarizer, gpt-3.5-turbo. As shown in Figure 5, three human judges are given an instruction for the evaluation and they are asked to assess whether the generated summaries are consistent with the prompt specified in Table 9.

## Appendix C Example of Summaries

In Tables 10 to 14, we provide more summaries generated by the LLM, gpt-3.5-turbo.





## 6 Conclusion

In this paper we have presented a new method for the detection of events in the presence of a signal. The proposed approach is to detect events with a given background, and it has been shown that the detection of events from the same event can be used as an alternative to search for events with a given background. We also showed that the detection of events from the same event will not be possible at all. This method allows us to determine the background by comparing the background and background of the background. In addition, we found that the background of the background is expected to be less than 10%.

The background of the background is estimated using the method described in Section 4.2.3. The background of the background is calculated using the method described in Section 5.3. The background of the background is estimated using the method described in Section 5.3. The background of the background is estimated using the method described in Section 5.



You are shown a conversation between a virtual assistant on a phone and a user. You have to summarise what the user wants at this particular point of the conversation. You summary should contain the user intent and the slots he mentioned. However, the summary should be a delexicalized abstract sentence, which means it should not contain actual slot values. Note that it is possible that not all conversation history is relevant and you need to summarise based on what is relevant to the most recent user turn. If the user does not have a goal at this point or his goal gets completed by the system, just summarize that "The user wants nothing more".

\(<\)fictional_example>

USER: make an alarm for 6 SYSTEM: I have created an alarm at 6 USER: Also, send a message to my wife SYSTEM: What would you like the message to say? USER: ehm... happy birthday SYSTEM: I can do that. What message service do you want to use USER: whatsapp What does the user want at this point in the conversation? The user wants to send a message to a recipient with a given text using a specified app \(<\)/fictional_example\(>\)

\(<\)fictional_example\(>\)

USER: make an alarm for 6 SYSTEM: I have created an alarm at 6 USER: thanks you and goodbye What does the user want at this point in the conversation? The user wants nothing more \(<\)/fictional_example\(>\)

Now it's your turn. \(\{\)test_example\(\}\)

What does the user want at this point in the conversation?

**Conversation**

**USER**: I am looking for a hotel in the east , and i would like free wifi please.

**SYSTEM**: I have found express by holiday inn Cambridge. 1517 Norman way, Coldhams business park

**USER**: Does that hotel include free parking and have 2 stars?

**SYSTEM**: No, thank you , that s all I need.

**USER**: Sure thing what's the area and or name?

**SYSTEM**: Thank you for contacting Cambridge Towninfo centre. Have a nice trip! Good Bye!

**USER**: Actually, I also need a moderate priced restaurant in the same area

**Summary**: The user wants to find a moderate priced restaurant in a specific area.

## Appendix A

\begin{table}
\begin{tabular}{l l} \hline \hline  & \multicolumn{1}{c}{**Construction**} \\ \hline
**USER**: I am looking for a hotel in the east , and i would like free wifi please. \\
**SYSTEM**: I have found express by holiday inn Cambridge. 1517 Norman way, Coldhams business park \\
**USER**: Does that hotel include free parking and have 2 stars? \\
**SYSTEM**: No, thank you , that s all I need. \\
**USER**: Sure thing what’s the area and or name? \\
**SYSTEM**: Thank you for contacting Cambridge Towninfo centre. Have a nice trip! Good Bye! \\
**USER**: Actually, I also need a moderate priced restaurant in the same area \\ \hline \end{tabular}
\end{table}
Table 9: Prompt with two exemplars for summarizing a conversation.

 



**Conversation**

\begin{table}
\begin{tabular}{l l} \hline \hline  & **Conversation** \\ \hline
**USER**: Please help me reserve a taxi from saint catharine s college \\
**SYSTEM**: Sure! When would you like to arrive? \\
**USER**: I would like to arrive by 01:15. \\
**SYSTEM**: Great! And what is your destination? \\
**USER**: I am going to Loch Fyne. \\ \hline
**Summary**: The user wants to reserve a taxi from a specific location to a specific destination at a specific time. \\ \hline \hline \end{tabular}
\end{table}
Table XIII: Example of conversation summary.

\begin{table}
\begin{tabular}{l l} \hline \hline  & **Conversation** \\ \hline
**USER**: Are there any concert halls in the west? \\
**SYSTEM**: I am sorry there are no listings for concert halls in the west. Is there something else \\ I can assist you with today? \\
**USER**: What about colleges in the west \\
**SYSTEM**: There are 5 colleges in that area. 3 have no entrance fees. \\ Would you like additional information on any of these? \\
**USER**: No thank you. I am looking for information El Shaddai hotel. \\
**SYSTEM**: The El Shaddai is a guest house in the centre part of town. It has free internet , parking and has 0 stars. \\ Would you like me to book it for you? \\
**USER**: Could you? That would be great. There are 5 of us and we plan to arrive on Thursday. We’d like to stay for 5 nights. \\ \hline
**Summary**: The user wants to book a hotel for a group of 5 people for a specific duration of time. \\ \hline \hline \end{tabular}
\end{table}
Table XII: Example of conversation summary.