Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples
Authors: Haesung Pyun, Yoonah Park, Yohan Jo
Accepted at: acl25

Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples

Haesung Pyun1   Yoonah Park2   Yohan Jo1*
1Graduate School of Data Science, Seoul National University
2Department of Computer Science and Engineering, Seoul National University
{haesung.pyun, wisdomsword21, yohan.jo}@snu.ac.kr

Abstract
In dialogue state tracking (DST), in-context learning comprises a retriever that selects labeled dialogues as in-context examples and a DST model that uses these examples to infer the dialogue state of the query dialogue. Existing methods for constructing training data for retrievers suffer from three key limitations: (1) the synergistic effect of examples is not considered, (2) the linguistic characteristics of the query are not sufficiently factored in, and (3) scoring is not directly optimized for DST performance. Consequently, the retriever can fail to retrieve examples that would substantially improve DST performance. To address these issues, we present CombiSearch—a method that scores effective in-context examples based on their combinatorial impact on DST performance. Our evaluation on MultiWOZ shows that retrievers trained with CombiSearch surpass state-of-the-art models, achieving a 20× gain in data efficiency and generalizing well to the SGD dataset. Moreover, CombiSearch attains a 12% absolute improvement in the upper bound DST performance over traditional approaches when no retrieval errors are assumed. This significantly increases the headroom for practical DST performance while demonstrating that existing methods rely on suboptimal data for retriever training.1

1 Introduction
Dialogue state tracking (DST), a core task for task-oriented dialogue systems and tool agents, tracks user intentions and relevant constraints throughout the dialogue. For example, if a user says “I want to book a guest house” and later adds “I want to book it for four people”, the dialogue state is updated with the slot-value pairs “[<hotel-type: guest house>, <hotel-book people: 4>]”.

Recent work shows that large language models can perform DST through in-context learning (ICL) (Brown et al., 2020; Zhao et al., 2023). By supplying query–response exemplars in the prompt, models learn task-specific patterns from in-context examples, thereby eliminating the need for costly fine-tuning or extensive annotation (Brown et al., 2020; Zhang et al., 2022; Chowdhery et al., 2023; Hoffmann et al., 2022; Touvron et al., 2023). However, since performance depends on the relevance of these exemplars (Liu et al., 2022), recent studies propose retrievers that dynamically select examples tailored to each test query, boosting the accuracy and robustness of ICL performance across tasks (Levy et al., 2023; Lu et al., 2023; Ye et al., 2023).

To select effective examples, it is crucial to train an ICL example retriever on high-quality data. The most widely adopted approach for preparing retriever training data follows two steps (Hu et al., 2022): (1) pool construction selects dialogues similar to the target query, and (2) re-ranking prioritizes dialogues whose dialogue states substantially

![A diagram comparing conventional and our data preparation methods for dialogue state tracking. The conventional method shows overlap between example pools and scored pools, while our method integrates diverse pool construction and combinatorial re-ranking based on DST performance.](page_1016_370_388_312.png)

Figure 1: (A) Conventional data preparation method. (B) Our data preparation method that integrates diverse pool construction and combinatorial re-ranking based on DST performance.
overlap with the query’s dialogue state (Figure 1-A). However, what constitutes effective in-context examples for DST and how to systematically select them remain understudied and particularly challenging, as DST requires a comprehensive understanding of various conversational aspects (Chen et al., 2023). For instance, coreference resolution is both crucial and error-prone, requiring the system to correctly identify anaphors (e.g., “it”) and their corresponding antecedents (e.g., “guest house”), which may span multiple dialogue turns. Our analysis identifies prominent errors and challenges in prior approaches (Appendix A.1).

We focus on three key limitations in the prior data preparation process and address them in this paper. First, the re-ranking step scores each example individually, which can result in selecting redundant in-context examples that offer little new information. Second, relying solely on dialogue state similarity may neglect important contextual information within the dialogue itself and retrieve examples whose conversational characteristics differ significantly from the query dialogue. Such examples are suboptimal for handling complex linguistic phenomena, such as coreference resolution, present in the query dialogue. Furthermore, re-ranking examples based on dialogue state similarity does not directly optimize for DST performance, resulting in only “indirect supervision”.

To address these limitations, we introduce CombiSearch, a new method for preparing training data for in-context example retrievers in DST. By combining diverse pool construction with combinatorial re-ranking (Figure 1-B), CombiSearch has three key characteristics. First, the pool construction step collects diverse dialogues that share similar linguistic characteristics with the query—such as conversational flow, style, and structure—beyond just dialogue states. This ensures that more relevant examples are considered during re-ranking, improving the system’s ability to handle the linguistic phenomena present in the query. Second, rather than re-ranking individual examples in isolation, CombiSearch scores examples based on how they function together in tandem. By evaluating their combinatorial impact on DST performance, it accounts for synergistic effects that might be overlooked when scoring examples separately. Finally, the re-ranking step scores examples by directly optimizing Joint Goal Accuracy (JGA), a standard metric for DST, rather than relying on the indirect signal of dialogue state similarity between each example and the query.

According to our evaluation, CombiSearch outperforms existing state-of-the-art methods on both the MultiWOZ 2.4 and Schema-Guided Dialogue (SGD) datasets. Notably, CombiSearch is highly data-efficient, enabling a retriever trained on CombiSearch-scored data to surpass all baseline models while using only 5% of the training data required by these models. Understanding the upper bound performance achievable with retriever training data \emph{without retriever errors} is crucial, as this sets a hard limit on what practical retrievers can achieve. Under this ideal assumption of no retrieval errors, using examples scored by CombiSearch outperformed earlier approaches by 12% JGA, highlighting that prior methods rely on suboptimal data for training retrievers and that a more effective data preparation approach like CombiSearch is imperative for breakthroughs in DST. Notably, all these advances from CombiSearch’s combinatorial nature come with a running time that is linear in the number of examples to score.

Our key contributions are as follows:
1. We present CombiSearch, an effective combinatorial method for constructing data to train retrievers of in-context examples in DST.
2. We demonstrate that CombiSearch outperforms baselines in DST, even with 20x data efficiency.
3. We reveal the significantly improved upper bound performance of CombiSearch when no retrieval errors are assumed, highlighting the importance of optimizing retriever training data.

2 Problem Setting

Dialogue State Tracking  DST focuses on predicting the dialogue state that captures the user’s intentions and goals, based on both the current user-agent interaction and the conversation history. Let \(A^t\) and \(U^t\) denote the agent’s and user’s utterances at turn \(t\), respectively. The dialogue history up to turn \(t\) is \(C^t = [(A^1, U^1), (A^2, U^2), \ldots, (A^t, U^t)]\). The objective is to infer a dialogue state \(y^t\) from \(C^t\), where \(y^t\) is a set of slot-value pairs: \(y^t = \{(s^1, v^1), (s^2, v^2), \ldots, (s^n, v^n)\}\).

Following Hu et al. (2022), we adopt a state-change formulation. Specifically, the model takes the previous dialogue state \(y^{t-1}\) and current utterances \((A^t, U^t)\) as input, denoted as \(x^t = [y^{t-1}, (A^t, U^t)]\), where \(y^{t-1}\) summarizes \(C^{t-1}\). It predicts the dialogue state change \(\Delta y^t = \{+(s^i, v^i), \ldots, -(s^j, v^j)\}\), with “+” indicating ad-
dition and “-” indicating removal of slot-value pairs. The t-th dialogue state is \( y^t = y^{t-1} \cup \Delta y^t \).

In-Context Learning for DST Given a test query \( x_{test} \), the ICL approach retrieves k labeled examples \( \{ (x_i, \Delta y_i) \}_{i=1}^k \) from the training set \( D_{train} = \{ (x_i, \Delta y_i) \}_{i=1}^N \). The goal is to predict the dialogue state change of the test query \( x_{test} \):

\[
\Delta \hat{y}_{test} \sim P_{LLM}(y_{test} | \{ (x_i, \Delta y_i) \}_{i=1}^k, x_{test})
\]

where \( \sim \) refers to a decoding method (e.g., greedy decoding, beam search, etc.).

In conventional retriever-based ICL for DST, two stages are involved. Example-Retrieval Training trains the retriever on query-example pairs: each query is paired with candidates ranked by dialogue state similarity, and a contrastive loss draws similar embeddings together while repelling dissimilar ones. Inference Phase uses the trained retriever to fetch top-ranked examples based on the test query; these examples are prepended to the prompt so the language model can exploit them to generate the query’s dialogue state.

3 Method

We present CombiSearch, an algorithm that aims to identify an optimal combination of in-context examples to improve overall DST performance. We first detail each step of CombiSearch (\S 3.1) and introduce a retriever training method based on CombiSearch-scored data (\S 3.2). Lastly, we describe our text-to-JSON prompt format (\S 3.3). The overall procedure is illustrated in Figure 2.

3.1 CombiSearch

CombiSearch is designed to generate training data for a retriever of in-context examples for DST. The resulting data consists of <query, scored examples> pairs which can be used to train a retriever to select high-scoring examples for a given query. CombiSearch unfolds in two complementary steps. First, for each query, it constructs a diverse pool of examples \( E \) by leveraging both semantic and lexical similarity to the query. Second, it employs a combinatorial mechanism to score the examples based on their collective contribution to successfully predicting the query’s dialogue state. The algorithm is presented in Algorithm 1. Please note that during data preparation and retriever training, each query is drawn from the training set.

Algorithm 1 Pseudo-code for CombiSearch
<table>
  <tr>
    <th colspan="2">Algorithm 1 Pseudo-code for CombiSearch</th>
  </tr>
  <tr>
    <td>1:</td>
    <td><b>Input:</b> \( D_{train} \): Training Dataset, \( D_{query} \): Query Dataset</td>
  </tr>
  <tr>
    <td>2:</td>
    <td><b>Output:</b> \( D_{CombiSearch} = \{ (x_i, y_i, E_i) \}_{i=1}^{|Q|} \)</td>
  </tr>
  <tr>
    <td>3:</td>
    <td><b>Parameters:</b> \( k \): Number of in-context examples, \( M \): Number of evaluations per example, \( P_i \): Input prompt</td>
  </tr>
  <tr>
    <td>4:</td>
    <td>for \( (x_{test}, y_{test}) \in D_{query} \) do \( \triangleright \) Pool Construction</td>
  </tr>
  <tr>
    <td>5:</td>
    <td>\( D_i \leftarrow \text{top-k}_{BM25}(D_{train}, N) \)</td>
  </tr>
  <tr>
    <td>6:</td>
    <td>\( D_j \leftarrow \text{top-k}_{SBERT}(D_{train}, N) \)</td>
  </tr>
  <tr>
    <td>7:</td>
    <td>\( D_{union} \leftarrow D_i \cup D_j \)</td>
  </tr>
  <tr>
    <td>8:</td>
    <td>for \( u \in D_{union} \) do</td>
  </tr>
  <tr>
    <td>9:</td>
    <td>\( u.\text{hybrid\_score} \leftarrow u.\text{TF-IDF} \times u.\cos\_sim \)</td>
  </tr>
  <tr>
    <td>10:</td>
    <td>end for</td>
  </tr>
  <tr>
    <td>11:</td>
    <td>\( E \leftarrow \text{top-k}_{\text{hybrid\_score}}(D_{union}, N) \)</td>
  </tr>
  <tr>
    <td>12:</td>
    <td>for \( i = 1 \) to \( M \) do \( \triangleright \) Example Scoring</td>
  </tr>
  <tr>
    <td>13:</td>
    <td>\( R_i \leftarrow \text{select\_random\_subset}(E, k) \)</td>
  </tr>
  <tr>
    <td>14:</td>
    <td>\( jga_i \leftarrow \text{JGA}(LLM(P_i, x_{test}), y_{test}) \)</td>
  </tr>
  <tr>
    <td>15:</td>
    <td>for \( e \in P_i \) do</td>
  </tr>
  <tr>
    <td>16:</td>
    <td>\( e.\text{CombiScore} \leftarrow e.\text{CombiScore} + jga_i \)</td>
  </tr>
  <tr>
    <td>17:</td>
    <td>end for</td>
  </tr>
  <tr>
    <td>18:</td>
    <td>end for</td>
  </tr>
  <tr>
    <td>19:</td>
    <td>\( D_{CombiSearch} \leftarrow D_{CombiSearch} \cup \{ (x_{test}, y_{test}, E) \} \)</td>
  </tr>
  <tr>
    <td>20:</td>
    <td>end for</td>
  </tr>
  <tr>
    <td>21:</td>
    <td><b>return</b> \( D_{CombiSearch} \)</td>
  </tr>
</table>

Pool Construction (Figure 2-1) Our motivating observation is that DST errors often arise when essential linguistic phenomena required for accurately predicting the query’s dialogue state (e.g., coreference resolution) are not adequately represented in the in-context examples. Hence, it is important to include diverse examples covering a broader range of linguistic phenomena relevant to the query. To that end, we adopt a hybrid search that integrates both semantic and lexical aspects of similarity using two retrievers: SBERT (Reimers and Gurevych, 2019) and BM25 (Robertson and Zaragoza, 2009). SBERT captures the semantic meaning of dialogue states while BM25 effectively captures linguistic properties—such as word choices, the use of coreferences, named entities, domains, and slots (Appendix D.2). We first retrieve the top-N examples from each retriever and merge them into a single list. We then re-rank them based on a hybrid score defined as the product of BM25 TF-IDF score and SBERT cosine similarity score (Appendix E.3). Finally, we retain the top-N examples to form the final pool \( E \).

Example Scoring (Figure 2-2) After constructing the candidate pool \( E \) with \( N \) examples per query, we score individual examples. Rather than scoring an example solely based on its similarity to the query as in existing work, we directly measure its impact on DST performance when used in ICL in tandem with other examples. This approach provides more direct supervision. Our objective is to
identify an optimal set of examples that collectively enhance DST performance.

However, exhaustively evaluating all possible combinations is impractical due to the combinatorial explosion. Instead, we conceptualize each example as a key “team player” if it consistently contributes to the success of the “teams” it belongs to (i.e., accurate DST) across different sampled combinations. CombiSearch implements this by iteratively sampling \( k \) examples and evaluating the DST model’s prediction using JGA. A JGA score of 1 increases the score of every example in the sampled combination by 1, while a score of 0 leaves the scores unchanged. Repeating this sampling process \( M \) times yields an accumulated score for each example—termed CombiScore—reflecting its contribution to DST. Notably, this algorithm allows CombiSearch to run in linear time with respect to the number of examples to score while preserving its combinatorial nature. The resulting dataset \( D_{\text{CombiSearch}} \) pairs each query with a pool of examples \( E \) ranked by their CombiScores.

3.2 Retriever Training (Figure 2-3)

We train a specialized retriever to identify effective in-context examples for each query. The retriever takes a dialogue (\( y^{t-1}, A^t, \) and \( U^t \)) as input and produces an embedding. The training objective is to maximize the embedding similarity between a query and its high-scoring examples in \( D_{\text{CombiSearch}} \).

We construct training instances as follows. For each query \( x_i \), we first construct a set of positive examples consisting of the top \( |P| \) examples ranked by CombiScore from pool \( E \); ties are broken by the degree of dialogue state overlap. Next, for every pair of \( x_i \) and a positive example \( e_j^+ \), a set of negative examples is constructed by selecting the bottom \( B \) examples from \( E \) and additional \( B - 1 \) examples randomly sampled from outside \( E \). As a result, the training data comprise tuples of the form \( \langle x_i, e_j^+, e_{j,1}^-, \ldots, e_{j,2B-1}^- \rangle \) for all \( j = 1, ..., |P| \).

Based on this data, we fine-tune our retriever using the InfoNCE loss (van den Oord et al., 2019):

\[
L(x, P, N) = \sum_{e^+ \in P} - \log \frac{\exp(\operatorname{sim}(x, e^+))}{\sum_{e' \in N \cup \{e^+\}} \exp(\operatorname{sim}(x, e'))},
\]

where \( x \) is the query, \( P \) is the positive set, and \( N \) is the negative set. Additional hyperparameter details are provided in Appendix B.2.

At inference time (Figure 2-4), for each query, we first retrieve \( k \) candidate examples from both our retriever and BM25 and merge the two lists. We then re-rank the merged candidates with the hybrid score introduced in §3.1, defined as the product of the two retrievers’ scores, and finally select the top \( k \) examples as in-context examples. The number of retrieved examples does not have to be equal to the combination size used in CombiSearch.

![Overall structure of CombiSearch.](page_186_120_1077_377.png)

Figure 2: Overall structure of CombiSearch.

3.3 Prompting with Text-to-JSON

This section describes our text-to-JSON prompt for DST, which we developed to avoid formatting errors observed when treating DST as a code generation task inspired by Liu et al. (2024b).

The prompt begins with a system message that instructs the model to act as a DST expert (Kong et al., 2024) and provides an explicit task definition (He et al., 2024). Next, domain-specific details—such as slots, types, and possible values for categorical slots—are provided. In-context examples follow, presented as tuples \( ([y^{t-1}, A^t, U^t], \Delta y^t) \) in which both the prior state \( y^{t-1} \) and the state changes \( \Delta y^t \) are represented in JSON format. The prompt ends with clear DST instructions and output format specifications, all formatted in Markdown for clarity (see Appendix G).
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Language Model</th>
    <th colspan="3">MultiWOZ 2.1</th>
    <th colspan="3">MultiWOZ 2.4</th>
  </tr>
  <tr>
    <th>1%</th>
    <th>5%</th>
    <th>100%</th>
    <th>1%</th>
    <th>5%</th>
    <th>100%</th>
  </tr>
  <tr>
    <td>DiDISTRICT (Venkateswaran et al., 2023)</td>
    <td>T5-small</td>
    <td>13.4</td>
    <td>41.3</td>
    <td>56.1</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SGPDST (Lee et al., 2021)</td>
    <td>T5-base</td>
    <td>32.1</td>
    <td>43.1</td>
    <td><b>56.7</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SM2-11B (Chen et al., 2023)</td>
    <td>T5-XXL</td>
    <td>38.4</td>
    <td>44.6</td>
    <td>-</td>
    <td>40.0</td>
    <td>51.1</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DS2 (Shin et al., 2022)</td>
    <td>T5-large</td>
    <td>33.8</td>
    <td>44.2</td>
    <td>53.3</td>
    <td>36.8</td>
    <td>49.9</td>
    <td>57.9</td>
  </tr>
  <tr>
    <td>IC-DST (Hu et al., 2022)</td>
    <td>code-davinci-002</td>
    <td>43.1</td>
    <td>47.1</td>
    <td>48.7</td>
    <td>50.7</td>
    <td>48.4</td>
    <td>55.4</td>
  </tr>
  <tr>
    <td rowspan="3">ICL</td>
    <td>SynthDST (Kulkarni et al., 2024)</td>
    <td><b>45.8</b></td>
    <td>44.9</td>
    <td>46.0</td>
    <td>51.0</td>
    <td>50.4</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td>RefPyDST* (King and Flanigan, 2023)</td>
    <td>gpt-3.5-turbo</td>
    <td>39.7</td>
    <td>43.7</td>
    <td>48.9</td>
    <td>44.9</td>
    <td>52.3</td>
    <td>58.0</td>
  </tr>
  <tr>
    <td>CombiSearch</td>
    <td>gpt-3.5-turbo</td>
    <td><b>45.8</b></td>
    <td><b>47.7</b></td>
    <td>50.0</td>
    <td><b>56.7</b></td>
    <td><b>59.8</b></td>
    <td><b>64.2</b></td>
  </tr>
</table>

Table 1: JGA scores in the closed-source setting, evaluated on the full test sets of MultiWOZ 2.1 and 2.4 using 1%, 5%, and 100% of the training data. Each score represents the mean of three runs. *RefPyDST was reproduced under our own settings because code-davinci-002 has been deprecated and the log-probabilities of input prompts are no longer provided by the API. More details are provided in Appendix B.1.

4 Experiments

In this section, we conduct ICL experiments for DST in various settings. The practical setting (\S4.1) uses a trained retriever and evaluates DST performance on in-domain data (\S4.1.1), unseen data (\S4.1.2), and a dataset that involves coreference resolution (\S4.1.3). On the other hand, the oracle setting (\S4.2) examines the upper bound DST performance without retrieval errors. In ablation studies (\S4.3), we assess the contribution of individual components in CombiSearch. Furthermore, we compare the combinatorial scoring method with an individual scoring baseline to measure the effect of modeling interactions among examples.

Common experimental settings are described below, with specific configurations detailed in their respective subsections. Further analysis of the experimental settings can be found in Appendix E.

Common Experiment Settings We evaluated our method on the MultiWOZ and Schema-Guided Dialogue (SGD) datasets. The MultiWOZ dataset comprises roughly 10,000 human-human dialogues across five domains (hotel, attraction, taxi, train, and restaurant) for training, with evaluation conducted on MultiWOZ 2.4. The SGD dataset contains 16,000 machine-machine dialogues across 26 services, with unseen domains in its test set used to assess generalization. Performance is measured using Joint Goal Accuracy (JGA), which requires exact matches between predicted and ground-truth dialogue state changes at each turn.

Our CombiSearch configuration uses a candidate pool size of \( N = 100 \) and \( k = 10 \) examples in each combination. We set \( M = 3 \) evaluations per example to balance accuracy with computational efficiency on large-scale datasets (Appendix E.2). During CombiSearch, we score candidate examples with Llama-3-8B-Instruct as the DST model. At inference time, we experiment with Llama-3-8B-Instruct for the DST model. To assess data efficiency, we explore using 1%, 5% and 100% of the training data, for both data construction for retriever training during CombiSearch and data retrieval at inference time. Further implementation details are in Appendix B.

4.1 Performance with Trained Retrievers

We evaluate CombiSearch’s DST performance using retrievers fine-tuned on \( D_{\text{CombiSearch}} \).

4.1.1 In-domain Performance

We cover two scenarios: a Closed-source setting using black-box LLMs and an Open-source setting where only open-source LLMs are available.

Settings The Closed-source setting is as follows. When we explore using only 1% and 5% of the training data, we run experiments three times with random samples of the training data (following Hu et al. (2022)). Unlike the common practice of training separate retrievers for each split (Hu et al., 2022; King and Flanigan, 2023), however, we train a single retriever on one split and use it for all three splits, which serve as retrieval corpora. This out-of-distribution (OOD) setup is more challenging than using a retriever trained specifically for each split, as the retriever must select effective examples from unseen data during inference. We use gpt-3.5-turbo as the DST model and report the mean performance across the three runs.

For Open-source experiments, we sample 1% and 5% of the training data once, which serve as both the retriever training data and the retrieval corpus. We experiment with Llama-3-8B-Instruct and a 4-bit GPTQ-quantized Llama-3-70B-Instruct (to accommodate resource constraints) as the DST model.
<table>
  <tr>
    <th>Retrieval</th>
    <th>DST Model</th>
    <th>1%</th>
    <th>5%</th>
    <th>100%</th>
  </tr>
  <tr>
    <td>RefPyDST</td>
    <td>Llama-3-8B-Instruct</td>
    <td>47.7</td>
    <td>50.6</td>
    <td>55.5</td>
  </tr>
  <tr>
    <td>CombiSearch</td>
    <td>Llama-3-8B-Instruct</td>
    <td><b>52.1</b></td>
    <td><b>56.2</b></td>
    <td><b>61.8</b></td>
  </tr>
  <tr>
    <td>RefPyDST</td>
    <td>Llama-3-70B-Instruct</td>
    <td>56.0</td>
    <td>57.1</td>
    <td>56.2</td>
  </tr>
  <tr>
    <td>CombiSearch</td>
    <td>Llama-3-70B-Instruct</td>
    <td><b>56.6</b></td>
    <td><b>59.3</b></td>
    <td><b>61.5</b></td>
  </tr>
  <tr>
    <td>RefPyDST</td>
    <td>gpt-3.5-turbo</td>
    <td>46.2</td>
    <td>52.7</td>
    <td>58.0</td>
  </tr>
  <tr>
    <td>CombiSearch</td>
    <td>gpt-3.5-turbo</td>
    <td>56.6</td>
    <td>59.3</td>
    <td>64.2</td>
  </tr>
</table>

Table 2: JGA scores in the open-source setting on MultiWOZ 2.4, using 1%, 5%, and 100% of the training data. gpt-3.5-turbo performance on the same training data is also provided.

Details about the baseline models used in the experiments are provided in Appendix B.1.

Results For the Closed-source setting, Table 1 shows that CombiSearch outperforms all baselines in data-constrained settings (1% and 5%) on both MultiWOZ 2.1 and 2.4. Notably, CombiSearch with 5% of training data even surpasses every baseline trained on the full dataset on MultiWOZ 2.4. Furthermore, with only 1% of training data, CombiSearch still exceeds SynthDST, which is trained on its full synthetic data. These results highlight the effectiveness and data efficiency of CombiSearch.

In the Open-source setting (Table 2), CombiSearch exceeds RefPyDST by more than 5% in JGA when Llama-3-8B-Instruct serves as the DST model, regardless of the amount of training data. A consistent—though smaller—average gain about 3% appears with the 4-bit-quantized Llama-3-70B-Instruct. The last two rows of the table show the performance of gpt-3.5-turbo when the same training samples are used for a fair comparison, which exhibit the consistent tendency.

Although constructing CombiSearch data for training a retriever takes more wall-clock time than RefPyDST for the same training data (Appendix C), CombiSearch achieves equal or higher JGA in less overall time because it requires scoring far fewer training data. With only 5% of the training data, CombiSearch reaches 59.3% JGA, already surpassing RefPyDST trained on the full training data (56.2% JGA). Moreover, the running time of CombiSearch increases only linearly with the number of examples to score, and this is only a one-time cost for retriever training that does not affect inference time. Therefore, in practical deployment settings where a retriever is updated only occasionally, the strong data efficiency and performance gains from CombiSearch compensate for its data construction cost.

<table>
  <tr>
    <th rowspan="2">Training Dataset</th>
    <th rowspan="2">Metric</th>
    <th colspan="3">Retrieval Method</th>
  </tr>
  <tr>
    <th>Random</th>
    <th>RefPyDST</th>
    <th>CombiSearch</th>
  </tr>
  <tr>
    <td rowspan="2">SGD</td>
    <td>\( y_{test} \)</td>
    <td>4.45</td>
    <td>15.81</td>
    <td>16.81</td>
  </tr>
  <tr>
    <td>\( \Delta y_{test} \)</td>
    <td>37.25</td>
    <td>53.17</td>
    <td>54.41</td>
  </tr>
  <tr>
    <td rowspan="2">MultiWOZ</td>
    <td>\( y_{test} \)</td>
    <td>4.45</td>
    <td>12.08</td>
    <td>12.24</td>
  </tr>
  <tr>
    <td>\( \Delta y_{test} \)</td>
    <td>37.25</td>
    <td>49.16</td>
    <td>50.16</td>
  </tr>
</table>

Table 3: JGA scores on the SGD test set. For “SGD” and “MultiWOZ” rows, retrievers were trained on 5% of SGD and MultiWOZ 2.1, respectively.

4.1.2 Generalization Ability

To verify that CombiSearch’s advantage generalizes beyond MultiWOZ, we additionally evaluate on SGD dataset. We consider two experimental settings: (1) in-domain, where a retriever is trained and tested on SGD, and (2) cross-dataset, where a retriever trained on MultiWOZ is applied to SGD without adaptation.

Settings We train two retrievers on 5% of the training data: one on SGD and the other on MultiWOZ 2.1. Both were evaluated on 30% of the SGD test set. During inference, in-context examples were drawn from the same 5% subset of SGD, for a fair comparison. DST performance was measured with JGA for both the cumulative state (\( y_{test} \)) and the per-turn state change (\( \Delta y_{test} \)). The Random baseline selects examples randomly. Llama-3-8B-Instruct serves as the DST model.

Results Table 3 shows that both CombiSearch and RefPyDST generalize beyond their training corpus, outperforming the Random baseline by large margins. When the retriever is trained in-domain (SGD), CombiSearch attains 54.41% JGA on \( \Delta y_{test} \), edging out RefPyDST by roughly one 1% JGA and achieving a similar improvement on \( y_{test} \) (16.81%). The same pattern holds under the cross-dataset setting: CombiSearch trained on MultiWOZ still surpasses its RefPyDST counterpart on both metrics. Though modest, this consistent edge under both domain-matched and domain-shift conditions underscores that CombiSearch is not merely overfitting to a single corpus. Rather, it preserves a performance lead across different domains.

4.1.3 Coreference Resolution

We evaluate the effectiveness of CombiSearch in dialogues that require coreference resolution, a critical challenge in multi-turn, cross-domain DST. Because speakers routinely employ ellipsis or pronoun-alization, a system must resolve such contextual references and extract precise slot values from ongoing dialogues.
Settings Experiments are conducted on MultiWOZ 2.3, which provides turn-level coreference annotations (Han et al., 2021). Following King and Flanigan (2023), we compute (i) slot-level accuracy on turns that require coreference resolution and (ii) JGA restricted to dialogues that contain at least one such turn. For few-shot evaluation, we use 5% of the training data both to train the retriever and as the retrieval corpus. Two retrievers—RefPyDST and CombiSearch—are compared.

In this experiment, our focus is primarily on examining the differences in retrieved examples between the two methods. Therefore, we control for several factors that differ between them by using only embedding-based retrieval (instead of hybrid retrieval) and excluding the diversity-promoting re-ranking employed by RefPyDST. Since they use different prompt formats—Text-to-JSON (‘JSON’) and Text-to-Python (‘Python’)—we further control for prompt format. In the zero-shot condition (0%), we follow Hu et al. (2022) and provide a single example to inform the output format. Llama-3-8B-Instruct serves as the DST model.

Results Table 4 shows that, in the 5% setting, CombiSearch surpasses RefPyDST on accuracy and JGA by up to 2.7% (JSON 83.6% vs 81.6%; Python 23.6% vs 21.3%). This suggests that CombiSearch retrieves examples that are more helpful for coreference resolution than RefPyDST. In the zero-shot setting, the ‘JSON’ format yields better performance than the ‘Python’ format. This improvement may be due to frequent formatting errors in the ‘Python’ prompt, as well as the superiority of our formatting instructions.

When a query requires coreference resolution, CombiSearch retrieved such examples slightly more often than RefPyDST (3.7 vs. 3.6 per query) and achieved about 2% higher JGA than RefPyDST (72.4% vs. 70.7%). These findings further highlight the strength of CombiSearch in retrieving effective examples for handling linguistic phenomena such as coreference resolution. Representative cases are provided in Appendix F.

<table>
  <tr>
    <th>Prompt</th>
    <th>Retriever</th>
    <th colspan="2">5%</th>
    <th colspan="2">0%</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Acc</th>
    <th>JGA</th>
    <th>Acc</th>
    <th>JGA</th>
  </tr>
  <tr>
    <td rowspan="2">JSON</td>
    <td>RefPyDST</td>
    <td>81.6</td>
    <td>39.9</td>
    <td>48.7</td>
    <td>12.2</td>
  </tr>
  <tr>
    <td>CombiSearch</td>
    <td><b>83.6</b></td>
    <td><b>42.6</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Python</td>
    <td>RefPyDST</td>
    <td>21.3</td>
    <td>36.0</td>
    <td>15.2</td>
    <td>19.5</td>
  </tr>
  <tr>
    <td>CombiSearch</td>
    <td><b>23.6</b></td>
    <td><b>39.4</b></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 4: Performance on coreference resolution data. Acc: the accuracy of slots for which coreference resolution is required. JGA: JGA for dialogue turns that require coreference resolution.

truth dialogue state. Although this setting is impractical in real-world applications, where the ground-truth dialogue states of queries are unavailable, it serves two key purposes: (i) estimating the maximal DST performance achievable from the retriever training data and (ii) assessing how effectively existing DST methods leverage available data.

Additionally, we conduct an error analysis using DST-EA, a tool we developed to categorize errors into seven types and twenty subclasses. This enables a deeper understanding of DST model behavior and contributes to further advancements in DST research. A detailed description of DST-EA and the error types is provided in Appendix A.1.

Settings Due to computational constraints, we use three separate 10% subsets of the test set and report the mean performance across these subsets to ensure statistical reliability. For each test query, we build \( D_{CombiSearch} \) from the training set, exploring the use of either 1% or 5% of the data to examine the impact of example diversity on DST performance. We compare our method against two baselines: RefPyDST and a “Hybrid” approach that employs the pool construction process in CombiSearch and ranks examples based on the hybrid score (from BM25 and SBERT) without applying combinatorial scoring. For fair comparison, RefPyDST and Hybrid also skip retrieval errors and are given access to the ground-truth dialogue states of test queries. We employ both Llama-3-8B-Instruct and the 4-bit-quantized Llama-3-70B-Instruct as the DST model.

Results As shown in Table 5, CombiSearch achieves 82.7% JGA with Llama-3-8B-Instruct (row 3) and 79.6% with the 4-bit-quantized Llama-3-70B-Instruct (row 6). These scores represent an improvement of roughly 13% points over RefPyDST for each model (rows 1 and 4). The higher upper bound signifies greater headroom for the practical setting of using trained retrievers. It also indicates that the existing methods relied on
<table>
  <tr>
    <th>Scoring Method</th>
    <th>Model Size</th>
    <th>1%</th>
    <th>5%</th>
    <th>100%</th>
  </tr>
  <tr>
    <td>RefPyDST</td>
    <td>8B</td>
    <td>58.0</td>
    <td>62.7</td>
    <td>69.7</td>
  </tr>
  <tr>
    <td>Hybrid</td>
    <td>8B</td>
    <td>60.4</td>
    <td>68.0</td>
    <td>75.9</td>
  </tr>
  <tr>
    <td><b>CombiSearch</b></td>
    <td>8B</td>
    <td><b>68.4</b></td>
    <td><b>75.1</b></td>
    <td><b>82.7</b></td>
  </tr>
  <tr>
    <td>RefPyDST</td>
    <td>70B</td>
    <td>63.8</td>
    <td>66.4</td>
    <td>67.9</td>
  </tr>
  <tr>
    <td>Hybrid</td>
    <td>70B</td>
    <td>60.2</td>
    <td>62.3</td>
    <td>67.6</td>
  </tr>
  <tr>
    <td><b>CombiSearch</b></td>
    <td>70B</td>
    <td><b>71.8</b></td>
    <td><b>72.7</b></td>
    <td><b>79.6</b></td>
  </tr>
</table>

Table 5: Oracle JGA scores across different scoring methods and models.

suboptimal data for retriever training. Furthermore, CombiSearch outperforms the Hybrid baseline by 8%–12% (rows 2 and 5), highlighting the importance of aligning the scores of examples with DST performance through a combinatorial manner.

Error Analysis CombiSearch markedly reduces the most common errors made by BM25- and SBERT-based systems—chiefly, missing explicit user confirmations and over-interpreting ambiguous user utterances—by modeling context more comprehensively. Notable improvements were observed in recognizing confirmation utterances across multiple turns and achieving more accurate utterance interpretation.

4.3 Ablation Study

We evaluated the contributions of pool construction and combinatorial search through component ablation. In this analysis, we not only assess the significance of each component but explore the best way to integrate these two steps to maximize DST performance (see Appendix E for further analysis).

Settings All the experiments use the oracle setting as in §4.2, accessing test queries’ ground-truth dialogue states and directly selecting the highest-scoring examples. We use 5% of the training data and Llama-3-8B-Instruct as the DST model.

Pool Construction The original pool construction step of CombiSearch employs both BM25 and SBERT to retrieve example candidates. To analyze the contribution of each of these retrievers, we explore several configurations and measure DST performance.

As shown in Table 6, using both BM25 and SBERT (row 3) yielded a 14% higher JGA score than composing the pool with random examples (row 1). Additionally, using SBERT alone (row 2) led to a performance drop of 3%–4% compared to the combined approach. This finding underscores the importance of example diversity within the pool. Employing both retrievers results in examples that capture a wider range of dialogue characteristics and enables the combinatorial scoring step to effectively consider and recognize useful examples.

<table>
  <tr>
    <th>Pool Construction</th>
    <th>Combinatorial Search</th>
    <th># of Inference</th>
    <th>JGA (%)</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>CombiSearch</td>
    <td>31</td>
    <td>56.9</td>
  </tr>
  <tr>
    <td>SBERT</td>
    <td>CombiSearch</td>
    <td>31</td>
    <td>67.7</td>
  </tr>
  <tr>
    <td>BM25&SBERT</td>
    <td>CombiSearch</td>
    <td>31</td>
    <td><b>70.7</b></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>N/A</td>
    <td>1</td>
    <td>19.2</td>
  </tr>
  <tr>
    <td>SBERT</td>
    <td>N/A</td>
    <td>1</td>
    <td>60.8</td>
  </tr>
  <tr>
    <td>BM25&SBERT</td>
    <td>N/A</td>
    <td>1</td>
    <td>58.4</td>
  </tr>
  <tr>
    <td>-</td>
    <td>Se<sup>2</sup></td>
    <td>151</td>
    <td>58.8</td>
  </tr>
</table>

Table 6: Oracle JGA scores across different pool-construction and combinatorial search methods.

Combinatorial Search We compare CombiSearch with the another combinatorial search method Se<sup>2</sup> (Liu et al., 2024a), which greedily selects the example that maximally enhances ICL performance when combined with previously selected examples. For a fair comparison, we use the same three-shot setting as Se<sup>2</sup> (\( k = 3 \)). While Se<sup>2</sup> employs a pool size of 50 examples, we use a smaller pool size of 30 (\( N = 30 \)) for CombiSearch, which serves as a handicap. See Appendix B.1 for details about Se<sup>2</sup>.

As shown in Table 6, CombiSearch outperformed Se<sup>2</sup> by 12% (rows 3 and 4). At the same time, CombiSearch is also more cost-efficient, requiring only 30 evaluations to construct \( D_{CombiSearch} \) for a given query (i.e., 30 examples divided into 10 groups and evaluated 3 times), compared to the 150 evaluations needed by Se<sup>2</sup> to select three examples. The results demonstrate that while each component provides distinct benefits, their optimal integration substantially enhances the overall effectiveness of CombiSearch.

Error Analysis We observed a notable behavior for the DST model that was not present in the 10-shot ICL setting. The model initially predicted all domain-related slots as <i>dontcare</i> in the first turn before updating them in later turns. This suggests that, with fewer examples, the model overinterprets unspecified slot values as a lack of user preference.

4.4 Individual vs. Combinatorial Scoring

We test whether scoring examples in groups (CombiSearch) yields higher accuracy than scoring each example independently (\( k = 1 \) in CombiSearch).

Setting In the oracle setting, for every query, we draw 100 candidate dialogues from the full MultiWOZ training set (using ground-truth dialogue
<table>
  <tr>
    <th>Scoring Method</th>
    <th>Model</th>
    <th>JGA (%)</th>
    <th>Calls/query</th>
  </tr>
  <tr>
    <td>Individual (100 cand.)</td>
    <td>8 B</td>
    <td>79.9</td>
    <td>100</td>
  </tr>
  <tr>
    <td>CombiSearch (\( M=3 \))</td>
    <td>8 B</td>
    <td>82.7</td>
    <td>30</td>
  </tr>
  <tr>
    <td>CombiSearch (\( M=9 \))</td>
    <td>8 B</td>
    <td>85.3</td>
    <td>90</td>
  </tr>
</table>
Table 7: Oracle JGA scores on the MultiWOZ test set across varying scoring methods.

<table>
  <tr>
    <th>Scoring Method</th>
    <th>Model Size</th>
    <th>JGA (%)</th>
  </tr>
  <tr>
    <td>Individual scoring</td>
    <td>8 B</td>
    <td>54.3</td>
  </tr>
  <tr>
    <td>CombiSearch (\( M=3 \))</td>
    <td>8 B</td>
    <td>56.2</td>
  </tr>
  <tr>
    <td>Individual scoring</td>
    <td>70 B</td>
    <td>56.9</td>
  </tr>
  <tr>
    <td>CombiSearch (\( M=3 \))</td>
    <td>70 B</td>
    <td>59.3</td>
  </tr>
</table>
Table 8: Oracle JGA scores by individual and combinatorial scoring (5% training data).

states) and score them with (i) Individual (100 calls/query), (ii) CombiSearch (\( M=3 \), 30 calls), and (iii) CombiSearch (\( M=9 \), 90 calls). In the practical setting, we use 5% of the training data under the open-source regime described in §4.1. Example scoring employs Llama-3-8B-Instruct as the DST model, and both Llama-3-8B-Instruct and 4-bit–quantized Llama-3-70B-Instruct are tested during inference.

Results In the oracle setting (Table 7) CombiSearch with \( M=3 \) improves JGA by 2.8% over Individual while cutting compute to one-third. With \( M=9 \), the gain widens to 5.4% at a comparable budget. The practical setting mirrors this pattern (Table 8): retrievers trained on CombiSearch data surpass their counterparts by roughly 3%, irrespective of the DST model’s size. Hence, exploiting interactions among examples not only reduces computational cost but also results in higher-quality training data for retrievers.

5 Related Work

Dialogue State Tracking To ensure efficient domain transfer in DST, zero-shot methods have been explored (Wu et al., 2019; Hosseini-Asl et al., 2020; Gupta et al., 2022). But since they struggle when domains differ significantly, few-shot DST has been studied, where models are often fine-tuned to follow in-context demonstrations (Shin et al., 2022; Venkateswaran et al., 2023), with additional approaches exploring instruction tuning (Feng et al., 2023) and meta-training (Chen et al., 2023). However, these approaches require extensive training and annotation as the DST schema changes.

Another approach is to train retrievers to select in-context examples per query. Hu et al. (2022) labeled positive and negative examples and fine-tuned a retriever with the data. They proposed representing a dialogue history as a summary state and tracking state changes at each turn. Expanding on this, King and Flanigan (2023) applied maximum marginal relevance to diversify retrieved examples, while Kulkarni et al. (2024) proposed synthetic data generation for fine-tuning retrievers.

In-Context Example Selection Traditional methods often examples based on their impact on task performance (Nguyen and Wong, 2023; Chang and Jia, 2023). However, many studies use a fixed set of examples across different queries, limiting query relevance. To address this, other works train retrievers to select examples per query (Hu et al., 2022; King and Flanigan, 2023; Rubin et al., 2022). Our work presents a data construction method for training retrievers in this setting.

For data construction, studies have explored scoring functions to distinguish positive and negative examples. Common methods include measuring semantic similarity to the query, applicable to DST via dialogue state similarity (Liu et al., 2022; Hu et al., 2022; King and Flanigan, 2023). Other methods leverage the model’s generation probabilities (Rubin et al., 2022; Iter et al., 2023; Peng et al., 2024). These generation probability-based methods suffer from surface-form competition (Holtzman et al., 2021). To mitigate this, while accounting for the model’s impact, we optimize for end-task performance instead. Furthermore, some studies employ combinatorial search as in our work (Liu et al., 2023; Gupta et al., 2023). For instance, Liu et al. (2024a) suggest iteratively selecting the best example at a time in a greedy manner. We demonstrated that our method achieves a lower computational cost and higher effectiveness.

6 Conclusion

We presented CombiSearch, a combinatorial method for scoring in-context examples in DST. We explored the oracle upper bound of performance and demonstrated that CombiSearch-scored examples achieve a substantially higher JGA score than the SOTA model. In practical settings, the retriever trained on CombiSearch-scored examples also outperformed the SOTA model. Our error analysis tool sheds light on common error types made by LLMs. Our work highlights the importance of utilizing high-potential data for retriever training and identifies several areas for improvement in LLMs to reduce common errors.
7 Limitations

Our method facilitates effective example selection and training data preparation for DST retrievers. It also demonstrates high ideal performance, defining the upper bound for retriever training. This highlights its potential and suggests that existing methods rely on suboptimal data preparation in DST.

Applying in-context example selection to DST inevitably introduces computational overhead—a well-known challenge in such methods. Since existing approaches require evaluating thousands of candidates against a small example set, computational cost remains a major concern. To address this, we optimized efficiency while ensuring selection aligned with task objectives. As a result, we achieved strong discriminative power with only 30 evaluations on a large example set. Despite these optimizations, high computational overhead remains a key challenge for the field.

Additionally, practical retriever training did not fully reach the estimated upper bound. Our research primarily focuses on example selection and data preparation rather than optimizing retriever training itself. While designing advanced training methodologies is beyond our scope, recent studies have explored reinforcement learning-based iterative retrieval and contrastive learning for ranking. Future work could integrate such approaches to improve retriever performance. Advancements in this direction would not only enhance DST but also refine industrial applications.

8 Ethical and Social Impacts

Social Implications  Our approach enhances dialogue state tracking efficiency and accuracy by optimizing the retrieval process via CombiSearch. This reduction in computational cost renders high-performing DST systems more accessible, particularly in resource-constrained settings, and promises to improve user experiences in domains such as customer service, healthcare, and education.

Potential Risks and Challenges  Despite these benefits, risks remain. If the retrieval process selects irrelevant or low-quality examples, the DST system may yield misleading outputs, potentially propagating misinformation. Moreover, while our method is more efficient than traditional combinatorial approaches, the need for multiple evaluations still poses scalability challenges—issues common to systems leveraging LLMs and example selection methods.

Mitigation Strategies  To address these challenges, future work should focus on enhancing retriever robustness to consistently select high-quality, diverse examples, thereby reducing misinformation and improving domain generalization. Additionally, developing techniques to adapt to unseen or out-of-domain dialogue scenarios—such as augmenting training data or designing retrievers to prioritize variability—will be essential. Investigating adaptive evaluation strategies that minimize computational overhead without compromising performance is also a key direction.

By systematically addressing these concerns, our method aims to maximize positive impact while mitigating inherent risks, thus advancing the field of DST.

Acknowledgments

This work was supported by the National Research Foundation of Korea (NRF) grant (RS-2024-00414981), by the Institute of Information & Communications Technology Planning & Evaluation (IITP) under the Leading Generative AI Human Resources Development grant (IITP-2025-RS-2024-00397085), and by the IITP grant (RS-2025-02215122, Development and Demonstration of Lightweight AI Model for Smart Homes), all of which were funded by the Korean government (MSIT). This research was also supported by the Korea Institute of Science and Technology Information (KISTI) in 2025 (No.(KISTI) K2SL1M1C1), aimed at developing KONI (KISTI Open Neural Intelligence), a large language model specialized in science and technology.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
Ting-Yun Chang and Robin Jia. 2023. Data curation alone can stabilize in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8123–8144, Toronto, Canada. Association for Computational Linguistics.

Derek Chen, Kun Qian, and Zhou Yu. 2023. Stabilized in-context learning with pre-trained language models for few shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1551–1564, Dubrovnik, Croatia. Association for Computational Linguistics.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaeck Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.

Gordon V. Cormack, Charles L. A Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 758–759, New York, NY, USA. Association for Computing Machinery.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yujie Feng, Zexin Lu, Bo Liu, Liming Zhan, and Xiaoming Wu. 2023. Towards LLM-driven dialogue state tracking. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 739–755, Singapore. Association for Computational Linguistics.

Raghav Gupta, Harrison Lee, Jeffrey Zhao, Yuan Cao, Abhinav Rastogi, and Yonghui Wu. 2022. Show, don’t tell: Demonstrations outperform descriptions for schema-guided task-oriented dialogue. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4541–4549, Seattle, United States. Association for Computational Linguistics.

Shivanshu Gupta, Matt Gardner, and Sameer Singh. 2023. Coverage-based example selection for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13924–13950, Singapore. Association for Computational Linguistics.

Ting Han, Ximing Liu, Ryuichi Takanobu, Yixin Lian, Chongxuan Huang, Dazhen Wan, Wei Peng, and Minlie Huang. 2021. Multiwoz 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation. Preprint, arXiv:2010.05594.

Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. 2024. Does prompt formatting have any impact on llm performance? Preprint, arXiv:2411.10541.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. 2022. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems.

Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.

Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In-context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627–2643, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu, and Chenguang Zhu. 2023. In-context demonstration selection with cross entropy
difference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1150–1162, Singapore. Association for Computational Linguistics.

Brendan King and Jeffrey Flanigan. 2023. Diverse retrieval-augmented in-context learning for dialogue state tracking. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5570–5585, Toronto, Canada. Association for Computational Linguistics.

Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024. Better zero-shot reasoning with role-play prompting. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4099–4113, Mexico City, Mexico. Association for Computational Linguistics.

Atharva Kulkarni, Bo-Hsiang Tseng, Joel Ruben Antony Moniz, Dhivya Piravipermal, Hong Yu, and Shruti Bhargava. 2024. SynthDIST: Synthetic data is all you need for few-shot dialog state tracking. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1988–2001, St. Julian’s, Malta. Association for Computational Linguistics.

Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with a language model using schema-driven prompting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4937–4949, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Itay Levy, Ben Bogin, and Jonathan Berant. 2023. Diverse demonstrations improve in-context compositional generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1401–1422, Toronto, Canada. Association for Computational Linguistics.

Haoyu Liu, Jianfeng Liu, Shaohan Huang, Yuefeng Zhan, Hao Sun, Weiwei Deng, Furu Wei, and Qi Zhang. 2024a. se2: Sequential example selection for in-context learning. In Findings of the Association for Computational Linguistics ACL 2024, pages 5262–5284, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.

Jinxin Liu, Li He, Yachen Kang, Zifeng Zhuang, Donglin Wang, and Huazhe Xu. 2023. Ceil: generalized contextual imitation learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.

Michael Xiyeang Liu, Frederick Liu, Alexander J. Finnacca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. 2024b. “we need structured output”: Towards user-centered constraints on large language model output. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’24, New York, NY, USA. Association for Computing Machinery.

Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurkarit, Peter Clark, and Ashwin Kalyan. 2023. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In ICLR.

Tai Nguyen and Eric Wong. 2023. In-context example selection with influences. Preprint, arXiv:2302.11042.

Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2024. Revisiting demonstration selection strategies in in-context learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9090–9101, Bangkok, Thailand. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, Seattle, United States. Association for Computational Linguistics.

Jamin Shin, Hangyeol Yu, Hyeongdon Moon, Andrea Madotto, and Juneyoung Park. 2022. Dialogue summaries as dialogue states (DS2), template-guided summarization for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3824–3846, Dublin, Ireland. Association for Computational Linguistics.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenjin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Bourdais, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaoylov, Pushkar Mishra, Igor Moltybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharar Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation learning with contrastive predictive coding. Preprint, arXiv:1807.03748.

Praveen Venkateswaran, Evelyn Duesterwald, and Vatche Isahagian. 2023. DISTRICT: Dialogue state tracking with retriever driven in-context tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5101–5112, Singapore. Association for Computational Linguistics.

Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808–819, Florence, Italy. Association for Computational Linguistics.

Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher Deenan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2).

A Error Analysis Framework

We introduce a comprehensive error analysis tool that categorizes DST prediction errors into seven types, allowing us to better understand the underlying causes of mistakes made by LLMs and inform directions for improving the methods.

Figure 3 shows our classification of error types (error types are marked in boxes). Recall that the predicted dialogue state in the current turn \( t \) is \( y^t = \Delta y^t \cup y^{t-1} \). Hence, at the top level, we classify errors based on two sources: (1) misprediction of slot-value pairs from the utterances in the current turn \( \Delta y^t \) (\emph{delta}), or (2) propagation of mispredictions from past turns \( y^{t-1} \) (\emph{error propagation}). These two sources share the same set of specific error types.

At the second level, we classify errors by comparing slot-value pairs between the ground-truth dialogue state and the prediction: (1) a slot-value pair in the ground-truth is absent in the prediction (\emph{miss}), and (2) a slot-value pair in the prediction is absent in the ground-truth (\emph{hallucination}). These two categories contain seven specific error types, detailed as follows.

For the \emph{miss} category, an error may arise because the prediction (1) misses an entire slot-value pair (\emph{miss-total}), (2) misses a special token “delete” (\emph{miss-delete}), (3) misses “don’t care,” which indicates the user’s openness to any options (\emph{miss-dontcare}), or (4) misses a slot by confusing it with another slot (\emph{miss-confuse}).

In the \emph{hallucination} category, errors may arise when the prediction (1) hallucinates a slot-value pair that is not informed by the dialogue (\emph{hallucination-total}), (2) hallucinates a value with a correct slot that does not match the dialogue (\emph{hallucination-value}), or (3) incorrectly modifies the previous dialogue state (\emph{hallucination-overwrite}).

This categorization systematically identifies various sources of errors and provides an intuitive framework for understanding DST model weaknesses. Additionally, it highlights the challenges faced by LLMs in key DST capabilities, such as utterance comprehension, hallucination detection, and identifying critical slots. We release the framework as a Python library to facilitate its easy integration in future studies.
A.1 Error Analysis of Modern LLMs

To understand the weaknesses of modern LLMs in DST, we used our error analysis framework to evaluate the zero-shot and 10-shot DST performance of four models: RefPyDST (King and Flanigan, 2023), gpt-3.5-turbo, Llama-3-8B-Instruct, and the 4-bit quantized Llama-3-70B-Instruct (Dubey et al., 2024). RefPyDST, the current SOTA model, was reproduced with Llama-3-8B-Instruct (Dubey et al., 2024) because code-davinci-002 has been deprecated. It uses prompts in Python format, while the other models employ natural language prompts. The prompts used in the experiments are shown in Appendix G.

In the 10-shot setting, RefPyDST retrieves in-context examples using its own fine-tuned retriever. For the other models, we randomly selected examples from the full training dataset.

Overall, error propagation errors were significantly more frequent than delta errors. In the zero-shot setting, error propagation hallucination-total was the most common, while in the random setting, error propagation miss-total dominated. For RefPyDST, an exception was observed where format errors were the most frequent due to Python code formatting, with hallucination-total being the next most prominent type of error.

In the 10-shot setting, miss errors outnumbered hallucination errors in all models except gpt-3.5-turbo. In the zero-shot setting, hallucination errors were generally more prevalent. Across all models, miss-total and hallucination-total errors—where the correct slot and value are either missed or incorrectly generated—were the most frequent. Miss-total errors affected subsequent dialogue turns more extensively, especially for name-related slots, due to issues like user confirmation and flexible preferences not being properly interpreted. Meanwhile, hallucination-total errors often stemmed from oversensitivity to system descriptions, incorrect coreference resolution, or user question misinterpretation, notably affecting area-related slots.

To reduce both miss and hallucination errors, addressing delta errors and ensuring accurate understanding of current dialogue turns are essential. We now incorporate both the dialogue state and conversation similarity in retrieval and provide explicit instructions in the prompts for better information extraction.

B Experiment Settings

B.1 Baselines

DiSTRiCT (Venkateswaran et al., 2023) Retrieves highly relevant examples based on semantic similarity within dialogues to enhance in-context tuning of T5-small for predicting slot values, eliminating the need for hand-crafted templates.

SGPDST (Lee et al., 2021) Applies schema-driven prompting by providing both dialogue context and schema information into the encoder of a pre-trained T5-base. Its prompts consist of candidate slot labels and their natural language descriptions.

SM2 (Chen et al., 2023) Adapts a meta-ICL framework by training T5-XXL on support dialogue datasets in DST format and evaluating on MultiWOZ 2.4. It aims to stabilize few-shot performance and reduce variance across prompts. It fine-tunes SBERT as an example retriever and uses the predicted dialogue state as a summary of dialogue history.

DS2 (Shin et al., 2022) Reframes dialogue state tracking as a summarization task, fine-tuning T5-large and BART-large to generate rule-based synthetic summary templates.

IC-DST (Hu et al., 2022) Applies ICL for DST by reformulating the task as a text-to-SQL generation, using SQL to represent dialogue states. An example retriever is trained by scoring examples based on the similarity of their dialogue state changes, and fine-tuning SBERT. Additionally, prior dialogue states are used as context instead of the full dialogue history.

SynthDST (Kulkarni et al., 2024) Builds on Hu et al. (2022), which reformulated DST as a text-to-SQL task. Unlike Hu et al. (2022), who fine-tuned retrievers for semantically similar dialogues, SynthDST generates high-quality synthetic datasets using LLMs without retriever fine-tuning. With code-davinci-002 deprecated, gpt-3.5-turbo is used to maintain system performance.

RefPyDST (King and Flanigan, 2023) Follows the ICL framework for DST introduced by Hu et al. (2022). It reformulates DST as a text-to-Python task and trains a retriever using the two-step data generation process from Hu et al. (2022). To enhance inference diversity, it employs maximum
Figure 3: Error tree to divide error type by slot and value

![Error tree to divide error type by slot and value](page_186_120_1077_495.png)

Figure 4: Statistics of error types made by different models in §4.2. From left to right: CombiSearch, Hybrid, and RefPyDST. Formatting errors in RefPyDST are not included.

![Bar charts showing statistics of error types made by different models](page_186_624_1077_180.png)

marginal relevance for example decoding and PMI-based decoding to mitigate surface form competition. With code-davinci-002 deprecated, we replaced it with gpt-3.5-turbo and substituted PMI\(^{\beta}\) decoding with beam search.

Se\(^2\) (Liu et al., 2024a) Selects in-context examples sequentially through beam search rather than the traditional ‘select and organize’ approach. This method leverages LLM feedback to capture inter-example relationships and sequential patterns, improving the contextual relevance of ICL prompts. For DST, we applied this approach using Joint F1 as the evaluation metric and Llama-3-8B-Instruct as the LLM to provide task-specific feedback. To reduce computational overhead, we set the beam size to 1 for beam search.

B.2 Hyperparameters

All hyperparameter tuning was conducted on the 5% validation set and performed manually. The parameters used in the CombiSearch process are presented in the “Common Settings” paragraph of §4. During inference, we configured both BM25 and SBERT to retrieve the top 10 highest-scoring examples. For LLM inference, beam search was applied as the decoding strategy, with the beam width set to 4. For retriever training, the batch size was set to 32. For each instance, we set the number of hard negatives \( B \) to 16, the number of negatives to 15, and the number of positives to 1. Consequently, each training instance consists of 1 positive, 16 hard negatives, and 15 negatives.

C Computing Details

<table>
  <tr>
    <th>Train Data Size</th>
    <th># of instance</th>
    <th>Running Time (h)</th>
  </tr>
  <tr>
    <td>1%</td>
    <td>524</td>
    <td>5</td>
  </tr>
  <tr>
    <td>5%</td>
    <td>2731</td>
    <td>26</td>
  </tr>
  <tr>
    <td>100%</td>
    <td>54971</td>
    <td>550</td>
  </tr>
</table>

Table 9: The number of instances corresponding to 1%, 5%, and 100% training data and the time required to generate \( D_{CombiSearch} \) by applying CombiSearch to these datasets.

Hardware All experiments were run on a single NVIDIA A6000 for Llama-3-8B-Instruct and a single NVIDIA A100 for 4-bit–quantized Llama-3-70B-Instruct. Inference speed was accelerated with vLLM; the 70 B model was additionally compressed with GPTQ (4-bit, group size
<table>
  <tr>
    <th>Training Method</th>
    <th>1% (hour)</th>
    <th>5% (hour)</th>
    <th>100% (hour)</th>
  </tr>
  <tr>
    <td>CombiSearch</td>
    <td>4</td>
    <td>20</td>
    <td>260</td>
  </tr>
  <tr>
    <td>RefPyDST</td>
    <td>4.5</td>
    <td>18</td>
    <td>36.5</td>
  </tr>
</table>

Table 10: Total training time for training retrievers based on each method. We report the total time required to train a retriever, so the running time of RefPyDST (King and Flanigan, 2023) for 1% and 5% is the aggregated training time of the three retrievers.

128) to fit on one GPU without noticeable accuracy loss.

Inference cost Table 9 lists the total wall-clock time required to construct \( D_{\text{CombiSearch}} \) under the 1%, 5%, and 100% training regimes.

Retriever training Training is performed on a single NVIDIA A100. For each dialogue \( x_i \) we create 10 positive instances and 31 negatives (\S 3.2). Because the large negative pool increases per-epoch cost, we train for 15 epochs on the 1% and 5% splits and for 7 epochs on the 100% split. Detailed statistics are given in Table 10.

Comparison with RefPyDST. For the same amount of training data, CombiSearch takes 4.5–16.2x as much time as RefPyDST. The computation time for CombiSearch increases linearly with the training data size as expected, because CombiSearch is not designed to explore all possible combinations. On the other hand, RefPyDST turns out to benefit from a speed boost as training progresses. RefPyDST trains three separate retrievers—one per data shard—whereas CombiSearch trains a single model on the full split. This design choice makes CombiSearch slower to train (e.g. 260 h vs. 36.5 h at 100%), yet the method still reaches higher Joint Goal Accuracy with only 5% of the data. In practice, its stronger generalization shortens the overall *time-to-target-accuracy* despite the longer training phase. Crucially, however, the extra compute buys markedly better sample efficiency: with only 1% of the data, CombiSearch already exceeds RefPyDST trained on 5%, and with open-source inference models it even beats RefPyDST trained on the full corpus. Since pre-training is a one-off expense and retrieval latency at inference time is identical across methods, the longer training phase is quickly amortised; in realistic deployments, months-to-years of serving time dominate the few hundred hours needed to build the CombiSearch retriever, making it the faster route to any target Joint Goal Accuracy.

D Retriever Details

In this section, we present the details related to our retriever.

First, in §D.1, we provide a detailed explanation of how each retriever’s input is formatted during both training and inference. In Appendix D.2, we provide experimental justification for the approach in which BM25 and SBERT are assigned different input components for the same query, as described in §3.1.

D.1 Embedding Model Input

There are three key stages where the embedding model’s input plays a crucial role: (1) when applying CombiSearch, (2) when training on data obtained from CombiSearch, and (3) when using the trained retriever for inference. We detail how the retriever’s input is structured in each case.

During the CombiSearch process, all examples are embedded using only their state change information. Specifically, to represent an example \( \{x_i, \Delta y_i\} \), we use only \( \Delta y_i \) for embedding. This applies equally to both training instances and query instances, which are sampled from the training set.

We trained an embedding model on \( D_{\text{CombiSearch}} \) to select examples that enhance DST performance when combined in the prompt (Figure 2-3) 2. Each query in \( D_{\text{CombiSearch}} \) is paired with a candidate pool of size \( N \), where each example is a dialogue context and dialogue state change pair, \( (e_i = (x_i, \Delta y_i)) \). Following prior work (Hu et al., 2022; King and Flanigan, 2023), we concatenate the dialogue context \( y^{t-1} \) and recent utterances \( A^t, U^t \) before encoding a turn context \( x_i \) (or query) with the embedding model \( emb \). To measure relevance between two input instances \( x_i \) and \( x_j \), we compute \( \cos(emb(x_i), emb(x_j)) \).

At inference time, we similarly embed a test instance \( x_t \) and perform a nearest-neighbor search to retrieve the most relevant examples for the test dialogue context.

D.2 Optimizing Retriever Inputs for Hybrid Search in CombiSearch

We conducted experiments to determine the optimal input format for each retriever.

Setting Few-shot DST was evaluated on a 20% validation set using 5% of the training data, assuming oracle settings with access to the gold state.

2We use all-mpnet-base-v2 (Song et al., 2020)
<table>
  <tr>
    <th>Index</th>
    <th>History</th>
    <th>Recent Utterances</th>
    <th>Gold State</th>
    <th>JGA (%)</th>
  </tr>
  <tr>
    <td>i</td>
    <td>O</td>
    <td>O</td>
    <td>\( \Delta y_{slot} \)</td>
    <td>55.6</td>
  </tr>
  <tr>
    <td>ii</td>
    <td>X</td>
    <td>O</td>
    <td>\( \Delta y_{slot} \)</td>
    <td>55.7</td>
  </tr>
  <tr>
    <td>iii</td>
    <td>X</td>
    <td>X</td>
    <td>\( \Delta y_{slot} \)</td>
    <td><b>66.8</b></td>
  </tr>
  <tr>
    <td>iv</td>
    <td>O</td>
    <td>X</td>
    <td>\( \Delta y_{slot} \)</td>
    <td>50.9</td>
  </tr>
</table>
Table 11: The JGA using BM25 varied its input. ‘O’ indicates inclusion, while ‘X’ indicates exclusion. \( \Delta y_{slot} \) represents the slots of state change.

<table>
  <tr>
    <th>Index</th>
    <th>History</th>
    <th>Recent Utterances</th>
    <th>Gold State</th>
    <th>JGA (%)</th>
  </tr>
  <tr>
    <td>i</td>
    <td>O</td>
    <td>O</td>
    <td>\( \Delta y \)</td>
    <td>44.4</td>
  </tr>
  <tr>
    <td>ii</td>
    <td>X</td>
    <td>O</td>
    <td>\( \Delta y \)</td>
    <td>44.2</td>
  </tr>
  <tr>
    <td>iii</td>
    <td>X</td>
    <td>X</td>
    <td>\( \Delta y \)</td>
    <td><b>67.3</b></td>
  </tr>
  <tr>
    <td>iv</td>
    <td>O</td>
    <td>X</td>
    <td>\( \Delta y \)</td>
    <td>42.0</td>
  </tr>
</table>
Table 12: The JGA using SBERT varied its input. ‘O’ indicates inclusion, while ‘X’ indicates exclusion. \( \Delta y \) represents the state change.

We assessed suitable inputs for BM25 and SBERT in hybrid search (Table 11, Table 12). Based on these results, we selected the two best input formats for each retriever and applied CombiSearch with combinatorial example scoring to evaluate overall performance (Table 13). Both retriever scores were z-score normalized and multiplied for re-ranking, following the procedure in §4.

Result As shown in Table 11, BM25 performed best when retrieving examples based only on the slot values of the query’s gold state change (iii), followed by using only the utterance and state change slots without history (ii). Similarly, Table 12 shows that SBERT achieved the highest performance when retrieving examples solely based on the query’s gold state change (iii), with the next best approach incorporating history, utterance, and state change (i).

To assess hybrid search, we combined the top two input formats from each retriever, resulting in four experimental conditions (Table 13). The best-performing combination used BM25 with utterance and state change slots (excluding history) (ii) and SBERT with examples retrieved based on the query’s gold state change (iii). We adopted this combination for pool construction in CombiSearch.

E Further Analysis

Here, we conduct a more in-depth analysis. First, we investigate the effect of the number of in-context examples in Appendix E.1. Next, we conduct an experiment by increasing the number of evaluations (\( M \)) for each example in the example scoring process, as described in Appendix E.2. Finally, we examine the score aggregation method that integrates the scores returned by BM25 and SBERT in Appendix E.3.

<table>
  <tr>
    <th>Input of BM25</th>
    <th>Input of SBERT</th>
    <th>JGA (%)</th>
  </tr>
  <tr>
    <td>ii</td>
    <td>i</td>
    <td>57.7</td>
  </tr>
  <tr>
    <td>ii</td>
    <td>iii</td>
    <td><b>64.8</b></td>
  </tr>
  <tr>
    <td>iii</td>
    <td>i</td>
    <td>58.8</td>
  </tr>
  <tr>
    <td>iii</td>
    <td>iii</td>
    <td>58.8</td>
  </tr>
</table>
Table 13: The oracle JGA using SBERT and BM25 varied its input. We aggregated the two scores using multiplication. Ultimately, to determine which input to use in the CombiSearch data preparation process, we conducted experiments under the same settings. The Roman numeral (e.g., i) in the inputs of BM25 and SBERT refers to the index of input combinations presented in the previous tables. Refer to Table 11 for BM25 input and Table 12 for SBERT input.

E.1 The Effects of the Number of In-Context Examples

Here, we modify the number of in-context examples included in the prompt under both the oracle setting, where the query’s gold state change is accessible for retrieval, and the practical setting.

Setting In both settings, we use 100% of the training data as the corpus and evaluate performance on a 20% validation set. We vary the number of in-context examples from 1 to 10 and measure the corresponding performance. We compare BM25, SBERT, and BM25&SBERT, where SBERT is used without fine-tuning. In the oracle setting, the slot values of the gold state change are used as input for BM25, while SBERT takes the gold state change as input. In the practical setting, both BM25 and SBERT take the dialogue context, represented as the dialogue state and the most recent utterance, as input.

Result The results in Figure 5 show that performance tends to improve as more shots are included, without noticeable saturation up to 10 shots. This trend is observed across BM25, SBERT, and BM25&SBERT. The performance gap between the oracle and practical settings was larger for SBERT than for BM25, suggesting that SBERT is more dependent on access to the gold state. Additionally, SBERT’s influence appears to have significantly contributed to the high performance of BM25&SBERT in the oracle setting.
Figure 5: Changes in JGA according to the number of shots. From left to right: BM25, SBERT, and BM25&SBERT. The “ideal” setting refers to a scenario where access to the gold state is available, while the “practical” setting refers to a scenario where it is not. Interestingly, when using BM25&SBERT, the performance gap between the oracle and practical settings was the largest.

<table>
  <tr>
    <th>Evaluations per example</th>
    <th>1% (JGA)</th>
    <th>5% (JGA)</th>
    <th>Running time (h)</th>
    <th>Est. build time (h)</th>
  </tr>
  <tr>
    <td>1</td>
    <td>65.02</td>
    <td>69.67</td>
    <td>2.3</td>
    <td>180</td>
  </tr>
  <tr>
    <td>2</td>
    <td>68.98</td>
    <td>72.81</td>
    <td>4.2</td>
    <td>330</td>
  </tr>
  <tr>
    <td>3</td>
    <td>69.12</td>
    <td>75.27</td>
    <td>6.0</td>
    <td>470</td>
  </tr>
  <tr>
    <td>5</td>
    <td>69.94</td>
    <td>78.14</td>
    <td>10.0</td>
    <td>769</td>
  </tr>
  <tr>
    <td>10</td>
    <td>76.91</td>
    <td>80.87</td>
    <td>19.5</td>
    <td>1515</td>
  </tr>
</table>

Table 14: Oracle-mode JGA and compute cost as a function of the evaluation budget \( M \). “Running time” measures the 10% test slice; “Est. build time” extrapolates to the full training set.

E.2 Trade-off Between Accuracy and Cost of the Evaluation Budget \( M \)

We assess how the per-example evaluation budget \( M \) influences oracle setting performance and compute cost, with the goal of identifying an \( M \) that offers the best balance between JGA and wall-clock time.

Setting The experiment follows the CombiSearch configuration in §4 under oracle setting that has access to the gold dialogue state change (oracle mode). Few-shot DST is evaluated on 1% and 5% subsamples of the MultiWOZ training set using a 10% test slice. The estimated build time for the full training set is obtained by multiplying the number of training dialogues by the average latency of scoring each instance \( M \) times.

Results Table 14 shows that accuracy rises linearly with \( M \), while runtime scales almost linearly. At 5% data, increasing \( M \) from 3 to 5 adds +2.87 absolute point JGA but inflates runtime by 66% and build cost by 64%. With only 1% data, the same change yields a negligible +0.82% absolute point. Hence, \( M = 3 \) emerges as the most cost-effective point, capturing the bulk of the attainable accuracy without incurring prohibitive compute.

E.3 Score Aggregation Methods

We compared the performance of two retrievers based on different score aggregation methods.

Setting Few-shot DST was conducted using 5% of the training data and evaluated on three test sets, each comprising 10% of the full test data. The final performance was determined by averaging the results across these sets. This experiment was conducted after all design choices were finalized to assess the effectiveness of our aggregation method. Since the focus was on evaluating score aggregation, we assumed access to the gold state, eliminating retrieval errors from consideration as in §4.2.

We re-ranked retrieved examples using SBERT and BM25 scores through the following methods:

Multiplication The approach selected prior to this experiment, Multiplication, combines SBERT and BM25 scores by multiplying them. Due to scale differences, direct multiplication is infeasible. Instead, both scores are first normalized using z-score transformation before multiplication.

Weighted Sum This method sums SBERT and BM25 scores after applying z-score transformation to account for scale differences. Equal weighting is used, setting \( \alpha = 0.5 \).

RRF (Cormack et al., 2009) Reciprocal Rank Fusion (RRF) assigns a score based on the reciprocal of an example’s rank in each retriever’s list. These scores are then summed across both retrievers. Unlike z-score normalization, RRF does not alter score distributions, making it a widely used and simple re-ranking approach.

Union Top-K This method selects the top-K examples by alternating between SBERT and BM25’s ranked lists in a round-robin fashion. If a duplicate
<table>
  <tr>
    <th>Aggregation Method</th>
    <th>JGA (%)</th>
  </tr>
  <tr>
    <td>Multiplication</td>
    <td>75.063</td>
  </tr>
  <tr>
    <td>Weighted Sum</td>
    <td>71.0</td>
  </tr>
  <tr>
    <td>RRF</td>
    <td>70.8</td>
  </tr>
  <tr>
    <td>Union Top K</td>
    <td>67.7</td>
  </tr>
</table>

Table 15: The result of score aggregation method.

appears earlier in the ranking, the next available example is selected instead.

Result  Multiplication delivers markedly better performance than the alternative methods, confirming that our design choice was appropriate.

F  An Example of Retrieved Example

We showcase an example in Table 16. We observe substantial overlap between the query’s previous dialogue state and that of the CombiSearch example. Both dialogues contain complex prior states involving two entities—restaurant and attraction in the query, restaurant and hotel in the example—before the user transitions to a new request (booking a taxi). Notably, the CombiSearch example mirrors the query’s dialogue history and therefore presents a user utterance ("i also need a taxi..") that exhibits a similar structural shift: it refers back to two previously mentioned points of interest (departure and destination) using coreference. Resolving such coreferences is challenging in DST, and in-context examples that display this pattern can enhance model performance. By contrast, although the RefPyDST example also contains a taxi request between two entities, its preceding dialogue state overlaps far less with the query’s rich context and lacks the nuanced history captured by CombiSearch.

This case illustrates that our hybrid CombiSearch retriever finds examples sharing not only dialogue-state similarity but also a parallel conversational flow and structure, leading to significant performance gains.

G  Prompt for Few-shot DST experiments
<table>
  <tr>
    <th>Example Type</th>
    <th>Category</th>
    <th>Example</th>
  </tr>
  <tr>
    <td rowspan="4">Query</td>
    <td><b>Previous Dialogue State</b></td>
    <td>restaurant-book day: saturday, restaurant-book people: 7, restaurant-book time: 17:15, restaurant-food: spanish, restaurant-name: la raza, restaurant-area: centre, attraction-name: saint barnabas press gallery</td>
  </tr>
  <tr>
    <td><b>System Utterance</b></td>
    <td>‘saint barnabas press gallery s postcode is cb13ew . would you like more information ?’</td>
  </tr>
  <tr>
    <td><b>User Utterance</b></td>
    <td>‘no that s it . i also need a taxi taking me from the gallery to the restaurant .’</td>
  </tr>
  <tr>
    <td><b>Dialogue State Change</b></td>
    <td><i>taxi-destination: la raza, taxi-departure: saint barnabas press gallery</i></td>
  </tr>
  <tr>
    <td rowspan="4">CombiSearch Example</td>
    <td><b>Previous Dialogue State</b></td>
    <td>restaurant-book day: wednesday, restaurant-book people: 6, restaurant-book time: 16:15, restaurant-food: italian, restaurant-name: prezzo, restaurant-area: west, hotel-book day: wednesday, hotel-book people: 6, hotel-book stay: 1, hotel-name: huntingdon marriott hotel, hotel-area: west, hotel-pricerange: expensive</td>
  </tr>
  <tr>
    <td><b>System Utterance</b></td>
    <td>“i was able to successfully book you for 1 night . your reference number is bxwli8k3 . is there anything more i can help you with ?”</td>
  </tr>
  <tr>
    <td><b>User Utterance</b></td>
    <td>‘i also need a taxi to get from the restaurant to the hotel .’</td>
  </tr>
  <tr>
    <td><b>Dialogue State Change</b></td>
    <td><i>taxi-destination: huntingdon marriott hotel, taxi-departure: prezzo</i></td>
  </tr>
  <tr>
    <td rowspan="4">RefPyDST Example</td>
    <td><b>Previous Dialogue State</b></td>
    <td>restaurant-name: pipasha restaurant, attraction-name: club salsa,</td>
  </tr>
  <tr>
    <td><b>System Utterance</b></td>
    <td>‘pipasha is in the east . the full address is newmarket road fen ditton postal code cb58pa . can i help you with anything else ?’</td>
  </tr>
  <tr>
    <td><b>User Utterance</b></td>
    <td>‘i am gonna need a taxi to get from the club to the restaurant . can you schedule that for me ?’</td>
  </tr>
  <tr>
    <td><b>Dialogue State Change</b></td>
    <td><i>taxi-destination: pipasha restaurant, taxi-departure: club salsa</i></td>
  </tr>
</table>

Table 16: Representative example retrieved by each retriever for the target query. The CombiSearch example shares substantial Previous Dialogue State with the query. This indicates that CombiSearch effectively retrieves examples that share not only dialogue state change but also a similar conversational flow and structural context.
<table>
  <tr>
    <th>Role</th>
    <th>Message</th>
  </tr>
  <tr>
    <td><b>System</b></td>
    <td>**Task:** You are an expert in Dialogue State Tracking (DST) focused on managing and updating the dialogue state change based on system-user interactions. The dialogue state represents the user’s preferences and booking details across different domains: Hotel, Train, Attraction, Restaurant, and Taxi.</td>
  </tr>
  <tr>
    <td><b>User</b></td>
    <td>
      **Guidelines:**
      <ol>
        <li>
          **Hotel:**
          <ul>
            <li>
              - **Slots:** name (string), pricerange (PriceRange), type (HotelType), parking (Option), book stay (integer), book day (DayOfWeek), book people (integer), area (Area), stars (integer between 0 and 5 or "dontcare"), internet (Option).
              <ul>
                <li>
                  - **Valid Values:**
                  <ul>
                    <li>PriceRange: "dontcare", "cheap", "moderate", "expensive".</li>
                    <li>HotelType: "hotel", "guest house", "dontcare".</li>
                    <li>Option: "yes", "no", "dontcare".</li>
                    <li>DayOfWeek: "monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday".</li>
                    <li>Area: "dontcare", "centre", "east", "north", "south", "west".</li>
                  </ul>
                </li>
              </ul>
            </li>
          </ul>
        </li>
        <li>
          **Train:**
          <ul>
            <li>
              - **Slots:** destination (string), departure (string), day (DayOfWeek), book people (integer), leaveat (hh:mm or "dontcare"), arriveby (hh:mm or "dontcare").
            </li>
          </ul>
        </li>
        <li>
          **Attraction:**
          <ul>
            <li>
              - **Slots:** name (string), area (Area), type (AttractionType).
              <ul>
                <li>
                  - **Valid Values:**
                  <ul>
                    <li>AttractionType: "architecture", "boat", "church", "cinema", "college", "concert hall", "entertainment", "hotspot", "multiple sports", "museum", "nightclub", "park", "special", "swimming pool", "theatre", "dontcare".</li>
                  </ul>
                </li>
              </ul>
            </li>
          </ul>
        </li>
        <li>
          **Restaurant:**
          <ul>
            <li>
              - **Slots:** name (string), food (string), pricerange (PriceRange), area (Area), book time (hh:mm or "dontcare"), book day (DayOfWeek), book people (integer).
            </li>
          </ul>
        </li>
        <li>
          **Taxi:**
          <ul>
            <li>
              - **Slots:** destination (string), departure (string), leaveat (hh:mm or "dontcare"), arriveby (hh:mm or "dontcare").
            </li>
          </ul>
        </li>
      </ol>
    </td>
  </tr>
</table>

**Example 1 of Dialogue State Change Update Task:**
**Previous Belief State:**
{
  'train-destination': 'cambridge', 'train-day': 'thursday', 'train-arriveby': '15:45',
  'train-departure': 'kings lynn'
}

**Latest Conversation Between System and User:**
**System:** "It leaves at 14:11 and lasts 47 minutes. Is there anything else?"
**User:** "Yes, I am looking for a restaurant that serves North American food, preferably in the expensive price range."

**Instructions:**
- Based on the user’s latest input, update the belief state by correctly identifying and filling in the relevant domain(s), slot(s) and value(s).
- Provide your output strictly in the Required Output Format below.

**Required Output Format:**
**Dialogue state change after Latest Conversation Between System and User:**
{
  "DOMAIN_1": "SLOT_1": "VALUE_1", "DOMAIN_2": "SLOT_2": "VALUE_2", ...
}
Assistant
{
  'restaurant-food': 'north american', 'restaurant-pricerange': 'expensive'}
Table 17: An example of prompt for few-shot DST. For brevity, this example only includes one example without query. However, we set ten shots as a default setting for fair comparison with baseline models.

Reviewer paper summaries: This paper proposes a novel example selection method for Dialogue State Tracking (DST) called CombiSearch, designed to optimize the example retrieval process in In-Context Learning (ICL)-based DST tasks. In DST, training a retriever to select high-quality context examples is crucial for improving performance. Existing methods score each example independently and rank them primarily based on their similarity to the query dialogue state. However, this approach has three significant limitations: (1) Examples are scored independently, without considering the overall impact of multiple examples in combination; (2) Relying solely on dialogue state similarity may overlook contextual characteristics, leading to selected examples that fail to support complex linguistic phenomena (e.g., coreference resolution); (3) The scoring criteria are not directly optimized for DST performance but rely on indirect signals.
To address these issues, this paper introduces CombiSearch, which constructs a diverse example pool and employs a combinatorial scoring mechanism to optimize example selection from a global perspective, thereby maximizing DST performance improvements. Experimental results demonstrate that CombiSearch outperforms existing methods on both the MultiWOZ and SGD datasets, exhibiting remarkable data efficiency—achieving superior performance over all baselines using only 5% of the data. Furthermore, under the ideal assumption of perfect retrieval, CombiSearch improves Joint Goal Accuracy (JGA) by 12% over traditional methods, further proving the critical role of optimized example selection in enhancing DST performance.
Summary of strengths: The Dialogue State Tracking (DST) task often involves complex scenarios such as reasoning, implicit mentions, and coreference resolution. However, existing large-scale language models still face certain bottlenecks under zero-shot or low-resource conditions. Meanwhile, the rapid advancement of large-scale pretrained language models continues to enhance natural language understanding and generation capabilities, pushing the boundaries of dialogue system technology. Therefore, a key challenge in improving DST performance is how to fully leverage the strong generalization ability of large models while maintaining compatibility with mainstream DST modeling approaches.
In this paper, we introduce CombiSearch, a novel example selection method that integrates cutting-edge large-model paradigms with the modular design of task-oriented dialogue systems. Specifically designed for In-Context Learning (ICL)-based DST tasks, CombiSearch overcomes the limitations of traditional independent example scoring by incorporating combinatorial search, optimizing example selection from a global perspective to effectively capture inter-example synergy, thereby significantly enhancing DST performance. Moreover, beyond DST, our approach provides a new perspective on ICL example selection, offering valuable insights for example retrieval and combinatorial optimization in broader applications.
Summary of weaknesses: 1. While the current abstract clearly describes the method and results, it does not sufficiently emphasize the nature of the problem and its impact. Strengthening this aspect would make the contribution more compelling.
2. The Introduction does not provide a conceptual introduction to In-Context Learning (ICL), which might make it difficult for readers unfamiliar with the concept to follow the discussion. Consider adding a brief explanation to improve readability.
3. In the Introduction, the paper mentions identifying similar dialogue states to the query during the pool construction and re-ranking stages based on features such as conversational flow, style, and structure. However, there is no experimental evidence or citations supporting the necessity of these features or their direct impact on addressing the stated limitations.
4. The paper does not discuss the criteria for selecting M when computing CombiScore. It is recommended to include an ablation study to demonstrate that the chosen M is optimal.
5. Compared to the previous submission, the paper has added more baseline models. However, to further enhance credibility, it is suggested to include additional baselines for a more comprehensive comparison.
Comments, suggestions and typos: 1. It is recommended to add a **brief overview of the core concept of In-Context Learning (ICL)** in the **Introduction** to help readers better understand its relevance to the paper.  
2. In the **Problem Setting** section, consider providing a **complete workflow description** of **In-Context Learning for DST**, detailing how the process is structured and implemented.  
3. There is an **inconsistency in data sampling** between the **Closed-source** and **Open-source** settings. In the **Closed-source** setting, multiple independently sampled datasets (1%, 5%) are used, whereas in the **Open-source** setting, only a single training set (1%, 5%) is utilized. This discrepancy may lead to **distributional differences** in training data, potentially affecting the **comparability of experimental results** across the two settings.
Overall assessment scores: 3