DiSTRICT: Dialogue State Tracking with Retriever Driven In-Context Tuning
Authors: Praveen Venkateswaran
Evelyn Duesterwald
Vatche Isahagian
Venue: EMNLP 2023 Main (2023)
Keywords: Dialogue state tracking
in-context tuning
semantic retrieval

Abstract: Dialogue State Tracking (DST), a key component of task-oriented conversation systems, represents user intentions by determining the values of pre-defined slots in an ongoing dialogue. 
Existing approaches use hand-crafted templates and additional slot information to fine-tune and prompt large pre-trained language models and elicit slot values from the dialogue context. Significant manual effort and domain knowledge is required to design effective prompts, limiting the generalizability of these approaches to new domains and tasks. 
In this work, we propose DiSTRICT, a generalizable in-context tuning approach for DST that retrieves highly relevant training examples for a given dialogue to fine-tune the model without any hand-crafted templates. 
Experiments with the MultiWOZ benchmark datasets show that DiSTRICT outperforms existing approaches in various zero-shot and few-shot settings using a much smaller model, thereby providing an important advantage for real-world deployments that often have limited resource availability.

DiSTRICT: Dialogue State Tracking with Retriever Driven In-Context Tuning
Praveen Venkateswaran, Evelyn Duesterwald, Vatche Isahagian IBM Research
(praveen. (praveen.venkateswwrrr venkateswaran, duester, vatchei 10 @ibm com
Abstract
kewyChaagea Diolog State:
User: Iree hotel NYC for people rights
hotel-location: NYC horal-paople hocal-day- hocel-stay :2
Dialogue State Tracking (DST), key compo- nent of task-oriented conversation systems, rep- resents user intentions by determining val- ues of pre-defined slots in ongoing dialogue. Existing approaches use hand-crafted templates and additional slot information fine-tune and prompt large pre-trained language models and elicit slot values from the dialogue context. Sig- nificant manual effort and domain knowledge is required to design effective prompts, limiting the generalizability these approaches new domains and tasks. In this work, we propose DiSTRICT, a generalizable in-context tuning approach for DST that retrieves highly rele- vant training examples for given dialogue to tne-tune the model without any land-crafted templates. Experiments with MultiWOZ benchmark datasets show that DiSTRICT out- performs existing approaches various zero- shot and few-shot settings using much smaller model, thereby providing an important advan- lage real-world deployments that often have limited resource availability.
System: Okey, Thvee you booked at the Eix Eourtyrd Marrictt Reeerr anna number 7onbat
User:
I need resarvation for an Ind
New//hanged Dialog State:
group at 22:0 zamee day
rastaurant--yoo Inxlaan
System: successfully r860 vation the myal spice
restourant- patpuurant-cpapee:.. restaurant- 1:2:45
restaurant Jull
Actually, ww 11 to che the restaurant reservation to July
User:
New//hhanged Dialog State:
System: Okay, have modifled successfully
resteurant-day July
Figure I: An example of multi-domain dialogue state tracking (DST) in a conversation to book hotel stay and reserve a table at restaurant.
these challenges, DST solutions need to be gen- eralizable to new zero-shot and few-shot settings with minimal overhead while also maintaining a small model footprint to enable compute- and cost-effective deployments.
Recent advances in DST have leveraged pre- trained language models (LMs) to elicit slot val- ues from dialogues using primarily methods fine-tuning and in-context learning. However, both methods suffer from several shortcomings
1 Introduction
Task-oriented dialogue systems are increasingly used to enable users to perform tasks through multi- turn conversations in various domains such as travel reservations, banking transactions, online shop- ping. Dialogue state tracking (DST) is critical component of these systems that tracks user re- quirements by determining key information each turn in the dialogue (Jacqmin et al., 2022). Given a predefined schema of task parameters (i.e., slots), DST models identify and represent the dialogue state pairs of slots and their corresponding val- shown in Figure 1.
Fine-tuning LMs - Most existing approaches con- dition LMs for DST by fine-tuning their param- eters using prompts derived from historical dia- logue data. However, they typically rely on hand- crafted prompt templates that include slot-specific questions, value based functions, or even text-to- code snippets (Lin et al., 2021b: Cao and Zhang, 2021). In addition to the manual effort required, these templates are customized to specific domains and slots, and hence have low generalizability new domains. Some approaches also extend fine- tuning first train models on additional natural language tasks using external datasets which ex- pensive and requires access to significantly more data. They also often rely on additional information
In real-world deployments, and slots are frequently added improve user func- tionality. Hence, periodic updates to the models may be required, even when the new domains offer little to no dialogue data for training. To address
task domains

---

from the schema such as slot descriptions and task instructions (Mi et al., 2022). However, real-world datasets may not always have this necessary infor- mation, which again limits their generalizability. In-context learning - LMs have shown remark- able performance through in-context learning of new tasks (Brown et al., 2020), where raw LM (i.e. pre-trained LM without fine-tuning task- specific data) is prompted during inference using onput-output task examples to condition the gener- ated output. For DST, approaches have leveraged this to craft prompts containing examples of di- alogue history and slot values (Hu et al., 2022). However, similar to fine-tuning approaches, these prompt examples are hand-crafted, customized, and require significant effort. Additionally, as result of using raw LMs, these approaches have rely on extremely large models limiting their practical use. Importantly, has been shown that prompting raw LMs without fine-tuning is often oversensitive to example choices and instruction wording (Chen et al., 2022), and can demonstrate undesirable biases that significantly reduce perfor- mance (Zhao et al., 2021; Liu et al., 2021).
and few-shot settings, while using much smaller model, thus demonstrating its practi- cality and applicability to real-world deploy- ments.
Related Work
Dialogue state tracking (DST) is a critical yet chal- lenging task for task-oriented dialogue systems (Williams et al., 2014), and several multi-domain benchmark conversation datasets have been pro- posed (Budzianowski al., 2018; Eric et al., 2019; Rastogi et al., 2020) evaluate research efforts. A majority state-of-the-art approaches fine- tune language models using hand-crafter templates containing descriptions or questions related to the dialogue slots. Shah et al. (2019) used slot descrip- tions and examples of slot values to create tem- plates while Lin et al. (2021b) and Lee et al. (2021) provided different types of manually annotated slot descriptions to the model. Mi al. (2022) extended this by also including task instructions and other constraints. In contrast, our approach does not re- quire hand-crafted templates for fine-tuning and is hence more easily generalizable.
In this work, we address these challenges and present DiSTRICT, generalizable approach for di- alogue state tracking that fine-tunes LM using rel- evant examples (i.e., in-context uming). For a given input dialogue and slot to be tracked, DiSTRICT retrieves semantically similar dialogues and slots from available historical data in zero-shot and few- shot settings, and concatenates them into prompt with no hand-crafted template requirements. We first fine-tune the language model using in-context examples and input dialogues from the training set, and subsequently perform similar inference on test inputs as shown in Figure 2. Specifically, we make the following contributions -
Another set of approaches aim to improve zero- shot performance by exploiting external knowledge and datasets from other natural language tasks be- fore fine-tuning a model for DST. For instance, Gao et al. (2020); Li et al. (2021); Lin et al. (2021a) pre-train models on reading comprehension data, Shin et al. (2022) reformulate DST as dialogue summarization task with external annotated data, and Hudeček et al. (2021) use semantic analysis and named entity recognition to identify slots. In contrast, our approach does not require any extra datasets or training efforts.
In-context learning (ICL) for DST has been ex- plored as part of larger set of few-shot generative tasks (Madotto et al., 2021; Xie et al., 2022), but the lack of task-specific prompt design resulted in low performance. Hu et al. (2022) and Gupta et al. (2022) achieved improved performance using extremely large models, where the former reformu- lated DST as a text-to-SQL. task by using semantic matching to identify relevant examples that were subsequently crafted into SQL queries, and the lat- ter manually created example dialogues containing combinations of slots in the schema.
To the best of our knowledge, DiSTRICT is the first DST approach use in-context tun- ing by fine-tuning LM with in-context exam- ples.
DiSTRICT avoids shortcomings of prior ap- proaches by leveraging relevant existing di- alogue and slot examples without requiring hand-crafted prompts or external datasets, thereby improving generalizability and avoid- ing manual overhead.
Recent efforts have shown that the shortcomings of ICL (Liu et al., 2022; Min et al., 2022) can be overcome through in-context tuning of LMss. Gu-
Our evaluation shows that DiSTRICT outper- forms existing approaches on most zero-shot

---

Test Ingut
Dialog Contact
Azziss to esammpe dislags from same domain
Promot
Predicted Dielog State
User: 22:45
Inciion
Example a::
In-cortext netr ever (Fewi Jil-- shatt
User Book me restas tabla for
Fine-tunad language Modcl
Systeme have suctesstully mace mocee
Monday at 25:30
reseroation in the rogal spice
System: Thes bensiing lus beten compleeted
restauratt peocle
Query recsurant--oeoell
Stat: sosta zant peoplec
Exomple al2
Diolog condict, Query stot
Prompt
In-cortext Betr ever
Exampic #2:
User: need hotel in NDC for people
Fine-tuned language
In-Context Examples (single-tum dialogues from training
12eroshee2
System: people:
rights confirm number
restaurant-pespee
simlar dialogs and from other
Search semuntically
Ficinng Query
Excmple #2:..
Figure 2: Overview of DiSTRICT. Given input dialogue and slot to be tracked, high-relevance examples are first identified by the retriever and used to create the prompt. In (1) zero-shot settings, the retriever must search for semantically similar dialogue contexts and slots from other domains, while in (2) few/full-shot settings, the retriever additionally has access to some example dialogues from the same domain that could also contain the query slot.
rurangan al. (2020) and Liu al. (2022) demon- strate the improved performance of models fine- tuned with examples compared to ICL over a va- riety of language tasks. In this work, we leverage this concept specifically for DST.
examples from the available training set of dia- logues. Intuitively, historical labeled dialogues contain information about slots and their values under different conversational contexts. Hence, for an input dialogue and given query slot, condition- ing the model during fine-tuning using example dialogues that are semantically similar and addi- tionally contain the same similar slots, enables the model better learn the association between slots, their values, and the context.
3 Approach
We first present the background and some defini- tions for dialogue state tracking before describing our approach.
As shown in Figure 2, the retriever in DiSTRICT performs semantic matching of the input dialogue and slot with single-turn training set conversations as examples (i.e. pair of user-system utter- ances). This design choice is due the fact that large prompts require additional memory and com- pute, significantly increasing the training time of the model. Furthermore, real-world dialogues can be lengthy, and the context needed to find the value of a particular slot can often be limited to sin- gle sentence. Hence, constraining the in-context examples to single-turn conversations reduces the prompt size, enables the addition of more examples, and removes irrelevant dialogue context.
3.1 DST Background
A task-oriented dialogue consists of a multi-turn conversation between a user U and the system A. Given a dialogue context C2 as the se- quence of utterances until turn t, (i.e.) C. = [Ai, UI,.., Ar, Uz), the goal of DST is to predict the dialogue state yt, defined as a set of (slot, value) pairs:
Ut -ー ( C. Vs' ES}
where denotes the set possible slots prede- fined in an ontology or schema. In a multi-domain setting, the schema can comprise of different do- mains or topics, each corresponding to a service such as restaurant booking or banking. The slots as- sociated with each domain can be either categorical with set of candidate values (c.g. restaurant-open 'True' / 'False"), ), or non-categorical, where the value is a span in the dialogue context (e.g. hotel- name = 'Courtyard Marriott').
Formally, we define a dataset D = {(e), sf,, uf)} consisting of single-turn dialogue examples ej, con- taining observed slot s) and its corresponding value u/- For given input dialogue context C, and query slot s", we retrieve the à most relevant examples &, E DD based on the similarity between their text embeddings
3.2 In-Context Retriever
& max max((axillllll (sim saaemh, Demb &
The key concept behind our approach is the identifi- cation of the most semantically relevant in-context
where @ denotes concatenation.

---

3.3 Applicability to Zero-shot and Few-shot
4 Experiments
To illustrate the generalizability of the retriever zero-shot and few-shot settings, use the ex- input dia-
Model
#Parameters
Size Difi.
ample shown in Figure 2. Given logue from the restaurant domain and the query slot restaurante -peope,,,,, the retriever identifies the k most relevant single-tur in-context examples derived from the training set.
STARC
355M
17x
TSDST
60M
1x
TransferQA
770M
13x
DS2-BART
406M
7x
DS2-T5
770M
13x
IC-DST
175B
3Kx
In zero-shot setting (Figure 2-(1)), dialogue and slot examples from the restaurant domain would not be available. Hence, the retriever iden- tifies semantically similar examples from other domains. For instance, conversations about ho- tel reservations have similar contexts, and the slot hotel- ppople is semantically similar to the query slot. The retrieved example thus conditions the model to look for the number of people men- tioned in the dialogue.
DISTRICT
60M
'able Comparison baselines and their pre-trained model size. TRADE (Wu et al., 2019) does not use a pre-trained model.
4.1 Datasets and Evaluation
MultiWOZ (Budzianowski al., 2018) is multi- domain task-oriented dialogue benchmark dataset that consists of around 10k multi-turn dialogues over 7 domains. The dataset has been refined and erroneous annotations have been corrected over multiple versions. To enable comparisons with most existing approaches, we use the MultiWOZ 2.0 and MultiWOZ 2.1 (Eric et al., 2019) datasets in our evaluation, and follow the same data pre- processing and domain selection steps as prior work (Wu et al., 2019; Gao er al., 2020; Lin et al., 2021b). To make consistent comparisons to prior work in zero-shot and few-shot settings, we use the same Joint Goal Accuracy (JGA) metric to evaluate our approach. For given turn, JGA compares pre- dicted dialogue states to the corresponding ground truth states and considers the prediction accu- rate if and only if all predicted values match the ground truth values (Wu al., 2019; Wen et al., 2017; Kumar et al., 2020).
In few-shot and full-shot settings (Figure 2-(2)) the set of available examples would include other dialogues from the same domain which could also contain the query slot. Hence, the most similar ex- amples retrieved would demonstrate the values of the query slot when used in similar restaurant reser- vation contexts (e.g. aestauratt-- peerrranttppllees. 8). We note that our approach requires no changes for the different settings, and can be easily extended to include additional information like slot descrip- tions to further enhance the semantic retrieval.
3.4 In-context Tuning
We fine-tune the language model by retrieving in- context single-turn dialogue examples for each di- alogue in the training set. shown in Figure 2, to create the input to the model, we annotate prefixes to each of the in-context examples Ek, dialogue context C, and the query slot 8 to enable the model to distinguish between them, and then concatenate them into single input sequence. We then fine-tune encoder-decoder based language model, where the input is passed to the encoder, and the decoder generates the corresponding value for the query slot. The model in-context tuning ob- jective to minimize the negative lieeliil loss
4.2 Implementation
DiSTRICT uses T5-small (60M parameters) (Raf- fel et al., 2020) which is one of the smallest pre- trained models available' It has 6 encoder-decoder layers and the size of each layer is 512. We fine-tune using an AdamW optimizer (Loshchilov and Hutter, 2018) with initial learning rate of le-4. For the retriever, we use SeRR (Reimers and Gurevych, 2019) and perform seman- tic search with cosine similarity using the FAISS library (Johnson et al., 2019). Unless specified oth- erwise, we set the number of in-context examples to be k = 3. We use a single NVIDIA V100 GPU
12
((A) = Σ logp(r; Ex
1 C, +
where n the total number of slots in the ontology and + denotes concatenation.
httpps:/ /tl:://hugg I inge co/ transf Foreers/ v2. 9. ttppetraineeddd..mmem....... modeiieeeeeeemeeeeeeeeeeee st.mll

---

Model
Attraction
Hotel
Restaurant
Taxi
Train
Avg.
MultiWoz 2.0
TRADE
19.87
13.70
11.52
60.58
22.37
25.76
TSDST
33.09:11.60
21.21:10.61
21.82:10.91
05.09:10.12
35.42::1.42
35.20=20.59
DISTRICT
33.6110073
22.82.10.43
21.3011
66.53.10.21
46.6111.56
38.17
MultiWoz 2.1
TRADE
20.06
14.20
12.59
59.21
22.39
25.69
TransferQA
31.25
22.72
26.28
61.87
36.72
35.77
DISTRICT
33.74:10.80
22.29:10.29
24.17:10.94
66.34..0.79
46.93:1...4
38.69-10.89
Table Zero-shot joint goal accuracy (JGA) on MultiWOZ 2.0 and 2.1. Results for TRADE and T5DST from Lin et al. (b)2 and TransferQA from Lin et al. (2021a).
for our experiments and provide further details in
4.4 Experimental Settings
the appendix.
Zero-shot - Similar to prior work (Lin et al., 2021a Wu al., 2019), the retriever and model have ac- cess to training data from all domains except from one 'unseen' domain, on which the model is eval- uated. We note that our retriever does not result in any information leakage since no examples and slots are included from the unseen domain.
4.3 Comparison Baselines
We evaluate DiSTRICT against existing DST baselines as shown in Table 1. The table shows that, except for TSDST which also uses T5-small, prior DST approaches use models that are significantly larger compared to DiSTRICT.
Cross-domain few-shot We include three few- shot settings, where the retriever and model addi- tionally have access to 1%, 5%, and 10% of train- ing data from the unseen domain, similar to Shin et al. (2022); Wu et al. (2019).
TRADE (Wu et al., 2019) uses slot and domain embeddings well copy mech- anism to track slot values across domains.
Multi-domain few-shot We follow the multi- domain scenario from Shin al. (2022); Wu et al. (2020), where 1%, 5%, and 10% of the entire train- ing data are sampled for model training and re- trieval.
STARC (Gao et al., 2020) prompts two different instances of the RoBERTa-large (Liu et al., 2019) model with separate natural language questions for categorical and non-categorical slots.
Note: We do not include the zero-shot results from prior in-context learning work (Hu et al., 2022) (IC- DST) since their prompt examples are designed to include information from the 'unseen' domain, which results in information leakage to the model, and hence does not reflect the traditional zero-shot learning setting. Additionally, we include results from the other comparison approaches where avail- able.
T5-DST (Lin al., 2021b) fine-tunes T5-small model (Raffel et al., 2020) with multiple hand-crafted prompts that include questions and different descriptions of slots.
TransferQA (Lin et al., 2021a) represents DST QA task, where the model is pre-trained six external QA datasets and individual questions are manually crafted for each slot in the ontology to use in the prompt.
4.5 Results
Zero-shot DST
DS2 (Shin et al., 2022) treats DST as a dialogue summarization task, and fine-tunes T5-large and BART models with synthetic summary templates.
Table 2 shows the dialogue state tracking perfor- mance of DiSTRICT in zero-shot settings along with the available baseline results for TRADE, TSDST, and TransferQA. We observe that DiS- TRICT outperforms the baseline approaches most domains across both datasets. DiSTRICT achieves an 8% improvement in JGA on average
IC-DST (Hu et al., 2022) reformulates DST text-to-SQL task and transforms relevant in-context examples to SQL queries and prompts a Codex model without any fine-tuning.

---

Attraction
Hotel
Restaurant
Taxi
Train
Model
1%
5%
10%
1%
5%
10%
1%
5%
10%
1%
5%
10%
1%
5%
10%
TRADE!
35.88
57.55
63.12
19.73
37.45
41.42
42.42
55.70
60.94
63.81
66.58
70.19
59.83
69.27
71.11
STARC
40.39
65.34
66.27
45,91
52.59
57.37
51.65
60.49
64.66
72.58
75.35
79.61
65.67
74.11
75.08
TransferQA"
52.3
63.5
68.2
43.4
52.1
55.7
51.7
60.7
62.9
75.4
79.2
80.3
70.1
75.6
79.0
TSDSTT
58.77
65.72
69.54
43.07
50.71
54,86
57.63
61.86
63.47
70.12
73.67
74.70
70.82
74.18
77.57
DS2 TS*
65.26
69.40
70.89
44.34
52.16
53.79
58.94
64.12
64.65
74.15
77.18
78.50
74.21
76.96
78.60
DISTRICT
72.14
74.61
75.91
48.07
58.84
59.29
61.17
69.31
70.07
82.65
85.37
85.89
79.77
81.55
81.74
Table 3: MultiWZZ 2.0 per-domain few-shot joint goal accuracy (IGA). Results from Lin et al. (2021b). "Results
from Shin et al. (2022).
Attraction
Hotel
Restaurant
Taxi 5%
Train 5%
Model
1%
5%
10%
1%
5%
10%
1%
5%
10%
1%
10%
1%
10%
TransferQA"
50.25
60.92
64.28
32.46
39.02
41.99
47.12
59.16
62.24
71.12
74.47
76.07
69.01
73.17
75.46
DS2 T5"
60.04
68.74
70.31
43.02
48.44
50.35
56.54
65.11
67.26
76.41
79.81
80.62
73.07
76.18
77.14
DISTRICT
70.71
74.98
75.30
47.35
55.37
57.78
61.03
68.65
70.41
84.06
86.06
87.68
80.31
81.41
81.91
Table 4: MultiWO7...1 per-domain joint goal accuracy (IGA). "Results from Shin et al. (2022).
over the next best approach on both datasets (i.e, TSDST in MultiWoz 2.0 and TransferQA in Multi- Woz 2.1), and obtains improvements up to 23% on the "Train' domain.
Additionally, the availability of even a few la- beled examples significantly improves the perfor- mance of our retriever, as evidenced by 46% im- provement in JGA on average across all domains over the zero-shot setting from Table 2 with just 1 1% of available few-shot data in MultiWOZ, 1, compared to a 34% improvement by TransferQA. This improvement stems from the increased rele- vance of available in-context examples, since the retriever now has access to a few (i.e. 1%-5%-10%) dialogues from the few-shot domain.
Both TransferQA and T5DST use hand-crafted prompts, where the former annotates all slots in the form of questions, and the latter uses individ- ual slot descriptions. In the zero-shot setting, this implies that the query-slot is not truly "unseen" since semantic information about the slot is being provided to the model in the hand-crafted prompt. Furthermore, the addition of new domains and slots would first require crafting new prompts, thereby limiting generalizability.
Cross-domain few-shot and full-shot DST
From Table 5, we see that DiSTRICT achieves the best performance in the full-shot (100%) setting, obtaining ~ 10% improvement in JGA on average over the other approaches. However, we observe a significant drop in performance in the cross-domain few-shot setting, when the total available training data is reduced. DiSTRICT suffers from a 75% drop in JGA when only 1% of training data is avail- able in MultiWOZ 2.1, compared to 35% drop for DS2-T5 which suffered the smallest drop in performance.
In contrast, DiSTRICT does not possess any ad- ditional information about the unseen query slot and instead relies on identifying other semantically similar slots and dialogues from the data avail- able from other domains to enable model reason- ing. The improved performance hence reflects the effectiveness of our retriever driven approach in zero-shot settings, and also demonstrates the gener- alizability of our solution.
Per-domain few-shot DST
This performance drop can be attributed to the limited diversity of in-context examples arising from the unavailability of training data. For in- stance, the 1% setting in MultiWOZ 2.1 translates to the availability of only 84 training examples. Hence, the retriever is limited repeatedly us- ing the same examples, restricting the model's rea- soning capabilities. In contrast, the hand-crafted prompts used by the baseline approaches appear
Tables 3 and 4 show the per-domain few-shot per- formance on MultiWOZ 2.0 and MultiWOZ 2.1 re- spectively. DiSTRICT outperforms the baseline ap- proaches across across all domains and across both datasets. For MultiWOZ 2.1, DiSTRICT achieves a JGA improvement of over 15% in the best-case (" l'Attraction" " domain at 1 1%) and 9% on average, compared to the next best approach, DS2-T5.

---

MultiWoz 2.0
MultiWoz
Model
1%
10%
100%
100%
TRADE'
11.74
32.41
37.42
48.62
12.58
31.17
36.18
46.00
DS2 BART"
28.25
37.71
40.29
46.86
DS2-
36.15
45.14
47.61
54.78
33.76
44.20
45.38
52.32
TSDST
28.87
42.03
46.49
53.42
28.23
44.41
47.12
52.21
IC-DST Codex
43.13
47.08
48.67
50.65
DiSTRICT
17.13
41.88
50.39
$7.02
13.39
41.31
49.73
56.08
Table S: Cross domain few-shot and full-shot joint goal accuracy (IGA) on MultiWOZ 2.0 and 2.1. "Results from
Shin al. (2022), "Results from local implementation based on Lin et al. (2021b). Results from Hu al. (2022).
to provide sufficient additional information to the model, reducing performance drop in limited data settings.
Zerp-shat
Few-shot
Tel
4.6 Additional Experiments
Test input
Impact of number of in-context examples
We study the effect of varying the retrieved num- ber of in-context examples used in the prompt on DiSTRICT's performance in all our experimental settings. From Figure 3, we observe that using no examples (i.e.) model fine-tuning and inference using only input dialogues, results in very poor per- formance that is worse than all the baseline com- parison approaches. This shows that the addition of relevant examples has significant impact conditioning the model for dialogue state tracking. We also observe an improvement in performance as the number of in-context examples increases, highlighting the potential of using larger number of examples as part of future work. However, this involves a trade-off, since the improvement is not linear and has diminishing returns, and using larger number of examples would require more memory, compute resources, and increased training time.
Tea
Han
Tran
In-context example
Figure 4: Distribution in-context examples picked across domains by the retriever in zero-shot and per- domain few-shot (10% data) settings for MultiWOZ 2.1. Darker color reflects larger number of examples.
Retriever design
As described in Section 3.2, DiSTRICT uses the entire dialog context until the current turn as the query to the retriever to identify relevant examples. We compare the performance of this design choice against using only the utterances of the current turn to obtain examples.
Table 6 shows that using the entire context per- forms better than the single-turn approach for var- ious settings in the MultiWOZ 2.1 dataset. We observed several examples, where the system's re- sponse in the current turn was incorrect (e.g. rec- ommending italian restaurants instead of indian as asked by the user in prior turns). Hence, using only the current turn resulted in low-relevancy exam- ples being retrieved (involving italian restaurants), whereas using the entire dialog context ensured the retrieval of more relevant examples (involving indian restaurants), thereby demonstrating the im- portance of using the dialog context.
/ernsshot
Zero-soat Fep-shot Ful shot
-B- Few-shot Fullishot
Jant Gral Accurato (GCA
Accurrry
60.
60
50.
50
60
40
10
10
ecamples
Figure 3: Impact of the number of in-context training examples on joint goal accuracy (JGA) for Zero-shot, Few-shot, and Full-shot settings using the Train domain
Retriever performance
We examined the effectiveness of the retriever by analyzing the domains and slots that were selected

---

Retriever Query
Full-shot
Few-shot Restaurant
Zero-shot Restaurant
sero-shot
Train
Single-tum
$2.13 56.08
67.11 70.07
22.93 24.01
45.29
Whole context
47.66
Table Performance comparison using single-turn utterances to retrieve examples vs. using whole context.
Case 1
Slot similarity
Input dialogue
[System] is there anything else that can do you
[User] can you find me an expensive place to stay should only stars include free wifi. hotel-price range: expensive [System] there are several restaurants. what type of food would you like [User] want somewhere cheap in the centre please
Query slot gold label Retrieved example
Example slot
restaurant-price range cheap
Case 2
Understanding dialogue context
Input dialogue
[System] is there anything else can help you with ?
[User] thank you also need to get taxi get me to the restaurant by 17:30.
Query slot gold label Retrieved example
taxi-arrive by: 17:30
[System] would you like a phone number well
[User] not today i just need get train that will arrive by 08:45 in cambridge
Example slot
train-arrive by 08:45
Case 3
Unrelated slots
Input dialogue
[System] am happy to help you find something. do you have a certain area of town mind [User] centre, please want a type of hotel and free parking and free wifi please.
Query slot gold label
hotel-internet: yes
Retrieved example
[System] it is cheap restaurant located in the centre.. can book for you" ? [User] ok book it for 6 on sunday at 15:30 and need a reference too
Example slot
restaurant-area. centre
Table 7:
In-context examples selected the retriever for different inputs in the zero-shot setting.
for the input dialogues in the test set. Figure shows the heatmaps for zero-shot and per-domain few-shot settings, depicting the relative number of examples picked from each domain for test inputs across all domains.
use of semantic matching.
Impact of model size
Model
T5-small (60M)
T5-buse (220M)
TS-laree (770M)
In the zero-shot setting, since data from the un- seen test domain is unavailable, the main diagonal is empty and we observed that examples were rela- tively evenly picked across the other available do- mains. In particular, as illustrated by the examples in Table 7, we found that the retriever identified examples containing semantically similar slots or having similar dialogue contexts, thereby demon- strating the effectiveness of our approach.
TSDST 65.09
66.00
68.78
DiSTRICT
66.58
67.23
70.09
Table 8: Impact of model size on JGA for the "Taxi' domain zero-shot setting in MultiWOZ 2.0. TSDST results for larger models obtained from local imple- mentation hased on Lin et al. (2021b).
Finally, we studied the effectiveness of using larger models for DST. We evaluate the perfor- mance of DiSTRICT and TSDST, as both ap- proaches use the T5-small model (Table 1), with multiple sizes of T5 in the zero-shot setting with the "Taxi' domain. As shown in Table 8, in both approaches the larger T5-base and T5 T5-lerge models achieve modest improvements over the T5-small model. However, these improvements may be too limited to justify the potentially significant increase in compute resources required to support larger model sizes in real-world deployments.
In the few-shot setting, we observed that the ma- jority of examples were selected from the same do- main as the input (i.e. darker diagonals), reflecting the higher semantic and contextual match between intra-domain dialogues. We also studied the dis- tribution of examples at an individual slot level, shown in the appendix, and observed the same pat- terns. In particular, for the few-shot setting, the retriever prioritized examples containing the same slot, followed by the same domain, validating the

---

Model
Attraction
Hotel
Restaurant
Taxi
Train
DiSTRICT
33.74:10.80
2..29
24.17:10.94
66.34:10.79
46.93.11.64
Variant w/ fine-tuning
3.98.10.21
1.72.10.15
1.1010.10
4.8810.38
2.6910.111
Variant w/ random examples
19.83:10.64
9.79:10.66
14.49:1.008
19.57::0.21 31.49::1.55
Variant BM25 retriever
13.91+0.822
11.16+0.27
11.54+0.77
33.87+00.9
16.71+0.36
Table
Ablation study DiSTRICT using different variants for the zero-shot setting in MultiWOZ 2.1.
Ablation study
out requiring hand-crafted templates or additional schema information, overcomes drawbacks of prior DST approaches and also reflects the high gener- alizability of DiSTRICT new task domains and slots, Our experiments show that DiSTRICT out- performs existing baselines in different zero-shot and few-shot experiments despite using a smaller and lower-resource model. We also demonstrate the effectiveness of our semantic-search based re- triever for the DST task and highlight several trade- off's between model performance and resource quirements that impact real-world use. As part of future work, we intend improve robustness dialogue quality and distribution-drifts.
We compare the performance of DiSTRICT against variants that use the same model, but have differ- ences in the in-context tuning pipeline shown in Figure 2. We evaluate the variants on the zero-shot setting, using the MultWWOZ, 2.1 dataset. We define three variants the first does not perform any fine- tuning (i.c.) in-context learning, using examples from the same retriever as DiSTRICT. The second and third variants perform in-context fine-tuning, but instead use random retriever (i.e.) select examples randomly, and a non non-parametri BM25 retriever respectively.
The results (Table 9) show that smaller mod- els like 'T5-small' cannot learn to perform DST without any fine-tuning, further highlighting the tradeoffs between performance, model size, and resource requirements. The performance drop with the use of random examples can be attributed to both biasing the model towards incorrect answers corresponding to other domains/slots, and also withholding valuable information that would have been present in the relevant examples used in DiS- TRICT. Furthermore, identifying relevant dialogue examples is heavily reliant on their semantic mean- ing. Hence, functions like BM25 which is based on bug-of-worde retrieval, also perform poorly since they ignore semantic similarity and instead rely on word frequency that often does not accurately re- flect the meaning of the dialogue. This serves to show that DiSTRICT's retriever driven in in-context tuning approach (Figure 2) plays a big role in en- abling effective dialogue state tracking.
Limitations
The performance of DiSTRICT hinges on the effec- tive retrieval of relevant in-context examples from the training data. This results in our approach being sensitive to issues with data quantity and quality. As shown in our results, when the amount of train- ing data is limited, the retriever often has to select from pool of examples that have low diversity and semantic similarity to the input, thereby adversely impacting performance.
Additionally, data quality issues such as poorly named slots (i.e. not sufficiently descriptive) and incorrect/mislabeled slot values would also impact the semantic matching and performance of our ap- proach. Also, our zero-shot learning relies on se- mantic relationships between the unseen samples and the known data. However, if the new task domains are highly disparate from the existing do- mains, this relationship may not hold, presenting a challenge for zero-shot learning.
5 Conclusion
Recently, research efforts have studied domain generalization in the context of model robust- ness under data distribution shifts (i.e.) out-of- distribution (OOD) generalization (Gulrajani and Lopez-Paz. 2020; Venkateswaran al., 2021a,b. 2023) which can also occur real-world task ori- ented dialogue systems. We did not address this part of our work, and intend to explore OOD model robustness as part of future work.
We present DiSTRICT, a novel approach for di- alogue state tracking using in-context tuning of language models (LMs). For input dialogue instance and slots, DiSTRICT retrieves the most relevant examples from the training data through semantic matching, and uses these examples as part the input to the LM to obtain the dialogue state. The fully automated prompt construction, with-

---

References
Vojtech Hudeček, Ondřej Dušek, and Zhou Yu. 2021. Discovering dialogue slots with weak supervision. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Pupers), pages 2430-2442.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Léo Jacqmin, Lina M Rojas Barahona, and Benoit Favre. 2022. "do you follow me?"" A survey of recent approaches in dialogue state tracking. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 336-350.
Pawet Budzianowski, Tsung-Hsion Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gasic. 2018. Multiwoz-a large- scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):335-547.
Jie Cao and Yi Zhang. 2021. A comparative study on schema- schema-guided dialogue state tracking. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pages 782- 796.
Adarsh Kumar, Peter Ku, Anuj Goyal, Angeliki Metalii- nou, and Dilek Hakkani-Tur. 2020. Ma-dst: Multi- attention- scalable dialog state tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8107-8114.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2022. Meta-learning via language model in-context tuning. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 719-730
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with a language model using schema-driven prompting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4937-4949.
Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi. Peter Ku, Anuj Kumar Goyal, San- chit Agarwal, Shuyang Gao, and Dilek Hakkami- 2019. Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state track- ing baselines. arXiv preprint arXiv: 1907.01669.
Shuyang Li, Jin Cao. Mukund Sridhar, Henghui Zhu, Shang-Was Wen Li, Wael Hamza, and Julian McAuley. 2021. Zero-shot generalization in dialog state track- ing through generative question answering. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1063-1074.
Shuyang Gao, Sanchit Agarwal, Di Jin, Tagyoung Chung, and Dilek Hakkani-Tur. 2020. From ma- chine reading comprehension to dialogue state track- ing: Bridging the gap. In Proceedings of the 2nd Workshop on Natural Language Processing for Con- versational AI, pages 79-89.
Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul A Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho. Rajen Subba, et al. 2021a. Zero-shot dialogue state tracking via cross- task transfer. In Proceedingy of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7890-7900.
Ishaan Gulrajani and David Lopez-Paz. 2020. In search of lost domain generalization. In International Con- ference on Learning Representations.
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul A Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho. and Rajen Subba. 2021b. Leveraging slot descriptions for zero-shot cross-domain dialogue statetracking. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5640-5648
Raghav Gupta, Harrison Lee, Jeffrey Zhao. Abhinav Rastogi, Yuan Cao, and Yonghui Wu. 2022. Show, don't tell: Demonstrations outperform descriptions for schema- -uidedd task-oriented dialogue. arXiv preprint 2rXhv:2204.04322
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo. Iz Beltagy. Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficies fine-tuning is better and cheaper than in contentet learning. arXiv preprint arXiv:2205.05633
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu,
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2
Noah A Smith, and Mari Ostendorf. 2022. In-context learning for few-shot dialogue state tracking, arXiy preprint arXi: 200.00888...

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zet,lemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: (tt::997..1122.
Praveen Venkateswaran, Vinod Muthusamy, Vatche Isa- hagian, and Nalini Venkatasubramaniin.. 2021a. En- vironment agnostic invariant risk minimization for classification of sequential datasets. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1615-1622
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Praveen Venkateswaran, Vinod Muthusamy, Vatche Isa- hagian, and Nalini Venkatasubramanian. 2021b. Ro- bust and generalizable predictive models for business processes. In International Conference on Business Process Management, pages 105-122. Springer.
Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt- based learning for dialogue systems. arXiv preprint 10.0818..
Tlsung-Hsinn Wen, David Vandyke, Nikola Mrkšić, Mil- ica Gasic, Lina M Rojas Barahona, Pei-Hao Su, Ste- fan Uless, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Volume Long Papers, pages 438 449.
Fei Mi. Yasheng Wang. and Yitong Li. 2022. Cins: Comprehensive instruction for few-shot learning in task-oriented dialog systems. In Proceedings of the AAA/ Conference on Artificial Intelligence, vol- ume 36, pages 11076-11084.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hamnanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? arXiv preprint arx:: 220.:2222..1833 12837.
Jason D Williams, Mattew Henderson, Antoine Raux. Blaise Thomson, Alan Black, and Deepak Ramachan- dran. 2014. The dialog state tracking challenge series. Al Magazine, 1214):1
Chien-Sheng Wu, Steven CH Hoi, and Caiming Xiong- 2020. Improving limited labeled dialogue state track- ing with self superviiie, In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020. pages 4462-4472.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. Peter Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140))1-67.
Chien-sheng Wu, Andrea Madotto, Ehsan Hosseini-ash, Ciaming Xiong, Richard Socher, and Pascale Ngan Fung. 2019. Transferable multi-domain state genera- tor for task- oriented dialogue systems. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedingg of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 18966.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengeheng Yin, Sida Wang, et al. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text lan- guage models. arXiv preprini arXiv:2201.00 05966.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR
Darsh Shah, Raghav Gupta, Amir Fayazi, and Dilek Hakkani- Tur. 2019. Robust zero-shot cross-domain slot filling with example values. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5484-5490.
Jamin Shin, Hangyeol Yu, Hyeongdon Moon, Andrea Madotto, and Juneyoung Park. 2022. Dialogue sum- maries dialogue states (ds2). template- guided sum- marization for few-shot dialogue state tracking. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 3824-3844 3846.
Praveen Venkateswaran, Vatche Isabagian. Vinod Muthusamy, and Nalini Venkatasubramannian 2023. Fedgen: Generalizable federated learning for sequen- tial data. In 2023 IEEE 16th International Confer- ence on Cloud Computing (CLOUD), pages 308-31 318. IEEE.

---

A Implementation
For the zero-shot and the full-shot experiments, we train our model for 3 epochs and use early stopping based on the loss in the validation set. For the few- shot experiments, we use the models from the zero- shot experiments that were trained on 4 domains (out of 5), and then train them the fifth domain with 1%, 5%, 10% of the target domain data for 10 epochs. We also use early stopping based on the validation loss.
B Retriever Performance
We analyzed the distribution of in-context exam- ples slot level for different test inputs. Figures 5 and 6 show the heatmap depicting the slots within the in-context examples that were picked for each test query slot for zero and few-shot settings.
For the zero-shot setting, we observed that when- ever possible, the retriever prioritized examples containing slots that had similar semantic mean- ing as the query slot (e.g.) restaurant area hotel-price range and restaurant-price range, train-arrive by and taxi-arrive by. cases where the query slot had no similar example slots (e.g.) hotel-internes the retriever picked examples based on the dialogue context similarity.
Figure 5: Slot-based heatmap of in-context examples
and attraction-ares
picked by retriever in zero-shot settings
For the few-shot setting, we observed that the retriever prioritized examples containing the same slot the query, reflected by the dark diagonal in the heatmap. Additionally, the retriever also typically picked examples from the same domain as the test input, which is shown clearly by the clusters within the heatmap. This serves to show that identifying examples using semantic matching is a viable and effective approach.
Figure 6: blot-based heatmap of in-context examples
picked by retriever in few-shot settings


Reviews:
[{"review_id": "X5BORPBySM", "reviewer": "Reviewer_sWot", "summary": "", "questions": "Question A. When selecting zero-shot examples from various domains, are you excluding cross-domain dialogs? From Figure 4, I observed that in the zero-shot scenario, the Taxi domain test samples mostly contained examples from the Restaurant and Hotel domains. This occurrence is possibly due to the presence of cross-domain dialogs where users initially book a hotel or a restaurant and subsequently request a taxi. If the model considers these as in-context examples and the gold states are not filtered to exclude taxi slots, the true zero-shot nature might be compromised. However, if the taxi slots are removed from the states, it should still be acceptable even if the model encounters taxi domain discussions in the in-context examples.\n\nQuestion B. Since you are employing in-context fine-tuning (0-3), I assume the training process is faster. However, why didn't you conduct the experiments multiple times with different seeds? It is essential to observe confidence intervals to ensure the reliability of the results and convince the readers about their trustworthiness.\n\nQuestion C. The same applies to the ablation part. Why did you only perform it for the train domain? Especially considering that this domain performed the best, it might give the readers the wrong impression that you cherry-picked the results. It would be more comprehensive to conduct the ablation experiments across multiple domains to provide a more balanced and accurate assessment.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and questions, and will include the provided references in the paper. \n\nQuestion A: \n\nThis is a great point, and thank you for pointing this out! \n\nWe do indeed filter the gold states to prevent any knowledge leakage of cross-domain slots and maintain the true zero-shot nature. As you have mentioned, we do not filter out the corresponding sentences (like taxi from your example) from the example dialogues.\n\nWe will add this clarification to the paper \n\nQuestion B: \n\nWe would like to clarify that with in-context fine-tuning, the model is trained on the exact same number of training examples as with traditional fine-tuning and the difference is the in-context examples that are included with each training example. Hence, the training process takes essentially the same amount of time as traditional fine-tuning (such as those done by prior work). \n\nThis is unlike in-context learning which does not involve any additional training at all, but instead just queries the pre-trained model directly. Table 8 shows a performance comparison between in-context fine-tuning and in-context learning. \n\nWe acknowledge the reviewer’s comments about confidence intervals, and will be adding them to all our experiments in the paper by re-running with different seeds. \nBelow, we updated Table 2 with confidence intervals to demonstrate that the observed performance is not due to chance. Since TRADE and TransferQA do not provide confidence intervals in their published results, we were unable to include those. \n\n\nMultiWOZ 2.0\n\n| **Model** | **Attraction**     | **Hotel**          | **Restaurant**     | **Taxi**           | **Train**          | **Avg.**           |\n|-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|\n| TRADE     | 19.87              | 13.70              | 11.52              | 60.58              | 22.37              | 25.76              |\n| T5DST     | 33.09$\\pm$1.60     | 21.21$\\pm$0.61     | **21.82$\\pm$0.91** | 65.09$\\pm$0.12     | 35.42$\\pm$1.42     | 35.20$\\pm$0.59     |\n| DiSTRICT  | **33.61$\\pm$0.73** | **22.82$\\pm$0.43** | 21.30$\\pm$1.08     | **66.53$\\pm$0.21** | **46.61$\\pm$1.56** | **38.17$\\pm$0.80** |\n\n\nMultiWOZ 2.1\n\n\n| **Model**  | **Attraction**     | **Hotel**      | **Restaurant** | **Taxi**           | **Train**          | **Avg.**           |\n|------------|--------------------|----------------|----------------|--------------------|--------------------|--------------------|\n| TRADE      | 20.06              | 14.20          | 12.59          | 59.21              | 22.39              | 25.69              |\n| TransferQA | 31.25              | **22.72**      | **26.28**      | 61.87              | 36.72              | 35.77              |\n| DiSTRICT   | **33.74$\\pm$0.80** | 22.29$\\pm$0.29 | 24.17$\\pm$0.94 | **66.34$\\pm$0.79** | **46.93$\\pm$1.64** | **38.69$\\pm$0.89** |\n\n\nQuestion C:\n\nWe acknowledge the reviewer’s feedback and have performed the ablation study on the other domains and present the results below. As stated in the paper, we continue to see that fine-tuning and choice of retriever are both important for DST performance. We will update the paper with these results.\n\nWe would also like to address the reviewer’s comment regarding Table 2, and clarify that the comparison approaches (TRADE, T5DST, and TransferQA) all perform fine-tuning with different strategies. Hence, this comparison does not have a direct bearing on our ablation study comparison against a non-fine tuning strategy. \n\n| **Model**                  | **Attraction**     | **Hotel**          | **Restaurant**     | **Taxi**           | **Train**          |\n|----------------------------|--------------------|--------------------|--------------------|--------------------|--------------------|\n| DiSTRICT                   | **33.74$\\pm$0.80** | **22.29$\\pm$0.29** | **24.17$\\pm$0.94** | **66.34$\\pm$0.79** | **46.93$\\pm$1.64** |\n| Variant w/ no fine-tuning  | 3.98$\\pm$0.21      | 1.72$\\pm$0.15      | 1.10$\\pm$0.10      | 4.88$\\pm$0.38      | 2.69$\\pm$0.11      |\n| Variant w/ random examples | 19.83$\\pm$0.64     | 9.79$\\pm$0.66      | 14.49$\\pm$1.08     | 31.49$\\pm$1.55     | 19.57$\\pm$0.21     |\n| Variant w/ BM25 retriever  | 13.91$\\pm$0.82     | 11.16$\\pm$0.27     | 11.54$\\pm$0.77     | 33.87$\\pm$0.93     | 16.71$\\pm$0.36     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It is the first paper to apply in-context tuning by fine-tuning the LM parameters using the in-context examples retrieved.\n- The experiments conducted in the study are thorough and consistently demonstrate incremental enhancements on both Multiwoz 2.0 and 2.1 datasets compared to the baselines.\n- The model displays superior performance in both few-shot scenarios and cases with full-shot training.", "reasons_to_reject": "- The primary hypothesis of the paper, which is in-context fine-tuning, lacks strong support from the experiments conducted. The ablation study demonstrates that fine-tuning improves results only within the 'train' domain. Looking at Table 2, we can observe that the train domain shows the most significant improvement with this method, while other domains exhibit either substantial or no improvements (notably, the differences in JGA between the previous SOTA and DISTRICT model are as follows: Train: +11.1, Attraction: +2.2, Hotel: -0.3, Restaurant: -2.2, Taxi: +4.8). To increase confidence in the findings, it would be more convincing to present how the average results over all domains change in the ablation study. It is unlikely that the authors have chosen to focus on only one domain due to concerns about training time since it should take considerably shorter to do in-context fine-tuning compared to full fine-tuning.\n- Moreover, the experiments do not report confidence intervals. This omission raises concerns that the observed improvements might be due to chance rather than an effect of the fine-tuning method.", "questions_to_authors": "Question A. When selecting zero-shot examples from various domains, are you excluding cross-domain dialogs? From Figure 4, I observed that in the zero-shot scenario, the Taxi domain test samples mostly contained examples from the Restaurant and Hotel domains. This occurrence is possibly due to the presence of cross-domain dialogs where users initially book a hotel or a restaurant and subsequently request a taxi. If the model considers these as in-context examples and the gold states are not filtered to exclude taxi slots, the true zero-shot nature might be compromised. However, if the taxi slots are removed from the states, it should still be acceptable even if the model encounters taxi domain discussions in the in-context examples.\n\nQuestion B. Since you are employing in-context fine-tuning (0-3), I assume the training process is faster. However, why didn't you conduct the experiments multiple times with different seeds? It is essential to observe confidence intervals to ensure the reliability of the results and convince the readers about their trustworthiness.\n\nQuestion C. The same applies to the ablation part. Why did you only perform it for the train domain? Especially considering that this domain performed the best, it might give the readers the wrong impression that you cherry-picked the results. It would be more comprehensive to conduct the ablation experiments across multiple domains to provide a more balanced and accurate assessment.", "paper_topic_and_main_contributions": "This research paper investigates a method for in-context fine-tuning for the DST task. The proposed framework, DiSTRICT, begins by retrieving dialogs and slots from historical data that are semantically similar, considering both zero-shot and few-shot scenarios. In the zero-shot setup, similar candidates are selected from source domains, while in the few-shot setup, they are directly chosen from the target domain. These retrieved dialogs are then combined into a prompt without requiring manual template creation. The model is subsequently fine-tuned using these in-context examples and evaluated on test inputs using a similar approach.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {"Missing_References": "- [Slot Dependency Modeling for Zero-Shot Cross-Domain Dialogue State Tracking](https://aclanthology.org/2022.coling-1.42) (Wang et al., COLING 2022)\n- [Prompter: Zero-shot Adaptive Prefixes for Dialogue State Tracking Domain Adaptation](https://aclanthology.org/2023.acl-long.252) (Aksu et al., ACL 2023)\n- [SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking](https://aclanthology.org/P19-1546) (Lee et al., ACL 2019)"}}, {"review_id": "ktvBbFWWgs", "reviewer": "Reviewer_nejq", "summary": "", "questions": "Question A: \nHow is the retrieval unit designed? Are the querying and retrieval examples both the concatenation of a query slot and a pair of utterances? \n\nQuestion B:\nIn Table 5, DISTRICT has surprisingly low performance in the 1% setting (13% VS. T5DST 28%). Both DISTRICT and T5DST share a similar decoding strategy which is predicting value for a single slot at each pass. It doesn't seem right to have this large gap even if the retrieval pool size is limited (the hypothesis the author makes to explain the drop). Is the problem stemming from the retriever being unable to retrieve relevant examples? 1% is 100 dialogues and breaking into turns you should have 700 turn examples to retrieve, right? If retrieval is essential to the success of ICT then finetuning of the retriever is worth considering.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and feedback. \n\nQuestion A : \n\nThe retrieval examples are a concatenation of a single-turn pair of utterances with the slot and its associated ground truth value. The query is the concatenation of the dialogue history till that turn along with the query slot. \nWe tried a few different design choices, but this approach gave us the best results. \n\nQuestion B : \n\nThis is a great question! The primary issue we observed was with the **diversity** of dialogues within the 1% setting. With a random selection of dialogues, there were many slots that did not have representative examples within the 1%, and many examples that had the same (slot, value) pairs. This issue, combined with the use of top-k retrieval meant that the model was often shown the **same** set of few-shot examples during fine-tuning. Hence, the model was not learning new information thereby impacting its performance. \n\nOn the other hand, since T5DST provides hard-coded information about **all** slots as part of its prompt (irrespective of 1% or 100% dialogues), there were essentially no unseen slots.\n\nThis meant that despite using the same decoding strategy, the T5DST model had a lot more information to use during fine-tuning compared to DiSTRICT, and hence its performance did not suffer as much. \n\nWe believe that if the 1% dialogues were cherry picked, to be diverse and minimize repetition, the results would be far better. We had not done this, since it would be a biased comparison with prior work. \n\nWe also agree with the reviewer’s idea about fine-tuning the retriever, but feel that this might likely require a dataset that has sufficient diversity of slots and domains to ensure the retriever’s effectiveness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. In-context tuning seems to outperform previous systems in various zero and few-shot settings, suggesting the proposed approach can be useful for building real-world DST systems where limited data is available. \n\n2. The paper has compared to appropriate baselines and conducted sufficient experiments to demonstrate the effectiveness.", "reasons_to_reject": "1. Despite no one has applied in-context tuning to DST before, I personally find the novelty is somehow limited. It's still a new application scenario but it probably falls into the \"who does it first\" type of paper. I wouldn't position this as a reason to reject, but it's definitely a weakness.", "questions_to_authors": "Question A: \nHow is the retrieval unit designed? Are the querying and retrieval examples both the concatenation of a query slot and a pair of utterances? \n\nQuestion B:\nIn Table 5, DISTRICT has surprisingly low performance in the 1% setting (13% VS. T5DST 28%). Both DISTRICT and T5DST share a similar decoding strategy which is predicting value for a single slot at each pass. It doesn't seem right to have this large gap even if the retrieval pool size is limited (the hypothesis the author makes to explain the drop). Is the problem stemming from the retriever being unable to retrieve relevant examples? 1% is 100 dialogues and breaking into turns you should have 700 turn examples to retrieve, right? If retrieval is essential to the success of ICT then finetuning of the retriever is worth considering.", "paper_topic_and_main_contributions": "This work tries to address the challenge of training a DST system in low-resource scenarios. It proposes to finetune a language model with retrieved in-context exemplars. The authors find improvements compared to previous zero/few-shot approaches.  \n\nThe author seems to be the first to apply in-context tuning to DST (quite surprisingly). The system design is of minimal prompt crafting effort and is generalizable to different domains.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vayRjjD2qY", "reviewer": "Reviewer_LNYL", "summary": "", "questions": "No particular questions.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and positive comments, and would like to address some of their concerns -\n\n3a. We acknowledge the reviewer’s concern regarding prompt size. We believe that our design choice of using single turn dialogues as in-context examples rather than the entire dialogue history helps with this issue. We computed the average number of tokens in the prompt for different values of $K$ below, and we see that even with $K=10$, the prompts are within the context window sizes of most popular models. \n\nPrior work [1][2][3] has shown that the addition of more in-context examples in the prompt does not always help and can even cause model performance to drop from SOTA to near-random. \n\nWe believe this explains the diminishing returns observed in Figure 3. Additionally, more examples could lead to over-biasing of certain (slot, value) pairs, and we will investigate the performance of our approach with a larger number of examples and update Figure 3 to include these results. \n\n| **K** | **Avg. Token length** |\n|:-----:|:---------------------:|\n| 3     | 181.19                |\n| 5     | 302.08                |\n| 10    | 608.71                |\n\n\n3b. This is an interesting point,  and we believe that our approach could also potentially leverage the strategy of fine-tuning the model to predict multiple slots simultaneously. Given that most turns provide information about multiple slots, DiSTRICT’s prompt could be modified to include the ground truth value of all these slots. There could also be an interesting tradeoff between efficiency and example quality, and we will add this discussion to the paper. \n\n4.In some domains, we agree that people could be employed to develop slot descriptions (e.g. using Mechanical Turk). However, in our experience, more complicated domains like some enterprise use-cases require significant domain knowledge, and incorrect descriptions can lead to poor results which is also supported by some of the results shown in the linked paper. \n\nAdditionally, the results would rely on the quality of the paraphraser, which can produce incorrect / invalid semantic paraphrases in certain domains. We have ourselves encountered certain situations with the Pegasus model not realizing that Python refers to a programming language and assuming it referred to the snake. This could require further manual annotations to ensure correctness. \n\nFinally, the use of real user utterances as examples could also improve model robustness with the same concept as the paper, since different users use different writing styles, thereby reducing model sensitivity. We thank the reviewer for pointing us to this paper, and will include a discussion and also look at performance on the SGD dataset. \n\n[1] Reynolds, Laria, and Kyle McDonell. \"Prompt programming for large language models: Beyond the few-shot paradigm.\" Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 2021.\n\n[2] Min, Sewon, et al. \"Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\n[3] Zhao, Zihao, et al. \"Calibrate before use: Improving few-shot performance of language models.\" International Conference on Machine Learning. PMLR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing and experiments are basically clear.\n2. The method is straightforward, easy to replicate, and works well. The proposed system outperforms several recent auto-regressive DST systems.\n3. The good performance does not rely on exhaustive training data and huge sets of model parameters, which is a good attempt in this domain.", "reasons_to_reject": "1. The method is not novel and not surprising. In-context learning/tuning is not a novel approach and has been attempted in other work and other domains. As a straightforward approach inspired by recent progress in literature, it achieves good performance without much excitement.\n2. The system was evaluated on MultiWOZ only. MultiWOZ is slightly outdated in terms of dataset size, data diversity and label soundness. The system should be evaluated on some more recent DST benchmarks which are more challenging (e.g. Schema-guided DST).\n3. The formulation itself has some obvious limitations. \n    1. The length of model input can be overwhelming since the input length increases significantly when more examples are used in in-context inference. In the work, K>3 is not attempted (Fig.3) possibly due to this issue; \n    2. The system only predicts one slot value at one time, which is a very computationally-expensive setup when a lot of slots need to be classified. Some recent DST systems shifted to predicting multiple slot values at one go (e.g. D3ST) which are far more efficient in practice. The proposed in-context learning approach may work well with this setup (retrieving similar turns with respect to one single slot query), while it may not be efficient enough when predicting multiple slot values given multiple slot queries. More discussion is needed.\n4. The statement regarding hand-craft templates is not very convincing. In practice, it is not very difficult to leverage some automated approaches to significantly reduce time and effort in preparing domain/slot descriptions. For example, [1] proposed to leverage paraphrasing to prepare diverse slot descriptions for description-drive DST systems. In practice, it is practically possible to employ several people to work out some slot descriptions for newly-added services in 1 hour. Thus, more discussions are needed to strengthen the advantage of the proposed system relative to those description-drive few-shot/zero-shot DST systems (particularly systems evaluated on the Schema-guided DST dataset).\n\n[1] ****More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking**** https://arxiv.org/abs/2303.09905", "questions_to_authors": "No particular questions.", "paper_topic_and_main_contributions": "This paper introduces DiSTRICT, a DST system that utilises in-context fine-tuning to achieve better DST performance. The system first retrieves semantically-similar dialogue examples (single turn conversation with ground-truth slot labels) from the training corpus, and then uses them as in-context learning examples during training and testing. The system performs few-shot evaluation by retrieving a small portion of training examples from the domain being evaluated. MultiWOZ 2.0 and 2.1 are used for assessing system performance and several recent pre-trained DST systems are used for comparing zero-shot/few-shot performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The discussion on more recent description-drive DST systems is slightly weak.", "Typos_Grammar_Style_and_Presentation_Improvements": "No obvious issues were found."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents DiSTRICT, a method to improve DST performance through in-context tuning with relevant retrieved training examples. This is the first work to apply in-context tuning method to the dialogue state tracking task. The selection of baselines are appropriate and the conducted experiments are generally sufficient to demonstrate the effectiveness of the method, while the domain generalizability can be further verified through more detailed analysis, which the authors addressed in the rebuttal.
Decision: Accept-Main