Continual Dialogue State Tracking via Example-Guided Question Answering
Authors: Hyundong Justin Cho
Andrea Madotto
Zhaojiang Lin
Khyathi Chandu
Satwik Kottur
Jing Xu
Jonathan May
Chinnadhurai Sankar
Venue: EMNLP 2023 Main (2023)
Keywords: dialogue state tracking
dialogue
natural language processing
continual learning
TL;DR: We propose a simple reformulation for continual dialogue state tracking as an example-guided question answering task to improve continual learning performance.

Abstract: Dialogue systems are frequently updated to accommodate new services, but naively updating them by continually training with data for new services in diminishing performance on previously learnt services. Motivated by the insight that dialogue state tracking (DST), a crucial component of dialogue systems that estimates the user's goal as a conversation proceeds, is a simple natural language understanding task, we propose reformulating it as a bundle of granular example-guided question answering tasks to minimize the task shift between services and thus benefit continual learning. Our approach alleviates service-specific memorization and teaches a model to contextualize the given question and example to extract the necessary information from the conversation. We find that a model with just 60M parameters can achieve a significant boost by learning to learn from in-context examples retrieved by a retriever trained to identify turns with similar dialogue state changes. Combining our method with dialogue-level memory replay, our approach attains state of the art performance on DST continual learning metrics without relying on any complex regularization or parameter expansion methods.

Continual Dialogue State Tracking via Example-Guided Question Answering
Hyundong Cho Andrea Madotto² Zhaojiang Lin² Khyathi Raghavi Chandu²
Satwik Kottur² Jing Xu²
Jonathan May' Chinnadhurai Sankar²
Information Sciences Institute, University of Southern California, MMeta Al
hd.
Abstract
DST
Example-guided QA
Dialogue systems are frequently updated to ac- commodate services (e.g. booking restau- rants, setting alarm clocks, etc.), but naive up- dates with new data compromises performance on previous services due to catastrophic for- getting. To mitigate this issue, we propose simple but powerful reformulation for dialogue state tracking (DST), key component of dia- logue systems that estimates user's goal as a conversation proceeds. We restructure DST to eliminate service-specific structured text and unify data from all services by decomposing each DST sample to bundle of' fine-grained example-guide question answering tasks. Our reformulation encourages model to learn the general skill of learning from an in in-context ex- ample to correctly answer a natural language question that corresponds state. With retriever trained to find exam- ples that introduce similar updates to dialogue states, we find that our method can significantly boost continual learning performance, even for a model with just 60M parameters. When com- bined with dialogue-leve level memory replay, our approach attains state-of-the-art performance continual learning metrics without relying on any complex regularization or parameter ex- pansion methods.
① Restaurant
"Id like to make reservation for 2 at Milpitas."
How many people is the user reserving for?
city: Milpitas party size: 2
Music
"I'm looking for by Jimmie Allen from the album Mercury Lane."
Which artist the user searching for?
artis:: Jimmie Allen album: Mercury Lane
Jimmie Allen
X Payments
"I need to make payment of $50 from my debit cord."
slot in dialogue
What is the amount that user wants to pay?
amount:
method: debit
S50
Figure I: Left: When continually learning with the orig- inal DST format, DST models need to memorize new slot keys when learning each subsequent service, Right: Instead, reformulating DST into bundle of granular question answering tasks with help from similar exam- ples (symbolized by the light bulbs) makes training data uniform across all services. Learning new services ef- fectively becomes additional training for the general task of example- cample-guided question answering and is more conducive to continual learning.
1 Introduction
As conversational digital assistants are becoming increasingly popular and versatile, is important to continuously update them to accommodate more services.' One of their key components is a dia- logue state tracking (DST) model that estimates the user's goal, i.e. the dialogue state (Williams et al., 2013). The dialogue state is used for queries sent to application programming interfaces to retrieve
information that grounds the dialogue model's 's sponse.
Unfortunately, naively updating model for new service by training with new data causes catastrophic forgetting (McCloskey and Cohen, 1989; French, 1999): upon learning from new data, the model's performance for previous services re- gresses. To mitigate this issue while also avoiding the impracticality of training a model from scratch
"This work was done while Meta
'In this work, we use services und domains interchange- ably to denote high-level services supported by digital assis- tants, c.g. setting an ularm or booking a restaurant. Task refers to lower-level functions, e.g. question answering, sentiment classification, and dialogue state tracking.

---

We discover that this is indeed the case, as our approach leads to significant gains in CL perfor- mance without using any of the aforementioned CL approaches or data augmentation methods. Specifi- cally, we transform DST into the TransferQA (Lin al., 2021) format and add examples from re- triever that is trained to identify turns that result in similar dialogue state updates (Hu et al., 2022). In addition, our approach does not require complex partitioning of the full training set into training samples and retrieval samples. We find that we can use each sample in the training set both tar- get samples and examples in the retrieval database without causing any label leakage. Also, we ex- periment with a wide array of retrievers and find that models trained to perform DST-EGQA can be effective even with lower quality retrievers by inten- tionally training it with subpar examples such that it can learn when to leverage good examples and with data from all services each time new data be- comes available, three main approaches have been established as generally effective approaches to continual learning (CL): memory replay, regular- ization, and parameter expansion. Variations and combinations of the three have been applied for DST in previous work (Liu et al., 2021; Madotto et al., 2021; Zhu et al., 2022).
ignore bad ones. Lastly we simply tweak the sam- pling approach for memory replay sample at the dialogue-level instead of the turn-level and achieve significant gains to CL performance even with a single dialogue sample, resulting in state-of-the- art performance on the Schema Guided Dialogue (SGD) dataset (Zhu et al., 2022). In summary, main contributions are:
However, most previous work has focused on improving CL performance with service-specific inputs or outputs, a paradigm that limits knowledge transfer between services (left side of Figure 1). This approach introduces large distribution shift from one service another since the model needs memorize service-spa service-specific slots that needs to predict as part of the output. However, DST can become a significantly more consistent task across services by simply reformulating it as a collection of example-guides question answering tasks. Our approach, Dialogue State Tracking as Example- Guided Question Answering (DST-EGQA' trains model to learn to answer natural language ques- tions that correspond dialogue state slots (right side of Figure 1) with the help of in-context ex- amples instead of predicting servcce-specffie struc- tured outputs all at once without any explicit guid- ance (left side of Figure 1). We hypothesize that DST-EGQA benefits continual learning because it transforms the DST task to become more granular, easy, and consistent across services.
We show that simply reformulating DST as a fine-grained example-guided question answer- ing task (DST-EGQA) significantly improves continual learning performance by enhancing task consistency across services.
2. We propose simple but highly effective dialogue-level sampling strategy for choos- ing memory samples that leads state-of-the- art performance when combined with DST- EGQA.
3.
We share a thorough analysis on DST-EGQA to establish its effectiveness, robustness, and limitations method for continual learning.
Dialogue State Tracking as Example-Guided Question Answering (DST-EGQA)
The goal of continual learning for DST is to sequen- tially train on stream of n services Ti...h, with the goal of minimal degradation, i.e. catastrophic forgetting, of peak performance that was achieved when the model was trained on for each service T, In this section, we motivate and elaborate on the methodology of DST-EGQA for attaining this goal. Figure 2 presents an illustrated overview.
2.1 DST as question answering
Dialogue state tracking (DST) is defined as esti- mating the beliefs of user's goals at every turn in a dialogue. It was traditionally formulated as a slot-filling task (Wu et al., 2020; Heck et al., 2020), and more recently as a structured text generation task (Hosseini-Asl et al., 2020; Peng et al., 2021; Su et al., 2022), shown in (0) in Figure 2. If a user were to say "Find me 3 star hotel." ", the goal is to deduce hotel-staz = 3. However, we can also indirectly achieve the same predictions by reformulating DST as collection of per-slot questions to answer (Gao et al., 2019; Lin et al., 2021). Given the same user request, we can ask our model to answer "What is the hotel star rating the
*Code available at https:/ wel.thub. hhb..com/ Bacebbokkresearch/SSSt.

---

(0) Domein-sprcific Structured Text Generation
(1) TransferQA: Domain-specific Granular Question Answering
Per dialogue tmm
per dialogue turn
Target Dialogue
Target
T5
Service Dialogue State
Slot Question
Dialogue
Answer
Sample
(2) DST-EGQA: Domain- -agnosicc Example- Guiddee-Guieed Question Answering
Iindd to stay in hotel for meeting. Pleuse three star hotel.
Target
Per slot per dialogue
Service
hutel-stor-3
Slot Question
Example Dialogue
Example Slot Value
Target Dialogue
T5
Answer
D5
Question
What the hotel star rating Me usey words?
Retrieve
Query
Answer
DB (-Trainning data)
Example
please.
Answer
Figure 2: DST-EGQA overview. We factor (0) the original dialogue state tracking task into (1) granular question answering task with the TransferQA format (Lin et al., 2021) and extend it to (2) pair each question with retrieved examples that are provided in-context such that the domain-shinn answering task. In TransferQA, the original dialogue slot key and value pair, which in aggregate request the equivalent information. DST-EGQA applies TransferQA for continual learning uses the target dialogue query retrieve similar examples from database, which is formed from the training excluding the target.
reduced further to an srample-guides question is mapped to templated questions that correspond each
user wants?" and have it predict 3. We hypothe- size that this question answering approach is more conducive to continual learning because it lever- ages general skill that is understandable through natural language. We only need to ask different questions to predict slot values we are interested in. On the other hand, directly predicting a struc- tured dialogue state requires training the model to generate slots that is has never generated before.
becomes NT granular question answer pairs:
{Q(st,) @ H. + vn i i
(2)
where @ denotes simple text concatenation. A dif- ference from the original TransferQA approach is that since will be finetuning the model, we skip the step of training with external question answer- ing datasets and not take any special measures to handle none, i.c., empty slots, because our models will learn to generate none as the answer for these slots. Further detail on the TransferQA format and additional examples of the fully constructed inputs shared in Appendix A.1.
To transform DST into question answering as shown in (1) in Figure 2, we leverage the Trans- ferQA (Lin et al., 2021) format. Given DS, the dialogue state of a dialogue until turn expressed (key, value) pairs {(St,,, vt,s) E /} for slot i, I (1,..., Nr). where Nr is the number of slots of interest for domain T, each st is transformed into a question with a manually pre-defined tem- plate Q 8; + q:- The overhead creating these templates is minimal as only has to be done once and is as simple as transforming the name slot in the hotel domain to natural text question equiv- alent, c.g. "What is the name of the hotel that the user wants?" Thus, with dialogue history until turn as H, {u,, .... 14-1, bi-1, ue). where u; is the user's utterance on the ith turn and by that of the bot's, the original single input output pair
2.2 Fine-tuning with in-context examples
Adapting to new services can be made even more seamless by providing in-context examples (Wang et al., 2022: Min et al., 2022: Ouyang et al., 2022). Even when faced with question it has never seen before, the examples provide guidance on how it should be answered. This kind of task refor- mulation enables the development of models that achieve Mate-of-the-att zero-shot performance and generalizability even with small models (60M pa- rameters) by explicitly fine-tuning with instructions and in-context examples. Since most recent work that focus on generalizabilty and zero-shot models leverages generation models because their open
H, O T + {(S1, = v.s) liE1}
(1)

---

vocabulary, we also place our focus on generation models.
ances (b, - 1, ue). With our changes, we were able to observe much better top k match, which we verified manually with 100 random sam- ples. We denote examples retrieved with this new SCS+BM25 score as the Oracle because getting A DS requires knowing the DS that we would like to predict ahead of time, and therefore cannot be used at test time. However, the Oracle score is useful for training retriever that can retrieve ex- amples with similar ADS and for estimating the upper bound for DST-EGQA.
Motivated by the results from Tk-instruct (Wang et al., 2022) and MetalCL (Min et al., 2022) that showed even relatively small models can general- ize well if explicitly trained to follow instructions with examples, we explore whether we can pre- vent model from overfitting domain-specific questions and instead continually develop example- based question answering capabilities to enhance continual learning performance. Therefore, we ex- tend Equation 2 to include in-context examples that are retrieved from the training set, as shown in (2) in Figure 2. To retrieve relevant examples, we use H, to form query that retrieves the top le samples (H21) & k} to use as in-context examples. 3 By inserting the retrieved examples and their relevant slot values for each slot question qi, the final format becomes:
Using the Oracle score, for each sample in the training set, we calculate its similarity with other training samples and select the top 200 samples. From the selected samples, we pair the top ten and bottom ten as hard positíve and hard negative samples, respectively, to train a SentenceBERT. based (Reimers and Gurevych, 2019) retriever us- ing contrastive loss. We call the resulting retriever IC-DST-retriever v2 (IDR2). This is the same con- figuration for creating the dataset that was used to train the original retriever used for IC-DST, but instead of using x% of the entire training data, we use the entire training set of the first domain Ti to train separate retrievers for each of the five do- main orderings. We impose this constraint such that conduct our experiments under the practi- cal assumption that we are only provided data for Ti at the beginning and we do not want to extend the continual learning problem for training the re- triever. More details of IDR2's training procedure can be found in Section A.3.
(0(s.) (H) H. on o l 1)
(3)
Throughout this work, we use i unless other- wise specified.
2.3 Retrieving relevant in-context examples
The goal of the retrieval system is to find exam- ple turn Hy, that requires similar reasoning for an- swering the target sample H,, such that fine-tuning with it as an in-context example will help enable the model to apply the same reasoning for answering the question for the target sample. Hu et al. (2022) found that instead of matching for dialogue state overlap, matching for similar dialogue state change ADS, i.e. state change similarity (SCS), yields more relevant examples. State changes are simply subset of DS that is different from the previous turn: ADS { vt,i) i E I, Vt,i A ut 1,1). We found that computing similarity with this def- inition of state change results in many ties that leads to less relevant examples being lumped into the same rank as more relevant ones, we make minor modifications by including the ADS operations, e.g. INSERT, DELETE, and UPDATE, as part of the slot key: ADSours {31 @ 01, .......ss,, (Sm + Om: vm,)), where o the slot operation. To resolve ties that still remain with this modification, we use the BM25 (Robertson et al., 2009) score berween the target and example's 's last bot and user utter-
2.4 Dialogue-level sampling for memory
The approaches that we outlined thus far not orthogonal to existing continual learning methods. Therefore, they can be combined further boost performance. One of the simplest methods is mem- ory replay, which samples training data from previ- ous tasks and adds them to the current training set so that the models forget less. For memory replay to be effective, it is important to select representa- tive and nonredundant training samples.
In DST, training sample is single turn in dialogue, since dialogue state is predicted for every turn. To reduce redundant instances, we propose a simple change to selecting training samples. In- stead of combining turns from all dialogues and then randomly sampling turns, we propose sam-
*Refer to Appendix A.2 for the details of the original definition of state change similarity and the reasoning hehind our modification details.
'Note that * t' hecause the retrieved example may not occur at the same tth turn.

---

pling at the dialogue-level first and then including all turns from the sampled dialogues form the memory. The motivation is that there are rarely the same type of dialogue state updates within a dialogue, but there a high chance that frequent dialogue state updates across dialogues may be sampled multiple times when using turn-level sam- pling.
the continual learning setup, (ii) Forward Trans-
fer (FWT) - T 1 how much training
2=2
on the current domain boosts JGA on future un- seen domains, and (iii) Backward Transfer (BWT)
OT - how much the training on
T - the current domain reduces JGA data from previ- ously seen domains. We place the most importance on Final JGA, while FWT and BWT provide addi- tional signal on how different approaches provide more transferability, and hence task consistency, between domains.
The simple difference between the sampling strategies are clearer when observing their code snippets in Python 3:
Turn-level sampling.
flatten () turns a nested 1ist into single-lew list.
chosen
crossn__tur__ae turn_ samples - zandom.samgaa sample flatten (dl a ogue), memory size)
3.3 Baselines
Dialogue-level sampling.
We replicate the baseline results from Zhu et al. (2022) using their implementation, which include approaches from Madotto et al. (2021):
samples random. sample (dialogue, memory size/ /10); chosen_turr__ampp = random. samplot flatter. (samples), memory size)
SimpleTOD (Hosseini-Asl al., 2020); per- form DST as structured text generation task, predicting the full state as a single sequence. As was done in Zhu et al. (2022), we modify the SimpleTOD format to append the domain name at the end of the dialogue history as described in Equation 1.
3 Experimental Setup
3.1 Data
We use the continual learning setup proposed by Zhu et al. (2022), which uses 15 single domains from the Schema Guided Dialogue dataset (Ras- togi et al., 2020), and aggregate our results over the same five domain orders make the most reliable comparisons with their results. Comparing results with the same order crucial as we find that re- sults can have significant variance depending on the chosen domains and their order. For multi-task training, there is only a single permutation, and therefore aggregate results over runs with three different seed values. Our formulation described in Section 2.2 shows that we are operating under the assumption that the domain of interest will be known ahead of time.
Memory: randomly select M turns from the training data for each previous domain and in- clude it in the current domain's 's training data.
EWC: use the same samples selected for mem- ory replay to regularize with the Fisher infor- mation matrix (Kirkpatrick et al., 2017)
AdapterCL. (Madotto et al., 2021): freeze the base model and train parameter efficient adapters for each domain with number of weights that are equivalent to 2% of that of the pretrained model.
Continual Prompt Tuning (Zhu al., 2022): freeze the base model and continually train soft prompts after reformulating DST as a masked-span recovery task (Raffel et al., 2020). We include their best results, which take advantage of a memory buffer for replay and for memory-guided backward transfer, a form of regularization that prevents updates if doing would increase the current model's loss on the memory samples by computing gradients on them.
3.2 Evaluation
DST performance is mainly measured by joint goal accuracy (IGA), which indicates the percentage of turns for which all slot values are correctly predicted. For CL, given JGA for domain af-
ter training up the hh domain az and the to-
tal number of domains T, we compare our ap- proaches with three metrics from Zhu et al. (2022):
T
Σ ar. the average of IGA
(i) Average JGA- T d=1
For DST-EGQA, we compure various configura- tions to better understand the strengths and weak-
on each domain after training on all domains in

---

nesses of our approach. We vary the retriever used during training and combine with other memory re- play strategies. We also show CPT Multi-task and DST-EGQA Multi-task to show the multi tasting upper bound performance for average JGA.
4 Experiments and Analysis
4.1 Main results
TransferQA's format is more CL-frriedlly The results for only transforming the DST from prior work (Equation 1) to that of granular question an- swering using the TransferQA (Equation 2) format shown the row for DST-EGQA In-context examples Table Without the help of any in- context examples, the transformation alone yields a dramatic improvement in CL performance, increas- ing average JGA from 14.4 to 43.2, and also im- proving on both FWT and BWT. These results sup- ports our hypothesis that question answering task that understandable through natural language is more conducive better continual learning than learning to generate service-specific structured out- put.
3.4 Retrieval baselines
We also experiment with various retrieval tech- niques as baselines to IDR2:
Random sampling
BM25 (Robertson et al., 2009)
OpenAl's fext-embedding ada 002 model
SentenceBERT (Reimers and Gurevych, 2019): the all-mpnet-base-v2
The original IC-DST retriever (oIDR): the re- triever from Hu al. (2022) that was trained with the original SCS formulation and pairs created from the MultiWOZ 2.1 dataset (Eric et al., 2020).
Example-guided question answering further en- hances CL performance. The subsequent rows for DST-EGQA shows that fine-tuning with in- context examples can further enhance all CL met- rics by large margin. Most notable is the boosts that are seen in the FWT, for which memory replay has almost negligible effect. Augmenting DST- EGQA with memory replay leads to even larger boosts, even exceeding the CPT Multi-task model, with most gains coming from BWT, which is ex- pected with memory replay methods. Using the Oracle retriever at test time leads to statistically in- significant improvements, indicating that IDR2 retrieve examples that are on par with the Oracle examples. Lastly, we can see that the relative gains in Average JGA and BWT from memory replay becomes less pronounced with models trained with in-context examples, indicating that memory re- play and example- -uided question answering have overlapping gains.
Other than random sampling and BM25, re- trieval ranking is based on the similarity between sentence embeddings, which is the dot product be- tween the query and the key. With the exception of oIDR, which was trained identify similarity with the last turn's dialogue state and last utterance pairs between the bot and user: {st = vt 1.4) i 1 B u4-1 @ ut, the query and key of the database uses only the last utterance pairs: w--1 @ u4 We found this approach to be better as it diminishes the undesirably high similarity assigned to examples from the same dialogue that have the same previous dialogue state.
3.5 Technical details
We conduct our experiments with the T5-small model (Raffel al., 2020). We train with a sin- gle GPU using the AdamW optimizer, learning rate of le-4, and batch size of 16. We train on each domain for ten epochs without early stopping. We select the checkpoint with the best validation set performance when moving on to the next do- main. Our experiments 1100 GPUs, based availability.
Double-dipping the training set retrieval database does not lead to overfitting. It is im- portant to note that, because our retrieval methods are commutative, target sample that is paired with an example will serve as an example when the example becomes the target sample. Therefore, the answers for all training samples are seen as part of the context during training with our setup described in Scction 2.3. This raises overfitting concerns that the model could easily memorize the
V100, A40, and
"nttps::/ppattoom. opt.ttor.o.ppei..........e openai com/docs guides dedee//bbediiigg
"https:/ http:://wwwwbbw.t/e/eeeeee .ebert.net/doce.
pretra ietraaieed_mme red__eddde modeooelss s.selss.hhe .html
'Our preliminary experiments with different GPU types
of GPU in final performance introduces minimal variability to the final result.
with otherwise identical configurations showed that the choice

---

Method
Retriever
Avg. JGA
FWT
BWT
+Memory
+Params
+Reg.
SimpleTOD (2020)
14.4
7.110
-42.5244
EWC (2017)
13.9h.
8.4
-50.84.3
Memory (2021)
38.6g.
10.9p.
-3.223
Adapter (2021)
49.81.7
CPT Memory (2022)
61.22.5
13.70.8
0.50.4
d
DST-EGQA
34.1g.3
22.81.8
-22.345
IDR2
+ Dialogue Memory
68.9g.a
21.21.5
-6.11.7
In-context examples
43.23.4
14.119
-31.04.
DST-EGQA
35.5g,5
23.62.1
-19.14.2
Oracle
+ Dialogue Memory
69.31.u
22.5
-5.91.
CPT Multi-task (2022)
64.01.9
DST-EGQA Multi-task
74.21.5
Table 1: CL metric results with checklist the reliance of other continual learning techniques. We compare models sequentially trained on 15 tasks from the SGD dataset and aggregate results across five different domain permutations. DST-EGQA achieves the best results without any additional parameters regularization methods. The last two rows provide the multi-tasking results, which serve upper bound. In this table, results with retrievers are with a single in-context example and the indicated retriever is used for training and test time, while the Oracle retriever used for the validation set. Memory here refers to samples that added for the training data of subsequent services for memory replay. All rows that use memory are with memory budget of M 50. indicates statistically significant at < 0.05 with the next best comparable value.
Dev
Test
Avg. JGA
FWT
BWT
4.2 Training dynamics
13.21.
14.1:
-1.0s
IDR2
IDR
46. 1ap
21.4.4
-31.9s4
In practical settings we don't have oracle triever, and our database may not contain the a perfect example for each case seen at test time. Thus, we may fact retrieve irrelevant examples. It is important for the model to be able to handle these situations. Specifically, it should be able to leverage relevant examples, yet ignore irrelevant ones. To become more robust to these realistic cir- cumstances, it may be useful intentionally mix in irrelevant examples during training for DST-EGQA. We vary the combination of IDR2 and Oracle used for training, validation, and test time. Results Table support our hypothesis, showing that align- ing the retrieval method from training time with the method used at test time leads to the best perfor- mance. Interestingly, best performance is achieved by using the Oracle retriever validation time, shown by the large gap between IDR2 IDR2 IDR2 and IDR2 + Oracle - IDR2 (second and third row). This somewhat surprising given that one may expect selecting a checkpoint that performs the best the same setting as test time would lead to better test time performance.
Oracle
54.1a.s
Oracle
Oracle
48.5ga
19.6
-2.11
53.74.4
24.12a
-21.3a1
Oracke
Oracle
IDR2
55.50.
23.61.s
-19.1.2
Table Train- maiin-vlidatioo--ee test retrieval method compari- son, Keeping the Training and Test-time retrieval meth- ods the same while keeping the development set the Oracle leads to the best results, except for the last row, which requires knowing the correct answer ahead of time. indicates statistically significant < 0.05 with the next best value.
answers for all samples and thus not Iearn general- izable question-answeringg Interestingly, this does not seem to be the case, training in this setup leads improved on-par final test set perfor- mance compared training without any examples. This implies that our approach does not impose additional data constraints of having to split the training set into dedicated training samples and retrieval samples for it to be effective.
4.3 Retrieval method sensitivity
However, shown in Table is that we find that DST-EGQA is sensitive to the training dynam- ics (Section 4.2) and the quality of the retrieved examples (Section 4.3).
The findings from Section 4.2 raises question on whether training with other retrievers that may provide a different mixture good and bad exam-

---

Train, Test
Dev
Avg. IGA
FWT
BWT
Train, Test
EEx.
Avg. JGA
FWT
BWT
43.254
14.1aa
-310
43.2s44
14.11a
-3.0c2
Randem
15. 15.5
14.222
-31.4..1
43.2g.
14.51.
-3.36.7
BM25
40.7:33
21.6..
20.1aaa
Random
2
45.24.5
16.51
-31.662
SemBERT
Oracle
26.
17.32m
29.76a
3
43.9ga
16.91.0
-31.40.7
GPT
47.81.
17.524
-27.0k7
45.9
2
-21.4.2
oDDR
19.2u7
19.9p1
-26.2s2
BM25
2
46.2g.1
23.31.
-17.1..5
IDR2 (ours)
54.1s.a"
22.818
-22.3
3
47 47.0s.a
20.820
-21.855
54.
22.81.a
22.34.5
Table Retrieval methods comparison. Although mix- ing in irrelevant examples boost performance training time, our results show that lacking reliable retrieval method test time is detrimental to perfor- mance. Our IDR2 model captures this balance the most effectively.
IDR2
2
50.2s7
22.0
-29.352
3
48.01.
21.819
-22.34.1
1
03.74.
21.126
-1.3
Oracle
2
54.85.0
28.726
-18.558
3
58.95a
30.51.5
-14.12
Table 5: Number of in-context examples analysis. Small models are unable to leverage more than one in-context example when explicitly finetuned to perform in-context learning.
Size
Method
Avg. JGA
FWT
BWT
43.2g.4
14.1..
-31.042
Turn
50. 50.1a.
15.0..4
-23.744
10
Dialogue
1..
15.22.r
-14.72a
number of samples from the exceeded set. Table 4 shows that dialogue-level sampling achieves significantly better performance for all equivalent memory budget sizes for tumnlevel sam- pling and is even par with the next budget size used for turn-level sampling. This is likely due dialogue-sampling leading sive set of samples that cover wider diversity of dialogue state updates in these smaller sizes of the memory budget as described subsection 2.4. As the memory budget becomes larger, however, the gap between turn-level sampling and dialogue-level sampling diminishes, since both methods converge to multi-task training when the memory budget is unlimited.
Tum
50.81.
15.61.7
-12.820
Dialogue
64.
15.021
-7.422²
Turn
63.9t
15.61.7
-8.7.a
100
Dialogue
66.81.5
15.42.1
-3.32s
Table 4: Memory size analysis for DST-EGQA. Sam- pling at the dialogue-level much more effective than sampling at the turn-level, especially for constrained memory budget.
more comprehen-
ples can lead to further boost performance with DST-EGQA. We apply all the retrievers defined in Section 2.3 and use the same training dynamics that led to best results previously to examine each retriever's effectiveness. As shown Table 3, our IDR2 model seems to capture this balance the most effectively, as is significantly better than all other retrieval methods.
4.5 Number of in-context examples
We also study the effect of having more than one in- context example and share the results in Table 5. In- cluding only example to learn from in-context creates single point of failure, which is especially risky for suboptimal retrieval methods. Having ad- ditional examples learn from can help mitigate this risk. Therefore, we repeat our experiments using multiple in-context examples. However, at least with small model sizes, the DST models are not able to effectively leverage additional examples. This is not surprising for the Oracle retriever, where in most cases the top example is the best example that can be leveraged from the training set.
4.4 Memory sampling strategy and size
We study the effect of the memory sampling strat- egy and the size of the memory budget. We do not use in-context examples for all configurations to study their effects in isolation. As hinted by the results in Table 1, dialogue-level sampling seems to be a superior sampling strategy to turn-level sam- pling. We take deeper dive into the relationship between the two sampling techniques and how both approaches scale with memory budgets by varying the memory budget sizes to 10, 50, and 100. Here, size refers the number of training samples. To make sure the comparison between turn-level and dialogue-level samples fair, we sample dialogues until the total number of turns in sampled dialogues exceed the target size, and then sample the targeted
Related Work
Continual learning Continual learning prolongs the lifetime model by training it further with
new incoming data without incurring the cost

---

catastrophic forgetting (McCloskey and Cohen, 1989; French, 1999). There are three main branches methods,
training. Building on T5, Sanh al. (2022) and Wei et al. (2021) used more elaborate and diverse set of instruction templates and showed that this can significantly boost zero-shot performance. Cho et al. (2022) applied a similar idea to a more se- lective set of pre-finetuning tasks before training on the target DST dataset to improve DST robust- ness. Tk-instruct (Wang et al., 2022) takes a step further by scaling up the amount of tasks included in TO and also provides positive and negative ex- amples in the context in addition to the instructions. Similarly, Min et al. (2022) introduced MetalCL, which explicitly trains a model with the few-shot in-context learning format used for large language models (Brown et al., 2020), and showed that it showed better in-context learning performance than larger models. Task reformulation has also been recently explored help the model better under- stand the task at hand and reduce domain- sppciffc memorization and thus boost zero-shot DST per- formance (Li et al., 2021; Lin et al., 2021; Gupta et al., 2022; Zhao et al., 2022).
of continual learning: architecture- replay-band based methods, and regularization based methods. architecture-basel methods propose dy- namically adding model weights when learning new data (Fernando et al., 2017: Shen et al., 2019). Replay-based methods mitigate catastrophic for- getting by keeping small sample of the previous data as part of memory budget to train with the new data (Rehuffi et al., 2017; Hou et al., 2019). These methods mainly experiment with sampling strategies and memory budget efficiency. Lastly, regularization-base methods place constraints on how the model becomes updated during training with the new data such that its performance on pre- vious data is maintained (Kirkpatrick et al., 2017; Li and Hoiem, 2018).
Dialogue state tracking Continual learning for DST has been explored by a series of recent work that applied a combination of methods mentioned above. Liu et al. (2021) expanded on SOM- DST (Kim et al., 2020) with prototypical sample selection for the memory buffer and multi-level knowledge distillation as regularization mecha- nism. Madotto et al. (2021) applied various contin- ual learning methods to end-to-end task-oriented dialogue models and found that adapters are most effective for the intent classification and DST while memory is most effective for response generation. More recently, Zhu et al. (2022) proposed Contin- ual Prompi Tuning (CPT), which is most related to our work. CPT improves continual learning perfor- mance by finetuning soft prompts for each domain and reformulating DST to align with T5's masked- span recovery pretraining objective (Raffel al., 2020). Compared CPT, we suggest a more gran- ular reformulation to facilitate the learning from examples and do not rely on any regularization nor additional weights.
6 Conclusion
In this paper, we propose Dialogue State Track- ing as Example-Guider Question Answering method for enhancing continual learning per- formance that factors dialogue state tracking into granular question answering tasks and fine-tunes the model to leverage relevant in-context examples to answer these questions. Our method is an ef- fective alternative to existing continual learning approaches that does not rely on complex regular- ization, parameter expansion, or memory sampling techniques. Analysis of our approach reveals that even models as small 60M parameters can be trained perform in-context learning for contin- ual learning and that complementing such a model with a randomly sampled memory achieves state- of-the-art results compared to strong baselines.
Limitations
Task reformulation and in-context learning Enhancing a model's generalizability to various tasks by reformulating input and/or outputs to become more uniform has become an increas- ingly popular method for massive multi-task Icarn- ing (Aghajanyan et al., 2021), even for tasks that are considered distant from one another. T5 (Raffel et al., 2020) accelerated this movement by pro- viding dataset or -sscifpcc labels or minimal instructions to the inputs and then doing multi-task
Using the TransferQA idea and retrieved examples for in-context fine-tuning adds a lot of configura- tions, which we have not been exhaustively ex- plored, in lieu of prioritization of, we judged, more important experiments. For example, explore sensitivity to the specific wording of ques- tions, as was done with TO (Sanh et al., 2022). We leave as future work the testing of the hypothesis that having more diverse questions per slot can lead
did not

---

even more generalizability between domains and bring even further improvements to DST-EGQA. Another limitation of DST-EGQA is that the re- trieval database stores all previously seen samples from training and thus can be considered a memory with infinite size in our current formulation. Al- though the samples in the retrieval database are not used as training samples and provided as in-context examples during inference after being trained with subsequent services, the memory requirement for maintaining the database may be quite high. How- ever, we believe that this memory requirement is still less restrictive than having the compute for fully retraining the model with all data whenever the model needs to learn new service, especially when the training data set is large.
Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani- Tur. 2020. Mul- tiWOZ 2.1; I: A consolidated multi-domain dialogue dataset with state corrections and state tracking base- lines. In Proceedings of the 12th Language Re- sources and Evaluation Conference, pages 422-428. Marseille, France. European Language Resources Association.
Chrisantha Fermando, Dylan Banarse, Charles Blundell. Yori Zwols. David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. 2017. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint
Robert M French. 1999, Catastrophic forgetting in con- nectionist networks. Trends in cognitive sciences. 3(4): 135.
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagy- oung Chung, and Dilek Hakkani-Turs 2019. Dialog state tracking: A neural reading comprehension ap- proach. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 264-273, Stockholm, Sweden. Association for Computational Linguistics.
Lastly, an important practical consideration is the varying technical overhead in implementation and portability of different approaches. Compared to other approaches, training and inference is rel- atively simple, as we use an autoregressive text generation objective without special modifications. However, while our approach does not require any additional parameters, it does require a database and retrieval model that is comparable in size to the DST model. Therefore, depending on the tech- nical constraints, managing these two components may be less desirable.
Raghav Gupta, Harrison Lee, Jeffrey Zhao, Yuan Cao. Abhinav Rastogi, and Yonghui Wu. 2022. Show, don't 't tell: Demonstrations outperform descriptions for schema- -unnedd task oriented dialogue. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4541-4549 Seattle, United States. Association for Computational Linguistics.
References
Michael Heck, Carel van Niekerk, Nurul Lubis, Chris- tian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. 2020. TripPy: A triple copy strategy for value independent neural dialog state tracking. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 35-44, 1st virtual meeting. Association for Computational Linguistics.
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-tineunning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 5799-5811. I, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguisties.
Ehsan Hosseini-Asi, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33:201 79- 20191.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. 2019. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 831-839.
Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Sadagopan, Shahin Shayandeh, Asli Celiky- ilmaz, Jonathan May. and Ahmad Beirami. 2022. Know thy strengths: Comprehensive dialogue state tracking diagnostics. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5345-5359, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In- context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 2627-2643, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi,

---

Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang- Woo Lee. 2020. Efficient dialogue state tracking by selectively overwriting memory. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 567-582, Online. Association for Computational Linguistics.
in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 2791-2809, Scattle, United States. Association for Computational Linguisties.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Barwe Barwinsaa, al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (161133:322135266
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2021. Soloist: Building task bots at scale with transfer learning and machine teaching. Transactions of the Association for Computational Linguistics, 9:807- 8244
Shuyang Li. Jin Cao, Mukund Sridhar, Henghui Zhu, Shang-Wes Wen Li, Wael Hamza, and Julian McAuley. 2021. Zero-shot generalization in dialog state track- ing through generative question answering. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume, pages 1063-1074, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67.
Zhizhong Li and Derek Hoiem. 2018. Learning without forgetting. IEEE Transactions Pattern Analysis and Machine Intelligence, 40( (0(12)2935-299
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages
Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul Crook, Zhiguang Wang. Zhou Yu, Eunjoon Cho, Rajen Subba, and Pascale Fung. 2021. Zero-shot dialogue state tracking via cross-task transfer. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 7890-7900, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: In- cremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 2001-2010.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lunguage Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP- (JCCLP, pages 3982-3992 Hong Kong, China. Association for Com- putational Linguistics.
Qingbin Liu, Pengfei Cao, Cao Liu, Jiansong Chen, Xunliang Cai, Fan Yang, Shizhu He. Kang Liu, and Jun Zhao. 2021. Domain-lifelong learning for dia- logue state tracking via knowledge preservation net- works. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 2301-2311, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 3(4):333-389
Andrea Madotto, Zhaojiang Lin, Zhempeng Zhou, Se- ungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eu- njoon Cho. Pascale Fung, and Zhiguang Wang. 2021. Continual learning trientriented dialogue systems. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7452-7467, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika. Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero- shot task generalization. In The Tenth International Conference on Learning Representations.
Michael McCloskey and Neal J Cohen. 1989. Catas- trophic interference in connectionist networks: The sequential learning problem. In Psychology of learn- ing and motivation, volume 24, pages 109-165. Else- vier.
Yilin Shen, Xiangyu Zeng, and Hongxia Jin. 2019. A progressive model to enable continual lcarning for semantic slot filling. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing (EMNLP- IJCNLP), pages 1279-1284, Hong Kong, China. As- sociation for Computational Linguistics.
Sewon Min, Mike Lewis. Luke Zettlemoyer, and Han- naneh Hajishirzi. 2022. MetalCI: Learning to learn

---

Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Zhang. 2022. Multi-task pre-training for plug-and-play task-oriented dialogue system. In Proceedings of the 60th Anmual Meer- ing of the Association for Computational Linguistics Dublin, Ireland. Association for Computational Linguisties.
(Volume I: Long Pupers), pages
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. SuperrNaturaallnssrutiioo
Generaliza- tion via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint 11X.v22109.01652.
Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Con- ference, pages 403-413, Metz, France. Association for Computational I Linguistics.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong, 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented di- alogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917-929. Online. Association for Computational Linguistics.
Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu. Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022. Description- driven task-oriented dialog modeling, arXiv preprint arXiv:2201.08900
Qi Zhu, Bing Li. Fei Mi. Xiaoyan Zhu, and Minlie Huang. 2022. Continual prompt tuning for dialog state tracking. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1124-1137, 137, Dublin, Ireland. Association for Computational Lin- guistics.

---

Appendix
on cosine similarity between the paired examples and an automatically calculated similarity thresh- old that results the best accuracy. The triplet evaluator, on the other hand, compares whether distance(q, p) > distance(q, n), where similar pair, is the negative pair, and q is the query that serves as the anchor between the similar and dissimilar samples. We find that the two evaluators yield statistically insignificant differences in the final performance for our approach and therefore we use the triplet evaluator for all the results in- cluded in this work. We exclude the previous turn's s dialogue state as the input query when fine-tuning SentBERT (Reimers and Gurevych, 2019).
Additional details
A.1 TransferQA format
the
The TransferQA format (Lin al., 2021) has two different formats depending on whether the ques- tion is multiple choice or extractive. Categori- cal questions with fixed answer choices are given as multiple choice questions where the options are provided as part of the context, prepended by [opt]. Extractive questions do not have spe- cial format. We remove the prefix Extractivn ive Question: and Multi-cheice QA the presence of [opt] already allows the model to distinguish between the two. In addition, when there are in-context examples, we add [target] ] and [example] so that the model can distinguish between the examples and the target. The original and our modifications are shown in Figure 3.
A.2 State change similarity
Given two sets of state changes represented as ADS, (1),, {m]} and ADS; (1)1,2),,... va)). where is the slot key and is the slot value to update the slot to, Hu al. (2022) defines state change similarity (SCS) as the average of the similarity between slot keys s, satt and the similarity between key value pairs, Fstot value:
SCS(ADS, ADS;) -F + value-
Similarity F between the two sets is measured by computing the average of two Fi scores using each set as the target. We the standard calculation: Fi P where P is precision and R is recall.
The resulting ties with this formula were as critical for IC-DST (Hu et al., 2022), because top examples, where a sizeable value that includes most ties, were all provided as in-context learning examples.
A.3 IC-DST retriever v2
We use the same hyperparameter settings as oIDR (Hu et al., 2022), which uses a learning rate of -3, 1000 warm-up steps, and the contrastive loss objective. We experimented with binary clas- sification and triplet evaluator test time for dis- criminating between similar and dissimilar samples and selecting the best checkpoint to use. The binary classification evaluator determines accuracy based

---

Extractive
Multichoice QA
Original
Extractive Question: Mhich city is the user planning to stay?
Hulci-Choice Question: Does the user want parking? Choices: [sep] [sep] no [sep] dontcare
Context:
[user] lookingg stars
context:
parking.
[user] lookingg
London
parking.
Mhich city is the user plamming to stay in?
Does: the user want parking?
Options:
[exusnle]
[opt]] yes [opt] no [opt] dontcare
Context:
[user] looking cheap hotel parking
Casbridge.
Context:
Answer: Casbridge
[user] looking for cheep hotel with free parking
Casbridge.
[Sarget]
Answer: yes
Contexe:
[user] looking stars hotel offers
[carget]
parking
[user]
horel
Answer::
Figure 3: An illustration of the original and modified version of the TransferQA format. The desired output green, which is the text that comes after the last Answer:


Reviews:
[{"review_id": "zNRaUiZzhL", "reviewer": "Reviewer_6XWq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review and for appreciating the core contributions of our work in restructuring the DST task to example-guided question answering to improve strong continual learning performance even with a small T5 model. We would like to address some of your concerns: \n- JGA of each domain should be included: These are intermediate results that are necessary to calculate the final results already included in the paper and therefore a minor fix. We will make by including these results in the appendix with analysis on various difficulty levels of each domain as it may provide insight into even more improved continual learning performance. Fortunately, \n- Only SGD is evaluated: Our preliminary experiments were actually conducted with MultiWOZ following the setup from previous work ([Liu et al. 2021](https://aclanthology.org/2021.emnlp-main.176/)) and we observed similar results (50.05% average JGA even with only using TransferQA format vs 40.14% average JGA for KPN, the best performing model in the previous work) but we found the setup to involve a smaller set of domains that were not independent from one another (`hotel` vs `hotel + restaurant` vs `restaurant` were considered separate domains). This makes MultiWOZ a poor test bed for continual learning performance for DST and so we pivoted to the SGD setup by [Zhu et al](https://aclanthology.org/2022.acl-long.80). However, we will repeat some of these experiments with MultiWOZ and include them in the appendix of the revised version of our paper for those interested and mention the caveats of using this dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written with thorough experimental results and analysis.\n\nThe proposed method, DST-EGQA, significantly improves the continual learning performance of DST on SGD dataset. DST-EGQA achieves state-of-the-art performance with a 60M model. This is quite remarkable especially in the era of large language models.", "reasons_to_reject": "JGA of each domain should be included. It is difficult to tell is certain domains are considered easier than others, or if this method has difficulty in adapting to certain domain. Such analysis would be important for the continual learning scenario.\n\nOnly SGD is evaluated. More dataset (MultiWOZ, etc.) and even cross-dataset domains should be included to consolidate the findings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes to improve continual learning in DST by eliminating service-specific structured text and decomposing each DST sample to a list of fine-grained example-guided question answering tasks. Specifically, the classical way of DST is reformatted as a QA problem asking for the value of each slot. The model input includes the target dialogue, the slot question, example dialogue, and example slot value. The model then outputs the answer to the slot question. Examples were retrieved from the training data, or the database. Ideal samples are those that require similar reasoning for answering the target sample. This is done via measuring the state change similarity and BM25 re-ranker.\n\nAuthors compared DST-EQGA with a range of models using SGD dataset under a continual learning set-up. Authors also proposed a dialogue-level memory sampling strategy and compared with other sampling approaches. Overall, the proposed DST-EQGA framework achieved state-of-the-art continual learning performance with a T5-small model (60M) parameters.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KivjRwoRsD", "reviewer": "Reviewer_88YD", "summary": "", "questions": "Overall, this paper is well-motivated and novel, I tend to see its appearance in EMNLP, but there are still some points that can be improved: \n1. I suggest that authors could try some larger models. \n2. Table 2 is unreadable. \n3. Some analysis on large models like ChatGPT can be included, even in Appendix. As this tells us how SOTA models performs on this task.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and for appreciating the core contributions of our work in restructuring the DST task to example-guided question answering to improve continual learning performance. \n- Larger models: We understand your interest to see larger models applied to this task and we will also include experiments with T5-large and Flan-T5 to shed light on the performance of larger models and also models that have been trained to follow instructions, which can be considered a more generalizable behavior of answering questions. \n- Models like ChatGPT are much larger and are used mostly for zero-shot scenarios due to the practical difficulty of fine-tuning them. Therefore, continual learning with ChatGPT-scale models will be challenging, but it would be interesting to compare its zero-shot performance using our setup and we will explore it. On a related note, [Heck et al. 2023](https://aclanthology.org/2023.acl-short.81/) suggests that it will likely achieve strong performance as it achieves similar results to IC-DST.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel perspective to view DST as a fine-grained example-guided question-answering task to improve continual learning performance. \n2. Propose a simple dialogue-level sampling strategy for choosing memory samples in the experience-replay approach.", "reasons_to_reject": "1. I suggest that authors could try some larger models like FlanT5/T5-large or so to test their continual learning approach for the dialogue state tracking problem.\n2. The motivation of this paper should be better described, especially its sampling strategy. \n3. The presentation, especially the tables and figures, can be improved.", "questions_to_authors": "Overall, this paper is well-motivated and novel, I tend to see its appearance in EMNLP, but there are still some points that can be improved: \n1. I suggest that authors could try some larger models. \n2. Table 2 is unreadable. \n3. Some analysis on large models like ChatGPT can be included, even in Appendix. As this tells us how SOTA models performs on this task.", "paper_topic_and_main_contributions": "Paper topic: Continual learning for dialogue state tracking. \nMain contributions: \n(1) reformulate dialogue state tracking as a fine-grained example-guided question answering task. \n(2) propose a novel dialogue-level sampling strategy for choosing memory samples.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jaggfM4g6E", "reviewer": "Reviewer_GgpV", "summary": "", "questions": "See \"Reasons To Reject\".", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our work and for appreciating the core contributions in restructuring the DST task to example-guided question answering to improve continual learning performance. You have raised important questions that will greatly improve the clarity of our paper: \n\n- Infinite memory: \n    - You are correct in that the retrieval database can be considered an infinite memory as the training samples are kept for retrieving examples at inference time. Therefore `+Memory = None` is misleading for the row with results of DST-EGQA without memory. We wanted to indicate that there are no additional **training** samples from previous domains. We will clarify this by creating separate columns for `+Memory` and `+Compute`. Considering that memory is cheap while compute is not, keeping samples for retrieval is a very minor disadvantage while keeping a large memory for training samples from previous domains to retrain on pushes the setting closer to full retraining at each new domain, which is antithetical to continual learning. However, we may make a stronger case for our approach if we see performance gains remain intact even after limiting the size of the retrieval database per domain. We will include this analysis in our revision. \n    - The IC-DST-ret retriever uses the previous turn’s dialogue state and also the previous bot utterance and user utterance ($\\{(s_{t-1,i}=v_{t-1,i})\\mid i \\in I\\} \\oplus u_{t-1} \\oplus u_t$), instead of just the last user utterance pairs for matching similarity. This is explained towards the end of Section 2.3 in lines 278-279, but will make this information more clear as it is an important detail. \n\n- Generalizability of retriever across domains: \n    - Our setup to train the retriever only on the first domain specifically serves to answer your question on whether such a retriever can still be useful for the subsequent domains that it has not yet seen. Our results, which incorporate five different domain order variations from [Zhu et al. 2022](https://aclanthology.org/2022.acl-long.80/), show that it can be. If we extend continual learning setup to the retriever as well, as in continually updating it for each new domain it encounters, we believe the performance of DST-EGQA will be enhanced further, but it introduces additional overhead in managing multiple models for continual learning. We will include a table on the specific domain ordering in the appendix for additional clarity. \n\n- Dialogue-level sampling is unexplained: \n    - Thank you so much for pointing out this oversight! Dialogue-level sampling for memory replay as opposed to turn-level sampling is that samples are chosen at the dialogue-level first and turns of the selected dialogues up to the memory size are put into memory. Pseudocode: \n`samples = random.sample(dialogue, memory size//10); chosen_turn_samples = random.sample(flatten(samples), memory size))`\n    - On the other hand, turn-level sampling means that the samples are flattened out to turns first and then randomly sampled from all turns (pseudocode: `samples = random.sample(flatten(dialogue), memory size`). Although this is a simple modification, we observe that this enables a wider diversity of dialogue state updates to be captured in smaller memory budgets without incorporating more complicated techniques to store prototypical samples. \n    - Your thorough review also let us realize that the performance gain of only using the TransferQA format was omitted from Table 1, making the numbers from the memory analysis table potentially confusing. We will clarify that the base number is for when only the TransferQA format is applied, i.e. without using any retrieved examples, and the following rows are only when various memory sampling methods are used. This lets us isolate the effect of memory from that of the retrieved examples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of reformulating DST to EGQA is effective for continual learning, which not only alleviates the catastrophic forgetting problem but also improves model generalization on unseen domains. Utilizing retrieval via in-context learning may also work well on tasks other than DST in the continual learning setting.\n2. The authors conduct detailed experiments to analyze the effects of different factors, such as the types of retrievers and the number of in-context learning samples.\n", "reasons_to_reject": "1. The main contribution of this paper is reformulating DST as EGQA. However, the process of retrieving in-context examples is not clear. I guess the database consists of all samples of the current domain (i.e., the domain of the target dialog), and the IC-DST-ret retriever compares the last utterance pairs ($u_{t-1}$, $u_t$) of $H_t$ and $H_t^\\prime$ (To the authors: please correct me if I'm wrong). In this way, EGQA needs to store all samples from all domains in the database, which should be regarded as using **infinite memory** (instead of \"None\" in Table 1) since all samples from previous domains are available. That would be a disadvantage of the proposed method. It can be improved by limiting the number of samples of each slot/domain in the database, like the memory replay method.\n2. The authors \"use the entire training set of the first domain\" to train the retriever and then use the retriever for all subsequent domains. The authors claim that \"we do not want to extend the continual learning problem for training the retriever.\" However, could that retriever work well on other domains? More evaluation and discussion are appreciated.\n3. \"Dialogue-level sampling\" is unexplained. As the authors claim it is one of the main contributions, it should be discussed explicitly. ", "questions_to_authors": "See \"Reasons To Reject\".", "paper_topic_and_main_contributions": "This paper proposes alleviating the catastrophic forgetting problem in continual learning by formulating the Dialog State Tracking (DST) task as an Example-Guided Question Answering (EGQA) task, which reduces the data distribution gap between domains and takes advantage of in-context learning. The authors transform DST to QA like TransferQA (Lin et al., 2021), and retrieve similar samples from the training set as in-context examples like IC-DST (Hu et al., 2022). Experiments on continual learning DST show the effectiveness of EGQA reformulation and analyze method robustness on different retrievers, memory types and sizes, and the number of in-context learning examples.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 183: \"we will be finetuning the model\" => \"we will finetune the model\";\n\nLine 380: The number \"43.2\" should refer to Table 2."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the "dialogue state continual learning" problem. The authors present a novel and straightforward example-guided QA solution, achieving SOTA on the SGD dataset. The solution is built on three foundational ideas:

1. **Reformulation of DST into QA (DST-EGQA)**: This approach considerably enhances continual learning performance by improving task consistency across domains.
  
2. **In-context Learning** by retrieving relevant in-context examples (from past domains) during both fine-tuning and testing. This method is akin to Memory-replay, given its exposure to past training examples. However, its performance (when combined with DST-EGQA as DST-EGQA + IC-DST-ret) doesn't measure up to Memory-replay (with a 50 turn-level example memory size). Despite this, the third pillar demonstrates that Memory techniques and In-context fine-tuning can collectively enhance the performance.
  
3. **Dialogue-level Memory-replay**: Sampling examples at the dialogue level proves superior to turn-level sampling. This is attributed to the richer dialogue state a dialogue offers compared to an individual turn.

While the original paper has moments of ambiguity in presenting its main contributions and experiments, the authors managed to clarify these areas during the rebuttal phase, providing reviewers and AC with a clearer understanding of their innovations. Nonetheless, there are instances of overstatements in the paper, and I recommend the authors reevaluate and modify their claims and experimental presentation.

Regarding the scores provided: Soundness scores are uniformly at (3, 3, 3), while excitement scores varied at (4, 4, 3). It's evident that all reviewers are enthused by the proposed solution due to its evident performance improvements in the experimental settings. Additionally, the simplicity of the proposed solutions is commendable.

However, reviewers expressed concerns about:

1. The single testing benchmark.
2. Limiting their experimentation to a single model size (60M).
3. A potentially skewed comparison with the Memory-replay method.

AC's feedback on these issues:

1. **Single Testing Benchmark**: Though multiple benchmarks typically provide a more comprehensive demonstration of a solution's generalizability, the SGD is arguably the most fitting benchmark for the proposed solution. This is because other dialogue benchmarks combine domains, which may not serve as the ideal "proof-of-concept" benchmark for this study. Authors might consider other benchmarks (like a sequence of mixed-domain tasks) to further fortify their experimental claims.

2. **Model Size**: In their rebuttal, the authors mention testing larger models and observing similar findings to the 60M size. But without presenting these specific results, this concern remains unresolved.

3. **Comparison with Memory-replay**: A reviewer pointed out a potential unfair in comparison with the Memory-replay method. However, after examining lines 333-335 of the paper, it appears the authors incorporated all past examples when using the Memory-replay method. Authors might want to clarify this for enhanced transparency.

In conclusion, this study comes across as a "proof-of-concept" endeavor, mostly justifying its contributions. I consider the work to be of **moderate sound and exciting**.
Decision: Accept-Main