Continual Dialogue State Tracking via Reason-of-Select Distillation (2024)

Abstract: An ideal dialogue system requires continuous skill acquisition and adaptation to new tasks while retaining prior knowledge. Dialogue State Tracking (DST), vital in these systems, often involves learning new services and confronting catastrophic forgetting, along with a critical capability loss termed the “Value Selection Quandary.” To address these challenges, we introduce the Reason-of-Select (RoS) distillation method by enhancing smaller models with a novel ‘meta-reasoning’ capability. Metareasoning employs an enhanced multi-domain perspective, combining fragments of metaknowledge from domain-specific dialogues during continual learning. This transcends traditional single-perspective reasoning. The domain bootstrapping process enhances the model’s ability to dissect intricate dialogues from multiple possible values. Its domainagnostic property aligns data distribution across different domains, effectively mitigating forgetting. Additionally, two novel improvements, “multi-value resolution” strategy and Semantic Contrastive Reasoning Selection method, significantly enhance RoS by generating DSTspecific selection chains and mitigating hallucinations in teachers’ reasoning, ensuring effective and reliable knowledge transfer. Extensive experiments validate the exceptional performance and robust generalization capabilities of our method. The source code1 is provided for reproducibility.

heading: 1 Introduction
text: Practical dialogue systems require continual adaptation to new services while maintaining previous task capabilities. However, previous research in dialogue systems has focused on domain-specific offline systems, lacking adaptation abilities (Ni et al., 2023). With recent advances in large language models (LLMs), LLM-based systems have
∗Corresponding author. 1https://github.com/WoodScene/RoS
shown significant superiority over previous methods (Hu et al., 2023). However, the bulky sizes of LLMs make retraining models from scratch prohibitively time-consuming and challenging (Liu et al., 2023). Thus, efficient continual learning (CL) is vital for dialogue systems to obtain new skills while retaining knowledge of previous tasks. Dialog State Tracking (DST), central to task-oriented dialogue systems, dynamically updates triplets (domain, slot, value) to manage user intents (Feng et al., 2023a). The growing necessity to expand DST models for new services has spurred interest in recent Continual DST task (Cho et al., 2023).
As a specific task of CL, Continual DST grapples with catastrophic forgetting (McCloskey and Cohen, 1989; French, 1999), where sequential learning of new tasks hinders retention of prior ones. This problem is more acute in DST due to significant data distribution shifts across domains in the CL process, exemplified by transitions from Hotel to Restaurant domains (see Figure 1). Recent advances in Continual DST, including memory replay and regularization (Zhu et al., 2022), and refor-
7075
mulation as a question-answering task (Cho et al., 2023), strive to address forgetting. However, these methods face challenges like reliance on past data and high computational demands during testing, which hinder real-time applications.
Recent LLMs show impressive DST performance (Heck et al., 2023; Feng et al., 2023a), but practical deployment faces hurdles like offline computational load and online data privacy concerns. Moreover, enabling LLMs for CL demands significant resources, leading to the exploration of smaller models. After a comprehensive analysis of the current smaller DST models, we have identified a critical loss of capability brought by domain shifts in CL that models fail to output correct value when facing a set of similar candidates (named “Value Selection Quandary” as elaborated in Section 2). For example, in Figure 1, when tracking the <services-appointment_date> slot, the model incorrectly chooses the most recent time mentioned, revealing its inability to grasp contextual subtleties and favoring direct value extraction over logical reasoning. This phenomenon is because, as the CL progresses, the model needs to understand the relevant knowledge of newly emerging domains and chooses to forget domain-irrelevant or weakly relevant knowledge, such as value selection.
This paper addresses these challenges by boosting smaller models’ reasoning ability, termed as meta-reasoning. Meta-reasoning, guided by the insight that domain-specific dialogues represent only a fragment of the underlying meta-knowledge, can be viewed as a form of multi-view augmentation from different domains in the CL process. This domain bootstrapping strategy facilitates the transfer and broadening of meta-knowledge, transcending traditional single-perspective reasoning. Meta-reasoning enriches smaller models’ reasoning abilities to dissect and interpret intricate dialogue scenarios, addressing the “Value Selection Quandary” and broadening their cognitive horizons. Moreover, this domain-agnostic feature aligns data distribution shifts across different domains, effectively reducing catastrophic forgetting.
Inspired by the powerful reasoning abilities of LLMs like LLaMA-2-70B (Touvron et al., 2023) and ChatGPT 2, we introduce Reason-of-Select (RoS) distillation approach, designed to graft these advanced reasoning capabilities onto smaller models. We present a “multi-value resolution” strategy
2https://chat.openai.com/chat
tailored to DST, prompting teacher LLMs to generate a “selection chain” that discerns and elaborates the best value choice from various options. Then, we distill this rationale into a smaller student model to boost its meta-reasoning capabilities, making it easier to deploy to terminal systems.
Moreover, to enhance faithful reasoning and reduce hallucinations in the teacher model, we integrate a schema-guided prompt, strengthening source-target correlation and semantic coherence. Our innovative Semantic Contrastive Reasoning Selection method uses semantic similarity to select the most accurate teacher-generated rationale, suitable even in black-box scenarios. Extensive experiments with various teacher and student model sizes demonstrate our method’s exceptional Continual DST performance and robust cross-dataset generalization capabilities.
To summarize, our main contributions are:
• We propose a Reason-of-Select (RoS) distillation framework for Continual DST. Through enhancing domain-agnostic meta-reasoning capabilities, ambiguous value selection and catastrophic forgetting caused by data distribution shifts across domains can be effectively mitigated.
• To alleviate the hallucination outputted by a teacher model in our annotation-free reasoning framework, we present a Semantic Contrastive Reasoning Selection strategy to obtain trustworthy rationale from multiple candidates, ensuring faithful reasoning transfer.
• Comprehensive experiments with two teacher models and four student models of varying sizes demonstrate superior performance and robust generalization of our proposed method.

heading: 2 Motivation for Boosting Meta-reasoning Capability in Continual DST
text: Through an initial experiment with the same setups from prior work (Zhu et al., 2022) (details in Section 4), we employed two distinct backbone models, T5-small and LLaMA-7B, and observed a performance decline in Continual DST with turn ids increasing (Figure 2 (a)). This excites two fundamental questions for obtaining better continual DST performance: (i) What crucial knowledge do current models lack for continual DST? (ii) How can this missing knowledge be effectively integrated?

heading: 2.1 Pinpointing the Value Selection Quandary
text: Upon analyzing the 874 incorrectly predicted slot samples in long dialogues (turn id > 10), we initially found that in 94.5% of cases, the model usually returns values semantically relevant to the requested slot from the dialogue. For example, a location-related slot often results in selecting any mentioned location. Upon closer examination, we found that the model failed to select the correct value due to a significant capability loss, termed the “Value Selection Quandary”. This issue occurs when the model struggles to reason out the correct value among multiple ambiguous or semantically similar values in a dialogue.
Examining 826 errors linked to this quandary, we observed a pattern: in 45% of cases, models favored the most recently mentioned value (Figure 1). In 30% of instances, they stuck with a previously chosen value, disregarding modifications in the dialogue. The remaining 25% involved random selections from available values. Moreover, Figure 2(b) classifies error rates across 49 slots, highlighting that slots regarding ‘date’, ‘from’, and ‘to’ are most prone to errors. This is logical, as time or location slots in dialogues often present multiple values, leading to selection challenges.
These findings highlight a critical gap in models’ ability to navigate the “Value Selection Quandary”, presenting a significant avenue for enhancement.

heading: 2.2 Bridging the Gap of Reasoning Ability
text: Inspired by the robust reasoning capabilities exhibited by models exceeding 70B parameters (Li et al., 2023; Wang et al., 2023), we evaluated their efficacy in DST task, specifically their ability to generate coherent rationales. As demonstrated in Table 1, ChatGPT’s rationalizations for the <appointment_date> slot confirm their proficiency in discerning and justifying value selections.
Our Reason-of-Select Distillation method, leveraging LLMs’ selection logic, boosts smaller models’ reasoning and decision-making abilities. This domain-agnostic approach effectively reduces forgetting in continual learning by teaching models to reason across diverse tasks and domains instead of merely imparting specific information or tasks. In addition, we revealed the hallucination in teacher models’ rationales, such as references to non-existent dialogue elements (Table 1), which motivates a novel Semantic Contrastive Selection method (Section 3.3) ensuring the accuracy and relevance of the logic transferred to student models.

heading: 3 Reason-of-Select Distillation
text: Problem Formulation In continual DST, we train a model f : X × T → Y across a series of dialogue domains T1, ..., TK . This model predicts the target y based on input x and task Tk ∈ T . Within a specific task Tk, a dialogue with T turns of interactions between the system and the user can be represented as XT = {(A1, U1) , (A2, U2) . . . , (AT , UT )}, where A represents the system response and U represents the user input. A predefined slot set3 S = {S1, . . . , SJ} is provided, where J is the total number of slots for task Tk. The objective of DST is to predict the dialogue state Bt based on the dialogue context Xt. The dialogue state, Bt, is represented as a set of (slot, value) pairs, denoted as Bt = {( S1, V t 1 ) , . . . , ( SJ , V t J )} , where V tJ is the value associated with slot SJ at turn t. Essentially, the DST problem is defined as training a dialogue state tracker model f : Xt ⊕ Sj → V tj , where ⊕ denotes simple text concatenation.
3To specify the domain to which a slot belongs, a slot is defined as the concatenation of the specific domain and the slot name, e.g., “<restaurant-area>”.

heading: 3.1 Overview
text: Our Reason-of-Select Distillation framework capitalizes on LLMs to create a faithful selection reasoning process through a ‘multi-value resolution’ strategy and Semantic Contrastive Reasoning Selection. This enriched knowledge is then imparted to smaller student models for training, as illustrated in Figure 3.

heading: 3.2 Teacher’s Reasoning Generation
text: The process begins with a dialogue-centric prompt that includes dialogue content X , the target slot Sj , and its value Vj to derive a reasoning process R. While traditional prompts like “Tell me why <Sj> is Vj” or the famous “Let’s think step by step” prompt (Kojima et al., 2022) generate reasonings that tend to merely highlight the location of the correct answer in the dialogue.
Our innovative “multi-value resolution” prompt PR fosters a more elaborate reasoning process, enhancing the model’s capability from mere location identification to an in-depth selection reasoning process, as illustrated in Figure 3. This methodological advancement from basic identification to complex process reasoning represents a substantial leap in the model’s cognitive capacities.
This stage’s input-output representation is fteacher : Xt⊕Sj⊕V tj ⊕PR → Rtj . However, neural LMs often exhibit hallucinations — generating text with tenuous ties to the input (Ji et al., 2023; Maynez et al., 2020), which our unsupervised an-
notation approach can exacerbate. To counter this, we introduce the innovative Semantic Contrastive Reasoning Selection method, pivotal for accurately aligning reasoning with corresponding answers.

heading: 3.3 Ensuring Faithful Teaching with Semantic Contrastive Reasoning Selection
text: Our novel strategy employs a newly schema-guided prompt (yellow background in Figure 3) to direct teacher models toward generating on-topic rationales, significantly improving the source-target correlation. Our Semantic Contrastive Reasoning Selection technique revolutionizes reasoning generation, creating a spectrum of candidates and choosing the most semantically aligned. Inspired by contrastive decoding (Li et al., 2022) in text generation task, our method adapts this concept to suit blackbox LLMs like ChatGPT. We introduce strategic input perturbations to simulate and correct reasoning errors, refining the teacher model’s output.
The procedure commences with the LLM generating G diverse reasonings (R1, . . . ,RG). We then introduce two forms of perturbation: value-level and slot-level, as illustrated in Figure 4. While value-level perturbation subtly modifies the ground truth value, slot-level perturbation involves completely replacing the slot-value pair. This latter proves particularly effective in inducing a range of reasoning responses, from logical to nonsensical, thereby enriching our negative sample pool with N perturbated reasonings (PR1, . . . ,PRN ). Positive samples are represented by the dialogue-centric
prompt, denoted as DC. Utilizing a fixed, pretrained contextual encoder, Sentence-transformers (Reimers and Gurevych, 2019), we then transform these R, PR and DC into semantic representations within space E , as illustrated in Figure 5.
The crux of our method lies in selecting the most appropriate reasoning from the generated G candidates, a decision based on their semantic proximity to the positive samples and divergence from the negative samples. This selection process is formulated as an elegant optimization problem:
minDistance(Ri,DC)
s.t.max N∑
n=1
Distance(Ri,PRn) (1)
To quantify the relationship between each reasoning and the sample sets, we introduce a sophisticated scoring mechanism:
Score(Ri) = exp (d (Ri,DC) /τ)∑N
n=1 exp (d (Ri,PRn) /τ) (2)
where d is a distance function and τ is a temperature scalar. By computing a score for each reasoning, we discern the optimal choice with the lowest score for training the student model based on its alignment with actual dialog context and its divergence from perturbated content.

heading: 3.4 Training Student Models via Reasoning-Enhanced Data
text: Armed with the annotated dataset {X , S,R, V }, we proceed to train a smaller student model within the self-rationalization framework, emphasizing both predictive and explanatory skills. This approach departs from prior post-rationalization models, where rationales are formulated post-prediction or those employing a multi-task format, treating rationale creation as an auxiliary task.
The student model is conditioned to generate a sequence that merges rationale tokens with the corresponding answer tokens in response to a given dialogue context and slot request, as demonstrated in the right part in Figure 3. This task is executed by fine-tuning a pre-trained language model, with “silver” data derived from the teacher model. We employ a standard language modeling loss termed factual reasoning loss:
Lfactual = − J∑
j
logP (Rj , Vj | X , Sj) (3)

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Dataset Our experiment employs the SchemaGuided Dialog dataset (SGD) (Rastogi et al., 2020), encompassing 44 services across 19 domains with slot descriptions. Adhering to the Continual DST setup by Zhu et al. (2022), we focus on singleservice dialogs, randomly selecting 15 tasks from 44. Each service varies in training sample size and slots. To ensure robustness, we experiment with five task orders via random permutations, aligning with prior studies. Appendix B details the data statistics, task selection, and orderings.
Evaluation Protocol We assess the DST performance using the widely adopted Joint Goal Accuracy (JGA) metric (Wu et al., 2019), which demands accurate predictions for all slot values. We denote aj,i as the JGA on the test set of task Ti right after training on task Tj . CL performance is assessed using three metrics from Zhu et al.
(2022): (i) Avg.JGA = 1
K
K∑
i=1
aK,i, represent-
ing the average JGA across all tasks after training on the final task TK . (ii) Forward Transfer (FWT)
= 1 K − 1 K∑
i=2
ai−1,i, evaluating generalization by
measuring zero-shot performance, and (iii) Back-
ward Transfer (BWT) = 1 K − 1 K−1∑
i=1
aK,i − ai,i,
quantifying resistance to forgetting by assessing the influence of new learning on previous tasks.
Baselines We evaluate our model against existing Continual DST baselines: Fine-tuning: Continuously fine-tune the model on new task data. Replay: Stores |M | instances per task Ti in memory Mi for joint training with new tasks. EWC (Kirkpatrick et al., 2017): Maintain a memory but leverage it to compute the Fisher information matrix for regularization. AdapterCL (Houlsby et al., 2019): Freeze the pre-trained model and independently train a residual Adapter for each task. Continual Prompt Tuning (CPT) (Zhu et al., 2022): Freeze the backbone model and continually train soft prompts with knowledge transfer in both forward and backward directions. DST-EGQA (Cho et al., 2023): Reformulate DST as a Question-Answering task using retrieval-augmented in-context learning.
We also include our method with memory replay and multi-task learning as performance caps.
Training Details ChatGPT (using the gpt-3.5turbo API) and LLaMA-2-70B serve as teacher models for fine-tuning smaller student models including T5-small, T5-base, FlanT5-XL, and LLaMA-7B, utilizing generated rationales. The teacher model’s temperature is set to 0.7, generating five candidate reasonings (G = 5) alongside three value-level and three slot-level perturbed negative reasonings (N = 6). In Eq.2, the parameter τ is set to 0.8, employing the Euclidean distance metric. The memory size per task |M | is maintained at 50, in line with previous studies (Zhu et al., 2022). Detailed training specifications are provided in Appendix C.

heading: 4.2 Main Results
text: Table 2 presents the results from various methodologies. Key findings include:
Injecting Reasoning Knowledge Effectively Enhances CL Performance Our RoS method significantly surpasses standard fine-tuning on T5small, raising the Avg.JGA from 44.1% to 59.0%, and showing gains in both FWT and BWT. This highlights the value of integrating reasoning into Continual DST. Compared with the erstwhile topperforming DST-EGQA, we achieve a new SOTA performance in all metrics, notably increasing Avg.
JGA from 55.5% to 59.0% without extra memory, demonstrating our approach’s effectiveness in mitigating historical task forgetting. Furthermore, when memory is available, RoS jumps from 59.0% to 72.1%, even outstripping the CPT’s multi-task performance. Among various student models, RoS with LLaMA-7B, fine-tuned with ChatGPT’s reasoning, stands out, substantially lifting the Avg. JGA from 59.0% to 68.7%, with a remarkable FWT improvement from 25.5% to 51.9%.
Figure 6 shows the models’ ability to mitigate catastrophic forgetting by assessing their performance on the initial task post successive task learnings. Models with RoS reasoning skills exhibit a slower forgetting rate, with a 15% average drop in performance in both T5-small and LLaMA-7B. In contrast, vanilla backbone models show a steeper decline, averaging a 28% performance dip, highlighting the importance of domain-agnostic reasoning skills in sustaining historical task performance.
RoS Distillation Exhibits Strong Generalization Ability Our method significantly improves the FWT metric, indicative of robust generalization and zero-shot learning capabilities. It bridges the distribution gap across domains by creating a systematic reasoning chain, evident in handling semantically similar but differently described slots, like <services-appointment_date> and <hotels-check_in_date>. Unlike traditional models, ours utilizes reasoning to recognize similarities, enhancing adaptability to unseen slots.
To further evaluate this generalization, we introduce a new 16th task using MultiWOZ 2.4 data (Ye et al., 2022). The zero-shot performance on this task follows the sequential training of the first 15 tasks from the SGD, as shown in Table 3.
Compared to similar parameter-sized backbones, RoS (T5-small) outperforms the T5-base CPT
baseline, increasing average JGA by 3.7%, from 10.92% to 14.62% across domains. This is particularly notable in the Attraction domain, with JGA rising from 10.05% to 15.74%. Compared to various vanilla backbones, the improvement due to domain-agnostic reasoning averages a 6% increase, further confirming RoS’s robust generalization.
To assess the hallucination rate of the teacher model, we utilized SelfCheckGPT (Manakul et al., 2023) and employed BERTScore for evaluation. BERTScore provides scores ranging from [0.0, 1.0], where higher values indicate a greater likelihood of non-factual content, implying an increased probability of hallucination. Our evaluation included the average BERTScore for the initial generation of five candidate reasonings by the teacher models, the BERTScore after applying the Semantic Contrastive Reasoning Selection method, and the hallucination rate of the distilled student. The summarized results are presented in the Table 4.
Table 4 illustrates that for both teacher models, ChatGPT and LLaMA-2-70B, the average
BERTScore for the initial five reasoning candidates were 19.3% and 21.7%, respectively. After applying the Semantic Contrastive Reasoning Selection method, the BERTScore decreased to 14.2% and 15.9%, affirming the effectiveness of our proposed approach in enhancing the factual of reasoning. Regarding the student models, there is no significant difference in BERTScore when distilled with different teacher models. Additionally, we observed that the quality of reasoning generated by the student models improves as the model size increases.

heading: 4.3 Ablation Study
text: The Role of Model Size Variation in Teacher and Student Models Our study investigates the effect of teacher and student model sizes on performance. Figure 7 shows a clear trend: larger student models, like LLaMA-7B, outperform smaller ones, such as T5-small. We first evaluate LLaMA-7B’s continual learning in DST, finding it outperforms most T5-small or T5-base-based methods. However, applying our RoS method with reasoning is the most notable enhancement, especially in smaller models. For example, integrating reasoning into T5-small
yields an impressive 15% average increase in all evaluation metrics. Contrary to our initial assumption that students guided by ChatGPT might surpass those instructed by LLaMA-2-70B, the results indicate equal reasoning quality from both teacher models, suggesting their similar effectiveness in enhancing student model learning.
Effectiveness of Semantic Contrastive Reasoning Selection Method To assess our method’s ability to reduce reasoning hallucinations, we compared three reasoning selection strategies: random selection from G candidates and selecting the highest and lowest scoring (ours) reasonings, as per Equation 2. The results are detailed in Table 5.
Initial results show that all methods significantly outperform the vanilla LLaMA, with even random selection improving Avg. JGA by 4.94%. This highlights the inherent reasoning quality in current teacher models. Yet, refining the selection to eliminate hallucinatory reasonings and aggregate higher-quality reasonings further enhanced performance. Compared to random selection, our method shows additional gains of 3.2%, 0.9%, and 2.8% in the evaluated metrics. In contrast, selecting the highest-scoring result, i.e., favoring less effective reasoning, resulted in reduced performance. These findings robustly affirm the effectiveness of our semantic similarity-based selection approach.

heading: 4.4 Case Study
text: Table 6 presents a key test case and various methods’ predictions to showcase student models’ ad-
vanced reasoning capabilities. The dialogue involves two values for the <pickup_time> slot: “17:15” and a later update to “half past 1 in the afternoon”. The vanilla LLaMA model incorrectly selects “17:15”, while our model effectively identifies all possible values and accurately determines the correct one. Interestingly, our RoS T5-small model’s output of “1:30 pm” mirrors the ground truth but is marked incorrect due to current evaluation criteria. This highlights a challenge in matching model outputs with expected formats, suggesting a direction for future refinement.

heading: 5 Related Work

heading: 5.1 Knowledge Distillation from LLMs
text: Knowledge distillation in NLP, crucial for transferring insights from larger teacher models to smaller student models, has evolved recently, especially in extracting reasoning from LLMs. Recent studies (Hsieh et al., 2023; Wang et al., 2022; Yang et al., 2023; Xu et al., 2023b) highlight LLMs’ role in bolstering smaller models. Li et al. (2023) furthered this with a symbolic Chain of Thought (CoT) distillation, enhancing smaller models through CoT prompting. Despite these advancements, the faithfulness of generated rationales, critical for student models’ behavior, often must be addressed. Wang et al. (2023) proposed a contrastive decoding method to reduce hallucinations, but its reliance on teacher model logits limits its applicability for black-box LLMs. Current reasoning distillation methods are unsuitable for DST task and lead to unproductive reasoning processes.
To overcome these issues, our method focuses on the Value Selection Quandary in DST, introducing
a ‘multi-value resolution’ prompt and a Semantic Contrastive Reasoning Selection method, enhancing knowledge transfer accuracy and relevance.

heading: 5.2 Continual Dialogue State Tracking
text: Continual Learning in task-oriented dialogue systems, focusing on mitigating catastrophic forgetting, has employed various methods such as architecture-based (Shen et al., 2019; Geng et al., 2021; Xu et al., 2023a), rehearsal-based (Rebuffi et al., 2017; Hou et al., 2019; Lu et al., 2021b), and regularization-based (Li and Hoiem, 2017; Feng et al., 2024; Lu et al., 2021a). In DST, contributions like Madotto et al. (2020) and Liu et al. (2021) have utilized these CL strategies, with Zhu et al. (2022) introducing Continual Prompt Tuning (CPT) to fine-tune domain-specific soft prompts. Furthermore, DST-EGQA (Cho et al., 2023) employs a question-answering framework based on exampleguided learning. However, its reliance on fixed QA templates may limit adaptability across diverse domains. Our method, in contrast, offers enhanced time efficiency and flexibility during testing, eliminating the need for sample retrieval, thus presenting a more efficient solution.

heading: 6 Conclusion
text: This study introduces the Reason-of-Select (RoS) distillation method to address the Continual DST task’s catastrophic forgetting and “Value Selection Quandary” challenges. Enhancing smaller models with a novel domain-agnostic ‘meta-reasoning’ capability effectively broadens their reasoning horizon and aligns data distribution shifts across different domains. Two innovative methods, the “multi-value resolution” strategy and Semantic Contrastive Reasoning Selection, further strengthen RoS by ensuring reliable knowledge transfer. Extensive experiments demonstrate our method’s superior performance and robust generalization.
Limitations
In contrast to the conventional knowledge distillation process, our approach necessitates additional computational resources during the preparation of training data and the training phase of the student model. Firstly, our Contrastive Reasoning Selection method requires the generation of multiple candidate reasonings, along with several value-level and slot-level perturbation reasonings. This process is more time-consuming than standard rational
distillation, as it involves extensive data generation by the teacher model. Secondly, training student models with rationales introduces a minor increase in computational overhead during the training period. However, it’s important to note that these expenses are one-time costs. When it comes to testing, our model, already adept at reasoning, does not incur extra time for predictions, and the time involved in generating reasoning in the test phase is almost inconsequential.

heading: Acknowledgments
text: We thank the anonymous reviewers for their valuable feedback. This research was partially supported by the grant of HK ITF ITS/359/21FP.

heading: A Description of Prompt Templates
text: A.1 Prompt Templates for Generating Teacher Model Reasonings
Below, we present a specific example of the prompt used to elicit reasoning generation from the teacher model. { “instruction”: Just return a concise reasoning process. “input”: Performing the dialogue state tracking task. Consider the dialogue content: “[Previous dialogue omitted ...] [USER]: I would like an appointment at 10:45 am. [SYSTEM]: When is the appointment for? [USER]: The appointment is for the 11th of March. [SYSTEM]: Booking appointment with Rigg Christie on March 11th at 10:45 am. [Remaining dialogue omitted ...]”, the answer to slot <services-appointment_date> is ‘10:45 am’.
Some information about the slot: services indicates Discover the right therapist for you and make reservations easily, appointment_date indicates Date of the appointment. Can you tell me why and your reasoning
process? In the context of dialogue state tracking, there are often multiple possible values associated with the requested slot. Please provide a concise explanation of how to select
the most appropriate value for the requested slot by carefully analyzing the dialogue context, user and system intent, and taking into consideration any confirmation or rejection information.
}
A.2 Prompt Templates for Fine-Tuning the Student Model
Below, we provide a specific example of the prompt used for fine-tuning the student model. { “instruction”: Given the following dialogue, track the state of the dialogue and predict the value of the slot <alarm_1-new_alarm_name>.
“input”: [USER]: I want to check the alarms I have. [SYSTEM]: There are 2 alarms which you have set currently, with one of the alarms being at 6:30 am, and it is called Wake Up. [USER]: Alright, that is good. [SYSTEM]: Are you interested to add another alarm? [USER]: Actually I am, I do want to add another alarm. I want the alarm to be called Grocery run.
[slot] alarm_1-new_alarm_name, it indicates Name to use for the new alarm. So the value of slot <alarm_1-new_alarm_name> is
“output”: grocery run
}

heading: B Dataset Statistics
text: Here, we offer a detailed description of the dataset used in Continual DST (Feng et al., 2022). Table 7 displays the number of slots for each of the 15 services used in our experiments and the count of samples in the training, validation, and test sets. Table 8 illustrates the training sequence for these 15 tasks in the context of continual learning.
C Implementation
For reasoning generation by the teacher model (Feng et al., 2021), we utilized the following hyperparameters: • LLaMA-2 (70B): Model set as Llama-2-70B-
chat-GPTQ, with temperature at 0.7, top_p at 0.9, top_k at 40, and a maximum of 512 new tokens.
• ChatGPT: Generation was conducted from October 21th to 30th, 2023, using the ‘gpt-3.5-turbo’ API. The settings were temperature = 0.7 and max_tokens = 256. For training the student models, we applied these hyperparameters: • T5-small (60M) and T5-base (220M): Training
was conducted with a learning rate of 3e-4, batch size of 8, maximum input length of 512 (1024 for reasoning fine-tuning), maximum target length of 128 (512 for reasoning fine-tuning), and 5 epochs.
• T5-XL (3B): To optimize training time, we used LORA with a learning rate of 3e-4, batch size of 2 (8 for reasoning fine-tuning), maximum input length of 512 (1024 for reasoning fine-tuning), maximum target length of 128 (512 for reasoning fine-tuning), and 5 epochs. Lora settings included r = 8, alpha = 16, dropout = 0.05, targeting modules [ ‘q’, ‘v’ ]. For testing, we set max new tokens to 128 (512 with reasoning).
• LLaMA (7B): Also utilizing LORA for efficiency, with a learning rate of 3e-4, batch size of 128, a cutoff length of 512 (1024 for reasoning fine-tuning), and 5 epochs. Lora settings were r = 8, alpha = 16, dropout = 0.05, targeting modules [[q_proj,k_proj,v_proj,o_proj]]. For testing, settings included temperature = 0.02, top_p = 0, top_k = 1, num_beams = 1, max new tokens = 128 (512 with reasoning). Experiments are carried out using Nvidia RTX 3090 GPUs. Training durations for various models on total 15 tasks are as follows: T5-small takes about 1 hours on a single 3090 GPU, T5-base takes nearly 2 hours on a single 3090 GPU, FlanT5-XL takes about 16 hours on a single 3090 GPU, and LLaMa-7B takes 21 hours on two 3090 GPUs. Results are averaged across five different task orders and include the standard error in the tables and plots provided (Feng et al., 2023b).
Details on Perturbation Operation: We altered the ground truth value with a random value from the dataset for the same slot for value-level perturbation. However, given the robustness of the teacher model, which can often detect errors and provide correct explanations even if the misleading value does not cause significant confusion, we introduced a slot-level perturbation. This involves completely changing the slot and substituting it with a random slot-value pair from the current task.
To address the “Value Selection Quandary” com-
mon in long dialogues and to save the time and cost of using teacher LLMs, we engage teacher models to generate reasoning for samples with turn id greater than 10, aligning with findings from Figure 2(a). For briefer dialogues, where typically only one possible value appears, our examination indicates teacher models usually return direct reasoning, like “In the given dialogue, the user explicitly mentions their destination as <1150 Webster street>. Therefore, the answer to the slot <ridesharing-destination> is 1150 Webster street.”. Thus, for uniformity across all short dialogues, we’ve standardized the prompt to “In the given dialogue, the value of the requested slot is explicitly mentioned,” enabling student models to learn from this consistent template.

heading: D Time Complexity Comparison
text: Firstly, we examined the training time of the student model. When using LLaMA-7B as the back-
bone, the inclusion of additional rationales led to an approximately 1.5 times increase in training time compared to the original training. For smaller student models like T5-small, the training time remained consistent with no inclusion of reasoning. In terms of inference speed, a comparison is presented in the table 9 below: The table 9 illustrates
a ninefold increase in time consumption due to generating rationales. For instance, during the generation of rationales, the LLaMA-7B experiences a decrease in samples per minute from 41 to 5. Despite the reduction in inference speed, this change contributes to a significant performance improvement.

heading: Future Work
text: This highlights a challenge in matching model outputs with expected formats, suggesting a direction for future refinement