TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidation (2024)

Abstract: A practical dialogue system requires the capacity for ongoing skill acquisition and adaptability to new tasks while preserving prior knowledge. However, current methods for Continual Dialogue State Tracking (DST), a crucial function of dialogue systems, struggle with the catastrophic forgetting issue and knowledge transfer between tasks. We present TaSL, a novel framework for task skill localization and consolidation that enables effective knowledge transfer without relying on memory replay. TaSL uses a novel group-wise technique to pinpoint task-specific and task-shared areas. Additionally, a fine-grained skill consolidation strategy protects task-specific knowledge from being forgotten while updating shared knowledge for bi-directional knowledge transfer. As a result, TaSL strikes a balance between preserving previous knowledge and excelling at new tasks. Comprehensive experiments on various backbones highlight the significant performance improvements of TaSL over existing state-of-the-art methods. The source code1 is provided for reproducibility.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1266–1279 August 11-16, 2024 ©2024 Association for Computational Linguistics
A practical dialogue system requires the capacity for ongoing skill acquisition and adaptability to new tasks while preserving prior knowledge. However, current methods for Continual Dialogue State Tracking (DST), a crucial function of dialogue systems, struggle with the catastrophic forgetting issue and knowledge transfer between tasks. We present TaSL, a novel framework for task skill localization and consolidation that enables effective knowledge transfer without relying on memory replay. TaSL uses a novel group-wise technique to pinpoint task-specific and task-shared areas. Additionally, a fine-grained skill consolidation strategy protects task-specific knowledge from being forgotten while updating shared knowledge for bi-directional knowledge transfer. As a result, TaSL strikes a balance between preserving previous knowledge and excelling at new tasks. Comprehensive experiments on various backbones highlight the significant performance improvements of TaSL over existing state-of-the-art methods. The source code1 is provided for reproducibility.

heading: 1 Introduction
text: With the rising popularity of conversational digital assistants, it is imperative for dialogue systems to integrate new services while sustaining proficiency in prior tasks seamlessly. Traditional research, often conducted within specific domains offline, falls short in adaptability to new scenarios (Ni et al., 2023). Retraining pre-trained language models (PLMs) from scratch is both challenging and resource-intensive (Liu et al., 2023), highlighting the necessity for efficient continual learning (CL) approaches in dialogue systems (Ke and Liu, 2022). Dialogue state tracking (DST), crucial for taskoriented dialogue systems, dynamically updates
∗Corresponding author. 1https://github.com/WoodScene/TaSL
(domain, slot, value) triplets to capture user intentions precisely. The urgent demand for advancing DST models to accommodate emerging services has catalyzed the development of the Continual DST task (Cho et al., 2023).
An effective Continual DST system must address the issue of catastrophic forgetting (McCloskey and Cohen, 1989), where a model’s proficiency in old tasks diminishes after learning new ones. It should also promote knowledge transfer (KT) (Ke et al., 2021) across domains2 to enhance end-task performances. Knowledge transfer includes forward transfer, which improves new task performance using knowledge from previous tasks, and backward transfer, which enhances performance on previous tasks after learning a new relevant task. Striking a balance between retaining previous knowledge and excelling in new tasks is vital for success.
However, current Continual DST methods (Madotto et al., 2020; Liu et al., 2021; Cho et al., 2023; Feng et al., 2024) mainly focus on mitigating forgetting through memory replay or regularization, overlooking the advantages of KT that can be derived from the inherent correlations between different DST domains.
Task correlation in Continual DST is quite ev-
2In this work, different domains in Continual DST are equivalent to different tasks in CL.
1266
ident. For instance, domains like “Hotel” and “Restaurant” share semantically similar slots, such as “area” and “bookday”, highlighting the need for models to identify and handle common information types. The similarity in these domain-shared slots is crucial for enabling KT. However, learning domain-specific slots like “food” for “Restaurant” could introduce unique information that disrupts the retention of previously acquired knowledge, leading to catastrophic forgetting.
To address these challenges, we introduce Task Skill Localization and Consolidation (TaSL), a framework designed to improve KT between tasks without relying on memory replay. This is achieved by identifying and consolidating the importance distribution of model parameters across tasks. TaSL initially employs a group-wise importanceaware skill localization technique that utilizes gradient trajectories to pinpoint tiny regions in the model that store crucial knowledge for the current task. By comparing the importance distribution with those of previous tasks, we can differentiate between task-specific and task-shared parameters, as illustrated in Figure 1. Our innovative skill consolidation phase then categorically integrates weights from previous tasks with the current one, enabling effective KT while minimizing forgetting.
In detail, the importance-aware skill localization method employs a new group-wise metric to compute importance scores, effectively quantifying the significance of each “skill unit”3 for the current task. Our approach, focusing on parameter space rather than dataset-driven categorization of domain-shared and domain-specific slots, offers a more robust solution that accurately identifies taskspecific and task-shared knowledge, overcoming inaccuracies caused by dataset noise.
Our skill consolidation stage, then based on a fine-grained model averaging strategy, effectively manages different types of knowledge. We enable forward KT to new tasks by starting with a model initialized with weights from previously fine-tuned tasks, thus using past knowledge to improve learning for new tasks without restrictions. For backward KT, we merge knowledge from both current and past accumulated tasks into localized task-shared skill units, thereby enhancing their capability. To prevent catastrophic forgetting, we
3The definition of skill units is model-specific. For example, the Query matrix within the self-attention layer may be regarded as a skill unit. For detailed definitions, please refer to Appendix B.
consolidate the integrity of skill units containing previous task-specific knowledge, ensuring they remain unaffected by new task learning. Through extensive experiments on different parameter-level backbones (from 60M to 7B), TaSL exhibits superior performance in mitigating forgetting and showcases remarkable capabilities for KT, significantly outperforming state-of-the-art methods.
Our main contributions include: • We propose a novel task skill localization and
consolidation (TaSL) framework for CL. • We develop new group-wise skill localization and
fine-grained skill consolidation techniques. • Extensive evaluation on continual DST tasks
shows TaSL effectively enables knowledge transfer, resulting in a 3.1% absolute increase in Avg. JGA and an 8.8% absolute boost in BWT metrics compared to previous SOTA methods.

heading: 2 Related Work

heading: 2.1 Continual Dialogue State Tracking
text: Continual Learning (CL) in task-oriented dialogue systems focuses on perpetually integrating knowledge from data streams for future application. Three kinds of CL methods have been developed. Architecture-based methods propose dynamically adding model weights when learning new data (Geng et al., 2021; Lu et al., 2021b; Yang et al., 2023). Replay-based methods store and replay some training samples from previous tasks (Hou et al., 2019; Lu et al., 2021a; Xu et al., 2023a). Regularization-based methods employ additional loss functions to solidify new knowledge (Li and Hoiem, 2017; Xu et al., 2023b).
In the realm of Continual DST, pioneering efforts by Madotto et al. (2020) and Liu et al. (2021) have leveraged these CL strategies to set benchmark performance using PLMs. The DST-EGQA approach by Cho et al. (2023) reformulates the DST task to an example-guided question-answering task, aiming to align distribution shifts across different domains to mitigate forgetting. However, these methods overlook DST task correlations that could enhance knowledge transfer. The recent Continual Prompt Tuning (CPT) method by Zhu et al. (2022) attempts knowledge transfer via domain-specific soft prompts but depends on inefficient memory replay and extensive retraining. This dataset-driven approach is inefficient and lacks robustness.
Our TaSL innovates by distinguishing between domain-specific and domain-shared knowledge
within the parameter space, then leveraging the skill consolidation process for effective knowledge transfer and forgetting mitigation.

heading: 2.2 Task Skill Localization
text: Research indicates that model parameters contribute unevenly to performance (Michel et al., 2019). Panigrahi et al. (2023) introduced the concept of “skill localization” to identify crucial parameters within PLMs, suggesting that fine-tuning critical parameters nearly matches the effect of full fine-tuning. However, their method requires additional time for identifying and retraining key parameters post-fine-tuning, lowering efficiency.
Drawing inspiration from the pruning community, previous studies have used gradient-based metrics to identify important parameters during fine-tuning. Sensitivity-based scoring (Sanh et al., 2020; Zhang et al., 2022b) assesses the impact on training loss and sensitivity smoothing, as applied by Zhang et al. (2022a), eliminates unnecessary parameters for more efficient fine-tuning. However, these approaches, focusing on individual parameter importance, often lead to element-wise pruning with huge computational and storage burdens (Feng et al., 2023a). Based on these advances, we introduce a new importance-aware skill localization method, for the first time, that distinguishes between task-specific and shared parameters to mitigate forgetting in CL.

heading: 3 Proposed Method: TaSL
text: Problem Formulation In continual DST, we aim to train a model f : X ×T → Y across a sequence of dialogue domains T1, . . . , TK . Each dialogue domain has its dataset Dk for Tk. This model predicts the target y based on input x and task Tk ∈ T . The notation fk refers to the model after training on task Tk, while f̂k denotes the model after averaging for f̂k−1 and fk. Within a given task Tk, a dialogue with M turns of interaction between the system and the user is denoted as XM = {(A1, U1) , (A2, U2) . . . , (AM , UM )}, where A and U represent the system’s response and the user’s input, respectively.
Each task Tk is associated with a predefined slot set4 S = {S1, . . . , SJ}, where J is the total number of slots. The objective of DST is
4To specify the domain to which a slot belongs, a slot is defined as the concatenation of the specific domain and the slot name, e.g., “<restaurant-area>”.
to predict the dialogue state Bm based on the dialogue context Xm. The dialogue state is a collection of (slot, value) pairs, expressed as Bm = {(S1, V m1 ) , . . . , (SJ , V mJ )}, where V mJ is the value for slot SJ at turn m. DST involves training a model f : Xm ⊕ Sj → V mj , with ⊕ denotes simple text concatenation.
Overview TaSL includes two key components: (i) Skill Localization, utilizing a new group-wise importance metric to accurately identify the importance distribution of parameters across tasks, and (ii) Skill Consolidation, which employs a novel fine-grained model averaging strategy to integrate model weights from both current and past tasks for effective knowledge transfer. Figure 2 provides a comprehensive overview of TaSL, with the following subsections detailing each component.

heading: 3.1 Importance-aware Skill Localization
text: To address the substantial computational and storage demands imposed by previous parameter-level importance calculation methods (Konishi et al., 2023), we propose a new group-wise metric for evaluating the importance of each skill unit u:
I(u) = 1 d1 × d2
d1∑
i=1
d2∑
j=1
s(wij) (1)
where wij denotes the trainable parameters, and d1 × d2 represents the total parameter count in a skill unit u. I(u) measures the collective importance of all parameters within each skill unit, where higher values signify increased importance. The function s(·) is a designated importance function for individual parameters, defined as the magnitude of the gradient-weight product:
I (wij) = ∣∣wij∇wijL ∣∣ (2) This approximates the loss change when a parameter is zeroed out. If removing a parameter has a significant influence, then the model is sensitive to it, and we should retain it (Liang et al., 2021).
However, sensitivity in Eq. (2) may not reliably indicate importance (Zhang et al., 2022b). This metric, calculated from a sampled mini-batch, suffers from variability due to stochastic sampling and training dynamics, introducing large uncertainty in estimating sensitivity. To mitigate this, we apply sensitivity smoothing and uncertainty quantification (Zhang et al., 2022a):
Ī(t) (wij) = α1Ī (t−1) (wij) + (1− α1) I(t) (wij)
(3)
Ū (t) (wij) = α2Ū (t−1) (wij)+
(1− α2) ∣∣∣I(t) (wij)− Ī(t) (wij) ∣∣∣ (4)
where α1 and α2 are smoothing factors, and t is the iteration number. Ī(t) represents smoothed sensitivity by exponential moving average and Ū (t) is the uncertainty term quantified by the local variation between I(t) and Ī(t). Importance is then defined by multiplying Ī(t) and Ū (t), providing a more accurate importance assessment for s(·):
s(t) (wij) = Ī (t) (wij) · Ū (t) (wij) (5)
Calculating current task importance scores. To compute the importance score of each skill unit for task Tk, we employ Eq. (1) during finetuning. The model f with n skill units is denoted as U = {u1, . . . , un}, with their importance scores for task Tk denoted by I(Uk) ∈ Rn. The detailed computation process is provided in Algorithm 1.
Computing accumulated importance scores for previous tasks. After computing importance scores for each skill unit at the current task Tk, it is essential to compare these with scores from all previously learned tasks to distinguish between task-specific and task-shared parameters. To avoid the inefficiency of storing scores for each past task, we aggregate importance scores from all prior tasks into a cumulative score for tasks up to Tk−1. This method allows for the iterative refinement of accumulated scores without separately saving past task scores. The skill units with these cumulative scores
up to Tk−1 are denoted as Ûk−1, calculated using:
I(Ûk−1) = βNorm(I(Ûk−2))+ (1− β)Norm(I(Uk−1))
(6)
where β ∈ [0, 1], and Norm(·) normalizes importance scores to the [0, 1] range, thus resolving discrepancies across models. The initial scores, I(Û1), are set to be equal to I(U1). Following this, the importance distribution for skill units up to task Tk−1 is combined with that of the current task, Tk, to facilitate the skill consolidation process.

heading: 3.2 Skill Consolidation
text: After skill localization, the subsequent vital phase involves consolidating this knowledge into a unified framework. This process demands a sophisticated model averaging approach considering various factors to optimize task performance. Traditional coarse-grained model averaging assumes that all model weights are equally important for the training task (Kirkpatrick et al., 2017; Eddine Marouf et al., 2023), which can be written as the following iterative computation format:
f̂k = λf̂k−1 + (1− λ) fk (7)
However, this method may overemphasize weights irrelevant to the current task, contaminating previously acquired task-specific knowledge and leading to forgetting. To counteract this, we introduce a fine-grained averaging strategy focusing
Algorithm 1 Importance-aware Skill Localization Input: Training dataset Dk for task Tk; total train-
ing iterations T ; hyperparameters α1, α2. for t = 1, . . . , T do
Sample a mini-batch from Dk and compute the gradient∇L; Compute the sensitivity I (wij) via Eq. (2); Update Ī(t) via Eq. (3) and Ū (t) via Eq. (4); end for Compute the importance score I(uki ) for each skill unit uki by Eq. (1), for i = 1, . . . , n.
Output: fk and importance scores I(Uk) for Uk.
on skill units rather than the entire model. Our approach distinguishes between task-shared and taskspecific skill units, categorically applying weighted averaging to parameters within each skill unit.
We initially set importance thresholds δ using quantiles to select the top 20% of skill units based on importance scores. A skill unit uki is deemed important (denoted as (uki )
+) if its score I(uki ) is above δk, and unimportant ((uki )
−) otherwise. Our fine-grained averaging strategy customizes parameter combination for each skill unit, based on its importance under different tasks, as follows:
ûki =    γûk−1i + (1− γ)uki , if (ûk−1i )+, (uki )+ ûk−1i , if (û k−1 i ) +, (uki ) − uki , if (û k−1 i ) −, (uki ) +
1 2(û k−1 i + u k i ), if (û k−1 i ) −, (uki ) −
(8) This strategy performs the element-wise adjustment of parameters within each skill unit based on its relevance to previous and current tasks, using hyperparameter γ to control their influences.
In the scenario where a skill unit ui is significant for both past and present tasks (case 1), we integrate newly acquired knowledge into this taskshared skill unit to enable backward KT. If a skill unit ui is crucial solely for previous tasks (case 2), we maintain the knowledge within this previous task-specific skill unit untouched to prevent the contamination of historical knowledge with taskirrelevant information. In contrast, for a skill unit important only to the current task (case 3), since the model fk is trained on the initialization of f̂k−1, the historically learned knowledge is utilized to enhance the performance of the current task, enabling forward KT. Thus, we ensure the integrity of parameters within this current task-specific skill unit,
preserving essential knowledge for excelling in the new task. We adopt a straightforward averaging for units not pertinent to either task (case 4).
Skill consolidation is performed before starting a new task in CL, utilizing the averaged model for subsequent task initialization. Only the importance scores of Ûk−1 and Uk are retained for use between tasks, starting with Û1 = U1 estimated from f1 on D1. Detailed implementation of TaSL algorithm can be found in the Appendix (Algorithm 2).

heading: 4 Experiments and Analysis
text: Dataset We use the continual learning for DST setup proposed by Zhu et al. (2022), which uses 15 single domains from the Schema-Guided Dialog dataset (SGD) (Rastogi et al., 2020). We aggregate our results over the same five domain orders to make the most reliable comparisons with prior works. Comparing results with the same order is crucial as the results can have significant variance depending on the chosen domains and their order. More details about data statistics, task selection, and orderings can be found in the Appendix A.
Evaluation Protocol We evaluate DST performance using the widely adopted Joint Goal Accuracy (JGA) metric (Wu et al., 2019), which indicates the percentage of turns for which all slot values are correctly predicted. We denote aj,i as the JGA on the test set of task Ti right after training on task Tj . The performance of Continual DST is assessed using three metrics from Zhu et al. (2022):
Avg. JGA = 1
K
K∑
i=1
aK,i (9)
FWT = 1 K − 1 K∑
i=2
ai−1,i (10)
BWT = 1 K − 1 K−1∑
i=1
aK,i − ai,i (11)
Avg. JGA represents the average JGA across all tasks after training on the final task TK . Forward Transfer (FWT) evaluates a model’s generalization ability by measuring the averaged zero-shot performance. Backward Transfer (BWT) assesses the impact of learning on subsequent tasks on a previous task. Negative BWT indicates the model lost some previously acquired knowledge.
Baselines We evaluate our method with the following Continual DST baselines: Fine-tuning: Continuously fine-tune the backbone model on new task data. Replay: Randomly save |M| samples from the training set of each previous task Ti in memory Mi and jointly train the model on new task data DK and memoryM<k. EWC: Maintain a memory but leverage it to compute the Fisher information matrix for regularization (Kirkpatrick et al., 2017). AdapterCL: Freeze the pre-trained model and independently train a residual Adapter (Houlsby et al., 2019) for each task (Madotto et al., 2020). Continual Prompt Tuning (CPT) (Zhu et al., 2022): Freeze the backbone model and continually train soft prompts with memory-guided knowledge transfer in both forward and backward directions. DST-EGQA (Cho et al., 2023): Reformulate DST as a QA task to mitigate forgetting with retrieval-augmented in-context learning. RoS (Feng et al., 2024): Utilize knowledge distillation to enhance the meta-reasoning ability of student models, thereby mitigating forgetting.
Training Details We utilize four distinct parameter-level backbones for experiments: T5-small (Raffel et al., 2020), T5-base, FlanT5-large (Chung et al., 2022), and LLaMA-7B (Touvron et al., 2023). For the T5 series model, we perform full fine-tuning across all parameters. For LLaMA-7B, we adopt the Parameter-Efficient
Fine-Tuning technique, specifically Low-Rank Adaptation (LoRA) (Hu et al., 2021), to expedite the training process. For TaSL, we set the hyperparameters α1 and α2 in Eq. (3) and Eq. (4) to 0.85, and set β in Eq. (6) and γ in Eq. (8) to 0.7. The memory size per task |M| is maintained at 50, aligning with previous studies. Detailed training settings are provided in Appendix B.
Following this, we compare TaSL with baselines in Sec. 4.1, and present a comprehensive ablation study in Sec. 4.2. Subsequently, we delve deeper into the underlying success of our proposed importance-aware skill localization (Sec. 4.3) and skill consolidation techniques (Sec. 4.4), and get some insightful findings from this exploration.

heading: 4.1 Main Results
text: Overall CL results of different methods at the same T5-small backbone are summarized in Table 1.
TaSL demonstrates superior CL performance through effective knowledge transfer. Unlike vanilla fine-tuning, which suffers from catastrophic forgetting, our approach demonstrates a substantial improvement in Avg. JGA, increasing it from 44.1% to 62.1%, and shows marked advancements in both FWT and BWT.
TaSL not only exceeds the CPT method, which relies on memory replay, advancing the Avg. JGA from 61.2% to 62.1%, but also establishes a new
SOTA across all metrics against the top baseline, DST-EGQA. We achieve an impressive increase in Avg. JGA from 59.0% to 62.1% (3.1% absolute improvement) and elevate BWT from -17.9% to -9.1%, exceeding it by more than 8% and displaying robust backward KT capabilities. Additionally, TaSL obtains the highest FWT scores across all baselines under various conditions.
Remarkably, without relying on memory replay, our method nearly matches the performance of DST-EGQA with memory replay, particularly in BWT, with a minimal difference of 3.2% (-9.1% vs. -5.9%). Moreover, our Avg. JGA score closely approaches the upper bound performance set by the CPT multi-task strategy at 64%, underscoring the effectiveness of our fine-grained model averaging strategy that meticulously accounts for domainshared and domain-specific parameters.
TaSL consistently demonstrates superior performance across various backbones. To further substantiate the effectiveness of our framework, we conducted experiments using a variety of parameter-level backbones, illustrated in Figure 3, highlighting performance gains with increasing model size. Notably, TaSL achieves a breakthrough by recording a positive BWT score on LLaMA7B, without employing any memory replay techniques. Across different backbones, our method consistently outperformed traditional approaches. For instance, in Flan-T5-large, TaSL significantly boosts the Avg. JGA metric from 56% to 74%, also achieving the most substantial improvements in both FWT and BWT metrics — rising from 33% to 43% and improving from -28% to -13%, respectively. These results further validate the generality of our proposed framework.
Fine-grained model averaging can effectively mitigate catastrophic forgetting. To rigorously evaluate our model’s effectiveness in counteracting forgetting, we analyzed its performance on the
initial task after training on subsequent tasks. Figure 4 illustrates that our method results in a notably slower forgetting rate, manifesting as a nearly 8% average decrease in performance after training on the last task. This contrasts sharply with vanilla backbones, which display a substantial performance reduction of 20% on average, thereby underscoring our method’s superior capacity to mitigate forgetting. Moreover, an intriguing observation is the enhancement in performance on task 1 after training on task 3, highlighting our model’s effective backward KT ability.

heading: 4.2 Ablation Study
text: This section we assess the effects of importanceaware skill localization and fine-grained skill consolidation, with the results discussed below. For hyperparameter sensitivity, see Appendix D.
Various importance scoring methods for skill localization. Our method calculates importance scores by Eq. (1). As shown in Table 2, we explore alternative importance scoring approaches: (i) modifying s(·) in Eq. (1) only to include sensitivity, as in Eq. (2); and (ii) using absolute gradients, ∣∣∇wijL ∣∣, for importance assessment (Michel et al., 2019). The results indicate that using moving averages for importance scoring outperforms the alternatives, with the other two variants leading to a maximum performance decrease by 3.26%, 2.24%, and 4.11% across the three metrics. This highlights the value of accurate skill localization in improving model performance.
Fine-grained vs. coarse-grained model averaging for skill consolidation. We compared our fine-grained averaging strategy against two coarsegrained strategies: (i) Weight-Ensemble, which averages weights uniformly as per Eq. (7), and
(ii) Exponential Moving Average (EMA) (Szegedy et al., 2016), applying a running average of parameters at each fine-tuning iteration. Results are detailed in Table 3.
Weight-Ensemble significantly improves upon the vanilla model, highlighting coarse-grained averaging’s benefits for Continual DST. EMA generally surpasses Weight-Ensemble but falls short of our fine-grained approach due to its overuse of averaging, with frequent parameter adjustments within the same task possibly resulting in less optimal outcomes. Our method solely averages weights after each task, enhancing computational efficiency.

heading: 4.3 Visualization of Skill Units
text: We visualized the distribution of importance scores for the skill units across tasks and models, as shown in Figure 5, leading to several critical insights:
• There is a noticeable variation in the importance of skill units for the same task, with important skill units in LLaMA-7B making up about 20% of all trainable LoRA parameters.
• The distribution of important skill units is taskdependent, indicating both task-shared and specific parameters, confirming TaSL’s validity.
• Lower layers, nearer to the input, are more crucial for the DST task compared to upper layers.
• Within each layer, the importance of the attention layer, especially the V (value) and O (output) matrices, consistently exceeds that of the Q (query), K (key) matrices, and the MLP layer.

heading: 4.4 Improved Balance in Knowledge Transfer
text: This section evaluates the effectiveness of our fine-grained model averaging method in achieving the optimal balance between preserving previous knowledge and excelling at new tasks in CL, comparing it to coarse-grained approaches.
Table 4 shows that for a sequence of two tasks, vanilla fine-tuning on LLaMA-7B results in a notable decline in historical Task 1’s performance (from 92.3% to 73.1%), indicating notable forgetting. Coarse-grained averaging mitigates this to an extent, reducing the decline to 82.3% but impacting new task performance to 71.3%. Our method effectively lessens forgetting (improving to 86.9%) while also maintaining better performance on Task 2, with less than a 3% reduction.
As tasks increase to three, our method more effectively compensates for losses on new tasks with gains on historical tasks. Vanilla fine-tuning on T5small results in a combined 55.6% drop in Tasks 1 and 2, while our approach only shows a 16.6% decrease and the loss on task 3 is less than 1% due to TaSL’s effective KT ability.

heading: 5 Conclusion
text: In this paper, we introduce a novel TaSL method to enhance Continual DST performance by facilitating effective knowledge transfer across tasks. Our approach leverages an innovative importanceaware skill localization technique and a skill consolidation strategy to differentiate between domainspecific and domain-shared parameters, mitigating forgetting. Comprehensive experiments showcase our method’s exceptional ability to balance preserving past knowledge and excelling in new tasks.
Limitations
TaSL excels at precisely distinguishing between task-specific and shared parameters through importance-aware skill localization. However, the current importance scoring criteria, relying on firstorder gradients, may lack precision. The Hessian matrix often captures the actual importance, but computing these second-order gradients incurs significant computational costs. Therefore, future improvements should focus on developing more accurate and efficient skill localization methods.
In addition, in skill consolidation, the challenge lies in better integrating model parameters. Under our fine-grained model averaging strategy (Eq. (8)), selecting different weighted combinations could impact the overall performance. Although we investigated the model’s sensitivity to various hyperparameter settings (Appendix D), with results showing stable and consistently strong performance across different combinations, there is still potential for further improvement. For instance, developing adaptive methods to select optimal weights or devising more efficient model averaging strategies could further enhance model performance.

heading: Acknowledgments
text: We thank the anonymous reviewers for their valuable feedback. This research was partially supported by the grant of HK ITF ITS/359/21FP.

heading: A Dataset Statistics
text: Here, we offer a detailed description of the dataset used in Continual DST. Table 5 displays the number of slots for each of the 15 services used in our experiments and the count of samples in the training, validation, and test sets. Table 6 illustrates the training sequence for these 15 tasks in the context of continual learning.
B Implementation Details
Definition of Skill Units In our Importanceaware Skill Localization technique, to circumvent the high computational resource demands of parameter-level localization, we introduced a novel group-wise metric, redefining the skill unit u as the basic element for computing importance scores. However, the division of skill units varies across different backbone models due to variations in parameter quantities and architectural designs (e.g.,
decoder-only and encoder-decoder architectures). The specific distinctions are as follows: • Encoder-decoder architecture backbones. For
these backbones (Feng et al., 2021), such as T5small, T5-base, and T5-large, we perform fullparameter fine-tuning during training. For organizational simplicity, we divided the models based on the different functionalities within the transformer blocks, as depicted in Table 7. For instance, in T5-small, with both encoder and decoder comprising 6 transformer blocks each, the total comes to 131 skill units for T5-small, 257 skill units for T5-base, and 558 for T5-large.
• Decoder-only architecture backbone. The LLaMA-7B model we utilized falls into this category (Feng et al., 2023b). Leveraging Parameterefficient Fine-tuning Techniques (PEFT) to expedite training, we treat the matrices A and B in LoRA as individual basic skill units. And we add LoRA adapters to attention layers in LLaMA. Each layer, as detailed in Table 8, comprises 8 distinct skill units. Given that LLaMA-7B consists of 32 layers, it is thereby segmented into 256 skill units in total.
Model training details For different backbones, we utilized the following hyperparameters: • T5-small (60M), T5-base (220M), and FLAN-
T5-lare (780M): Training was conducted with a learning rate of 3e-4, batch size of 8, maximum input length of 512, maximum target length of 128, and 5 epochs.
• LLaMA (7B): Utilizing LORA for efficiency, with a learning rate of 3e-4, batch size of 128, a cutoff length of 512, and 5 epochs. Lora settings were r = 8, alpha = 16, dropout = 0.05, targeting modules [[q_proj,k_proj,v_proj,o_proj]]. For testing, settings included temperature = 0.02, top_p = 0, top_k = 1, num_beams = 1, max new tokens = 128. Experiments are carried out using 2 NVIDIA A100 with 80GB memory. Results are averaged across five different task orders and include the standard error in the tables and plots provided (Feng et al., 2022).

heading: C Additional Results
text: To further validate TaSL’s effectiveness in more complex continual learning scenarios, we have conducted additional experiments to verify its performance on transitioning between different datasets,
specifically from SGD to MultiWoz. This involved including another 5 distinct domains from the MultiWoz 2.1 dataset, in addition to the 15 domains in SGD, resulting in a total of 20 domains (i.e., tasks). Table 9 presents the performance of various methods, utilizing T5-small as the backbone.
The findings align with the observations from Table 1, although there is a noticeable decrease in the efficacy of all evaluated methods, due to the significant discrepancies in data distribution across the datasets examined. As a memory-free method, our TaSL still significantly outperforms the strongest baseline (i.e., the memory-free version of DSTEGQA) and even surpasses some memory-based methods like “Replay”. These findings demonstrate TaSL’s robustness and effectiveness, showcasing its capability to handle complex continual learning scenarios.

heading: D Sensitivity Analysis for Hyperparameters
text: The proposed framework incorporates three key hyperparameters, including the α for computing importance scores in Equations (3) and (4), the β for calculating cumulative importance scores in Equation (6), and the γ for performing weighted averaging within skill units as outlined in Equation (8). Our analysis aims to assess the impact of varying these hyperparameters on our method’s performance, testing on the T5-small backbone model.
As evidenced in Table 10, we determine that the optimal setting for α is 0.55. An α value too low results in a performance decline, indicating that the calculated importance scores are not sufficiently accurate. Furthermore, as depicted in the results of Tables 11 and 12, we also find that β and γ values within a normal range do not significantly affect performance. However, excessively high or low values for β and γ may skew the model towards favoring either past or current task knowledge, thereby disrupting the desired balance.
Nonetheless, the model’s performance remains relatively stable across most conditions, indicating a low sensitivity to hyperparameter variations.
About the selection of threshold for important skill units, the Table 13 below shows the model’s performance with varying thresholds δ on T5small.
It can be seen that setting a high threshold (50%) reduces model effectiveness by categorizing less significant skill units as important, which can contaminate historical knowledge and lead to forgetting. Conversely, a 1% threshold still maintains strong performance owing to our effective skill consolidation approach, which effectively preserves task-specific knowledge and prevents forgetting. Considering that the heatmap in Figure 5 displays approximately 20% of the area in darker shades, signifying greater importance, we opted for a 20% threshold to differentiate between important and unimportant skill units.

heading: E Detailed Algorithm
text: In this section, we provide the detailed implementation of TaSL algorithm (see Algorithm 2).

heading: 1% 62.0 26.3 -9.4

heading: 5% 63.4 25.8 -10.1

heading: 10% 62.2 26.2 -9.5

heading: 20% 62.1 26.6 -9.1

heading: 30% 62.7 26.5 -10.0

heading: 40% 60.9 24.6 -10.2

heading: 50% 54.8 23.4 -10.3
text: Algorithm 2 TaSL Input: Dataset Dk for task k = 1, . . . ,K; initial
pre-trained model f0; hyperparameters β, γ. 1: # sequential tasks. 2: for task k = 1, . . . ,K do 3: Get fk and calculate Uk by Algorithm (1); 4: Calculate δk based on I(Uk); 5: if k = 1 then 6: # initialization at beginning task. 7: f̂1 ← f1, Û1 ← U1, δ̂1 ← δ1; 8: else 9: # fine-grained model averaging.
10: for skill unit i = 1, . . . , n do 11: Calculate ûki by (8); 12: end for 13: Get the averaged model f̂k based on Ûk; 14: Calculate accumulated importance score I(Ûk) according to (6); 15: Calculate δ̂k based on I(Ûk). 16: end if 17: end for

heading: Future Work
text: Therefore, future improvements should focus on developing more accurate and efficient skill localization methods. In addition, in skill consolidation, the challenge lies in better integrating model parameters. Under our fine-grained model averaging strategy (Eq. (8)), selecting different weighted combinations could impact the overall performance. Although we investigated the model’s sensitivity to various hyperparameter settings (Appendix D), with results showing stable and consistently strong performance across different combinations, there is still potential for further improvement. For instance, developing adaptive methods to select optimal weights or devising more efficient model averaging strategies could further enhance model performance