Overcoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System (2024)

Abstract: Intelligent task-oriented dialogue systems (ToDs) are expected to continuously acquire new knowledge, also known as Continual Learning (CL), which is crucial to fit everchanging user needs. However, catastrophic forgetting dramatically degrades the model performance in face of a long streamed curriculum. In this paper, we aim to overcome the forgetting problem in ToDs and propose a method (HESIT) with hyper-gradient-based exemplar strategy, which samples influential exemplars for periodic retraining. Instead of unilaterally observing data or models, HESIT adopts a profound exemplar selection strategy that considers the general performance of the trained model when selecting exemplars for each task domain. Specifically, HESIT analyzes the training data influence by tracing their hyper-gradient in the optimization process. Furthermore, HESIT avoids estimating Hessian to make it compatible for ToDs with a large pre-trained model. Experimental results show that HESIT effectively alleviates catastrophic forgetting by exemplar selection, and achieves state-of-the-art performance on the largest CL benchmark of ToDs in terms of all metrics.

heading: 1 Introduction
text: Serving as a core technique of smart assistants, task-oriented dialogue systems (ToDs) are expected to continuously acquire new knowledge through time regarding user needs (Madotto et al., 2020), e.g., adding fresh slot-value pairs or handling dissimilar tasks. This ability is also known as Continual Learning (CL) (Mundt et al., 2020), which has recently attracted a surge of interest in NLP community (Biesialska et al., 2020), as well as other machine learning techniques (Qu et al., 2021).
In this setting, the central problem of CL is catastrophic forgetting (Kirkpatrick et al., 2017), as the data is streamed and the neural model inevitably forgets previously learned knowledge when fitting
new training data in the sequential order. Such a phenomenon is particularly conspicuous in ToDs tasks due to the obvious distributional shift between task scenarios, e.g., setting an alarm clock and booking a flight are completely irrelevant tasks. As a result, it is challenging for a single model that is learned from a unidirectional curriculum to simultaneously handle multi-domain tasks.
To mitigate the problem of forgetting, existing efforts (De Lange et al., 2021) follow the three lines: (i) adding regularization terms to consolidate learned knowledge, (ii) developing dynamic architectures for the task-specific domain, and (iii) applying historical data rehearsal. Previous works (Madotto et al., 2020) have demonstrated that regularization methods gradually lose effectiveness if faced with a long curriculum, and we confirm this belief with a relative experiment. Dynamic architecture methods (e.g. adapter tuning) are usually considered as an unfair baseline (Rusu et al., 2016; Maltoni and Lomonaco, 2019; Mi et al., 2020) due to multiple sets of parameters, and it requires a further step during inference to determine
48
the correct parameters resulting in high latency for ToDs. Therefore, this work focuses on rehearsal methods (Fig. 1 (a)) (Rolnick et al., 2019) that store a small number of past samples (a.k.a. exemplars) in episodic memory and replay them periodically, which is simple but effective. Then we raise our research question: which utterances are suitable to be selected as rehearsal exemplars for ToDs?
Although statistic methods (e.g., reservoir algorithm (Chaudhry et al., 2019)) conduce to select better exemplars than random sampling, according to the accredited influence chain “Data-ModelPerformance” (Sun et al., 2022), an exemplar selection strategy should further consider the general performance of models during testing. In other words, we should select the exemplars that positively influence the performance of the trained network on unseen data, as shown in Fig. 1 (b).
To explore the influence of training data from the performance perspective, one pioneering work uses the Influence Functions (IFs) based on the derivation chain rule (Koh and Liang, 2017) to quantify the contribution of an individual training sample, by observing its impact on the test loss when removing it. Though IFs have shown effectiveness in image classification tasks, this method requires estimating the inverse Hessian matrix, leading to high computing costs (Guo et al., 2020) and unstable results for large pre-trained models that are widely used in ToDs. Furthermore, IFs only measure the contribution around the model parameters at the final training epoch, which fails to trace the data contribution in the dynamic optimization process (Chen et al., 2021).
In this paper, we propose a rehearsal-based CL method, named HESIT (Hyper-gradient-based Exemplar Selection by Influence Tracing) to effectively overcome the catastrophic forgetting in ToDs. Specifically, instead of only focusing on the model parameters of the final epoch, HESIT traces and analyzes the hyper-gradient of training examples in the complicated optimization process. More importantly, its exemplar selection strategy roots in model performance on unseen data to measure the influence of these traced samples. In this way, influential and representative exemplars can be determined and stored, and then utilized to remind the model of learned knowledge. Moreover, HESIT is Hessian-free to avoid the instability when estimating Hessian matrix of a large pre-trained models, which has been widely used in ToDs. Experimental
results demonstrate that HESIT achieves state-ofthe-art performance on the largest CL benchmark of ToDs (Madotto et al., 2020) including 37 domain tasks in terms of all metrics. Furthermore, comparative experiments show that the exemplar selection in HESIT surpasses all other selection strategies from other perspectives.
The main contributions of this paper can be summarized as follows:
• We propose HESIT – a rehearsal-based CL method with a novel exemplar selection strategy that can effectively overcome catastrophic forgetting in ToDs.
• HESIT dynamically traces the hyper-gradient of candidate data in the training process and selects exemplars in terms of data influence to model performance.
• Compared with other influence functionbased methods, HESIT avoids estimating the Hessian matrix without degrading effect, which is compatible for ToDs with a largescale pre-trained model.
• Extensive experiments show that HESIT can achieve state-of-the-art performance on the largest CL benchmark of ToDs (37 domain tasks).

heading: 2 Related Work
text: Continual learning. In the past few years, CL has achieved remarkable progress in mitigating catastrophic forgetting (McCloskey and Cohen, 1989), which can be classified into three categories: (1) regularization-based methods (Kirkpatrick et al., 2017; Serra et al., 2018; Huang et al., 2021; Qin and Joty, 2022a; Qin et al., 2024), which focus on regularizing the parameters corresponding to be inherited from the old tasks and penalizing the feature drift. (2) architecture-based methods (Xu and Zhu, 2018; Li et al., 2019; Ermis et al., 2022; Qin et al., 2023), which develop dynamic parameter isolation or expansion during CL training, where each task domain learns a specific architecture. (3) rehearsalbased methods (Cui et al., 2019; Chaudhry et al., 2019; Verwimp et al., 2021; Qin and Joty, 2022b), which utilize a small replay buffer to retain a fraction of learned training data and utilize them to retain the task knowledge. One shortcoming of architecture-based and rehearsal-based methods
is that they require extra memory space for storing parameters and old data samples (De Lange et al., 2021), while regularization-based methods might be overwhelmed when handling many task domains (Mai et al., 2022). Continual learning in ToDs. Early work on CL for ToDs was introduced by Lee (2017), where the elastic weight consolidation (EWC) method was utilized to alleviate catastrophic forgetting on 3 sequential domains. Wu et al. (2019) studied CL for DST sub-task on the MultiWOZ dataset, where several baselines have been compared. Mi et al. (2020) combined data rehearsal and EWC on NLG sub-task and expanded the learned knowledge to 13 task domains. Furthermore, Madotto et al. (2020) developed a large benchmark for all INTENT, DST, and NLG tasks, where mainstream CL methods (e.g., L2, AGEM, and Adapter) were compared on a total of 37 domains. However, it shows that regularization-based methods lose effectiveness when ToDs encounter dozens of tasks (Madotto et al., 2020). Meanwhile, architecture-based methods require a further step to determine which set of parameters to use (Wortsman et al., 2020), thus blocking the real-time response from ToDs. In this work, we focus on rehearsal-based CL methods as it is simple and effective, with an acceptable cost of extra memory. Exemplar selection. To sample representative or informative exemplars from a large dataset, reservoir sampling has been introduced in (Isele and Cosgun, 2018; Chaudhry et al., 2019; Hu et al., 2024) so that the data distribution in the buffer follows the data distribution that has already been seen. Rebuffi et al. (2017) proposed a herdingbased strategy to maintain an online coreset. Similarly, gradient-based methods have also been proposed in (Broderick et al., 2013; Aljundi et al., 2019) to maximize the variance in the buffer. For ToDs with clear task boundaries, Mi et al. (2020) defined representative exemplars as a small set of utterances that fulfill a loss-based criterion on the current domain. The above-mentioned methods leverage models to choose high-quality data points, but fail to provide an insightful perspective in terms of model performance on unseen data.

heading: 3 Methodology

heading: 3.1 Background: End-to-end ToDs
text: We formulate task-oriented dialogue systems as a sequence-to-sequence generation problem that gen-
erates both API-calls and system responses. The API-call consists of the user intent and the current dialogue state, which can be empty or the system speech-act, to generate the system response. Thanks to the recent advance in ToDs, all groundtruth information at each turn is provided by existing annotated dialogue datasets. In this setting, the data format of API-call C is shown as follows:
C(H) = I︸︷︷︸ Intent (s1 = v1, . . . , sp = vp)︸ ︷︷ ︸ Slot-value pairs , (1)
where H is the dialogue history, I denotes the user intent, and (si, vi) stands for a slot-value pair from dialogue state tracking.
ToDs can be typically decomposed into different modules, including intent recognition (INTENT), history state tracking (DST), and natural language generation (NLG). In this paper, besides the submodules, we define the end-to-end (E2E) manner that directly generates the system response R:
H+ C(H)→ R (2)
where C(H) is often empty, and thus the model can directly generate a response according to dialogue history (H→ R). In addition, our method HESIT can work for both E2E and modular ToDs. More details are attached in Appendix B.

heading: 3.2 Rehearsal-based CL in ToDs
text: Given the training data D consisting of T ToDs task domainsD = {D1 · · · DT }, continual learning aims to train a neural model fθ on D in a sequence of tasks. In each task t, new data Dt is used to update model fθt−1 , while the updated model fθt needs to perform well on all tasks so far. In general, the crux of learning f is to overcome the catastrophic forgetting problem where learned knowledge is continually overwritten by streamed data.
To suppress it, the rehearsal-based CL methods construct a small size of memory bufferMt from each task domain Dt. When training task t, the data inM1:t−1 will be retrained together with Dt. Accordingly, a specific training step of task t is written as:
θ̂ = argmin θ
1
N
∑
(xi,yi)∈Dtrnt
ℓ((xi, yi), θ)
s.t. (xi, yi) ∈M1:t−1 ∪ Dtrnt (3)
where N is the total number of training data points from T tasks, ℓ denotes the empirical loss and
(xi, yi) is a training data point of task t. Considering the storage burden ofM, the retained samples, called exemplars, should be informative or influential for Dtrnt .

heading: 3.3 HESIT: Exemplar Selection
text: In this subsection, we present the details of the exemplar selection strategy to find influential utterances inMt, which also serves as the core technique in HESIT.
Intuitively, given a training sample zi := (xi, yi) ∈ Dtrnt , we want to quantify its influence on the validation loss L(Dvalt , θ̂) via the resulting model θ̂. One possible way is to find out the loss difference I(zi,Dvalt ) := L(Dvalt , θ̂z̄i)−L(Dvalt , θ̂) where θ̂z̄i would be the resulting model if zi were not present in Dtrnt . For this purpose, we introduce a weight variable ϵi and consider the new full-set training loss:
Ltrn(θ) = 1
N
∑
(xi,yi)∈Dtrnt
(ℓ((xi, yi), θ)+ϵiℓ(zi, θ)).
(4) Now θ̂ and I(zi,Dvalt ) are taken as functions of ϵi and I(zi,Dvalt ) can be approximated by the first order Taylor expansion:
I(zi,Dvalt ) ≈ − 1
N dL(Dvalt , θ̂) dϵi |ϵi=0
= − 1 N ∂L(Dvalt , θ̂) ∂θ̂ dθ̂ dϵi |ϵi=0.
(5)
While ∂L(D val t ,θ̂)
∂θ̂ can be exactly calculated using
gradient back-propagation on the validation data, computing∇i := dθ̂dϵi , which is also named hypergradient (Bengio, 2000), is not straightforward because the parameter ϵi is involved in the entire optimization process.
In gradient descent based optimization, for each iteration step r, we have θr = θr−1− γgr−1 where γ is the learning rate and gr−1 is the batch gradient. Then∇i can be computed recursively as:
∇i,r = ∇i,r−1 − γ(Hr−1∇i,r−1 + ∂gr−1 ∂ϵi ) (6)
where Hr−1 denotes the Hessian of the batch loss with respect to θr−1. Most related works impose some assumptions on the model to ensure that H is invertible and attempt to approach the inverse Hessian vector product by numerical methods (Koh and Liang, 2017). However, the assumptions are too
Algorithm 1 Influential Example Tracing
Input: Traced example set Zt = {zi}Ii=1 of task t, training dataset Dtrnt , validation dataset Dvalt , batch size B, training dataset size N , total iteration steps R, learning rate γ(t) Output: Model θ̂, influence values I(Z,Dvalt ) 1: Setup reproducible training environment 2: Train model with Dtrnt and obtain θ̂ 3: v ← ∂L(D val t ,θ̂)
∂θ̂ 4: Reset training environment and retrain model
with Dtrnt for second identical training 5: for i = 1 to I do 6: v∇i,0 ← 0 7: end for 8: for r = 1 to R do 9: if current batch contains zi then
10: v∇i,r ← γ(t)v∇i,r−1 − NB vgi,r−1 11: else 12: v∇i,r ← γ(t)v∇i,r−1 13: end if 14: end for 15: for i = 1 to I do 16: I(zi,Dvalt )← − 1N v∇i,R 17: end for
strong to be feasible in practice, and with increasing model and dataset size, the numerical methods are time-consuming and may lead to diverging results. Chen et al. (2021) propose a faster scheme by discarding the Hessian computations, which have a bounded error. Therefore, HESIT can recurrently update the∇i,r through the optimization trajectory with an acceptable time cost. We further design a validation experiment in Section 5.3 to demonstrate that such a manner can successfully estimate data influence without the estimation of Hessian.
As illustrated in Algorithm 1, the influence scores of training examples are computed via reproducible retraining to reduce the space complexity. In this way, the time complexity to trace hypergradient is O(I ·R · ω), where ω is the time spent in computing parameter gradient, but the space complexity has been reduced to O(I). Finally, a set of exemplars EKt are picked out according to the influence scores, with top-K strategy.

heading: 3.4 HESIT: Training Schedule
text: We present the training schedule of HESIT in Algorithm 2, which integrates the exemplar selection strategy and rehearsal-based CL into ToDs. Specifi-
Algorithm 2 HESIT Training Input: Sequential T -domain training data D = {(Dtrn1 ,Dval1 ), . . . , (DtrnT ,DvalT )} Output: Parameter θT that handles T-domain tasks
1: Initialize the model parameter θ0 and replay bufferM0 2: for t = 1 to T do 3: Sample Zt ∼ Dtrnt 4: Calculate I(Zt,Dvalt ) and θt−1 by Algo.1 5: Select EKt ⊂ Zt by top ∥∥I(Zt,Dvalt ) ∥∥ 6: while not converge do 7: Update θt−1 using {Dtrnt ∪Mt−1} 8: end while 9: Mt ← EKt ∪Mt−1
10: end for
cally, for each domain t, we select EKt from Dt using a hyper-gradient based strategy and feed them into the replay bufferM. ThenM will be involved in the subsequent training process for retraining the neural model.
To further accelerate training, we adopt two simplifications. Firstly, instead of tracing all training examples, we sample and trace a subset Zt ≫ EKt from each Dtrnt to reduce the cost. Secondly, tracing is performed only in the first R iterations rather than the entire training process, as the main benefits of optimization have been obtained in these iterations. In addition, the subsequent training (line 6-8) is depend on the size of domain t.

heading: 4 Experiment

heading: 4.1 Dataset
text: To evaluate the performance of our method, we employ the largest CL benchmark of the ToDs task developed in (Madotto et al., 2020). Specifically, four ToD datasets are merged: TaskMaster 2019 (TM19) (Byrne et al., 2019), TaskMaster 2020 (TM20) (Byrne et al., 2019), Schema Guided Dialogue (SGD) (Lin et al., 2021), and MultiWoZ (Budzianowski et al., 2018). Consequently, a curriculum of 37 domains is to be learned continuously in ToDs, where three modules (INTENT, DST, and NLG) are well annotated after pre-processing. We consider two settings which are Modularized setting that learns three modules separately, and E2E setting that learns these modules in a unified manner, as illustrated in 3.1.
In addition, we summarize the main statistics
and detailed sample numbers for each domain in Appendix D. It is noted that the amount of examples is highly imbalanced across different domains, which ranges from a few hundred to more than 36k. This distribution is more proximate to reality, as some task domains lack training data in practice.

heading: 4.2 Evaluation Metrics
text: Based on three modules, we employ the following well-defined metrics to evaluate system performance:
• INTENT recognition is directly measured in terms of accuracy between the predicted intent and ground-truth intent.
• DST is evaluated by Joint Goal Accuracy (JGA) (Wu et al., 2019) over the ground-truth dialogue.
• NLG is evaluated by two metrics: 1) BLEU score (Papineni et al., 2002) which is calculated by the distance between generated response and reference sentence, and 2) slot error rate (EER) (Wen et al., 2015) which is computed as the ratio between the total number of slots and the values not appearing in the response.
For all metrics except EER, a higher value denotes better performance. In addition, same as (Kale and Rastogi, 2020), we do not calculate EER count for the SGD dataset, since some slots of the SGD dataset have only binary values, e.g., yes or no, which is unfair to calculate average with others.

heading: 4.3 Baselines
text: In order to compare the effects of different CL methods, we investigate both regularization-based and rehearsal-based baselines in the ToDs benchmark. Architecture-based methods are excluded as they require a further decision step to determine the agnostic domain of the test set, resulting in a slow system response. Regularization baseline. We consider two kinds of regularization terms to constrain the model parameter update, which is identity function (L2) and Fisher information matrix (EWC) proposed in Kirkpatrick et al. (2017). Rehearsal baseline. We first employ 2 kinds of mainstream rehearsal-based CL methods, which are A-GEM (Chaudhry et al., 2018), and LAMOL (Sun et al., 2019). Since our contribution focuses on exemplar selection, we further
reproduce 4 coreset selection strategies in the ToDs benchmark, where 2 baselines are from dataperspective: (1) Random denotes that exemplars are randomly sampled from the train set, and (2) UNIFORM denotes that we uniformly select exemplars to ensure all user intents would be contained in the buffer. Meanwhile, the other two baselines are from the model perspective: (3) GSS (Aljundi et al., 2019) selects the examples that maximize the gradient variance in the replay buffer. (4) ARPER (Mi et al., 2020) selects exemplars from the training set that obtains minimum loss-based criterion. (1)∼(4) share the same training schedule with the proposed HESIT but they adopt different exemplar selection strategies respectively.
Additionally, we attach a lower-bound baseline, called VANILLA, which is trained on each task continuously without any anti-forgetting mechanism. We also provide the MULTI baseline which trains models with all data in an integrated curriculum simultaneously, which is widely viewed as the upper bound of CL methods.

heading: 4.4 Experimental Setup
text: For all experiments, we leverage the pre-trained GPT-2 (Radford et al., 2019) as the ToDs backbone. Each domain is trained for 10 epochs with early stopping over the validation set. The learning rate is set as 0.001 with a warm-up schedule. In L2 and EWC, the regularization weight is set as 0.001. Considering the buffer memory, all data replaybased methods sample 50 exemplars for each task domain. In HESIT, the Zt is set as 1000 for each domain, and the interested training examples are traced for the first 5 epochs. Task order in the curriculum usually has a slight impact on the final performance. To avoid contingency, all experiments are repeated three times and report the average results.

heading: 5 Result and Analysis

heading: 5.1 Main Comparison Results
text: E2E setting. We report the main E2E results on the full test set in Table 1, where all metrics are evaluated at the end of the curriculum (37 task domains). We observe that regularization baselines (L2 and EWC) can not alleviate catastrophic forgetting well, and even achieve worse performance than VANILLA in terms of most metrics. The learning curve of EWC is visualized in Fig. 4. The reason is that the regularization item gradually constrains
the optimization space of the neural model, making it difficult to handle subsequent tasks in a long curriculum. Similarly, due to clear task boundaries of ToDs, A-GEM and LAMOL baselines also lose effectiveness as their learning schedules depend on correlations across different domains. In this case, regardless of the selection strategy, storing some raw data and retraining the model periodically can obtain remarkable performance gain. As a performance-based selection strategy, our proposed HESIT performs significantly better than the RANDOM baseline, which respectively achieves 6.70%, 5.94%, 13.33%, and 7.86% relative improvement in terms of INTENT accuracy, JGA, EER, and BLEU. In addition, compared with the model perspective GSS baseline, HESIT respec-
tively achieves 2.04%, 2.93%, 4.0%, and 1.48% relative improvement. Modularized setting. We summarize modularized results in Table 2 that INTENT, DST, and NLG modules are evaluated separately. Compared with the E2E setting, we observe some result fluctuations in different evaluation metrics. On the one hand, each neural model focuses on a single module leading to performance gains. Especially in the NLG sub-task, the modularized model surpasses the E2E model by a large margin, where the EER of MULTI reduces nearly fourfold (12.56%→3.42%). On the other hand, the modularized model ignores the relevance of each sub-task, resulting in performance degradation according to INTENT accuracy (95.45%→87.50%). Regardless of E2E or modularized settings, HESIT shows superiority in alleviating catastrophic forgetting in most metrics. Forgetting curve. We visualize the learning curve of INTENT accuracy in the 37-domain curriculum to observe how HESIT overcomes catastrophic forgetting. To avoid the overlapping of the chart, partial baselines are selected for comparison and shown in Fig. 2. It is worth noting that we fix the task order in the curriculum, and each test point on the x-axis evaluates the model performance on the test set of learned tasks.
From the forgetting curve, we observe that 1) Catastrophic forgetting happened in the VANILLA baseline, and the model performance obviously degrades over time. 2) EWC, A-GEM, and LAMOL can alleviate catastrophic forgetting in the first few tasks, but gradually fail to handle learned tasks with an increase in task amounts. In addition, LAMOL achieves outstanding accuracy at the beginning of the curriculum and performs better than HESIT. 3)
For HESIT, there are some fluctuations in performance, but then stabilize at a high performance in terms of accuracy.

heading: 5.2 Analysis for Exemplar Selection
text: Amount of exemplar. Using different selection strategies, we vary the size of the memory buffer containing [20,30,40,50] exemplars for each domain. The results are shown in Table 3. We observe that 20 exemplars can significantly alleviate cartographic forgetting, benefiting all metrics compared with the VANILLA baseline. As the buffer size increases, all systems achieve better performance including each ToDs module. Specifically, GSS achieves the best NLG performance when the exemplar amount is 20. In other cases, HESIT performs the best regardless of the exemplar amount, demonstrating its superiority over the performance perspective methods. Case study. As each training sample is assessed using the Algo. 1’s Influence Functions to evaluate the impact of unseen data, we conduct a case study on several detrimental examples with low influence scores to examine how HESIT eliminates these negative examples. To this end, we simultaneously trace 1k training points of each domain and randomly sample some low-score examples in Table 4. The first two examples adhere to the format of APIcall, and the third example is a system response. In the first two cases, the HESIT successfully identifies the incorrect train_inform() intent and the missing value-slot pairs with extremely negative
scores. The third case is likewise eliminated by the HESIT with a negative score because the model is confused by the empty API-OUT from the history.

heading: 5.3 Effect of HESIT
text: In this part, we conduct experiments to demonstrate that HESIT can successfully measure the training data influence without estimating Hessian. To this end, we construct two Hessian-involved IFs baselines Conjugate gradients (CG) and Stochastic estimation (LISSA) (Agarwal et al., 2017) and one Hessian-free Hydra (Chen et al., 2021) for comparison. As the Hessian of a large pre-trained model is likely irreversible, we further employ a CNN backbone and CIFAR-10 dataset (Krizhevsky et al., 2009) to evaluate both computation time and performance. The details of the dataset, model and baselines are attached in Appendix E.
We vary the amount of training and validation data and report the time spent in influence analysis in Table 5. “Hes.” indicates whether the method needs to estimate the Hessian of the model. “Max_Iter.” denotes CG iteration in (Martens et al., 2010), and is set according to T. In LISSA, “depth” and “repeat” are two hyper-parameters that influence the computation time, and generally, their products should be equal to T. By comparing the baselines and HESIT on (T, V) training-validation dataset pairs of varying sizes, we find that HESIT, Hydra and LISSA require comparable calculation time, whereas baseline CG is significantly slower.
To further demonstrate that HESIT can still properly qualify data contribution even without estimating inverse Hessian, we then visualize the interclass and intra-class contributions in Fig 3, where diagonal elements denote the contribution of the intra-class training data to the test class. It is observed that without estimating inverse Hessian, HESIT can still successfully measure the intraclass data influence by hyper-gradient tracing. In addition, it is observed that compared with LISSA, HESIT provides a higher estimation of intra-class contribution due to iteratively tracing.

heading: 6 Conclusion
text: In this paper, we propose HESIT to address the catastrophic forgetting in ToDs. It selects exemplar for rehearsal-based CL methods that examines training examples in each domain from a performance perspective. Furthermore, HESIT tracks the hyper-gradient of training examples in an optimization method that is Hessian-free and compatible with large pre-trained models. Experiments show that HESIT effectively overcomes catastrophic forgetting and delivers state-of-the-art performance on the largest ToDs benchmark for CL.
Limitations
When selecting exemplars for the current domain, HESIT only evaluates the performance on the selfdomain validation set. However, we suggest that cross-domain influences should be taken into account, which means that the validation set of the later domain in the curriculum might be utilised to examine the picked example of the earlier domain. We would leave it as future work. In addition, we can only compare our model to the state-of-the-art work on ToDs from 2020, as there has been little work on ToDs employing CL in recent years compared to the extensive explorations of CL in computer vision, especially classification task.
Ethical Consideration
In developing and evaluating HESIT, our hypergradient-based exemplar strategy for Intelligent task-oriented dialogue systems (ToDs), we considered a number of significant ethical implications to ensure that our research conforms to generally accepted standards for the development of responsible AI.
Privacy and Data Security: While HESIT requires the selection of exemplars from the training data, we are mindful of user privacy and data security. The data used for training and testing HESIT are either publicly accessible or anonymized and used with the proper permissions, ensuring that no personally identifiable user information is exploited.
Bias and Fairness: Despite the optimistic results demonstrated by HESIT, it is essential to recognise the inherent possibility of bias in any AI system. We have attempted to mitigate this risk by assuring a diverse and representative training dataset, thereby preventing the model from favouring certain task domains disproportionately over others. Future work will include continued evaluations of potential biases with the objective of developing additional bias mitigation strategies.
Transparency and Accountability: The performance of HESIT is primarily dependent on the exemplars chosen from each task domain. This strategy is effective in terms of model performance, but if not managed carefully, it could contribute to a lack of transparency. We intend to maintain detailed documentation regarding the exemplar selection procedure and to make our algorithms, methods, and results accessible and interpretable to both practitioners and users.
Impact on Employment: The development of ToDs with the capacity for continuous learning may raise concerns regarding potential employment displacement. While our research contributes to the advancement of AI, it is not intended to supplant humans but rather to enhance their capabilities and productivity. To ensure a responsible transition to this technology, we encourage ongoing discussions regarding this issue.
Potential Misuse: As with all AI technologies, there is the potential for misuse with our method. If HESIT is used with nefarious intent or if the model is fed inappropriate data, there is a risk of abuse. We strongly advise establishing appropriate precautions and usage policies.
Long-term effects: We recognise that the longterm effects of HESIT and analogous technologies are indeterminate and require constant monitoring. As researchers, we are committed to continuous monitoring of our technology, its applications, and their societal implications.
In conclusion, we believe that the advantages of developing Continual Learning systems such as HESIT must be weighed against these ethical considerations. As a result, we do not perceive this work as a destination, but rather as a stepping stone on the path towards responsible and beneficial AI.

heading: 7 Acknowledgement
text: This work is funded by the Zhejiang Provincial “Jianbing” “Lingyan” Research and Development Program of China (2024C01135), National Natural Science Foundation of China (62302433, U23A20496), Zhejiang Provincial Natural Science Foundation of China (LQ24F020007) and CCFTencent Rhino-Bird Fund (RAGR20230122).

heading: A Large Language Model and ToDs
text: Large Language Models (LLMs) have emerged as an epistemic beacon in the field of natural language processing (NLP), endowing text-based tasks with substantial performance gains including ToDs (Bae et al., 2022). Furthermore, a foundation model with multiple adapters seems to be potential solution for overcoming catastrophic forgetting. However, this combination can not fit continual learning in ToDs, especially with increasing task domains. Firstly, the task domain of inference data is agnostic that requires a extra procedure to determine which of adapter should be activated for inference. Secondly, no matter what procedure is utilized, all of 37 combinations need to be traversed once, resulting in unacceptable delay for ToDs. Therefore, instead of using LLMs with adapter, we utilize single model to adapt to all domains.

heading: B Examples of E2E ToDs
text: We first give an example to elaborate INTENT, DST, NLG, and E2E. As shown in Fig 4, the grey box denotes the API-call C(H) process, where the user intents (INTENT) and slot-value pairs (DST) are both predicted according to dialogue history H. Then the system will generate a response (NLG) based on both H and C(H): H + C(H) → R. In addition, when C(H) is empty, the system can directly generate responses based on history: H→ R.

heading: C Methodology Supplement
text: For each task domain, Let Dt consists of training set Dtrnt , validation set Dvalt , and test set Dtstt . Assume Dtrnt contains N training-points which is defined as zi = (xi, yi). Our goal is to pick out those influential zis that benefit model performance on unseen test-point z′ ∈ Dvalt , as Dtstt is unavailable during training. Leveraging derivation chain rule,
the IFs (Koh and Liang, 2017) define the influence of z on z′ as:
I(z, z′) = −∇θℓ(z′, θ̂)TH−1θ̂ ∇θℓ(z, θ̂) (7)
where Hθ̂ = ∇2θℓ(z, θ̂) is the Hessian and is positive definite (PD) by assumption. In practice, Hθ̂ can not be materialized in memory, let alone be inverted by standard linear algebra.
Mainstream methods employ LISSA (Agarwal et al., 2017), which only samples a single point per iteration as an estimator of inverse Hessian. Compared with the standard transformation of matrix inversion (Martens et al., 2010), LISSA shows significant speedups in the following iteration r:
H−1r v = v + (1−H)H−1r−1v (8)
where H is approximated on random batch and v = ∇θℓ(z′, θ̂) is a gradient. However, this method faces two drawbacks. Firstly, only the final parameters are utilized to calculate the gradient, while the data influence is involved in a dynamic optimization process. Secondly, Hessian might not be positive definite for a large pre-trained model, and the estimation is not accurate enough. In practice, we found that using LISSA to estimate the inverse Hessian of GPT-2 is unstable and time-consuming.
To address the above defects, TracIn (Pruthi et al., 2020) traces the changes in the loss across all gradient steps, and avoids the estimation of inverse Hessian by the following definition:
ITracIn(z, z′) = 1
C
C∑
i=1
γi ∇θiℓ(z, θ̂i) · ∇θiℓ(z′, θ̂i)
(9) where C is the number of the checkpoints during training. Despite considering the optimization process, the computing complexity is C times slower than that of using exact gradient similarity and, as discussed in (Pruthi et al., 2020), care needs to be taken in selecting checkpoints. Another practical obstacle of TracIn in CL settings is that only several epochs are trained for each domain, resulting in insufficient C for influence analysis.

heading: D Dataset Statistics
text: The main dataset statistics are shown in Table 6. It contains 37 domains, 280 types of intents, and more than 31K training dialogues. The detailed example amounts of modules as well as training, validation, and test set are illustrated in Table 7.
Due to a serious imbalance of task domain, we observe that some CL methods (e.g., EWC) are vulnerable to the task order in the curriculum. If the model first learns on a small domain, then the regularization item would influence the subsequent learning on large domains. Meanwhile, we found the data replay methods are more insensitive to task order, as the data amount of each domain stored for retraining is equal. In addition, we have also attempted to sample different amounts of exemplars according to the size of the domain. However, it achieves similar results compared with sampling an equal amount of exemplars for each domain.

heading: E Experimental Details
text: E.1 CIFAR-10 Dataset The CIFAR-10 dataset (Krizhevsky et al., 2009) consists of 60000 32x32 color images in 10 classes, which can be split into 50000 training images and 10000 test images. There are 10 different classes in the CIFAR-10 dataset, including cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Each class contains 6000 images.
E.2 CNN backbone We employ a simple convolutional neural network, including 2×2D CNN layers with pooling layers, then a ReLU activation function and 2× linear layer are added to predict the labels.
In order to obtain Fig. 3, the CNN network is trained by an SGD optimizer with a momentum of 0.9. The learning rate is 1 × e−3 and 50 training epochs are repeated. Then we calculate the contribution of training data to test data regarding different categories.

heading: Future Work
text: We would leave it as future work. In addition, we can only compare our model to the state-of-the-art work on ToDs from 2020, as there has been little work on ToDs employing CL in recent years compared to the extensive explorations of CL in computer vision, especially classification task. Ethical Consideration
In developing and evaluating HESIT, our hypergradient-based exemplar strategy for Intelligent task-oriented dialogue systems (ToDs), we considered a number of significant ethical implications to ensure that our research conforms to generally accepted standards for the development of responsible AI. Privacy and Data Security: While HESIT requires the selection of exemplars from the training data, we are mindful of user privacy and data security. The data used for training and testing HESIT are either publicly accessible or anonymized and used with the proper permissions, ensuring that no personally identifiable user information is exploited. Bias and Fairness: Despite the optimistic results demonstrated by HESIT, it is essential to recognise the inherent possibility of bias in any AI system. We have attempted to mitigate this risk by assuring a diverse and representative training dataset, thereby preventing the model from favouring certain task domains disproportionately over others. Future work will include continued evaluations of potential biases with the objective of developing additional bias mitigation strategies. Transparency and Accountability: The performance of HESIT is primarily dependent on the exemplars chosen from each task domain. This strategy is effective in terms of model performance, but if not managed carefully, it could contribute to a lack of transparency. We intend to maintain detailed documentation regarding the exemplar selection procedure and to make our algorithms, methods, and results accessible and interpretable to both practitioners and users. Impact on Employment: The development of ToDs with the capacity for continuous learning may raise concerns regarding potential employment displacement. While our research contributes to the advancement of AI, it is not intended to supplant humans but rather to enhance their capabilities and productivity