SlowGuess/NAACL_2022_OCR huggingface dataset:
Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Xing Fan, Chenlei Guo, Yang Liu
Amazon Alexa AI Seattle, Washington, USA
zgchen,eunahch,jieha,derekliu
fanxing,guochenl,yangliud@amazon.com
Seq2seq language generation models that are trained offline with multiple domains in a sequential fashion often suffer from catastrophic forgetting. Lifelong learning has been proposed to handle this problem. However, existing work such as experience replay or elastic weighted consolidation requires incremental memory space. In this work, we propose an innovative framework, RMR_DSE that leverages a recall optimization mechanism to selectively memorize important parameters of previous tasks via regularization, and uses a domain drift estimation algorithm to compensate for the drift between different domains in the embedding space. These designs enable the model to be trained on the current task while keeping the memory of previous tasks, and avoid much additional data storage. Furthermore, RMR_DSE can be combined with existing lifelong learning approaches. Our experiments on two seq2seq language generation tasks, paraphrase and dialog response generation, show that RMR_DSE outperforms state-of-the-art models by a considerable margin and greatly reduces forgetting.
Seq2seq language generation is the essential framework for many tasks such as machine translation, summarization, paraphrase, question answering, dialog response generation. In these applications, models are typically trained offline using annotated data from a fixed set of domains. However, in real-world applications, it is desirable for the system to expand its knowledge to new domains and functionalities, that is, it has the capability of human-like lifelong learning (LLL) (Ring et al., 1994; Chaudhry et al., 2019) of acquiring new utterance patterns without forgetting what it has already learned. Neural networks struggle to learn continuously and experience catastrophic forgetting (CF) when optimized on a sequence of learning problems (McCloskey and Cohen, 1989; French, 1999).
Some past work in LLL has demonstrated that discriminative models can be incrementally learnt for a sequence of tasks (Kirkpatrick et al., 2017; Chen et al., 2020; Wang et al., 2019). In contrast, under generative settings there has been limited research. Recent work in this area includes (Mi et al., 2020; Madotto et al., 2020; He et al., 2021; Shin et al., 2017).
Existing work in LLL adopts the replay based methods (Pellegrini et al., 2019), such as Latent Replay, or regularization based methods (Huszár, 2018; Li and Hoiem, 2018), such as Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), Synaptic Intelligence (Zenke et al., 2017). Although they can reduce CF, they have some limitations. The replay-based methods require storing samples from previous tasks, and regularization methods often view all the model parameters as equally important and regularize them to the same extent. In addition, those approaches do not explicitly address the data distribution shift that causes the CF problem. The semantic gap between the embedding spaces of two domains is a leading reason of CF (Wang et al., 2021b). As illustrated in Figure 1, each data point and their cluster centers trained in Task 1 are shifted after training for Task 2. Yu et al. (2020) proposed to compensate this gap without using any exemplars via domain shift. However, that study focused on classification tasks.
In this work, we propose a novel method, regularized memory recall mechanism with additional domain shift estimation (RMR_DSE), to alleviate CF in continuous seq2seq language generation. The first RMR component improves the regularization-based method through adaptive regularization. We convert fisher information matrix deployed in EWC to a tunable hyperparameter constrained by a vocabulary-related hyperparameter. Further, we add a regularizer derived from the gradients of the generative function to tune model parameters. The second DSE component compen
Human Language Technologies, pages 5441 - 5454
Figure 1: Illustration of Domain Shift: (a) Data with three relevant topic/cluster in the embedding space after model trained on task 1. (b) Data with previous topics in the embedding space after the model trained on task 2, the arrow indicates the domain shift between two tasks, which is what our DSE aims to estimate.
sates the representation difference from the two domains by estimating the semantic gap between them. We obtain embeddings for the current task's data using the previous and current models, and group embeddings from previous models into clusters. Semantic shifts are computed for each cluster, and then used during inference time on previous test data to adjust its semantic representation to match the current model.
- We design a new regularized algorithm to consider parameters for the previous tasks while training for the current task for LLL seq2seq generation.
- We propose to estimate domain shifts in the embedding space of consecutive models via prototypical representations, thus alleviating the need for data storage.
- Our experiments on seq2seq generation benchmark datasets show that our model achieves state-of-the-art results in current task learning and reduces forgetting rates for previous tasks.
2.1 Life Long Learning (LLL)
Life long learning has been studied from a few perspectives, including data buffering, regularization and prototype keeping. Replay based methods can be used in data buffering or prototype keeping. They usually keep a small amount of real samples from old tasks or distill the knowledge from old data and recreate pseduo-data of old tasks for later training. Using these sampled data or pseudo data can prevent weights from deviating from previous status (Rolnick et al., 2019; Wang et al.,
2020; Lopez-Paz and Ranzato, 2017). The main idea of this approach is to assign a dedicated capacity inside a model for each task. After a task is completed, the weights are frozen as one prototype (Wang et al., 2021b; d'Autume et al., 2019; Wang et al., 2021a). Both data buffering and prototype keeping need storage of either data samples or model weights, i.e., they require extra memory to memorize important information of previous tasks. Another LLL method is regularization based, which adds a regularization term to weights when learning them for a new task in order to minimize deviation from previously trained weights. Most regularization based methods estimate the importance of each parameter and add them as a constraint to the loss function. Different algorithms have been designed to achieve this. For example, elastic weight consolidation (EWC) calculates a Fisher information matrix to estimate the sensitivity of parameters (Kirkpatrick et al., 2017); memory aware synapses (MAS) (Aljundi et al., 2018) uses the gradients of the model outputs; and episodic memory or gradient episodic memory (GEM) (Li et al., 2017; Lopez-Paz and Ranzato, 2017) allows positive backward transfer and prevents the loss on past tasks from increasing. These methods all attempt to slow down the learning of parameters that are important for previous tasks.
2.2 LLL in Seq2seq Language Generation
In Seq2seq language generation, not much work has been done in LLL. The most relevant work is from (Mi et al., 2020) where a framework of sequential learning is designed for task-oriented dialogs. Specifically, they replay prioritized exemplars together with an adaptive regularization technique based on EWC. They store representative utterances from previous data (exemplars), and replay them to the Seq2seq language generation model each time it needs to be trained on new data. They achieved good results on the MultiWOZ-2.0 dataset. However, their work needs to store data from previous tasks, and thus may not be scalable to large data environments. In addition, their system is specifically designed for the MultiWOZ task and lacks generalization to other tasks. In contrast, our proposed RMR_DSE method aims to fit different seq2seq language generation applications, therefore it is easy to be integrated to tasks such as summarization, translation, paraphrases, dialog response generation.
Figure 2: Overview of RMR_DSE for LLL Seq2seq Language Generation. Figure best viewed in color.
In this section, we introduce our proposed framework RMR_DSE, as illustrated in Figure 2. In the LLL scenario, models are trained for a sequence of domains (or tasks). For the first task, the model can be trained from scratch or using pretrained models. Starting from the second model, parameters are initialized with the previous model. Our RMR_DSE method is a combination of regularization and domain shift estimation (DSE). For the first part, we incorporate the mechanism of EWC to obtain a regularized memory recall mechanism (RMR) to optimize model training. For the second DSE part, we design an algorithm to integrate both K-means and mean shift. A shift of embedding representations is estimated using the previous and current models, and it is compensated to reduce forgetting when we evaluate the current model on test data from the previous domain. Although RMR_DSE is a generic mechanism, we evaluate on two seq2seq language generation tasks in this work. The underlying models in seq2seq language generation can be any models, including transformers, LSTM or variational auto-encoders. The following describes the RMR and DSE components in details.
3.1 Adaptive Regularization of Memory Recall
Although elastic weights consolidation (EWC) is generic enough to fit all tasks, it regularizes all the parameters to the same extent. In order to differentiate parameter importance, we propose an improved EWC, the regularized memory recall (RMR) mech
anism, where the training objective is:
$$
L o s s _ {t} = \lambda (\tau) L _ {t} (\theta) + (1 - \lambda (\tau)) \gamma F \sum_ {i j} \Pi_ {i j} \left(\theta_ {i j} - \theta_ {i j} ^ {*}\right) ^ {2} \tag {1}
$$
where  $L_{t}(\theta)$  is the loss for the current task. In our generation task, we use standard label smooth cross entropy.
In the regularization part,  $\theta^{*}$  represents the parameters from earlier models, e.g., that learned from task  $t - 1$ :
$$
\theta^ {*} = \arg \min  _ {\theta} \left\{- \log p \left(\theta \mid D _ {t - 1}\right) \right\} \tag {2}
$$
$\theta$  is for the current model, and indexes  $i$  and  $j$  are used to represent connections between pairs of neurons  $n_i$  and  $n_j$  in two consecutive layers. Adding the regularization term using the differences between the two models is expected to memorize important old parameters while updating values of current parameters.
In EWC,  $F$  is the diagonal element of the Fisher Information Matrix. It measures the importance of  $\theta$  after being updated with the set of data points in the current task to previous tasks. However, in our work, we suppose that we do not have access to data of previous tasks. Therefore, we cannot compute fine-grained values based on data, and thus convert  $F$  to a tunable hyperparameter without dependency on data from the previous task. It is used to penalize the quadratic function  $(\theta_{ij} - \theta_{ij}^{*})^{2}$ .
In order to make the parameters in a reasonable range, we add a hyperparameter  $\gamma$  to help tune  $F$ . For simplicity, the value of  $\gamma$  is determined by the ratio of the vocabulary size of the current corpus and the previous ones.
$$
\gamma = \gamma_ {\text {b a s e}} \sqrt {V _ {1 : t - 1} / V _ {t}} \tag {3}
$$
Furthermore, to control the learning between the loss for the current task and the regularization term, we add  $\lambda (\tau)$ , a sigmoid annealing function:
$$
\lambda (\tau) = \frac {1}{1 + e x p (- k) * (\tau - \tau_ {0})} \tag {4}
$$
where  $k$  and  $\tau_0$  are hyperparameters controlling the annealing rate and timesteps, and  $\tau$  refers to the update timesteps during fine-tuning.
Finally similar to MAS (Aljundi et al., 2018), to model the varying importance of individual parameters and the changes of model parameters, we integrate  $\Pi$  for fine-grained regularization:
$$
\Pi_ {i j} = \frac {1}{N} \sum_ {n = 1} ^ {N} \left\| g _ {i j} \left(x _ {n}\right) \right\| ^ {2} \tag {5}
$$
where  $g_{ij}(x_n) = \frac{\partial(G(x_n;\theta))}{\partial\theta_{ij}}$  is the gradient of the learned generative function (such as transformers, LSTM or VAE) with respect to parameter  $\theta_{ij}$  evaluated at the data point  $x_n$  of the current task. Parameters with small importance weights can be changed to minimize the loss for subsequent tasks while parameters with large weights are kept unchanged.
3.2 Deploying Domain Shift Estimation to Make Up Semantic Drift
It has been shown in previous studies (Yu et al., 2020; Wang et al., 2021b) that catastrophic forgetting is mainly due to the domain shift in the embedding space after the model is updated on new domains. When testing on previous domains, the embeddings derived from the model trained using new domains are suboptimal and lead to performance degradation. Hence, in addition to regularizing the model during training as described in the section above (RMR regularization), we propose to deploy domain shift estimation (DSE) to compensate the embedding drift to further reduce the forgetting.
Note that in our life long learning scenario, we do not rely on the data previously used for model training in new domains, but do have access to the trained model from the previous task. We thus approximate DSE with gaps between embedding representations of the current data based on both  $\text{model}_{t-1}$  and  $\text{model}_t$ , using the following steps.
First, for a data point  $i$  in the current training task  $t$ , its representation shift is:
$$
\delta_ {i} ^ {t - 1 - > t} = \mathbf {z} _ {i} ^ {t} - \mathbf {z} _ {i} ^ {t - 1} \tag {6}
$$
where  $\mathbf{z}_i^t$  and  $\mathbf{z}_i^{t - 1}$  refer to the embedding of point  $i$  based on model  $t$  and model  $t - 1$  respectively. In
the seq2seq language tasks, these are the encoder outputs.
Second, we deploy unsupervised clustering methods to identify some centers and mean shift (Anand et al., 2013) for the embeddings using model  $t - 1$ . Specifically, with K-means, we find  $K$  embedding centers, each of them represented as  $\mu_k^{t - 1}$ . Then, around each center, we find some number of samples to compute mean shift (we use  $3k$  in our experiments).
The mean shift  $\mathbf{M}_h(x)$  for each data point of each cluster is defined as:
$$
\mathbf {M} _ {h} (x) = \frac {\sum_ {i} ^ {n} G \left(\frac {x _ {i} - x}{h _ {i}}\right) w \left(x _ {i}\right) \left(x _ {i} - x\right)}{\sum_ {i} ^ {n} G \left(\frac {x _ {i} - x}{h _ {i}}\right) w \left(x _ {i}\right)} \tag {7}
$$
where  $G = e^{\frac{||x_i - x||}{2h^2}}$  is the Gaussian kernel,  $h$  is the bandwidth,  $x_{i}$  is the data belonging to the cluster containing  $x$ , and  $n$  is the number of data points in each cluster.
Finally for each cluster, we compute a domain shift vector as:  $\Delta_{dse_k}^{t - 1 - > t}$
$$
\Delta_ {d s e _ {k}} ^ {t - 1 - > t} = \frac {\sum_ {i} \mathbf {M} _ {h} \left(x _ {i}\right)) \delta_ {i} ^ {t - 1 - > t}}{\sum_ {i} \mathbf {M} _ {h} \left(x _ {i}\right)} \tag {8}
$$
where the summarization is performed over all the data points belonging to cluster  $k$ .
We use  $K$  such vectors as a domain shift estimate between the models trained for two different domains. When evaluating on a previous domain using the model trained for a new domain, for a test data point we first calculate the similarity between its embedding  $(e_{d_{t - 1}}^t)$  encoded by model and the stored cluster centers, and then the corresponding domain shift vector for that cluster is subtracted from  $e_{d_{t - 1}}^t$  before the generation decoding step. If we have multiple tasks ( $m$  for example), we can perform a series of subtractions, i.e.,
$$
\begin{array}{l} e _ {d _ {t - m}} ^ {t} = e _ {d _ {t - m}} ^ {t} - \Delta_ {d s e _ {k}} ^ {t - m - > t - m + 1} \\ - \Delta_ {d s e _ {k}} ^ {t - m + 1 - > t - m + 2} - \dots - \Delta_ {d s e _ {k}} ^ {t - 1 - > t} \\ \end{array}
$$
Algorithm 1 describes the domain shift estimation of RMR_DSE. We have input of embeddings encoded by  $model_{t-1}$  and  $model_t$ . Firstly, KMeans is employed to obtain  $K$  embedding centers for the training data (we obtained best results when  $K = 3$  in our experiments). Also, since KMeans is sensitive to the initialization of center points, we have to run multiple rounds (about 5 rounds in our experiments) before we can obtain the best ones. FAISS, the fast KNN-based embedding search tool is utilized to search relevant
Table 1: Dataset stats for paraphrase generation task (number of sentence paraphrase pairs).
Quora	Twitter	Wiki_Data	total
train	111,947	85,970	78,392	276,309
valid	8,000	1,000	8,154	17,154
test	37,316	3,000	9,324	49,640
samples for each embedding center. In the while loop, we implement mean shift method to estimate whether each sample needs a mean shift against their closest embedding center. We finally obtain a list of mean shift points and their values. Those values are used as weights to be multiplied to the difference between embeddings from  $model_{t-1}$  and  $model_t$ . In the inference stage, we select  $\Delta_{dse}^{t-1-t}$  based on the closeness of input test sentences to the embeddings.
4 Experiments on Paraphrase Generation
To test RMR_DSE's generalization, we apply it to two datasets that both follow the seq2seq generation setup but are quite different tasks. In this section, we focus on the paraphrase generation task.
Figure 3: Results of Meteor scores of QTW setting when evaluating on the current task.
For paraphrase generation, we use three existing paraphrase datasets, Quora, Twitter and Wiki_data, in a sequential fashion, that is, the model is first trained on the Quora data, then Twitter, then Wiki_data. We name this experimental setting as QTW. Statistics of the data are provided in Table 1.
We use a current SOTA generation model, BART, as the seq2seq backbone in our LLL framework and the other compared methods. We compare our approach with the following baselines.
- Finetune: for each task, each model is initialized with the model obtained until the last task, and then fine-tuned with the data of the current task.
- Full: we train a model with all the three data sets together.
- EWC: the model is trained with the base EWC model on the data from the current task with the initialization of the previous model.
For our proposed RMR_DSE, we also evaluate different configurations including MR, RMR, and DSE only for an ablation study. The details of parameters implementations are given in Appendix.
For evaluation metrics, we use BLEU-4, ROUGE-L and METEOR for the generation task. Because of space limit, we only report bar figures with METEOR scores and leave tables with full scores in the Appendix. To measure the forgetting rates of different methods, we apply models trained using new data to past data.
Figure 4: Results of Meteor scores of QTW setting while evaluating on previous datasets.
Evaluating on the Current Task
For QTW setting, Figure 3 shows results when models are evaluated on the data corresponding to the current task. Note that since DSE is only applicable when models are evaluated on the past data, we do not use DSE in this experiment. From left to right are domains for Quora, Twitter and Wik_data respectively.
Each of the domains has 8 results. The first three bars are results from independent models, that is, the BART models are trained on only one of the datasets in QTW. As expected, models trained on the matched domain achieve higher performance than otherwise. And there is a large performance drop when using models trained from mismatched domains. This is mostly because of the different writing styles of the three datasets. Wik_data is the most formal one, and Twitter is the most informal one.
In the fourth bar, the BART model is trained in finetune mode, i.e., in QTW order, the model is initialized with that trained in the previous domain and fine tuned using the subsequent domain. We can see that results on both Twitter and Wik_data test data are slightly lower than those when models are trained directly on the corresponding training data. Again, this suggests pretraining the model with mismatched data is not beneficial. The results from the EWC baseline are not consistently better than the finetune method, showing the limited effectiveness of EWC regularization. In contrast, our proposed approaches obtain better results than Finetune. Even for the first task, Quora, we observe around  $1\%$  better results. This demonstrates that
even for pretrained models, regularization shows positive effects. For the later tasks, there is about  $3 - 4\%$  performance increase on Twitter data and  $7 - 9\%$  for Wik_data. This shows the effectiveness of both MR and RMR. In addition, six out of nine results from RMR win about  $1\%$  over MR. This shows that further regularization with quadratic penalty has positive impact on selection of important parameters. The last bar is the results of Full. Since the model has seen all the data, it is not surprising that results for both Twitter and Wik_data are better than our models, and it may be partly because of some similarity in Quora and Wik_data.
Evaluating on Previous Tasks
Figure 4 shows the results when models trained on new domains are evaluated on data from past domains. Since we only report results of QTW setting in the main page, they are presented for evaluating on Quora and Twitter data. For the Quora test set, we show results after training with Twitter data, and then subsequently Wik_data. The first bar of each domain is the result of the BART model trained on only the corresponding data. The second bar uses the baseline fine tuning fashion. We show results using our proposed method, RMR_DSE, and its individual components, MR, RMR, and DSE. Each of them yields much better results than the fine-tune or EWC baselines, with much less drop rates. On all the datasets, we can see the incremental improvement from MR to RMR and to RMR_DSE. This shows each module can reduce forget rates. In addition, after the model is trained on Wik_data, forgetting rates for Quora Test (the first dataset) are even lower than the model trained on Twitter. This again indicates Wik_data and Quora are more similar in style than Twitter.
In Table 2, we show some generated samples from QTW setting using the baseline BART model and our RMR_DSE model. All examples are results generated by  $\text{model}_t$  on  $\text{data}_{t-1}$ . Among the five examples, the first one is from Quora, the last one from Wiki data and the other three from Twitter. The reason that we select more samples from Twitter is that we find Twitter is the most informal in style with quite many fragments. Hence, it is the hardest for the generation task and has lowest generation performance and forgetting reduction rates. In the four samples, the italicised parts are the key words. From the table, we can observe that com
pared to BART, RMR_DSE has better performances on all of the three datasets. The BART model misses all of them except drilling. In contrast RMR_DSE succeeds in all cases without forgetting the previously learned patterns.
5 Experiments on Dialog Response Generation
5.1 Task Definition
In task oriented dialogs, recent neural generation methods use seq2seq setup for response generation given the dialog act of the target response. A dialog act is defined as the combination of intent  $I$  and a set of slot-value pairs  $S(d) = (s_i, v_i)_{i=1}^p$ , where  $p$  is the number of slot-value pairs. Intent  $I$  refers to the utterance functionality, while slot-value pairs contain messages to express. For example, given input "Recommend (Addr=regent stree, Fee=free, Name=Downing College)", the system is expected to generate the response, "[Downing College] is my favorite. It is located on [regent street] and it's [Free] to get in" and the slot type "[Slot-Hotel-Area)". Slot values are composed of domain and relevant attributes (details are in (Eric et al., 2019)).