Does RoBERTa Perform Better than BERT in Continual Learning: An Attention Sink Perspective

Authors: xueying  bai
yifan  sun
niranjan  balasubramanian
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 20:25:33

Abstract: Continual learning (CL) aims to train models that can sequentially learn new tasks without forgetting previous tasks' knowledge. Although previous works observed that pre-training can benefit CL, it remains unclear whether a pre-trained model with higher downstream capacity also performs better in CL. In this paper, we observe that pre-trained models may allocate high attention scores to some 'sink' tokens, such as [SEP] tokens, which are ubiquitous across various tasks. Such attention sinks may lead to models' over-smoothing in single-task learning and interference in sequential tasks’ learning, which may compromise the models' CL performance despite their high pre-trained capabilities. To reduce these effects, we propose a pre-scaling mechanism that encourages attention diversity across all tokens. Specifically, it first scales the task's attention to the non-sink tokens in a probing stage, and then fine-tunes the model with scaling. Experiments show that pre-scaling yields substantial improvements in CL without experience replay, or progressively storing parameters from previous tasks.

Main paper: 
 introduction: 
 Machine learning applications in the real world often need to face continuous streams of data from different tasks or distributions (Lopez-Paz & Ranzato, 2017; Hou et al., 2019) . For such cases, it is important to develop continual learning (CL) models that can progressively learn new tasks without performance degradation in previous tasks (i.e., catastrophic forgetting).The pre-training and fine-tuning paradigm (Devlin et al., 2018) , which effectively learns downstream tasks by fine-tuning a pre-trained language model (LM), is widely used for general NLP tasks. Previous works (Wu et al., 2022; Mehta et al., 2023) observed that pretraining can also benefit CL. However, it remains unclear whether a pre-trained model that has better single-task performance also performs better in CL settings. For example, BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019b) are two pre-trained LMs with the same model structure. RoBERTa achieves generally better downstream performance than BERT, in part because it is pre-trained on more diverse data. In CL, however, RoBERTa does not always outperform BERT (Wu et al., 2022) . This motivates our study on the factors that may cause models' inferior performance in CL, besides their pre-trained capacity.In this paper, we show that the attention sink phenomenon can influence models' CL capacity. Attention sinks have been observed on autoregressive LLMs, where models tend to allocate high attention scores to specific tokens in the input ('sink tokens') regardless of their semantic significance (Xiao et al., 2024) . In Fig. 1 , we show that attention sinks appear after the first layers in both BERT and RoBERTa models. And the sink tokens are usually common tokens (e.g., special tokens like [SEP]), which are not semantically significant but are present in most NLP tasks (Fig. 3 right). However, unlike the previous work by Xiao et al. (2024) which focuses on the magnitude of attention scores on sink tokens, our work focuses on the attention deviations on sink tokens ('sink attention deviations') and their influence on CL. In particular, we connect the attention deviations on sink tokens to the over-smoothing phenomenon. Over-smoothing has been observed in pre-trained LMs, where models output nearly identical representations for all input tokens (Dong et al., 2021; Shi et al., 2022) . We show that over-smoothing is related to small sink attention deviations in pre-trained models.It can cause distortions of pre-trained features (Kumar et al., 2022) , which may make models less generalizable to out-of-distribution data (e.g., data from other tasks) and affect CL.Models' small attention deviations on common sink tokens can also cause unnecessary interference across tasks in CL. Specifically, representations of common sink tokens may carry the information of one task, which then influences another task's learning. This can be harmful if the new task is irrelevant to the previous one. We conduct a case study to show that such interference is hard to avoid when models have small sink attention deviations.To address the above problems, we propose a pre-scaling mechanism that encourages diverse attention scores on sink tokens. It introduces a scaling layer that is first learned to allocate diverse attention to tokens in a probing stage, and then tuned in a fine-tuning stage together with the pre-trained encoder. Experiments show that pre-scaling improves models' CL performance with reduced over-smoothing. Moreover, RoBERTa models consistently outperform BERT models after pre-scaling, which suggests that pre-scaling helps to better utilize models' pre-trained capacity in CL.In conclusion, we make the following contributions: (1) We characterize the attention sink phenomenon in pre-trained LMs and build a connection to over-smoothing. (2) We conduct a case study to show that the above attention sinks may propagate unexpected interference in CL.(3) We propose a pre-scaling mechanism that can significantly improve pre-trained LMs capacity in CL without experience replay or progressively storing parameters. 
related work: 
 Continual Learning. Models for CL can be divided into three main categories: (1) regularization-based models which constrain the deviation of new parameters from the older ones (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Lee et al., 2017) ; (2) replay-based models which reduce catastrophic forgetting by rehearsing on real or pseudo samples from previous tasks (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a) or generative models (Shin et al., 2017; Kemker & Kanan, 2017) ; (3) architecture-based models which learn evolving architectures for sequential tasks, with their capacities for each task carefully assigned (Rusu et al., 2016; Yoon et al., 2017) .CL in NLP is an emerging area (Liu et al., 2019a; Biesialska et al., 2020) . MBPA++ (d'Autume et al., 2019) uses experience replay and local adaptation to mitigate forgetting; LAMOL (Sun et al., 2019) generates pseudo samples for replay; IDBR (Huang et al., 2021a) disentangles task-agnostic and task-specific information; CTR (Ke et al., 2021) uses a capsule network for knowledge transfer. All the above models are based on pre-trained LM (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) . Recent works show that pre-training can alleviate catastrophic forgetting (Wu et al., 2022; Mehta et al., 2023; Lee et al., 2023) . Mehta et al. (2023) claims that the benefit may come from having less sharp minima. In this paper, we tackle the CL problem from an attention sink perspective, which provides an explanation of why sometimes RoBERTa underperform BERT in CL tasks (Wu et al., 2022) . To the best of our knowledge, we are the first to tackle CL problems from this angle.Over-Smoothing. In this paper, we connect attention sinks to an over-smoothing phenomenon, which is first proposed in graph neural networks (Oono & Suzuki, 2020; Huang et al., 2020; Cai & Wang, 2020; Rusch et al., 2023; Yang et al., 2020) . Over-smoothing refers to the problem that the models' performance deteriorates as representations of all the nodes in the graph become similar (Li et al., 2018; Xu et al., 2018) . For transformer-based models, Dong et al. (2021) claims that pure attention loses rank doubly exponentially with model depth. And Shi et al. (2022) characterize the oversmoothing problem in transformers by viewing the attention matrix as a form of adjacency matrix in the graph. In this paper, we connect the over-smoothing problem to attention sinks, to show that in some cases attention sinks will influence model's task learning and cause inferior performance in CL. 
attention sinks in language models: 
 We first empirically show that certain attention sinks exist in pre-trained LM layers. Then we study their impact by connecting them to an over-smoothing phenomenon, which may influence models' single-task overfitting and in turn can influence their CL abilities. The attention sinks can also cause cross-task interference in CL, which we discuss in section 4.We characterize the presence of attention sinks (Xiao et al., 2024) in LMs using data from NLI datasets SST and MNLI (Socher et al., 2013; Williams et al., 2018a; Wang et al., 2019) .Attention on sink tokens. Fig. 1 illustrates the presence of attention sinks using attention maps, which show high attention scores are allocated to specific input tokens (i.e., sink tokens). The figure also shows sink tokens might receive similar (high) attention scores from all tokens. To empirically quantify these observations, we devise the measurements below.Let A ∈ R n×n denote an attention matrix over n tokens for a single attention head. An element a ij ∈ A denotes the attention on the j-th key token for the i-th query token. For the i-th (query) token, we have the following measurements :Average outer degree:d i = ∑ n k=1 a ki n,Attention deviation:EQUATIONWe average d i and ∆ i across all attention-heads in each layer. d i is the averaged attention score allocated to the i-th token, and ∆ i is the per-degree attention score deviation to the i-th token's average outer degree. We study sink tokens with the largest average outer degrees, and calculate their attention deviations as sink attention deviations.Fig. 2 shows the layer-wise average outer degrees and sink attention deviations for BERT and RoBERTa models. In Fig. 2 (a), tokens with top-3 largest outer degrees obtain 60% attention scores from input tokens, while the top-1 tokens obtain over 20% attention. This shows that a small number of tokens obtain major attention in self-attention layers. In Fig. 2 (b), we observe that the sink attention deviations are mostly small, except for the first two layers. This shows that all tokens pay similar attention to the sink tokens in the top layers.Sink tokens are usually common tokens. We also find that sink tokens in pre-trained LMs turn out to be common tokens, ones that appear in many language tasks. These common tokens include special tokens such as ('[SEP]'), punctuation ('.'), or the initial tokens in The right shows the ratio of sink tokens that are common tokens across tasks. Here we consider special tokens, the punctuation '.' and the second token in the input as the common tokens. 'PT' stands for pre-training, and 'FT' stands for fine-tuning on 3k MNLI data.inputs, which are not semantically significant. The right side of Fig. 3 shows the ratio of sink tokens with the top-1 largest outer degrees that are also common tokens at each layer. Almost all sink tokens are common tokens in the first several layers of the pre-trained model.Even after fine-tuning, the models still have high attention on them in the middle layers.We show the impact of attention sinks on single tasks by connecting them to an oversmoothing phenomenon, where the attention deviations on sink tokens are a key factor.Over-Smoothing in transformers. Previous works have identified an over-smoothing phenomenon in transformer-based models: token representations become identical after several self-attention layers. Over-smoothing is closely related to models' task learning ability and their overfitting on a task (Shi et al., 2022) . There are several factors that can lead to over-smoothing, and we focus on the effect of self-attention matrices here.For a self-attention layer with an attention matrix A ∈ R n×n , let H ∈ R n×d be its input token representations and AH its output. The over-smoothing problem is described as:d M (AH) ≤ λ max d M (H).The distance d M (H) := ||(Iee T )H|| F measures the closeness between each token representation in H and the averaged token representation ee T H, where e = n -1 2 [1, 1, ..., 1] ∈ R n . λ max is the largest eigenvalue of A T (Iee T )A. When λ max is small, representations after the attention layer will be closer to the averaged representations, which causes over-smoothing.Connection to attention sinks. Over-smoothing has been identified in many transformerbased models. We analyze the eigenvalue λ max to see the property of the attention matrix A under the over-smoothing circumstances (i.e., λ max is small). With each attention score a ij and average outer degree d i defined in Section 3.1, λ max is lower bounded as:λ max ≥ max i ∑ n k=1 a ki -d i 2 .(2)The details are in Appendix A. When λ max is small, the RHS of Eq. ( 2) has to be small. In particular, the i-th token which has the largest outer degree must have its deviation ∑ n k=1 (a kid i ) 2 to be small. When d i is large (as shown in Fig. 2 ), to make the deviation small each attention a ki needs to be close to d i . Therefore, all tokens have similar (high) attention on the token with the largest outer degree, which is an attention sink phenomenon.We empirically show the connection between attention deviations ∆ and over-smoothing in Fig. 3 . The over-smoothing degree is reflected by the average cosine similarity between token representations (Shi et al., 2022) . Going from lower to higher layers, we observe that the attention deviation decreases while the representation similarity increases. This validates the connection between attention sinks and the over-smoothing phenomenon.Impact of over-smoothing with attention sinks. When over-smoothing occurs with attention sinks above, the sink token representations may dominate the averaged token representation, and make other token representations (including [CLS]) close to them. To learn a task, models may push sink token representations close to the task data representation (Fig. 4(a) ). Since sink tokens may be irrelevant to tasks (Fig. 3 right), this may distort pre-trained features and make models less generalizable to OOD data (Kumar et al., 2022) .Comparing BERT and RoBERTa, we observe that pre-trained RoBERTa suffers more from over-smoothing (i.e., high representation similarity), corresponding with low attention deviations on sink tokens at the second and last several layers (Fig. 2(b) ). Therefore, we hypothesize that RoBERTa may be more vulnerable to feature distortion in task learning, which is reflected by its distorted attention patterns (Fig. 1 and Fig. 3 right) after fine-tuning. This may also influence RoBERTa's CL capacity. 
attention sink and interference in continual learning: 
 In this section, we first conduct a case study to show that attention sinks above can cause unnecessary interference when learning across tasks in CL. Then we discuss a transfer vs. interference trade-off induced by attention sinks, which inspires our method in Section 5.We study the following CL problem: A model continually learns a sequence of tasks, with no previous tasks' data accessible when learning new tasks, and no storage of previous tasks' model parameters. The model has different predictors for different tasks, while the encoder is shared. Each task i consists of data D i = (X i , y i ) where X i is the input feature for task i, and y i ∈ R is its target output.When learning task j after task i, one way to quantify the cross-task interference on the shared parameter θ is through the dot product between its (vectorized) gradients on the two tasks' losses (Riemer et al., 2019) :EQUATION)where • is the dot product operator on the flattened gradients. The interpretation is that interference that leads to forgetting happens when I(θ; i, j) < 0, while the positive knowledge transfer may happen if I(θ; i, j) > 0. For tasks that do not have knowledge transfer, the interference is expected to be 0.We use a case study to show that attention sinks can propagate unexpected interference between irrelevant tasks. The study is based on the attention sink phenomenon characterized in Section 3, which showed: (1). models allocate high attention to sink tokens with small deviations; (2). sink tokens are usually common tokens shared across different tasks.Data. We consider two irrelevant NLP tasks in a CL setting. For task i, we have data instance (X i , y i ) where X i consists of embeddings of input tokens. We make the following assumptions about tasks and data:1. There is no knowledge transfer between the tasks and there should be no interference (positive or destructive) when learning one task after the other. 2. Assume there are k common tokens (e.g., special tokens) in two tasks' data instances.For all other tokens in a task, we assume they are irrelevant to non-common tokens in the other task, with corresponding embeddings being orthogonal.Model. We use a model consisting of two single-head self-attention layers. For each task i, the input X i ∈ R n i ×d consists of d-dimensional embeddings for n i tokens. Considering a regression problem (generalizable to classification), the prediction ŷi is calculated as:ŷi = b T i (A i X i W)v i ,where A i ∈ R n i ×n i is the attention matrix in the first attention layer, b i ∈ R n i is the attention vector in the second attention layer that integrates all hidden representations for the target task prediction. Both A i and b i are obtained under the self-attention mechanism (Vaswani et al., 2017) . W ∈ R d×d is a transformation matrix. v i ∈ R d×1 is the predictor that maps the representation to an output value ŷi for task i. The loss function is:L( ŷi , y i ) = E[ 1 2 ( ŷi -y i ) 2 ].For simplicity, we sort X i and A i to make the k common tokens have indices {1, ..., k} and others have indices {(k + 1), ..., n i }. We assume the common tokens are sink tokens in A i .Claim. The interference on the transformation matrix W between task 1 and task 2 mainly depends on the outer degrees of sink tokens and the sink attention deviations.We calculate interference on the shared parameter W based on Eq. ( 3) and the model above:EQUATIONwhereB T 1 = b T 1 A 1 X 1 and B T 2 = b T 2 A 2 X 2 .When both training losses are non-zero (which is the usual case in CL), interference in Eq. ( 4) depends on the correlation v T 1 v 2 between predictors and the correlation B T 1 B 2 between representations. Since the learned predictors may not be good enough to reflect the orthogonality between task 1 and task 2 (Kumar et al., 2022) , we have to consider the interference caused by the correlation B T 1 B 2 , discussed below.Step 1: decompose B 1 and B 2 . Generally, for any matrix M, we denote M (ij) as the ij-th element of M. And for any vector b, we denote b (i) as the i-th element of b.Denote d 1 as the vector of tokens' average outer degrees in attention matrices A 1 . For the attention A (ij) 1 , we define its deviation to the j-th average outer degree as: ϵ(ij) 1 = A (ij) 1 -d (j) 1 .Since the attention vector b 1 is row-stochastic, we decompose B T 1 as:EQUATIONEQUATIONSimilarly, we have B T 2 = r 2 + s 2 + δ 2 , where r 2 denotes the non-sink representations, s 2 the sink representation and δ 2 the sink representation deviations for task 2.Step 2: calculating B T 1 B 2 . Based on assumption 2 that non-sink token embeddings from two tasks are orthogonal, we have r 1 r 2 T = 0. Moreover, since sink tokens are common tokens that are supposed to be neutral to other tokens, we hypothesize that their embeddings are nearly orthogonal to other token embeddings (Appendix B). This makes (s1 + δ 1 )r 2 T + (s 2 + δ 2 )r 1 T ≈ 0. Then we have B T 1 B 2 = (r 1 + s 1 + δ 1 )(r 2 + s 2 + δ 2 ) T ≈ (s 1 + δ 1 )(s 2 + δ 2 ) T . Therefore, B T1 B 2 depends largely on tasks' sink representations s 1 , s 2 and their representation deviations δ 1 , δ 2 . Specifically, it is dominated by s 1 s T 2 when: (1) each attention deviation ϵ (ij) in Eq. ( 5) is close to 0; or (2) the attention score b (i) is close to 0 when the absolute deviation |ϵ (ij) | is large, which makes b (i) ϵ (ij) close to 0. When the sink tokens' outer degrees are large, the correlation s 1 s 2 T can cause high interference between the orthogonal tasks 1 and 2.Since attention sinks on common tokens can propagate unnecessary interference, should we exclude sink tokens that are common tokens ('common sink tokens') when calculating attention in CL? To answer this question, we first train models with and without attention on common sink tokens, and then compare their in-task and transfer learning capacities. Results in Fig. 4 (b) show that when discarding the attention on common sink tokens in training, models have a significant performance drop in the in-task evaluation.In addition, common sink tokens may benefit tasks with positive knowledge transfer. In Fig. 4 (b), we use models trained on MNLI data for zero-shot evaluation on SNLI data, which is for the same sentence entailment task but with a different data distribution. Results show that the models' transfer ability significantly drops after discarding the attention on common sink tokens. We hypothesize that this could be because sink tokens that are common across tasks can easily transfer knowledge on them.The analysis above motivates us to balance the transfer and interference caused by attention sinks in task learning. Specifically, when allowing models to preserve relatively high attention on sink tokens, we in turn encourage them to pay attention to non-sink tokens that have relatively low attention on sink tokens. This may increase sink attention deviations, which help reduce interference and oversmoothing. We develop a pre-scaling model to achieve this goal, detailed in Section 5. 
method: pre-scaling for diverse attention: 
 We introduce a pre-scaling mechanism that encourages models to allocate diverse attention to sink tokens and increase attention on non-sink tokens, when learning a downstream task.As shown in Section 3.1, attention sinks exist in pre-trained models. However, since sink tokens are usually common tokens that are not semantically significant, their pre-trained representations may not contain much information related to downstream tasks. On the other hand, pre-trained representations of non-sink tokens may contain more information for task prediction. This motivates us to first allocate task-specific attention on tokens based on their pre-trained representations, for increasing attention on non-sink tokens.Figure 5 : The scaling and regular model.We design a scaling layer to allocate attention scores on tokens based on their contributions to the task. To encourage diverse attention, we have different scaling of attention scores for different classes. The scaling layer is first learned through a probing stage to allocate high attention scores to non-sink tokens based on their pre-trained representations. Then we fine-tune the whole model.For each task, let H = {h 1 , ..., h n } be the pre-trained representations of the n input tokens, and V = {v 1 , ..., v c } the learnable class vectors for the c classes in that task. Each token representation h and class vector v are R d vectors. The scaling layer computes attention A c on tokens for classes as:EQUATION)where f : R d → R d is a learnable linear function. The output for class i is calculated by:p(i|H) = exp A (i•) c Hv i ∑ c i=1 exp A (i•) c Hv i ,whereA (i•)c is the i-th row of the attention A c , v i is the i-th class vector in V. We use the cross entropy loss to train the model with scaling layer.For each task, we use a two-stage training process: (1). probing: the encoder is fixed and we only learn the scaling layer (including class vectors); (2). fine-tuning: the whole model, including the encoder and the scaling layer, is learned for the target task.For sequential tasks in CL, we follow the task-incremental training where at each task, the loss is only computed over classes in that task. However, the scaling and prediction can be applied over classes in all tasks, and thus our model is general to the task-agnostic setting.Connection to probing then fine-tuning. Our pre-scaling mechanism has connections to the probing-then-fine-tuning mechanism proposed in Kumar et al. (2022) , since both use a similar two-step training process. However, our mechanism utilizes a scaling layer to gather diverse representations of all tokens instead of only using the representation of the [CLS] token for prediction. As shown in Fig. 5 (b), probing-then-fine-tuning under the regular model is a special case in our mechanism while the attention scores in Eq. ( 6) are 1 for [CLS] token and 0 for other tokens. As claimed in Kumar et al. (2022) , the two-stage training can reduce feature distortion by first learning good class vectors v. These good class vectors may further benefit our pre-scaling mechanism in CL. 
experimental settings: 
 Datasets. We evaluate four sequences of CL tasks: (1) Yahoo Split: a split of Yahoo dataset for news question-answer categorization (Zhang et al., 2015) with 5 disjoint tasks containing 2 classes each; (3) DB: a split of DBPedia data for Wikipedia article classification (Zhang et al., 2015) with 7 disjoint tasks containing 2 classes each; (4) News Series: a sequence of tasks on news-related data, including AG_news (news classification, 4 classes), MRPC (paraphrase detection, 2 classes) (Dolan & Brockett, 2005) , RTE (text entailment, 2 classes) (Williams et al., 2018b) and SST (sentiment analysis, 2 classes) (Socher et al., 2013) . For the above sequences, we randomly sample 1245 samples per class, which is the least number of class samples in our datasets.One category performs vanilla sequential learning for CL but has different training strategies on each single task, including (1) FT: a model where all parameters are sequentially updated;(2) PT+FT (Kumar et al., 2022) : a model first trains the classifier in the probing stage and then fine-tunes the whole model; (3) Prescale (ours): a model first trains the classifier and a scaling layer in the probing stage and then fine-tunes the whole model (with scaling).Another category is designed with specific CL techniques like experience replay, including (1) ER: a FT model storing all seen examples and performs sparse (1%) experience replay;(2) A-GEM (Chaudhry et al., 2019a ): a FT model constraining on gradients to prevent degrading performance of previous tasks; (3) MBPA++ (d'Autume et al., 2019 ): a FT model that stores and retrieves samples to locally adapt the model at inference time (Sprechmann et al., 2018) . (4). IDBR (Huang et al., 2021b) : a FT model with information-disentanglementbased regularization and replay. We also compare to IDBR without replay, denoted as IDBR(-R); (5) CTR (Ke et al., 2021) : an adapter-based task-incremental model with capsules Figure 6 : The last layer's representation similarity (for over-smoothing) and attention deviations (for attention sinks) for BERT and RoBERTa models on CL tasks. For an overall view, we average attention deviations over the tokens with top-5 largest outer degrees. and task transfer routing; (6) L2P (Wang et al., 2022) : a prompt-based model that learns to dynamically prompt for different data and tasks. We also compare models under multi-task learning (MTL) and separate learning for each task (Separate) as non-CL baselines to show the performance gap from CL models to them. Detailed settings are in Appendix C.We compare BERT-base and RoBERTa-base models in sequential learning, and use BERTbase for CL-specific models. For BERT models, we use learning rate 2e-5 to train 3 epochs for each task; and for RoBERTa models, we use learning rate 1e-5 to train 5 epochs per task.For probing and pre-scaling, we use the learning rate 5e-4 to train the classifier.Metrics. We train models in a task-incremental setting where task identifiers are given (task-aware). We evaluate models' CL performance by evaluating their average accuracy (ACC) and forgetting (FGT) on the sequence (Chaudhry et al., 2019b) , with or without task identifiers (task-agnostic). For analysis, we evaluate models' over-smoothing and attention deviations on sink tokens using metrics in Eq. (1).RoBERTa does not always outperform BERT in CL. Table 1 compares BERT and RoBERTa's CL performance under sequential learning. With fine-tuning, RoBERTa does not always outperform BERT in CL despite its high pre-trained capacity. Specifically, RoBERTa has a lower accuracy than BERT on Yahoo Split and a higher forgetting on News Series. This may relate to its higher over-smoothing and lower sink attention deviations than BERT, which make it more vulnerable to feature distortion and easier to propagate interference.Prescaling improves RoBERTa's CL capacity. With PT+FT, RoBERTa consistently outperforms BERT on CL tasks. We believe that is because PT+FT first learns good class vectors v, which reduces feature distortion in each single task and then benefits CL. After applying our prescaling method, BERT and RoBERTa achieve further improvements in CL tasks.Prescaling increases attention deviations. In Fig. 6 , we compare models' representational similarity and attention deviations after CL. After prescaling, RoBERTa's representation similarity decreases while the attention deviations increase. This suggests that our prescaling can encourage diverse attention, which reduces over-smoothing and benefits CL.Prescaling model outperforms CL models with replay. In Fig. 5 , we compare our prescaling model to CL-specific models to evaluate its overall CL capacity. Without experience replay or progressively storing model parameters, Prescale achieves overall best accuracies on CL tasks. Even for the News Series sequence which has knowledge transfer between tasks, Prescale outperforms replay-based models that are effective in this scenario. This validates the effectiveness of our prescaling mechanism in CL. Scaling strategies. In Table 3 , we compare our prescaling strategy (Full) to two other scaling strategies: one uniformly distributing attention to all tokens (Uniform); another learning to scale attention only on common sink tokens including special tokens, the punctuation '.' and the second token in the sentence (Sink). Results show that only scaling attention on common sink tokens does not yield improvements to PT+FT, and uniform scaling does not give as much improvement as full scaling. These suggest that the effectiveness of our prescaling strategy does not come only from having distributed attention on tokens.Scaling visualization. Fig. 7 shows a heapmap of the scaled attention on the SST data (with task classes {positive, negative}) after training the model on News Series. For the corresponding positive/negative classes, we observe that attention is also distributed on task-related tokens (e.g., charming, affecting) besides common tokens.Task-agnostic evaluation. In Table 4 , we evaluate models in the task-agnostic setting after task-aware training to show the separation of data representations across tasks. Both PT+FT and Prescale perform better than FT. This may relate to their larger attention deviations and less over-smoothing (Fig. 6 ), which make data representations contain more information from non-sink tokens with different distributions across tasks. On BERT, PT+FT outperforms Prescale. We hypothesize that this is because BERT is pre-trained with the next sentence prediction, and thus [CLS] may contain more general sentence-level information across tasks than the learned scaling. On the other hand, for RoBERTa which does not have sentence-level pre-training, Prescale performs better than PT+FT. 
conclusion: 
 In this paper, we study an attention sink phenomenon that can cause pre-trained models to perform inferior in CL tasks, despite their pre-trained capacity on downstream tasks. Specifically, we find that the small attention deviation on sink tokens and the fact that sink tokens are usually common tokens across tasks can cause the model having an over-smoothing problem and easily propagate interference during CL. To mitigate such interference, we propose a prescaling mechanism which first learns a scaling layer during probing to allocate diverse attention on non-sink tokens, and then fine-tunes the model with scaling. Results show that our pre-scaling method outperform most CL models and other scaling strategies. 
 
 Appendix: Denote P = A T (Iee T )A and λ max is the largest eigenvalue of P. Denote the layer's attention matrix as A ∈ R n×n (n is the number of input tokens) and its ij-th element as a ij . The eigenvalue λ max is lower bounded by:λ max ≥ max i ∑ n k=1 a ki -d i 2 ,where d i = ∑ n k=1 a ki n is the average outer degree of the i-th token. Derivation. Each element in P can be written as:P ij = ∑ n k=1 a ki a kj -( 1 n ∑ n q=1 a qi 1 n ∑ n q=1 a qj ,where a ij represents the ij-th element in A. Because P is symmetric and positive semi definite, its max eigenvalue λ max is real and positive, which also satisfies:λ max = max ||x||=1 x T Px,where x ∈ R n . Set {x} = {x 1 , ..., x n } as a set of unit vectors, where each x i has the element as 1 at the i-th place and others as 0. Then λ max is lower bounded as:λ max ≥ max i ∑ n k=1 a ki -d i a ki + d i = max i ∑ n k=1 a ki -d i 2 + 2d i ∑ n k=1 a ki -d i = 0 by the definition of d i = max i ∑ n k=1 a ki -d i 2The RHS above can be further decomposed as max i d 2 i ∑ n k=1 a kid i 2 /d 2 i to reflect the effects of the outer degree and the per-degree attention deviations.There are several cases that can make λ max small. When the largest average degree d i is large, λ max is small when its per-degree attention deviations are small. On the other hand, when the largest average degree is small, λ max can be small even when the per-degree attention deviations are relatively large. In the paper, we focus on the case when the largest average is large, for the observed attention sink phenomenon.In Section 4, the interference B T 1 B 2 depends on correlations between representations of the sink and non-sink tokens, which is related to correlations between their embeddings in X 1 and X 2 . Here we empirically calculate the correlations (i.e., dot product) between embeddings of the sink and non-sink tokens in BERT and RoBERTa-base, to verify our hypothesis that (common) sink tokens' embeddings are nearly orthogonal to other tokens' embeddings. Results are shown in Fig. 8 .For both BERT and RoBERTa, embeddings of sink tokens (e.g, [CLS] and [SEP] ) have close to 0 correlations to most other token embeddings. On BERT, the punctuation '.' has negative correlations to many other tokens, while on RoBERTa the distribution of its correlations is also centered around 0. We also randomly sample non-sink tokens from the vocabulary and show their embeddings' correlation distributions as a reference. On BERT, the embedding of non-sink token 'exchange' tends to have positive (non-zero) correlation to other tokens' embeddings. On RoBERTa, although the embedding of the non-sink token 'aution' has centered to 0 correlations to other tokens' embeddings, it has large correlations up to 8. However, the correlation distributions of cls (
) and sep (<\s>) tokens' embeddings have a much smaller range.Compared to the correlation to other tokens' embeddings, we also compute the selfcorrelation on sink tokens' embeddings as shown in Fig. 9 . The embeddings' selfcorrelations can not be ignored on both BERT and RoBERTa. When the common sink 


Decision: The paper identifies the prevalence of attention sinks in Bert and Roberta, and hypothesises that they may lead to interference across tasks in continual learning set ups. To address this, the authors propose an approach to diversifying attention. Reviewers thought the work took an interesting new angle on continual learning, and demonstrated improvements in experiments The work could be further strengthened by using more recent mask language models, and evaluating on more tasks. Overall this is borderline, but leaning towards acceptance.

Reviews: review: paper_summary: paper_summary: The paper studies the attention sink and its effect on continual learning. It also compared BERT and RoBERTa for continual learning with attention sink.
strength_weakness: strength_weakness: It is sometimes confusing whether you are discussing class-incremental learning (CIL) or task-incremental learning (TIL) when you talk about the attention sink's impact. It is only clear in the experiment section that your work is about TIL. Please make the CL setting clear earlier. 

In TIL, if you use a parameter isolation-based method, there is not much impact. See [1] 

TIL is very much a solved problem. Its results mainly depend on individual task performance and knowledge transfer. It is unclear if the state-of-the-art TIL method is used, we can see any noticeable impact. 

What is FGT? Is it the forgetting rate? For TIL, there should not be forgetting. See  [1,2]

[1] "Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks." EMNLP-2023;
[2] "Beyond not-forgetting: Continual learning with backward knowledge transfer." NeurIPS-2022.
questions: questions: See the weaknesses section
paper_summary: paper_summary: The paper studies the attention patterns of BERT and ROBERTA and they evaluate if an over-smoothing pattern in the token representations is influencing the learning patterns in a continual learning setup. Furthermore, they propose a mechanism for encouraging the models to have a more diverse attention behaviour. This model change improves the adaptation capabilities of the models to new tasks int a continual learning framework.
strength_weakness: strength_weakness: I think that the paper requires a more comprehensive perspective on the impact of the studied phenomenon in the tasks that BERT and Roberta are generally used.

Second, I would like to see a better coverage of how this work can be extended to other models and tasks and more specifically on the new class of language models that would have a bigger capacity of encoding the information from multiple tasks. It would also help to provide resources that would facilitate applying the proposed techniques and evaluations to other models.
paper_summary: paper_summary: The paper presents a novel perspective on the performance of pre-trained language models (LMs) in continual learning (CL). The authors observe that despite higher single-task performance, RoBERTa does not consistently outperform BERT in CL settings. The paper introduces the concept of "attention sink" and its connection to over-smoothing, which may lead to interference in learning subsequent tasks. To address this, the authors propose a pre-scaling mechanism to encourage attention diversity and reduce interference.
strength_weakness: strength_weakness: 1. The performance advantage is not very pronounced, and it is hoped that verification can be done on more challenging tasks. Since most of the experiments in the paper are binary classification tasks, the absolute values of accuracy are already very high, making the differences between various comparison methods not significant enough.
2. Does the two-step training process significantly increase computational costs?
3. Is it reasonable to assume that tokens other than common tokens are definitely irrelevant in the unrelated tasks in Section 4.2?
4. Is it reasonable to assume that unrelated tasks in the continual learning tasks are irrelevant? If tasks are completely unrelated, then directly learning separate models for each task would be the best solution. Therefore, the trade-off between Transfer and Interference analyzed in Section 4.3 is more important.
questions: questions: 1. Is the analysis of single-head self-attention in Section 4.2 not without loss of generality for multi-head self-attention?
2. Is the phenomenon that Sink Tokens are Usually Common Tokens in Section 3.2 beneficial or harmful to CL? If harmful, then Fig. 2 right indicates that the proportion of Sink Tokens that are Common Tokens decreases after RoBERTa fine-tuning, which should mitigate the damage to CL.
3. Is the problem of RoBERTa in CL mentioned in the introduction caused by a lack of stability or plasticity? That is, is the performance low on new tasks or old tasks, or both? Specific experimental results analysis is hoped for.
paper_summary: paper_summary: This paper discusses continual learning from the perspective of "attention sink." It initially notes that attention often concentrates on specific tokens for a single task. The authors suggest that this could cause interference in continual learning, as the sink tokens might hold information specific to one task. Consequently, they introduce a pre-scaling mechanism to promote diverse attention scores on sink tokens.
strength_weakness: strength_weakness: 1. This paper focuses on task-incremental learning. As noted in [1], a significant non-CL baseline in TIL is to train each task separately. This can help determine whether the proposed methods effectively balance transfer and interference.

2. The BERT and RoBERTa models are quite outdated. I believe the community would be more interested in whether the proposed method remains effective with the latest LLMs

3. Given that the pre-trained model has been updated, how does the current method prevent forgetting the original pre-trained knowledge?

[1]: Continual Learning of Natural Language Processing Tasks: A Survey, Ke and Liu, arXiv 2023
questions: questions: See above

Comments: title: Gentle nudge on our author response.
comment: Dear reviewers, 

We thanks for your constructive feedback on our work. If you have had a chance to look at our responses, can you please let us know if it addresses your concerns? We are happy to provide any additional clarification as needed.