Does RoBERTa Perform Better than BERT in Continual Learning: An Attention Sink Perspective
Authors: Xueying Bai
Yifan Sun
Niranjan Balasubramanian
Venue: COLM (2024)
Keywords: Continual Learning, Attention Sink, Interference, Over-Smoothing, Pre-Scaling
TL;DR: We tackle the LM's interference in CL from an attention sink's perspective, where different models' concentration on common tokens (e.g., special tokens and punctuations) may influence their CL performance besides the models' single-task capacities.

Abstract: Continual learning (CL) aims to train models that can sequentially learn new tasks without forgetting previous tasks' knowledge. Although previous works observed that pre-training can benefit CL, it remains unclear whether a pre-trained model with higher downstream capacity also performs better in CL. In this paper, we observe that pre-trained models may allocate high attention scores to some 'sink' tokens, such as [SEP] tokens, which are ubiquitous across various tasks. Such attention sinks may lead to models' over-smoothing in single-task learning and interference in sequential tasks’ learning, which may compromise the models' CL performance despite their high pre-trained capabilities. To reduce these effects, we propose a pre-scaling mechanism that encourages attention diversity across all tokens. Specifically, it first scales the task's attention to the non-sink tokens in a probing stage, and then fine-tunes the model with scaling. Experiments show that pre-scaling yields substantial improvements in CL without experience replay, or progressively storing parameters from previous tasks.

Published as a conference paper at COLM 2024
Does RoBERTa Perform Better than BERT in Continual Learn- ing: An Attention Sink Perspective
Xueying Bai, Yifan Sun, Niranjan Balasubramanian Department of Computer Science
Stony Brook University (xubai, ysun, ni niranjan) wtteeccsttoooooo...... edu
Abstract
Continual learning (CL) aims to train models that sequentially learn new tasks without forgetting previous tasks' knowledge. Although pre- vious works observed that pre-training can benefit CL, it remains unclear whether pre-trained model with higher downstream capacity also per- forms better in CL. In this paper, we observe that pre-trained models may allocate high attention scores to some 'sink' tokens, such as [SEP] tokens, which are ubiquitous across various tasks. Such attention sinks may lead models' over-smoothing in single-task learning and interference in sequen- tial tasks' learning, which may compromise the models' CL performance despite their high pre-trained capabilities. To reduce these effects, we pro- pose a pre-scaling mechanism that encourages attention diversity across all tokens. Specifically, it first scales the task's attention to the non-sink tokens in a probing stage, and then fine-tunes the model with scaling. Experiments show that pre-scaling yields substantial improvements in CL without expe- rience replay, or progressively storing parameters from previous tasks.
1 Introduction
Machine learning applications in the real world often need to face continuous streams of data from different tasks or distributions (Lopez-Paz Ranzato, 2017; Hou al., 2019). For such cases, it is important to develop continual learning (CL) models that can progressively learn new tasks without performance degradation in previous tasks (i.e., catastrophic forgetting).
The pre-training and fine-tuning paradigm (Devlin et al., 2018), which effectively learns downstream tasks by fine-tuning a pre-trained language model (LM), widely used for general NLP tasks. Previous works (Wu et al., 2022; Mehta et al., 2023) observed that pre- training can also benefit CL. However, remains unclear whether a pre-trained model that has better single-task performance also performs better in CL settings. For example, BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019b) are two pre-trained LMs with the same model structure. RoBERTa achieves generally better downstream performance than BERT, in part because it is pre-trained on more diverse data. In CL. however, RoBERTa does not always outperform BERT (Wu et al., 2022). This motivates study on the factors that may cause models' inferior performance in CL, besides their pre-trained capacity.
In this paper, we show that the atlention sink phenomenon can influence models' CL capacity, Attention sinks have been observed on autoregressive LLMs, where models tend to allocate high attention scores specific tokens in the input ('sink tokens") regardless of their semantic significance (Xiao et al., 2024). In Fig. 1, we show that attention sinks appear after the first layers in both BERT and RoBERTa models. And the sink tokens are usually common tokens (e.g., special tokens like [SEP]), which are not semantically significant but are present in most NLP tasks (Fig. 3 right). However, unlike the previous work by Xiao et al. (2024) which focuses on the magnitude of attention scores sink tokens, our work focuses on the attention deviations on sink tokens ('sink attention deviations') and their influence on CL.
The code is available https::/ httss://ggtuuubccoo thwb. wttphbccwm/sttttgeeoooo/tttttttt sink-cl

---

Published as a conference paper at COLM 2024
Layer
Layer 12
Layer 12
Layer
ROBERRT
Pre- Prainng
Fine- Vine-Tuning
Prelinginng
Figure 1: Attention maps averaged from all attention heads after pre-training, fine-tuning, and our pre-scaling mechanism. Sink tokens (i.e., with dark blue columns) in pre-trained models obtain similar high attention scores. After fine-tuning, models (especially RoBERTa) have drastic attention changes, which may indicate feature distortion. After pre-scaling, models have diverse attention on sink tokens and preserve the pre-trained attention patterns.
In particular, we connect the attention deviations on sink tokens the ovey-smoothing phenomenon. Over-smoothing has been observed in pre-trained LMs, where models output nearly identical representations for all input tokens (Dong et al., 2021; Shi et al., 2022). We show that -eooshig is related to small sink attention deviations in pre-trained models. can cause distortions of pre-trained features (Kumar et al., 2022), which may make models less generalizable to out-of-distribution data (e.g,, data from other tasks) and affect CL.
Models' small attention deviations on common sink tokens can also cause unnecessary interference across tasks in CL. Specifically, representations of common sink tokens may carry the information of one task, which then influences another task's 's learning. This can be harmful if the new task irrelevant to the previous one. We conduct a case study show that such interference is hard to avoid when models have small sink attention deviations.
To address the above problems, we propose a pre-scaling mechanism that encourages diverse attention scores on sink tokens. It introducss scaling layer that is first learned to allocate diverse attention to tokens probing stage, and then tuned in a fine-tuning stage together with the pre-trained encoder. Experiments show that pre-scaling improves models' CL. performance with reduced over-smoothing. Moreover, RoBERTa models consistently outperform BERT models after pre-scaling, which suggests that pre-scaling helps to better utilize models' pre-trained capacity in CL.
In conclusion, we make the following contributions: (1) We characterize the attention sink phenomenon in pre-trained LMs and build a connection to over-smoothing. (2) We conduct case study to show that the above attention sinks may propagate unexpected interference in CL. (3) We propose a pre-scaling mechanism that can significantly improve pre-trained LMS capacity in CL. without experience replay progressively storing parameters.
2 Related Work
Continual Learning. Models for CL can be divided into three main categories: (1)
regularization-bast models which constrain the deviation of new parameters from the older ones (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Lee et al., 2017); (2) replay-based models which reduce catastrophic forgetting by rehearsing on real or pseudo samples from previous tasks (Lopez-Pazz & Ranzato, 2017; Chaudhry et al., 2019a) or genera- tive models (Shin et al., 2017; Kemker& Kanan, 2017); (3) architecture-basee models which learn evolving architectures for sequential tasks, with their capacities for each task carefully assigned (Rusu et al., 2016; Yoon et al., 2017).
CL in NLP is emerging area (Liu et al., 2019a; Biesialska et al., 2020). MBPA+ (d' Autume et al., 2019) uses experience replay and local adaptation to mitigate forgetting; LAMOL (Sun et al., 2019) generates pseudo samples for replay; IDBR (Huang et al., 2021a) disentangles
2

---

Published as a conference paper at COLM 2024
task-agnostic and task-specific information; CTR (Ke et al., 2021) uses capsule network for knowledge transfer. All the above models are based on pre-trained LM (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019). Recent works show that pre-training can alleviate catastrophic forgetting (Wu et al., 2022; Mehta et al., 2023; Lee et al., 2023). Mehta et al. (2023) claims that the benefit may come from having less sharp minima. In this paper, we tackle the CL problem from an attention sink perspective, which provides an explanation of why sometimes RoBERTa underperform BERT in CL tasks (Wu et al., 2022). To the best of our knowledge, we are the first to tackle CL problems from this angle.
Over-Smoothing. In this paper, we connect attention sinks to an over-smoothing phe- nomenon, which is first proposed in graph neural networks (Oono & Suzuki, 2020; Huang graph
et al., 2020; Cai & Wang, 2020; Rusch al., 2023; Yang et al., 2020). Over-smoothing refers to the problem that the models' performance deteriorates as representations of all the nodes in the graph become similar (Li et al., 2018; Xu al., 2018). For transformer-based models, Dong et al. (2021) claims that pure attention loses rank doubly exponentially with model depth. And Shi et al. (2022) characterize the oversmoothing problem in transformers by viewing the attention matrix form of adjacency matrix in the graph. In this paper, connect the over-smoothing problem to attention sinks, to show that in some cases attention sinks will influence model's 's task learning and cause inferior performance in CL.
Attention Sinks in Language Models
We first empirically show that certain attention sinks exist pre-trained LM layers. Then we study their impact by connecting them to an over-smoothing phenomenon, which may influence models' single-task overfitting and in turn can influence their CL abilities. The attention sinks can also cause cross-task interference in CL, which we discuss in section 4.
3.1 Empirical Analysis of Attention Sinks
We characterize the presence of attention sinks (Xiao et al., 2024) in LMs using data from NLI datasets SST and MNLI (Socher et al., 2013; Williams et al., 2018a; Wang et al., 2019).
Attention on sink tokens. Fig. 1 illustrates the presence of attention sinks using attention maps, which show high attention scores are allocated specific input tokens (i.e., sink tokens). The figure also shows sink tokens might receive similar (high) attention scores from all tokens. To empirically quantify these observations, we devise the measurements below.
Let E R denote an attention matrix over tokens for a single attention head. An element dij E A denotes the attention on the j-th key token for the i-th query token. For the i-th (query) token, we have the following measurements:
Average outer degree: d; En_1 aki/nz
Attention deviation: A; Ex (Au d;)2 (nd;)
(1)
We average d; and A; across all attention-heads in each layer. d; is the averaged attention score allocated to the i-th token, and A; is the per-degree attention score deviation to the i-th token's is average outer degree. We study sink tokens with the largest average outer degrees, and calculate their attention deviations as sink attention deviations.
Fig.2 shows the layer-wise average outer degrees and sink attention deviations for BERT and RoBERTa models. In 2 (a), tokens with top-3 largest outer degrees obtain 60% attention scores from input tokens, while the top-1 tokens obtain over 20% attention. This shows that a small number of tokens obtain major attention in self-attention layers. In Fig.2 (b), we observe that the sink attention deviations are mostly small, except for the first two layers. This shows that all tokens pay similar attention to the sink tokens in the top layers.
Sink tokens are usually common tokens. We also find that sink tokens in pre-trained LMs turn out to be common tokens, ones that appear in many language tasks. These common tokens include special tokens such ('[SEP]', punctuation (".), the initial tokens in
3

---

Published as a conference paper at COLM 2024
MNLI
SST
MNLI
0.8
63
0.00
0.50
Layer
Layer
Layer
Layer
(a) Average Degrees
(b) Sink Attention Deviations
Figure 2: Average outer degrees and attention deviations on MNLI and SST data. (a) The cumulative average outer degrees of tokens with the top-1, top-3 and top-5 largest outer degrees, (b). The attention deviation sink tokens with the top-1 largest outer degrees.
Over-Smoothing and Sink Attention Deviations
Ratio of Sink Tokens that Are Common Tokens
71 Simienfy
0.
Simertt
MNLI
sestarty Daciation IT Cevistion
Rattor (pneceentaee))
12
SS7
BERT
RoBERTa Layer
BERT
RoBERTa Layer
Figure 3: The left shows models' over-smoothing and attention deviation on sink tokens. The right shows the ratio of sink tokens that are common tokens across tasks. Here we consider special tokens, the punctuation" '' and the second token in the input as the common tokens. 'PT' stands for pre-training, and 'FT' stands for fine-tuning on 3k MNLI data.
inputs, which are not semantically significant. The right side of Fig. 3 shows the ratio of sink tokens with the top-1 largest outer degrees that are also common tokens at each layer. Almost all sink tokens are common tokens in the first several layers of the pre-trained model. Even after fine-tuning, the models still have high attention on them in the middle layers.
3.2 Connection between Over-Smoothing and Attention Sinks
We show the impact attention sinks on single tasks by connecting them to an over- smoothing phenomenon, where the attention deviations on sink tokens are key factor.
SmeotSmmo in transformers. Previous works have identified an over-smoothing phenomenon in transformer-based models: token representations become identical after several self-attention layers. Over-smoothing is closely related to models' task learning task (Shi et al., 2022). There are several factors that can lead
ability and their overfitting to over-smoothing, and we focus on the effect of self-attention matrices here.
For a self-attention layer with an attention matrix A E Rax" let H E R be its input token representations and AH its output. The over-smoothing problem described as:
d S VAmaxd (H).
The distance (H) -|[I-ee"
measures the closeness between each token repre- sentation in H and the averaged token representation ee H, where n 1 [1, 1, 1] E IR". Amax is the largest eigenvalue of A (T(I-ee") ) When Amax is small, representations after the attention layer will closer to the averaged representations, which causes over-smootiing.
Connection to attention sinks. Over-smoothing has been identified in many transformer- based models. We analyze the eigenvalue Amax to see the property of the attention matrix A under the over-smoothing circumstances (i.e., Amax is small). With each attention score ajj
4

---

Published as a conference paper at COLM 2024
and average outer degree d; defined in Section 3.1, Amax is lower bounded as:
Amax 2 max D (ax - -d))
(2)
The details are in Appendix A. When Amax is small, the RHS of Eq. (2) has to be small. In particular, the i-th token which has the largest outer degree must have its deviation [ d,)2 to be small. When d; is large (as shown in Fig. 2), to make the deviation small each attention agi needs to be close d, Therefore, all tokens have similar (high) attention on the token with the largest outer degree, which is an attention sink phenomenon.
We empirically show the connection between attention deviations A and over-smoothing in Fig. 3. The over-smog degree is reflected by the average cosine similarity between token representations (Shi al., 2022). Going from lower higher layers, we observe that the attention deviation decrenses while the representation similarity increases. This validates the connection between attention sinks and the over-smoothing phenomenon.
Impact of over-smoothing with attention sinks. When over-smoothing occurs with at- tention sinks above, the sink token representations may dominate the averaged token representation, and make other token representations (including [CLS]) close to them. To learn a task, models may push sink token representations close to the task data representa- tion (Fig. 4(a)). Since sink tokens may be irrelevant to tasks (Fig. 3 right), this may distort pre-trained features and make models less generalizable to OOD data (Kumar et al., 2022).
Comparing BERT and RoBERTa, observe that pre-trained RoBERTa suffers more from over-smoothing (i.e., high representation similarity), corresponding with low attention deviations sink tokens at the second and last several layers (Fig. 2(b)). Therefore, hypothesize that RoBERTa may be more vulnerable to feature distortion in task learning, which is reflected by its distorted attention patterns (Fig. and Fig. 3 right) after fine-tuning. This may also influence RoBERTa's CL capacity.
4 Attention Sink and Interference in Continual Learning
In this section, we first conduct a case study to show that attention sinks above can cause unnecessary interference when learning across tasks in CL. Then we discuss a transfer vs. interference trade-off induced by attention sinks, which inspires our method in Section 5.
4.1 Interference in Continual Learning
We study the following CL problem: A model continually learns sequence of tasks, with no previous tasks' data accessible when learning new tasks, and no storage of previous tasks' model parameters. The model has different predictors for different tasks, while the encoder shared. Each task consists of data D; = (X;, where X; is the input feature for task and y; R is its target output.
When learning task after task one way to quantify the cross-task interference on the shared parameter is through the dot product between its (vectorized) gradients the two tasks' losses (Riemer et al., 2019):
I(0;i, VoL(X;,)) VoL(X,,);)
where is the dot product operator the flattened gradients. The interpretation is that interference that leads to forgetting happens when i(0; < 0, while the positive knowl- edge transfer may happen if i( > 0. For tasks that do not have knowledge transfer, the interference is expected to be O.
4.2 Case Study: Attention Sink Can Cause Unnecessary Interference Between Tasks
We use a case study to show that attention sinks can propagate unexpected interference between irrelevant tasks. The study is based the attention sink phenomenon characterized in Section 3, which showed: (1). models allocate high attention sink tokens with small deviations; (2). sink tokens usually common tokens shared across different tasks.
5

---

Published as a conference paper at COLM 2024
Data. We consider two irreleoant NLP tasks in CL setting. For task i, have data instance (X,, y/) where X; consists embeddings of input tokens. We make the following assumptions about tasks and data:
1. There is no knowledge transfer between the tasks and there should be no interfer- ence (positive or destructive) when learning one task after the other.
2. Assume there are common tokens (egg, special tokens) in two tasks' data instances. For all other tokens in task, we assume they are irrelevant non-common tokens in the other task, with corresponding embeddings being orthogonal.
Model. We use model consisting of two single-head self-attention layers. For each task i, the input X; E RRixd consists of d-dimensional embeddings for n; tokens. Considering regression problem (generalizable to classification), the prediction 9; is calculated as:
yi = by WA,,,W)v;,
where A E R is the attention matrix in the first attention layer, b; E R" is the attention vector in the second attention layer that integrates all hidden representations for the target task prediction. Both A; and b, are obtained under the self-attention mechanism (Vaswani et al., W E Raxd is transformation matrix. E Rdx11 is the predictor that maps the representation to output value Oi for task i. The loss function is: L(9i, vi) El (0: yI
For simplicity, we sort X; and A; to make the k common tokens have indices {1, k} and others have indices {(K + 1),-., n:}. We assume the common tokens are sink tokens in A
Claim. The interference on the transformation matrix W between task and task 2 mainly depends on the outer degrees of sink tokens and the sink attention deviations.
We calculate interference on the shared parameter W based on Eq. (3) and the model above:
I(W) = (9) y1) - (B)
(4)
where B = by X1 and _ = byj A2X2. When both training losses are non-zero (which is the usual case in CL), interference in Eq. (4) depends on the correlation vī v2 between predictors and the correlation B B2 between representations. Since the learned predictors may not be good enough to reflect the orthogonality between task and task 2 (Kumar et al., 2022), we have to consider the interference caused by the correlation B B2, discussed below.
Step 1: decompose B1 and B2. Generally, for any matrix M, we denote M (ij) the ij-th element of M. And for any vector b, we denote ((0) as the i-th element b.
Denote d1 as the vector of tokens' average outer degrees in attention matrices A1. For the
attention A (1),, we define its deviation to the j-th average outer degree as: C (if) Since the attention vector b is row-stoc,astic, we decompose B
A (i)
(
T = 11 11 + b E"
Σ
(5)
ri: nun-sink representations
Sy sink representations
51: representation deviations
Similarly, we have B 72 s2 + 82, where denotes the non-sink representations, 52 the sink representation and 82 the sink representation deviations for task 2.
Step 2: calculating B B2. Based on assumption 2 that non-sink token embeddings from two tasks are orthogonal, we have T = 0. Moreover, since sink tokens are common tokens that are supposed to be neutral to other tokens, we hypothesize that their embeddings are nearly orthogonal other token embeddings (Appendix B). This makes (s1 + 8 6 + (s2
82)r1² T x 0. Then we have B B2 = (r1 + $ + 61 + $2 82) (s1 + 61) + 62)
Therefore, B B2 depends largely on tasks' sink representations $1 82 and their representation deviations 6,,02 02 Specifically, it is dominated by $1s; when: (1) each attention deviation (ii))

---

Published as a conference paper at COLM 2024
(8) Neurest neighbors of [CLS] token after fine-tuning
(b) Models' cupacities with and without sink tokens
SNLI
St: Two wome are embmeingg while holding go paciages
Data
MNLI
SNLI (landdrr
SNLI wio Sink (Transerr
(Ncutral)
52: The siters are lugging goodbye walle bolding to so packages ofter just coling lunc..
Model
(In- Task)
BERT
L6. are women aee package saters bugging while go to
[SEP] [SEP)... The are are
whlle go just go
BERT
69.2 0.2
46.4 1.6 50.9
L12: are The while huggnng aiter lust are. bolding goodbye saten caling (SEP) (S6P).
RoBERTa
The are packoges packagos dsters entrracing lmaginn go L12: womes digh Two dig The are afler sisters are goodaye legging while embroing
TA Memoman <iso... Ton
women. sta-- caa-
are hulding holding ofter
RoBERTa
0.7
2.11
39.3 1.4
Sink
4.3
45,8+
Figure 4: (a) After fine-tuning, sink tokens' representations can be close to data ([CLS))
representations, even though the sink tokens are irrelevant to the task. (b) Models trained on 3k MNLI data and evaluated on MNLI and SNLI data, with and without attention on common sink tokens. Sink tokens ensure models' capacity and their transfer to similar tasks.
in Eq. (5) is close to O; or (2) the attention score ((i) is close to 0 when the absolute deviation le(0) is large, which makes b (0f) close to O. When the sink tokens' outer degrees are large, the correlation $1$2 can cause high interference between the orthogonal tasks and 2.
4.3 Transfer vs. Interference
Since attention sinks on common tokens can propagate unnecessary interference, should we exclude sink tokens that are common tokens ('common sink tokens') when calculating attention in CL. To answer this question, we first train models with and without attention on common sink tokens, and then compare their in-task and transfer learning capacities. Results in Fig. 4(b) show that when discarding the attention on common sink tokens in training, models have a significant performance drop in the in-task evaluation.
In addition, common sink tokens may benefit tasks with positive knowledge transfer. Fig. 4(b), we use models trained on MNLI data for zero-shot evaluation on SNLI data, which is for the same sentence entailment task but with a different data distribution. Results show that the models' transfer ability significantly drops after discarding the attention common sink tokens. We hypothesize that this could be because sink tokens that are common across tasks can easily transfer knowledge them.
The analysis above motivates us to balance the transfer and interference caused by attention sinks in task learning. Specifically, when allowing models to preserve relatively high attention on sink tokens, we in turn encourage them to pay attention to non-sink tokens that have relatively low attention on sink tokens. This may increase sink attention deviations, which help reduce interference and oversmoothing. We develop pre-scaling model to achieve this goal, detailed in Section 5.
5 Method: Pre-Scaling For Diverse Attention
We introduce a pre-scaling mechanism that encourages models to allocate diverse attention to sink tokens and increase attention on non-sink tokens, when learning downstream task.
As shown in Section 3.1, attention sinks exist in pre-trained models. However, since sink tokens are usually common tokens that are not semantically significant, their pre-trained representations may not contain much information related to downstream tasks. On the other hand, pre-trained representations of non-sink tokens may contain more information for task prediction. This motivates us to first allocate task-specific attention on tokens based on their pre-trained representations, for increasing attention on non-sink tokens.
We design a scaling layer to allocate atten- tion scores on tokens based on their contri- butions to the task. To encourage diverse attention, we have different scaling of atten- tion scores for different classes. The scaling layer is first learned through a probing stage to allocate high attention scores to non-sink tokens based on their pre-trained represen- tations. Then fine-tune the whole model.
199
+Ppa
Taxefao
Scaling
Regular Modd
Figure 5: The scaling and regular model.
7

---

Published as a conference paper at COLM 2024
Scaling layer. For each task, let H (h,, h be the pre-trained representations of the imput tokens, and V (Vi,..,, VeJ the learnable class vectors for the c classes in that task. Each token representation h and class vector are R vectors. The scaling layer computes attention A₂ on tokens for classes as:
A. softmax (VS(H))/VE
(6)
where f: : R2 + Rd is learnable linear function. The output for class is calculated by:
( H) exp (A
Hv;) Σ exp (A Hvi),
where A is the i-th row of the attention Ae, vi is the i-th class vector in V. We use the cross entropy loss to train the model with scaling layer.
Two-Stage training. For each task, we use two-stage training process: (1). probing: the encoder is fixed and we only learn the scaling layer (including class vectors); (2). fine-tuning: the whole model, including the encoder and the scaling layer, is learned for the target task.
For sequential tasks in CL, we follow the fask-incremental training where at each task, the loss is only computed over classes in that task. However, the scaling and prediction can be applied over classes in all tasks, and thus our model is general to the task-agnostic setting.
Connection to probing then fine-tuning. Our pre-scaling mechanism has connections to the probing-then tihen-fine-tuning mechanism proposed in Kumar et al. (2022), since both use similar two-step training process. However, our mechanism utilizes a scaling layer to gather diverse representations of all tokens instead of only using the representation of the [CLS] token for prediction. As shown in Fig. 5(b), probing-then-fine fihnn-iinetuuning under the regular model is a special case in mechanism while the attention scores in Eq. (6) are 1 for [CLS] token and 0 for other tokens. As claimed Kumar et al. (2022), the two-stage training can reduce feature distortion by first learning good class vectors v. These good class vectors may further benefit our pre-scaling mechanism in CL.
6 Experiments
6.1 Experimental Settings
Datasets. We evaluate four sequences of CL tasks: (1) Yahoo Split: split of Yahoo dataset for news question-answerr categorization (Zhang et al., 2015) with 5 disjoint tasks containing 2 classes each; (3) DB: a split DBPedia data for Wikipedia article classification (Zhang et al., 2015) with disjoint tasks containing 2 classes each; (4) News Series: sequence of tasks on news-related data, including AG AG_news (news classification, 4 classes), MRPC (paraphrase detection, 2 classes) (Dolan & Brockett, 2005), RTE (text entailment, 2 classes) (Williams et al., 2018b) and SST (sentiment analysis, 2 classes) (Socher et al., 2013). For the above sequences, randomly sample 1245 samples per class, which is the least number of class samples in our datasets.
Baselines. We consider two categories of baselines:
One category performs vanilla sequential learning for CL but has different training strategies on each single task, including (1) FT: model where all parameters are sequentially updated; (2) PT+FT (Kumar et al., 2022): a model first trains the classifier in the probing stage and then fine-tunes the whole model; (3) Prescale (ours): model first trains the classifier and scaling layer in the probing stage and then fine-tunes the whole model (with scaling).
Another category is designed with specific CL techniques like experience replay, including (1) ER: FT model storing all seen examples and performs sparse (1%) experience replay; (2) A-GEM (Chaudhry et al., 2019a): a FT model constraining on gradients to prevent degrading performance of previous tasks; (3) MBPA++ (d' Autume et al., 2019): FT model that stores and retrieves samples to locally adapt the model at inference time (Sprechmann et al., 2018). (4). IDBR (Huang et al., 2021b): FT model with iilommationddisetaanleeeee based regularization and replay. We also compare to IDBR without replay, denoted as IDBR(-R); (5) CTR (Ke et al., 2021): adapter-based task-incremental model with capsules
8

---

Published as a conference paper at COLM 2024
Table 1: Comparison between sequential models on CL. We report ACC and FGT with their standard deviations (std) on five random seeds. Bold scores are the best scores.
Model
Yahoo Split
DB Split
Series
ACC
FGT
aCC
FCT
aCC
FGT
BERT
Probing
88.43
99.30
74.81
FT
86.19 082
6.70 LGs
66 22
9,,15
68.98 5.8
17.13
PT+FT
90.24
22
1.64
27
8,16230
Prescale (ours)
90.92
1.47 071
99
0.13
79.76 076
4,40
RoBERTa
Probing
88,06-20
99.33
68:27 1:38
FT
83.54 k.s
10.91 894
71.94
32.53 6:2
70.61 4:42
18.24
PT+FT
90.76
2.14
99.68
0.21
79.39
8.01 3:1
Prescale (ours)
90.92 007
1.95 1.0
99.78 c.
0.09 cx3
81.59 1:24
4.16
Figure 6: The last layer's representation similarity (for over-smoothing) and attention deviations (for attention sinks) for BERT and RoBERTa models on CL tasks. For an overall view, we average attention deviations over the tokens with top-5 largest outer degrees.
and task transfer routing; (6) L2P (Wang et al., 2022): a prompt-based model that learns to dynamically prompt for different data and tasks. We also compare models under multi-task learning (MTL) and separate learning for each task (Separate) as non-CL baselines to show the performance gap from CL models to them. Detailed settings are in Appendix C.
We compare BERT-base and RoBERTa-base models in sequential learning, and use BERT- base for CL-specific models. For BERT models, use learning rate 2e-5 to train 3 epochs for each task; and for RoBERTa models, we use learning rate le-5 to train 5 epochs per task. For probing and pre-scaling, we use the learning rate 5e-4 to train the classifier.
Metrics. We train models in a task-incremental setting where task identifiers are given (task-aware). We evaluate models' CL performance by evaluating their average accuracy (ACC) and forgetting (FGT) on the sequence (Chaudhry et al., 2019b), with or without task identifiers (task-agnostic). For analysis, we evaluate models' over-smoothing and attention deviations on sink tokens using metrics in Eq. (1).
6.2 Results
RoBERTa does not always outperform BERT in CL. Table 1 compares BERT and RoBERTa's 's CL performance under sequential learning. With fine-tuning, RoBERTa does not always outperform BERT in CL despite its high pre-trained capacity. Specifically, RoBERTa has lower accuracy than BERT on Yahoo Split and a higher forgetting on News Series. This may relate to its higher over-smoothing and lower sink attention deviations than BERT, which make it more vulnerable to feature distortion and easier to propagate interference.
Prescaling improves RoBERTa's CL capacity. With +T+ RoBERTa consistently outper- forms BERT on CL tasks. We believe that is because FT+ first learns good class vectors v, which reduces feature distortion in each single task and then benefits CL. After applying our prescaling method, BERT and RoBERTa achieve further improvements in CL tasks.
Prescaling increases attention deviations. In Fig. 6, we compare models' representational similarity and attention deviations after CL. After prescaling, RoBERTa's representation similarity decreases while the attention deviations increase. This suggests that our pre- scaling can encourage diverse attention, which reduces over-smoothing and benefits CL.
Prescaling model outperforms CL models with replay. In Fig. 5, we compare our pre- scaling model to CL-specific models to evaluate its overall CL capacity. Without experience replay or progressively storing model parameters, Prescale achieves overall best accuracies on CL tasks. Even for the News Series sequence which has knowledge transfer between tasks, Prescale outperforms replay-based models that are effective in this scenario. This validates the effectiveness of our prescaling mechanism in CL.

---

Published as a conference paper at COLM 2024
Table 2: Results for task-incremental learning. We report ACC and FGT with their std on five random seeds. Bold scores are the best scores and underline scores are the second best.
Yahoo Split
DB Split
News Series
Model
ACC.u
PGT
ACC.vd
PGT
ACC
FGT
CL
ER
87 87.42 0.22
5.61 c/
91.05 10:20
10.20 1014
75.47 3.8
7.81
A-GEM
89.43
2.95
94.71 4.70
5.98 5.3
a5.90
6.60
MBPA++
86.50
6.62..z
97.17 3m
3.09
72.55
9.64
IDBR (-R)
89.32 1M
2.74 1.25
96.47 Az
3.95 css
72.36
8.67 4:33
IDBR
90.48
1.32
99.84
0.04 003
26.90
3.24
CTR
87.06
1.28 cm
99.04 0.55
0.29 mus
75.12
3.40
90.82 0.5
0.60
99.63
0.29 n.s
73.99
3.43
Sequential
FT
86.19
6.70 u0
66 22
39.15 9:47
68.98
17.13
Prescale (ours)
90.92 ust
1.47 cm
99.74
0.13 puts
79.76
4.40
Non-CL
Separate MTL
92. nM
99.87 a.
83.72
92.27
99. 19.88
82.04
Table 3: ACC and FGT Table 4: ACC on task- of different scaling strate- agnostic evaluations for gies on News Series.
paraphrase
parastrace
entsilment
DB and Yahoo Split.
World
0
Sports
0
Model
ACC
FGT
Model
DB
Yalono
Businsss
ScUTech
000
BERT
Uniferm
78.98
5.32
BERT
FT
15.90
36.19
Prescale
Sink
76.08
9.27
PI+FT
72.41
53.34
ve
Full
Prescale
70.36
53.21
RoBERTa Prescale
Uniferm
79.44
7.07
RoBERTa
18.71
36.24
Sink
79.09
9.26
PT+FT
67.32
52.98
Figure 7: Attention for each class on tokens in SST (from News Series).
Full
81.59
4.16
Prescale
77.35
53.51
6.3 Ablation Study
Scaling strategies. In Table 3, we compare our prescaling strategy (Full) to two other scaling strategies: one uniformly distributing attention to all tokens (Uniform); another learning to scale attention only on common sink tokens including special tokens, the punctuation and the second token in the sentence (Sink). Results show that only scaling attention on common sink tokens does not yield improvements to +T+ and uniform scaling does not give as much improvement as full scaling. These suggest that the effectiveness of our prescaling strategy does not come only from having distributed attention on tokens.
Scaling visualization. Fig. 7 shows heapmap of the scaled attention on the SST data (with task classes (positive, negative)) after training the model on News Series. For the corresponding positive/ /negiie classes, we observe that attention is also distributed on task-related tokens (e.g., charming, affecting) besides common tokens.
Task-agnostic evaluation. In Table 4, we evaluate models in the task-agnostic setting after task-aware training to show the separation of data representations across tasks. Both +T+ and Prescale perform better than FT. This may relate to their larger attention deviations and less over-smoothing (Fig. 6), which make data representations contain more information from non-sink tokens with different distributions across tasks. On BERT, PT+ outperforms Prescale. We hypothesize that this is because BEKT is pre-trained with the next sentence prediction, and thus [CLS] may contain more general sentence-level information across tasks than the learned scaling. On the other hand, for RoBERTa which does not have sentence-level pre-training, Prescale performs better than PT+FT.
Conclusion
In this paper, we study an attention sink phenomenon that can cause pre-trained models to perform inferior in CL tasks, despite their pre-trained capacity on downstream tasks. Specif- ically, we find that the small attention deviation on sink tokens and the fact that sink tokens are usually common tokens across tasks can cause the model having over-smoothing problem and easily propagate interference during CL. To mitigate such interference, we propose prescaling mechanism which first learns scaling layer during probing to allocate diverse attention on non-sink tokens, and then fine-tunes the model with scaling. Results show that our pre-scaling method outperform most CL models and other scaling strategies.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We thank the anonymous reviewers for their helpful feedback to improve the earlier draft of this paper. This material is based on research supported in part by the Air Force Research Laboratory (AFRL), DARPA, for the KAIROS program under agreement number FA8750- 19-2-1003.
References
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), Pp. 134,-55,,00 2018.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà, Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6523-6541, Barcelona, Spain (Online), Decem- ber 2020. International Committee on Computational Linguistics. doi: 10.18653/v1 /222200.. coling-main.555 URL https://www aclweb. wwwpo//an/ooog///////2222
www.:o/ma..i........n....mm 574.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. ar Xiv preprint ar XrX::2005..14166 14165, 2020.
Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arxiv preprint ar Xie:2006.13316 2020.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Effi- cient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019a. URL httpt:tt/tt:seeeeeete http://wwww.e/ooooo////////////.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. 2019b.
Cyprien de Masson d' d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. arXiv preprint ar 19Xio:1906.01076, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprin! 1xXiv:1810.04805 2018
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005), 2005.
Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need:
Pure attention loses rank doubly exponentially with depth. In International Conference Machine Learning, pp. 2793-2803. PMLR, 2021.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on Computer Vision and Paliern Recognition, pp. 831-839, 2019.
Wenbing Huang, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Tackling over- smoothing for general graph convolutional networks. arXiv preprint ar K1w:2000.00864, 2020.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learn- ing for text classification with information disentanglement based regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 2736-2746, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/ /v155//v//// 2202 maaac-mmin.221 218, URL https://aclantholl. org/2211 naacl moallmmain.22 218.
11

---

Published as a conference paper at COLM 2024
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. arXiv preprint ar x7io:2104.005888 2021b.
Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. Achieving forgetting prevention and knowledge transfer in continual learning. Advances in Neural Information Processing Systems, 01122433-22455 2021.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. ar preprint ar X11::1171..0066,, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwar Barninska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-352 2017.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform -ut-of-distribution. In International Conference Learning Representations, 2022. URL https: http://////eeeeeeeeeeeeee/e net/
Kuan-Ying, Lee, Yuanyi Zhong, and Yu-Xiong Wang. Do pre-trained models benefit equally in continual learning? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6485-6493, January 2023.
Sang-Woo Lee, Jin-HWa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tie Tak Zhang Overcoming catastrophic forgetting by incremental moment matching. arXiv preprint ar arXiv:1703.08475, 2017.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Tianlin Liu, Lyle Ungar, and Joao Sedoc. Continual learning for sentence representations using conceptors. Xiv preprint ar arXiv:1904.09187, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ar Xiv preprint ar *xXi:1907.11692. 2019b.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research, 24 2023.
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference Learning Representations, 2020. URL https://openreviee ntt/teeunt Sid 1tt Id02EFPr.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ar Xiv preprint arXio:1910. 10683, 2019.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019. URL https: ShAct7.
Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmooth- ing in graph neural networks. Xiv preprint arXio:2303.1099) 2023.
12

---

Published as a conference paper at COLM 2024
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arv preprint ar (tte:16060046771 2016.
Han Shi, JIAHUI GAO, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen
M. S. Lee, and James Kwok. Revisiting over-smoothing in BERT from the perspective of graph. In International Conference Learning Representations, 2022.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. arviv preprint ar Xixx:1705.08690, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic com compositinnality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Pablo Sprechmann, Siddhant M Jayakumar, Jack W Rae, Alexander Pritzel, Adria Puig- domenech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell. Memory-based parameter adaptation. arXiv preprint ar 2018.
Fan-Keng Sun, Cheng-Hao Ho, and Hung- Yi Lee. Lamol: Language modeling for lifelong language learning. ar Xio preprint ar Xii::1909.0329, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Infernational Conference Learning Representations, 2019. URL https://openreviie
Zifeng Wang, Zizhao Zhang, Yuen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision Pattern Recognition, pp- 139-149, 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapler of the Association for Compulational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018a URL htLp: org/ anthology/NN (aaii/ggg///111111/111 11011.
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018b. Association for Computational Linguistics. doi: 10.18653 10.18653/v1/N18-1111 11/1181101 URL https: //www. aclweb. org/ http//nnoooool/N11g1g
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. Pretrained language model in continual learning: A comparative study. In International Conference on Learning Representations, 2022. URL https://openreviiwwnn forum?id=figzpGtrdd
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. The Twelfth International Conference on Learning Representations, 2024. URL
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference machine learning, pp. 5153-5456666 PMLR, 2018
13

---

Published as a conference paper at COLM 2024
Chaoqi Yang, Ruijie Wang, Shuochao Yao, Shengzhong Liu, and Tarek Abdelzaher. Revisit- ing over-smoothing in deep gens. arXio preprint ar #rXiv:2003.13663, 2020.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint Xxio:1708.01547,, 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995. PMLR, 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances neural information processing systems, 28, 2015.
14

---

Published as a conference paper at COLM 2024
A Connection between Over-Smoothing and Attention Sinks
Denote P A (I- A and Amax is the largest eigenvalue of P. Denote the layer's attention matrix as A E Rox=" (n is the number of input tokens) and its ij-th element dij. The eigenvalue Ama is lower bounded by:
Amax
max
(aKi d:)²
where d; -k=1 4u/n n is the average outer degree of the i-th token.
Derivation. Each element in P can be written as:
- Agi) 1 { -
Pij k= [AaAkj
where dij represents the ij-th element in A. Because P is symmetric and positive semi definite, its max eigenvalue Amax is real and positive, which also satisfies:
Amax |xX| Px,
where x R" Set {x] = (x1, ... ] as a set of unit vectors, where each x; has the element as at the i-th place and others as Then Amax lower bounded as:
Amax 2 max Li=1 (aka d;) (aks + d;)
= max Lk=1 (aki d:)² + 2d: Ex=1 aki -d)
by the definition d;
(au d
max
The RHS above can be further decomposed max; d? Ei_ (axa - / to reflect the effects of the outer degree and the per-degree attention deviations.
There are several cases that can make Amax small. When the largest average degree d; is large, Amax is small when its per-degree attention deviations are small. On the other hand, when the largest average degree is small, Amax can be small even when the per-degree attention deviations are relatively large. In the paper, we focus on the case when the largest average is large, for the observed attention sink phenomenon.
B Correlation between Sink and Non-Sink Token Embeddings
In Section 4, the interference B B2 depends on correlations between representations of the sink and non-sink tokens, which is related to correlations between their embeddings in X1 and X2. Here we empirically calculate the correlations (i.e., dot product) between embeddings of the sink and non-sink tokens in BERT and RoBERTa-base, to verify our hypothesis that (common) sink tokens' embeddings are nearly orthogonal to other tokens' embeddings. Results are shown in Fig.
For both BERT and RoBERTa, embeddings of sink tokens (e.g, [CLS] and [SEP]) have close to 0 correlations to most other token embeddings. On BERT, the punctuation" ' has negative correlations to many other tokens, while on RoBERTa the distribution of its correlations is also centered around We also randomly sample non-sink tokens from the vocabulary and show their embeddings' correlation distributions as reference. On BERT, the embedding of non-sink token 'exchange' tends have positive (non-zero) correlation to other tokens' embeddings. On RoBERIa, although the embedding of the non-sink token 'aution' has centered to correlations to other tokens' embeddings, it has large correlations up to However, the correlation distributions of cls (
) and sep (< (<\s>) tokens' embeddings have a much smaller range.
Compared to the correlation to other tokens' embeddings, we also compute the self- correlation on sink tokens' embeddings as shown in Fig. 9. The embeddings' self- correlations can not be ignored both BERT and RoBERTa. When the common sink
15

---

Published as a conference paper at COLM 2024
tokens are allocated high attention, the correlation between sink token representations from two tasks may be large, leading to interference in CL.
[CL.S]
[SEP]
exchennge
3130
31:0
BERT
313
300
1800
Commistonn
<ig>
aution
1400
NoBERRT
Figure 8: Correlation distributions between pre-trained sink token embeddings and all other tokens' embeddings based on BERT and RoBERTa. The right-most column shows the correlation distribution of randomly selected token embedding for reference.
RoBERTa
Frequency
Token
[SEP]
1303
102
Comslation
Correbation
(a) Distribation of loken's self-corrlations
(b) Sink loken's self-cuerelations
Figure 9: (a). Distribution of token embeddings' self-correlations BERT and RoBERTa. (b). Self-correlations sink tokens' embeddings.
C Detailed Experimental Settings
In Section 5, we train all models with task-incremental settings (i.e., training with the loss only on classes in that task), while evaluating them in both task-incremental and class- incremental settings. We perform all experiments on one Nvidia RTX A6000 machine.
We provide detailed experimental settings of baselines below:
Probing: We fix the encoder and only train the classifier. We train 5 epochs for each task in BERT and RoBERTa, with the learning rate 5e-4.
FT: We fine-tune the whole model, including the encoder and classifier. We train 3 epochs for each task in BERT, with the learning rate 2e-5; and train 5 epochs for each task in RoBERTa, with the learning rate 1e-5.
PT+FT: We first train the classifier with the same setting in Probing, and then train the whole model with the same setting in FT.
We first train the classifier and the scaling layer with the learning rate 5e-4 for 5 epochs, and then train the whole model with the same setting in FT.
Prescale:
IDBR: We train IDBR with the learning rate 3e-5 for 3 epoches per task. We follow the k-means memory selection rule, and the replay batch size is 16 (training batch size) x number of tasks in the memory.
CTR: We follow the settings in the original paper, training 5 epochs for each task.
16

---

Published as a conference paper at COLM 2024
L2P: We have the prompt pool with 100 prompt tokens and select 50 of them to prepend to the input. We train the model with the learning rate le-3 for 20 epochs for each task.
ER: We apply sparse experience replay with 1% replay ratio. At each replay time, we sample 32 samples from the memory and perform one-step gradient descent based on them.
A-GEM: We store all previous data in the memory. At each gradient step, we randomly extract 32 samples from the memory and apply the A-GEM gradient projection.
We fine-tune the model with ER and then adapt the model at the inference time. At the inference time, retrieve 32 nearest samples in the memory for local adaptation.
MBPA++:
17


Reviews:
[{"review_id": "LxgzWpeiEV", "reviewer": "Reviewer_JZWa", "summary": "The paper studies the attention sink and its effect on continual learning. It also compared BERT and RoBERTa for continual learning with attention sink.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Make the CL setting clear earlier**\n\nThanks for your suggestions. We will add a clarification of the CL settings earlier in the paper. \n\n**2. What is FGT?**\n\nSorry about the confusion. FGT is the forgetting rate defined in [3].\n\n**3. TIL is very much a solved problem and there should not be forgetting**\n\nFirst, we would like to clarify that our work focuses on the influence of an attention-sink phenomenon on CL, a characteristic of LM and its tuning, but not on the CL strategy outside the LM. Therefore, we study the CL model which does not have (*a*) experience replay and (*b*) storage of previous tasks’  parameters. Under the above restrictions, models may still forget in TIL, supported by recent CL works like [4]. We originally claimed this in Section 4.1 but it might not be clear enough. Thanks for pointing it out and we will further clarify this.   \n\nBoth methods in reference [1, 2] (from the review) are based on stored task models (masks) or task gradients, which do not satisfy our restriction above (and our method can be combined with them). In addition, we notice that forgetting is also observed for [2]’s method in [1]’s experiments, which may suggest that it is still an issue in TIL, even if we store the information of previous tasks. \n\n**4. Comparison to isolation-based method**\n\nAlthough isolation-based methods can have no forgetting in TIL, at inference time, they require specification of the task to select which isolated part to use. Therefore, they can not be directly applied to CIL for inference. However, our method focuses on a characteristic (attention sink) of LM and can be applied to CIL evaluation after the TIL training. And it can achieve improved CIL performance as well. The results are shown below.\n\n| Model | DB Split (CIL Eval) | Yahoo Split (CIL Eval) |\n|-|----------|----------|\n|  FT  |   15.90 ± 3.35  |   36.19 ± 4.81  |\n|  IDBR (w/ replay)  |   62.07 ± 12.68  |   40.20 ± 4.61  |\n|  Prescale (ours)  |   **70.38 ± 4.84**  |   **53.21 ± 4.46**  |\n|  FT-RoBERTa  |   18.71 ± 3.29  |   36.24 ± 4.37  |\n|  Prescale-RoBERTa (ours)  |**77.55 ± 10.99**|**53.51 ± 2.63**|\n\nThese results further support our claim that increasing attention diversity when tuning LM can help CL (for both CIL and TIL). \n\n[3] Continual learning with tiny episodic memories. Chaudhry et.al,  ICML WS 2019\n\n[4] An Empirical Investigation of the Role of Pre-training in Lifelong Learning. Mehta et.al, JMLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper studied continual learning from a new angle. Previous research on CL has not dealt with attention sink. \n\nThe results are ok although there is no major improvement.", "reasons_to_reject": "It is sometimes confusing whether you are discussing class-incremental learning (CIL) or task-incremental learning (TIL) when you talk about the attention sink's impact. It is only clear in the experiment section that your work is about TIL. Please make the CL setting clear earlier. \n\nIn TIL, if you use a parameter isolation-based method, there is not much impact. See [1] \n\nTIL is very much a solved problem. Its results mainly depend on individual task performance and knowledge transfer. It is unclear if the state-of-the-art TIL method is used, we can see any noticeable impact. \n\nWhat is FGT? Is it the forgetting rate? For TIL, there should not be forgetting. See  [1,2]\n\n[1] \"Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks.\" EMNLP-2023;\n[2] \"Beyond not-forgetting: Continual learning with backward knowledge transfer.\" NeurIPS-2022.", "questions_to_authors": "See the weaknesses section", "extra_scores": {}}, {"review_id": "xB1XvknkIy", "reviewer": "Reviewer_Gejt", "summary": "The paper studies the attention patterns of BERT and ROBERTA and they evaluate if an over-smoothing pattern in the token representations is influencing the learning patterns in a continual learning setup. Furthermore, they propose a mechanism for encouraging the models to have a more diverse attention behaviour. This model change improves the adaptation capabilities of the models to new tasks int a continual learning framework.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. A more comprehensive perspective on the impact of attention sinks**\n\nThanks for the suggestion. BERT and RoBERTa are generally fine-tuned for language understanding tasks, like those in the GLUE benchmark. We show BERT and RoBERTa’s attention deviation (related to attention sinks) and representational similarity (to measure the over-smoothing degree) on three more GLUE tasks in this [link]( https://drive.google.com/file/d/10QbuAjtZ5qJQqD1Ve0ATCooF3yZq-5qx/view?usp=drive_link) (Figure A). Results have a similar trend as shown in the main paper Figure 2.\n&nbsp;\n\nOn single tasks, we observe that if a model allocates too much attention to high-degree tokens (attention sinks), it may result in the over-smoothing phenomenon (reflected by representation similarity), which decreases the model’s single-task learning capacity [1]. When the attention sinks are on task-irrelevant tokens (e.g., common tokens including punctuations), there may be spurious correlations based on them during tuning, which degrade the model’s generalization ability [2]. \n\n**2. How to apply the proposed method and evaluations to the latest LLMs**\n\n- Our method and evaluation in this paper are generalizable to other transformer-based models. First, our pre-scaling layer is applied after the LM encoder, which can be easily adapted to different LMs. Second, our evaluation of attention sinks (attention deviations) are applicable to attention matrices of different LMs. As long as there are attention sinks on common tokens (e.g. as has been observed in new-generation LLMs [3]), our analysis of interference based on them is still valid.\n\n- We show the (pre-trained) attention matrices on LLama 3 8B of our classification tasks in this [link]( https://drive.google.com/file/d/10QbuAjtZ5qJQqD1Ve0ATCooF3yZq-5qx/view?usp=drive_link) (Figure B). The results show that LLama 3 also has attention sinks on the first token (common token), thus our analysis is still valid. However, for the latest (more capable) LLMs, we may need new tasks and benchmarks for CL evaluation, which are not well-explored in previous works. We will keep this in our future works.   \n\n[1]. Revisiting Over-smoothing in BERT from the Perspective of Graph. Shi et.al, ICLR 2022\n\n[2]. Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lens. Joshi et al, EMNLP 2022.\n\n[3]. Efficient Streaming Language Models with Attention Sinks. Xiao et. al, ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Firstly, the authors follow a principled approach to study the phenomenon and the issues it causes on the model behaviour and then they propose a sound and clean solution which also has a significant positive impact on model performance.\n\nThey also have a strong experimental design and framework that should be easily adaptable to other models. Also, the models are throughly evaluated and the results are interpreted in a clear manner.", "reasons_to_reject": "I think that the paper requires a more comprehensive perspective on the impact of the studied phenomenon in the tasks that BERT and Roberta are generally used.\n\nSecond, I would like to see a better coverage of how this work can be extended to other models and tasks and more specifically on the new class of language models that would have a bigger capacity of encoding the information from multiple tasks. It would also help to provide resources that would facilitate applying the proposed techniques and evaluations to other models.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wbafA5YMnU", "reviewer": "Reviewer_MZAH", "summary": "The paper presents a novel perspective on the performance of pre-trained language models (LMs) in continual learning (CL). The authors observe that despite higher single-task performance, RoBERTa does not consistently outperform BERT in CL settings. The paper introduces the concept of \"attention sink\" and its connection to over-smoothing, which may lead to interference in learning subsequent tasks. To address this, the authors propose a pre-scaling mechanism to encourage attention diversity and reduce interference.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. More experiments**\n \nWe show more experiments on (1). *Review* [1] including tasks with more classes; (2). *GLUE* including five GLUE tasks (more challenging), and two review tasks (more classes). \n|Model|Review Acc|Review FGT|GLUE Acc|GLUE FGT|\n|-|-|-|-|-|\n|FT|69.01±3.04|6.90±3.80|67.06±2.15|12.76±2.37|\n|IDBR|71.26±0.88|1.97±1.10|70.57±1.42|4.84±1.83|\n|Prescale|**72.56±0.66**|2.33±0.86|**72.20±1.55**|7.68±2.00|\n|FT-RoBERTa|68.90±3.33|8.92±4.16|65.07±3.60|18.20±4.26|\n|Prescale-RoBERTa|**74.11±0.48**|1.92±0.62|**74.45±1.54**|7.17±1.64|\n\nOur models consistently outperform baseline models on BERT and achieve further improvements on RoBERTa.\n\n**2. Computation**\n\nThe probing stage takes less than 40% training time and 30% GPU memory compared to full fine-tuning. \n\n**3. Assumptions in Sec 4.2**\n\nWe make assumptions in Sec 4.2 to show the effect of attention sinks in propagating interference. However, our Eq. (5) is generalizable to relevant tasks and the case where non-common tokens are relevant. In these cases, the interference can be caused by both common and other tokens, which depend on correlations between data of specific tasks and are hard to identify. Since we have observed the attention sinks on common tokens, we focus on their effects in the analysis. In experiments, we also test on relevant tasks (in News Series) with other related tokens.\n\nFor comparison to isolation-based models, please see Point 4 of the response to Reviewer JZWa.\n\n\n**4. Multi-head attention (MHA)**\n\nFor each head, our analysis holds and after concatenation for MHA, attention sinks will cause interference at corresponding dimensions. \n\n**5. Effect of common sink tokens**\n\nThe effect depends on the correlation between tasks. It may be harmful if sink tokens propagate unexpected interference between specific tasks. However, as in Sec 4.3, sink tokens may also make knowledge transfer easier. \n\nAs shown in the paper Figure 1, although the common sink tokens decrease after fine-tuning, the attention patterns deviate far from the pre-trained patterns, which may not well preserve the pretrained knowledge, causing the model to lose generalization and forget.\n\n**6. Stability vs. plasticity**\n\nIn our experiments, fine-tuning RoBERTa has higher averaged accuracy but more forgetting than BERT on News Series, which indicates it performs better on new tasks while forgetting more (lack of stability). \n\n[1]. Episodic Memory in Lifelong Language Learning. d'Autume et.al, Neurips 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a unique analysis of the attention sink phenomenon and its impact on CL, which is a novel contribution to the field of NLP.\n2. The proposed pre-scaling mechanism is theoretically sound and is supported by empirical evidence through extensive experiments on various CL tasks.\n3. The paper demonstrates improvements in CL performance without relying on experience replay or parameter storage from previous tasks, which is a substantial advancement in the efficiency of CL models.", "reasons_to_reject": "1. The performance advantage is not very pronounced, and it is hoped that verification can be done on more challenging tasks. Since most of the experiments in the paper are binary classification tasks, the absolute values of accuracy are already very high, making the differences between various comparison methods not significant enough.\n2. Does the two-step training process significantly increase computational costs?\n3. Is it reasonable to assume that tokens other than common tokens are definitely irrelevant in the unrelated tasks in Section 4.2?\n4. Is it reasonable to assume that unrelated tasks in the continual learning tasks are irrelevant? If tasks are completely unrelated, then directly learning separate models for each task would be the best solution. Therefore, the trade-off between Transfer and Interference analyzed in Section 4.3 is more important.", "questions_to_authors": "1. Is the analysis of single-head self-attention in Section 4.2 not without loss of generality for multi-head self-attention?\n2. Is the phenomenon that Sink Tokens are Usually Common Tokens in Section 3.2 beneficial or harmful to CL? If harmful, then Fig. 2 right indicates that the proportion of Sink Tokens that are Common Tokens decreases after RoBERTa fine-tuning, which should mitigate the damage to CL.\n3. Is the problem of RoBERTa in CL mentioned in the introduction caused by a lack of stability or plasticity? That is, is the performance low on new tasks or old tasks, or both? Specific experimental results analysis is hoped for.", "extra_scores": {}}, {"review_id": "Yus3pLjBwa", "reviewer": "Reviewer_ASC9", "summary": "This paper discusses continual learning from the perspective of \"attention sink.\" It initially notes that attention often concentrates on specific tokens for a single task. The authors suggest that this could cause interference in continual learning, as the sink tokens might hold information specific to one task. Consequently, they introduce a pre-scaling mechanism to promote diverse attention scores on sink tokens.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Non-CL baseline** \n\nWe list the results of training each task separately below.\n| Model | Yahoo Split | DB Split |News Series|\n|----------|----------|----------|----------|\n|  FT-BERT  |   92.25±0.04  | 99.87±0.01  | 83.72±0.53|\n|  FT-RoBERTa |   92.43±0.07 |  99.83±0.01  |85.01±1.54|\n\n We have also provided the MTL performance in the paper. \n\n**2. BERT and RoBERTa models are quite outdated**\n\nBERT and RoBERTa are still widely used in CL, especially for foundational analyses [1, 2]. We believe our analyses and methods are generalizable to larger models. First, our pre-scaling layer is applied after the LM encoder, which can be easily adapted to different LMs. Second, our analysis is general to different LMs. As long as there are attention sinks on common tokens (e.g. as has been observed in new-generation LLMs [3]), our analysis of interference based on them is still valid.\n\nWe show the (pre-trained) attention matrices on LLama 3 8B in our classification tasks in the [link]( https://drive.google.com/file/d/10QbuAjtZ5qJQqD1Ve0ATCooF3yZq-5qx/view?usp=drive_link) (Figure B). The results show that LLama 3 also has attention sinks on the first token (common token), thus our analysis is still valid. However, for the latest LLMs, we may need new tasks and benchmarks for CL evaluation, which are not well-explored in previous works, and we keep this in future works.   \n\n**3. Prevent forgetting the original pre-trained knowledge**\n\nThe current method preserves the pre-trained knowledge in the following ways:\n\n- The additional probing stage is proven to reduce the distortion of pre-trained representations [3], which in turn helps to preserve the pre-trained knowledge.\n\n- In our model, the pre-trained knowledge is used to pre-scale the weights of each token in the data. This helps to use the pre-trained knowledge for the current task’s learning, and may further reduce the distortion of pre-trained representations. As shown in the paper Figure 1, compared to fine-tuning, pre-scaling better preserves the attention patterns in the pre-trained model.\n\n[1]. An Empirical Investigation of the Role of Pre-training in Lifelong Learning. Mehta et.al, JMLR 2023\n\n[2]. Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study. Tao et.al, ICLR 2023 \n\n[3]. Efficient Streaming Language Models with Attention Sinks. Xiao et. al, ICLR 2024 \n\n[4]. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. Kumar et. al, ICLR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I believe the idea is logical. Interference is the primary reason for forgetting. Anything specific to a single task may cause this interference, including attention sink tokens. \n\n2. There are both empirical and theoretical analyses.", "reasons_to_reject": "1. This paper focuses on task-incremental learning. As noted in [1], a significant non-CL baseline in TIL is to train each task separately. This can help determine whether the proposed methods effectively balance transfer and interference.\n\n2. The BERT and RoBERTa models are quite outdated. I believe the community would be more interested in whether the proposed method remains effective with the latest LLMs\n\n3. Given that the pre-trained model has been updated, how does the current method prevent forgetting the original pre-trained knowledge?\n\n[1]: Continual Learning of Natural Language Processing Tasks: A Survey, Ke and Liu, arXiv 2023", "questions_to_authors": "See above", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper identifies the prevalence of attention sinks in Bert and Roberta, and hypothesises that they may lead to interference across tasks in continual learning set ups. To address this, the authors propose an approach to diversifying attention. Reviewers thought the work took an interesting new angle on continual learning, and demonstrated improvements in experiments The work could be further strengthened by using more recent mask language models, and evaluating on more tasks. Overall this is borderline, but leaning towards acceptance.
Decision: Accept