Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study
Authors: Mingxu Tao
Yansong Feng
Dongyan Zhao
Venue: ICLR 2023 poster (2023)
Keywords: Natural Language Processing
Probing Study

Abstract: Large pre-trained language models have helped to achieve state of the art on a variety of NLP tasks, nevertheless, they still suffer from forgetting when incrementally learning a series of sequential tasks. To alleviate this problem, recent works propose several models enhanced by sparse experience replay and local adaption, which yield satisfactory performance. However, in this paper we find that pre-trained language models like BERT have a potential ability to learn sequentially, even without any sparse memory replay. To verify the ability of BERT to maintain old knowledge, we adopt and re-finetune single-layer probe networks with the parameters of BERT fixed. We investigate the models on two typical kinds of NLP tasks, text classification and extractive question answering. And our experiments reveal that BERT can actually generate high quality representations for previous tasks in a long term, under extremely sparse replay or even no replay. We further introduce a series of methods to interpret the mechanism of forgetting and how memory rehearsal plays a significant role in task incremental learning, which bridges the gap between our new discovery and previous studies about catastrophic forgetting. Additionally, we provide both quantified and visualized results demonstrating that the representation space of BERT is always topologically organised, which guarantees its performance.

Published as a conference paper at ICLR 2023
CAN BERT REFRAIN FROM FORGETTING ON SEQUEN- TIAL TASKS? A PROBING STUDY
Mingxu T Yansong Feng'>, Dongyan Zhao
WWnnxuua Institute of Computer Technology, Peking University, China
CCetter for Data Science, Peking University, China
'The MOE Key Laboratory of Computational Linguistics, Peking University, China (thomastao, fengyansong, zhaody) @pku (ekok)@p.u.....oe..oomooo cn
ABSTRACT
Large pre-trained language models help to achieve state of the art on a variety of natural language processing (NLP) tasks, nevertheless, they still suffer from for- getting when incrementally learning a sequence of tasks. To alleviate this prob- lem, recent works enhance existing models by sparse experience replay and local adaption, which yield satisfactory performance. However, in this paper we find that pre-trained language models like BERT have potential ability learn se- quentially, even without any sparse memory replay. To verify the ability of BERT maintain old knowledge, we adopt and re-finetune single-layer probe networks with the parameters of BERT fixed. We investigate the models on two types of NLP tasks, text classification and extractive question answering. Our experiments reveal that BERT can actually generate high quality representations for previously learned tasks in long term, under extremely sparse replay or even no replay. We further introduce series of novel methods to interpret the mechanism of forget- ting and how memory rehearsal plays a significant role in task incremental learn- ing, which bridges the gap between new discovery and previous studies about catastrophic forgetting
INTRODUCTION
Continual Learning aims obtain knowledge from stream of data across time (Ring 1994 Thrun
1998 Chen & Liu 2018). As booming area in continual learning, iask-incremental learning re- quires model learn a sequence of tasks, without forgetting previously learned knowledge. is practical scene to train models on stream of tasks sequentially. avoiding to re-train on all existing data exhaustively once new task arrives. In natural language processing, although many large-scale pre-trained language models (PLMs) have ceaselessly achieved on new records on vari- ous benchmarks, they cannot be directly deployed in a ieskeincremental setting. These models tend to perform poorly previously seen tasks when learning model trained sequentially on text classification tasks may not be able to make any correct predic- tions for the first task after learning
ones. For instance, BERT BASE
ones, with almost-zero accuracy scores (d' A'ttumme al.) 2019). This phenomenon is known as catastrophic forgetting (McCloskey & Cohen 1989) French,
1999 Rosenstein et al. (2005). Many existing works design novel architectures or components to
alleviate the forgetting when learning incrementally (Kirkpatrick et al. 2017 Zenke et al. 2017]
Rebuffi al. 2017; Mallya Lazebnik
2018: d' d'Auttme al. 2019 Pfeiffer al.) [2200] /Sun
et 2020; Geng et al. [202]] et al. 2022, Qin et al. 2022). Among them, d Autume et al.
(2019) find that an NL. model augmented by sparse memory replay can refrain from forgetting to
great extent. Their method randomly samples 100 instances from old tasks for replay, after learn- ing every 10,000 unseen instances. Considering that their method can regain the ability to process previous tasks via merely 100 instances in 4 stepy] question comes our mind: Whether pre- trained language models like BERT really suffer from forgetting when learning a sequence of tasks?
!Code will be released /t/pttps:///ghhhb..c/////tt//tee
Tearners
WWith training batch of size 32, sampling 100 instances means it takes only 4 steps to recover the forgotten knowledge.

---

Published as a conference paper at ICLR 2023
Mehta et al. (2021) reveal that, under proper apper-parameter, models with pre-training can suffer less catastrophic forgetting than models without pre- pre-training. However, in this work, we specifically focus on the frequeney of memory replay. We wonder whether the BERT encoder can still maintain knowledge learned from previous tasks as it performs in d' Autume et al. 2019), with an extremely sparse replay frequency or even without replay.
Probing study has become a popular tool to investigate model interpretability (Tenney et al., 2019 Jawahar et al., 2019). For instance. Wu et al. (2022) probe the continual learning ability of model by comparing the performance of different PLMs trained with different continual learning strategies. In this paper, our main concern is to examine whether PLMs have an intrinsic ability to maintain previously learned knowledge in a long term. We track the encoding ability of BERT for specific tasks in a BERT before, during, and after learns the corresponding tasks. Comparing the probing results of models trained under different replay frequencies and trained without memory replay, we find that BERT itself can refrain from forgetting when learning sequence of tasks. This is somewhat contrary to existing studies about catastrophic forgetting, which further motivates to investigate how the representations of examples from different tasks are organized in the parameter space. Inspired by prior works (Gao et al. 2019 Wang et al., 2020a), we define the representation sub-space class as a convex cone, and provide an algorithm to acquire the narrowest solution. With this toolkit in hand, we find that: after learning several tasks without memory replay, the representation sub-spaces of classes from different tasks will overlap with each other. However, the sub-spaces of classes from the same task keep never-overlapping all along. The former explains the catastrophic forgetting in task mass-incremettll learning from a novel viewpoint of representations, while the the latter explains why BERT has a potential encode prior tasks even without replay.
Our main contributions in this work are:
(1) we conduct a thorough study to quantitatively characterize how the representation ability of PLM like BERT change when it continuously learns sequence of tasks. We are the first to track the encoding ability of previously learned tasks BERT when learning new tasks continuously. (2) Our findings reveal that BERT can actually maintain its encoding ability for already learned tasks, and has a strong potential to produce high-quality representations for previous tasks in a long term, under extremely sparse replay or even without memory replay, which is contrary to previous studies.
(3) We further investigate the topological structure of the learned representation sub-space within a task and among different tasks, and find that the forgetting phenomenon can be interpreted into two aspects, the intra-task forgetting and inter-task forgetting (Section 4 enabling us explain the contrary between our findings and previous studies.
BACKGROUND
Following prior work (Biesialska al. 2020), consider the musk-incrementll language learning setting as that a model should learn from a sequence of tasks, where samples of former tasks cannot be accessible during the training steps for later tasks, but samples of all classes in the current task can be acquired simultaneously.
Formally, the input training stream consists of K ordered tasks Ti, Tz. ..., TK. where we ob- serve ng samples, denoted by { (œ vt) training objective is general model fo + y which handles all tasks with a limited number of parameters by minimizing the negative log-likelihood averaged over all examples:
na drawn from distribution P& Y) of task Tg. Our J
1 N
((0)
In P (yk
x, 0)
where N EK ne is the number of all training examples.
2.1 INVESTIGATED MODEL
In Natural Language Processing, a model can be divided into two parts, a text encoder and task decoder, with parameters gene and gdee, respectively.
2

---

Published as a conference paper at ICLR 2023
Text Encoder Similar + (d Autume al. 2019) and Meta-MbPA (Wang et al. 2020b), we use BART (Devlin et al. 2019) as our text encoder, which produces vector representations according to given tokens.
In text classification, we take the representation of [CLS] token added at the first to aggregate in- formation of all tokens. For sequence of input tokens æ;, where 2;,0 [CLSS, BERT BASE will generate corresponding vectors { with L = | Therefore, we formulate the output of encoder model as: ( Vi,u u-
For extractive question answering, we take the task setting of SQuAD 1.1 (Rajpurkar al. 2016). as in previous work (d Autume et al. (019). The input tokens I; here are the concatenation of context x; ftx and query x: separated by a special token [SEP].
Task Decoder For text classification, add linear transformation and soft-max layer after BERT BASE encoder. Following dd Autume et al. (2019), we adopt a united decoder for classes of different tasks, and here Odee is the combination of (Wy)
exp (W foone (x+))
exp W vi
P = a(x;) EgEy exp (W, fooma fx)no(x4))
ΣyEn exp (,,) v2.0)
For question answering, the models extract span from the original context, i.e., determining the start and end boundary of the span. Our decoder for QA has two parts of linear layers Watart and Wond for the start and the end, respectively. The probability of the 1-th token in context as the start of the answer span can be computed as:
exp W
U.
(start o,, [
ctx x que
start
Lokk (WT ptxx
2j=1 exp
start
where _ is the length of context, and the probability of the end boundary has similar form. When predicting, we consider the probability distributions of two boundaries as independent.
2.2 SPARSE EXPERIENCE REPLAY
In reality, humans rely reviews keep long-term knowledge, which is based episodic mem- ories storing past experiences. Inspired by this, Gradient Episodic Memory (Lopez-Paz & Ranzato 2017) and other methods introduce memory module M the learning process. Training examples then can be stored in the memory for rehearsal at predetermined frequency.
Construction of Memory Every seen example added to the memory by a fixed rate y during training. If we sample N4 examples of the k-th task, in expectation there will be ym additional instances in M after learning from Tk-
Principles of Replay For experience replay, we need to set a fixed sparse replay rate r. Whenever the model has learned from Nor examples from current task, it samples [rNr]] ones from M and re-learns. We set storage rate 7 0.01 and replay frequency r 0.01 in all of our experiments to ensure comparability, the same prior work. In this paper, name model by REPLAY only if is enhanced by sparse memory replay without other modifications. We name a model trained on sequence of tasks without any memory replay by SEQ.
2.3 DATASETS
To provide comparable evaluation, we employ the same task incremental language learning bench- mark introduced by MbPA++. Its text classification part is rearranged from five datasets used by Zhang al. (2015), consisting of 4 text classification tasks: news classification (AGNews, 4 classes), classes), sentiment analysis (Amazon and Yelp. 5 shared classes), topic classification (Yahoo, 10 classes). Following d' A'ttum et al. (2019) and others, randomly choose 115,000 training and 7,600 testing examples to create a balanced collection. Since Ama- and Yelp both sentiment analysis datasets, their labels are merged and there 33 classes
ontology prediction (DBPedia,
3

---

Published as a conference paper at ICLR 2023
in total. In all our experiments, we evaluate model's performance on all five tasks and report the macro-averaged accuracy as prior work.
As for question answering, this benchmark contains 3 datasets: SQuAD 1.1 Rajpurkar al. 2016), TriviaQA (Joshi et al. 2017), and QuAC (Choi et al., 2018). Since TriviaQA has two sections, Web and Wikipedia, considered two different tasks, this benchmark totally consists of 4 QA tasks.
PROBING FOR INTRINSIC ABILITY AGAINST FORGETTING IN BERT
As mentioned in Section model can rapidly recover its performance of previously learned tasks, by memory replay on merely 100 instances (d Autume et al. 2019). If the model completely loses the ability encode prior tasks, it is counter-intuitive that the model can regain prior knowledge by 4 updating steps. We conjecture that BERT can actually retain old knowledge when learning new tasks rather than catastrophically forgetting. To verify this hypothesis, we first conduct pilot study.
We implement our pilot experiments on the text classification benchmark, employing BERT BASE with simple linear decoder as our model and training it under 4 different orders (detailed in Ap- pendix A).. Following previous probing studies (Tenney et al. 2019 Jawahar et al., 2019) to ex- amine BERT's encoding ability for specific tasks, we freeze encoder parameters after sequentially finetuning, re-initialize five new linear probing decoders and re-train them on five tasks separately. We find that evaluated on the corresponding tasks, every fixed BERT encoder combined with its new decoder can achieve a superior performance. Surprisingly. the macro- averaged accuracy scores of all tasks for orders are 75.87%+o 5..8% 73%. 76.76%+4..67 61%, 75. 19% 76. 76. which are close to the performance of multi-task learning model (78.89%40. 18%). However, previous works (Biesialska et al. 2020) show that sequentially trained models suffer from catastrophic for getting and sacrifice their performance experiments, in contrary to previous works, actually indicate that BERT may have the ability to maintain the knowledge learned from previous tasks in long term.
previous tasks when adjusting to new task. Our pilot
3.1 PROBING METHOD
To verify whether BERT can refrain from forgetting without the help of memory replay, we need tool systematically measure model's encoding ability for previous tasks when it incrementally learn sequence of tasks. One way is to compare the encoding ability of models at different learn- ing stages trained under two different settings, REPLAY and SEQ. For each setting, we consider to measure the performance before learning corresponding tasks can be regarded as baselines, which indicate BERT's inherent knowledge acquired from pre-training tasks. And then we can examine to what extent BERT forgets old knowledge, by comparing the results during and after learning corresponding tasks. Therefore, it is essential to track the change of BERT's task-specific encod- ing ability across time. We extract parameters of the encoder and save them as checkpoints at an assigned frequency during training. both REPLAY and SEQ, we record checkpoints every 5,000 training examples without regard to the retrieval memory subset.
For every checkpoint, we probe its encoding ability for every task Tk by following steps:
1. Add a reinitialized probing decoder to the parameters of BERT in this checkpoint.
2. Train the recombined model with all data in T 's training set Di with gene fixed, which means we adjust the parameters of probing decoder only.
3. Evaluate the score of re-trained models on the test set of Tg.
We re-train a compatible probing decoder on specific task without touching the encoder before evaluation. We use a linear decoder as probing network for text classification, and two linear bound- ary decoders for question answering, the same setting as MbPA++ (d Autume al., 2019) and Meta-MbPA (Wang et al.) (2020b). We have to mention that there still exist some controversies on whether we should use a simpler probing decoder or more complex one (Belinkov) 2022). Here, we adopt simple one-layer probing networks for two reasons. Firstly, a simpler probe can bring about less influence to the performance of re-trained models (Liu al.) 2019a] (Hewitt & Liang.
$ "Since every batch has 32 instances which is not divisible by 5,000, we save parameters at the closest.
hatches to scheduled points in order to refrain from unnecessary disturbance. We use accuracy scores for text classification, and F1 scores for extractive question answering.
4

---

Published as a conference paper at ICLR 2023
AGNews
Amazon
DBPeddia
Yahoo
Yelp
100
1.0
1.0
1.0
0.3
0.2
0.3
0.2-
30
Yap
ACloows
DiPrat
Amuren
Fabo
Y44
Acaloos
DiPeea
Amsrrc
Taloo
0.0
1elp
AGNooa
Amman
Falso
0.0
AGNoo
DRP
Anmano
0.3
Vol
AGWvas
1000
Valuo
Figure I: Probing results on five text classification tasks trained by Order 1, illustrated separately by the dataset: The abscissas indicate the training order of tasks.
TriviaQA (Wik))
ThiviaQA (Web)
QuAC
SQuAD
b.
0.4
0.3
0.5
1150000
0.6
0.5
0.6
0.6
0.4
0.2-
0.2
0.2
0.2
QuAC
9QUAD
Te-lish T.-Wii
THWiiii
Th-Wich ToWlil
Tak
Figure 2: Probing F1 scores on four tasks trained by Order 1, illustrated separately
2019), which enables us to focus on the encoding ability of BERT only. Secondly, our purpose in this paper is not to compare BERT's encoding ability among different tasks, but examine whether it forgets the knowledge of a specific task. Therefore, it is better to use the same single-layer decoder
as Autume et al. (2019) and Wang et (2020b), which can yield comparable results with them.
3.2 RETHINKING CATASTROPHIC FORGETTING
We are now able to quantitatively measure whether BERT model can maintain its encoding ability for previous tasks during iiskiincemmentl learning. by tracking the probing scores among check- points. It is also important investigate whether replay intervals have influence on BERT's encod- ing ability. We first set up a series of experiments on text classification described as below.
To compare with prior works (d' Autume al. 2019 Wang et al. 2020b), we retain consistent experimental setups with them, where the maximum length of tokens and batch size are set to 128 and 32, separately. We use the training settings of REPLAY ind''Autume al. (209)) the baseline, which samples 100 examples from M for replay every 10,000 new examples from data stream. As mentioned in Section 2.2 we control storage rate 7 and replay frequency both at 1%. To explore the impact of memory replay, we compare models trained under different replay intervals. We randomly select a subset S with [0.01N,r] samples from M after learning every Nor examples. Nor is set to (10k, 30k, 60k, 115k), and furthermore, we can consider Nor as too when training models sequentially. We employ Adam (Kingma Ba 2015) as the optimizer.
We use the method in Section 3.1 to evaluate the quality of the representations generated by BERT in every checkpoint. the set of BERT parameters have a stronger ability to encode specific task, we can observe a better probing performance. Here, for text classification, we depict the changes of accuracy scores on different figures according to task and training order. The results Order (de- tailed in Appendix A is shown in Figure I and the rest is illustrated in Appendix B Comparing the scores before and after the model learning specific tasks, we further obtain a new understand-
'The leftmost sub- figure depicts how a model's probing accuracy scores on the training set of AGNews are changing along with the training procedure. The following four sub-figures are for Amazon, DBPedia, Yahoo, and Yelp. We color the background into yellow since the model is trained corresponding task. Specially, Amazon and Yelp share the same labels, therefore, we color their background into light-yellow once the model is trained on the other task.
'The leftmost is TriviaQA (Wiki), followed by TriviaQA (Web). QuAC, and SQuAD. The Fl scores after re- re-training probing decoders are represented by blue lines. As comparison, we draw Fl scores of models with original decoders by red dashed lines since the models begin to learn new tasks. We color the background into yellow since the model is trained on corresponding task. Specially, TriviaQA (Wiki) and TriviaQA (Web) are actually subsets of one task, therefore, color their hackground into light-yellow when learning the other task.
5

---

Published as a conference paper at ICLR 2023
ing about the dusk-ineremental language learning: In spite of data distribution shift among tasks, BERT remains most of the ability to classify previously seen tasks, instead of catastrophic forgetting. This conclusion can also apply to SEQ, whose replay frequency is considered too. Although BERT's representation ability gets a little worse under a larger replay interval (such as 60k, 115k, +o0); it still maintains previous knowledge and can recover rapidly by sparse replay.
We also provide experimental results on question answering, which is more complex than text clas- sification. To examine whether BERT can still retain old knowledge on QA tasks, we adopt more strict experimental setting than d' Autume et al. (019). We train the model sequentially with different orders in Appendix A under the setting of SEQ without any memory replay. On each task, the model is finetuned for 15K steps. which is two times more than a Autume et al. (2019). We then evaluate the encoding ability of every BERT checkpoints by our probing methods. The results of Order 1 is illustrated in Figure 2 and others in Appendix [ Based on our experiment settings, the model is finetuned for enough steps to overfit on every task. However, the probing results (blue lines) are still much higher than the original scores measured before re-training de- coders (red dashed lines). Comparing the obvious gap between them we can find that BERT still keeps most of knowledge of previous tasks when learning new ones.
Additionally, we also investigate the ability of other pre-trained language models to retain old-task knowledge, which is detailed in Appendix D In general, all of these pre- language models have an intrinsic ability to refrain from forgetting when learning a sequence of tasks, although our investigated models have various attention mechanisms and various scales. Among different training orders, they still maintain the ability to encode the first learned task, even after learning 5 tasks.
A NEW VIEW OF FORGETTING
From the experiments in Section 3.2 we ob- serve that BERT has the potential to keep long-term ability to provide high-quality rep- resentations for task, once the model has learned it. Thus, it seems that we only need to finetune the decoder if we attempt recover the model's ability for previous task. But on the other hand, the SEQ models suffer from a seri- ous performance degradation on learned tasks, which known as catastrophic forgetting. To reconcile this contradiction, we employ t-SNE toolkit (van der Maaten & Hinton| (0088 and vi- sualize the representations after training on all tasks by SEQ or REPLAY (Figure B.. When learning sequentially, it shows the model pro- duces representations different tasks in overlapped space, In this circumstance, the task decoder identifies all vectors instances from new task, which leads to confusion but can be averted effec- tively by sparse replay.
(a) SEQ
(b) REPLAY
Figure 3: Visualization results of representation space after training on tasks by Order 1. Points of AGNews, Amazon & Yelp, DBPedia, Yahoo are colored by blue, orange, green, pink, respectively, while intersection areas of multiple tasks are grey.
All these observations push us make the assumption that the forgetting in inskeincremental learn- ing can be considered as two parts, intra-task forgetting and inter-task forgetting. The intra-task forgetting describes whether model can still generate meaningful representations for prior tasks af- ter learning new ones, while the inter-task forgetting refers to whether the representations produced for different tasks are distinguishable from each other. In this section, we first propose a toolkit to describe the representation (in Section 4.1 Then, we exhibit the changes of a model learning continuously under REPLAY settings, and provide a novel understanding for catastrophic forgetting in NLP models. Admittedly, question answering models usually involve interactions among repre- sentations of different granularities (from token-level to even document-level) (Wang et al., 2018)) thus is more challenging to analyze. Therefore, we will put more emphasis on analysing the results of text classification.
"I QA, the Fl scores on previous tasks will not decrease to zero when learning new tasks, since all QA tasks share the same answer boundary decoder. But different text classification tasks utilize different dimensions in the decoder, which leads more drastic deterioration on scores of old tasks.

---

Published as a conference paper at ICLR 2023
4.1 DEFINITION OF REPRESENTATION SUBSPACE
As claimed in Gao et al. 2019) and Wang al. (2020a), when trained by single-layer linear decoders, pre-trained language mod- els produce token-level embedding vectors in a narrow cone. We observe that this con- clusion applies to not only token-level rep- resentations but also sentence-leve represen- tations (more details in Appendix E). Repre- sentation vectors of the same class aggre- gated together, which enables us to use con- vex cone cover these vectors, whose ver- tex is the origin. To describe the vectors pre- cisely, the cone should cover all vectors and be as narrow as possible. Formally, we denote the surrounding cone as:
Algorithm 1: Calculating the Representation
Cone
Input: vector set V, input size n = //. initial central axis Co. learning rate a, termination
condition e
Output: central axis of the cone while [V] > [0.95n] do
Initialize Co
repeat
Compute optimization objective by Eq.
Obtain the gradient V fe V).
C c aVVcs V)
c c/c/|cll2 Adjust a by linear search.
until Vej in Ac₃- <
Calculate the cosine w, and c, denoting as
x E dR
8
{s:)[=) [2] Sort [s: { Ki
[1]
m + [(V] [0.95m 1)/2]
where c E Rd is the central axis of the cone, and 8 controls the filed angle.
Select lowest s; and their relevant vectors ydel
V V- Vdveel
To acquire the narrowest cone containing all vectors output by BERT, supposing the vector is {a } we solve the optimization objective described as below:
Co c c end while
ofc
2 6, minimize -5; s.t. Vo, E V, ||
||e||2 =
(2)
where || |122 means L2-norm. obtain a definite solution, we add a restriction |||||2 1, otherwise the equation implies the direction of only without length. The representation vectors clustered, so we can obtain a cone with tiny file angle (6 >> 0). Therefore, Eq. (2 is convex optimization objective, which can be solved by Sequential Least Square Programming (Krafil [988 Boggs & Tolle 1995). In iteration, we acquire the optimization gradient by following expression:
fo(c, {v:)__)) = max v
Vo(c,, (0:)12))
v
arg max
c
|
Furthermore, to reduce the interference from outliers caused by noisy annotations, we modify the constraint conditions that the cone only needs to cover no less than 95% training examples. Since it violates the convexity of the original objective, we employ an iterative method and get an approximate solution, which keeps every calculating step convexity-preservve outlines the detailed solving procedure. It is obvious that cone axis should be at the center of vectors, thus initialize co Σ, v. E vell2-
Algorithm
4.2 INTRA-TASK FORGETTING
From the results in Section we find that BERT can maintain previously learned knowledge in a long term. When working with a re-trained new decoder, BERT can still perform well on prior tasks, indicating that BERT rarely suffers from intra-lask forgetting. To investigate the mechanism preventing BERT from intra-task forgetting, forgeettin... we train BERT model on AGNews and Amazon as an example to analyse the changes within the BERT's representation space. We first train the model on all instances of AGNews, and then sample 30K instances from Amazon as the second task for task-incremental learning. Similar Figure [ BERT can still generate high-quality representations
CCoose dictionary order.

---

select its n nearest neighbors from D (0) by Euclidean distance, and record their indica- (0) Published as a conference paper at ICLR 2023
for AGNews after learning Amazon without Episodic Replay. We guess after learning new task, the representation sub-space of old tasks is still topologically ordered im
can see that, without Episodic Replay, the representation vectors of old-task instances will rotate to the overlapping sub-space of the new task, which causes the decoder cannot distinguish which task the input instance should belong to. On the other hand, if adopt a task-specific decoder (e.g., the probing decoder), it can effectively determine the class of a given instance. This may imply that the vectors of the same old-task class are still not far from each other, but they are far away to the vectors of other classes from the the same old task. Therefore, we guess if two representation vectors are trained be at adjacent positions, they will still be neighbors after learning new task.
As shown in Figure ((a)
To examine whether the rotating process of old-task representation vectors is topologically ordered, we first need a metric to define the relative positions among the representations of instances in the same class. Following our method in Section [4.]] we can describe the representation sub-space of a class convex cone, whose cone axis is Cy. Then, for instance i of class y, we can define the relative position of its representation vector By. i as the cosine between Vy and Cy.
Since we need to compare the relative positions of every instance at two checkpoints (before and after learning the second task), we distinguish the vectors at different checkpoints according to their superscripts. Formally, we denote the cone axis and the representation vectors before learning Amazon as
(1) respectively. e and v y. with the ones after learning Amazon as cy and v
(0)
For every 2 (0) y. in the y (the universal representation set of class y before learning Amazon), we v.a
tor set as N, It is reasonable to believe that these n neighbors have the most similar semantic
information uy. Then, we can check whether (..) and the vectors {
(1) I
y. REN,,. are still neigh- bors, to verify whether the representation sub-space of class y is topologically ordered. Here, we compute the the correlation between the relative positions of (1) and { (" KEN,,." which is esti-
mated by Pearson correlation coefficient between cos(c)) (1) o and ΣkEN, cos(c))) (1) yg.k). We list the results of all classes in AGNews with different scales of n in Table [ (where y (Class-1, Class-2, Class-3, Class-4), n E {5, 10, 25, 50, 100}). By comparing different n, we can see median size of neighbors brings a better correlation, which restrains randomness from tiny set and uncorrelated bias from huge set. Altogether, the influence of n is inessential and we y. and its neighbors are still close after learning new task, since the Pearson coefficients are no less than 0.483 (partly higher than 0.723).
can reach the conclusion that the positions of a !
In other words, if two examples are mapped to near positions before learning new tasks, they will remain close with each other after learning new tasks. Once BERT has learned a task, it will tend to generate representations of the same class at close positions, while generating representations of different classes at non-adjacent spaces. Therefore, if the rotating process of old-task representations can keep topologically ordered, the representation vectors of a class will always be separate to the vectors of other classes. This is why BERT exhibits an aptitude to alleviate intra-task forgetting in our study.
4.3 INTER-TASK FORGETTING
Neural network models always suffer from catastrophic forgetting when trained on a succession of different tasks, which is attributed to inter-task forgetting in this work. Similar to prior evaluation, we continue to use covering cones to investigate the role of memory replay when models resisting inter-task forgetting.
"iGven a won--nneenn con-e vector set V. we can cluster it into many disjoint sub-sets, Vi, ....... D* by the distances between vectors. After learning new task, the representation vectors of previous tasks will rotate to new directions. For any sub-set V and any new vector u within y", if any new vector ey E y" is closer to ef than any vectors og ' (g # p) in other sub-sets, we will think the rotating process of representation vectors is perfectly topologically ordered when learning new task.
8

---

Published as a conference paper at ICLR 2023
Table 1: Pearson correlation coefficient (x 100) of the angles of U1 and its neighbors to the cone axis. The highest scores are made bold, with the second underlined.
Class
Class 2
Class 3
Class 4
5
81.09
48.35 9.82
83.1 43.57
12.41
10
81.68
50.44
83.90
73.8043.22
25
81.10,3.19
51.46_10.27
83.7643.58
73.98
50
80.03+3.30
51.06+10.56
83.25+3.65
73.3943.12
100
78.51 43.49
50.16+10.58
83.27
72.35
When a model decodes representation vector e via linear layer connected by soft-max, the decoder can be regarded as set of column- vectors (i.e. {wy hyey in Section 2.1) and the predicting process is equal to selecting one hav- ing the largest inner product with There- fore, it is necessary to check whether the cones of previous task rotate to their corresponding column-vectors in decoder. In this section, we still examine the model trained on AGNews first and continuously trained on Amazon with a replay interval of 30K for three times.
0.3
Replay 1st
0.2
Replay 2nd
Replay 3rd
0.1
Rotation Angle
0.0
-0.1
-0.2
-0.3
-0.4
We observe that there is no significant change of veltonnvectoor decoder before and after memory replay, since their rotation angles are less than 10-3, which are negligible. For each time t, denote the cone axis of class & before and after replay as cT and ct. respec-
2
4
2
3
5
AGNows
Amazon
Figure 4: Bar chart for rotation angles during re- play, clusters by task label and colored according to replay time.
tively, and its corresponding column-vector in decoder as Wy. Then, the rotation angle of the k-th be estimated as: AÇe, cos(C, wx) cos(c, w&). If AGr, >> 0, it means cones rotate closer to the direction of w during replay. The results illustrated in Figure reveal that memory replay obliges the vectors of previous tasks rotating to their corresponding column-vectors in decoder efficiently, while dragging those of current task deviate from optimal position. Further- more, this dual process weakens along with the increase of replay times. Since the representation space of BERT is nigh-dimensional while our tasks finite, alternately learning memory and current tasks can separate encoding vectors by mapping them to different sub-spaces.
In Appendix F we provide more visualization results about how memory replay reduces inter-task forgetting, in other words, catastrophic forgetting in the traditional sense.
5 CONCLUSION
In this work, we conduct probing study to quantitatively measure a PLM's encoding ability for pre- viously learned tasks in task-incremental learning scenario, and find that, different from previous studies, when learning a sequence of tasks, BERT can retain its encoding ability using knowledge learned from previous tasks in a long term, even without experience replay. We further examine the topological structures of the representation sub-spaces of different classes in each task produced by BERT during its musk-incremental learning. We find that without memory replay, the representation sub-spaces of previous tasks tend to overlap with the current one, but the sub-spaces of different classes within one task are distinguishable to each other, showing topological invariance to some ex- tent. Our findings help better understand the connections between new discovery and previous studies about catastrophic forgetting.
Limited by the number of tasks, we have not discussed the capacity of BERT when continuously learning more tasks. As far as we know, there no existing method yet to measure whether a model has achieved its learning capacity and cannot memorize any more knowledge. In the future, will extend our probing method to a longer sequence or different types of tasks and explore what amount of knowledge large pre-trained language model maintain.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This work is supported by the National Key R&D Program of China (No.20204AA0106660), and the NSFC Grants (No.62161160339)
REFERENCES
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational doi: (0.1162/coli..00422)
Linguistics, 48(1):207-216 March 2022.
URL https:/7
aclanthology. wwtp:22220099222222222
Magdalena Bicsialska, Katarzyna Biesialska, and Marta Costa-jussà. Continual lifelong learning in natural language processing: A survey, In Proceedings of the 28th International Conference on Computational Linguistics, pp. 1523-6541, 2020.
Paul T. Boges and Jon W. Tolle. Sequential quadratic programming. Acta Numerica, 4:4-12, 1995.
Zhiyuan Chen and Bing Liu. Lifelong supervised learning. In Ronald J. Brachman and Peter Stone (eds.), Lifelong Machine Learing, pp. 35 54. Morgan Claypool Publishers, 2nd edition, 2018.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2174-2184, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/11/D18-1 1/411-121. URL
ht.tps: a/c antho 1aoeoooogp ooy............................. wgt.olgg..ooooo1222/0
Kevin Clark, Minh- Than-Thang&A Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020. URL [
https: pettpps:/oppereeevee
www.nn/fooommdd--ooo xMH BtvP
Cyprien de Masson d' A'ttumee Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 13132-131 13142, 2019.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp.
2019.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3:128-135. 135, 1999.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In International Conference Learning Representations, 2019.
Binzong Geng, Min Yang, Fajie Yuan, Shupeng Wang, Xiang Ao, and Ruifeng Xu. Iterative network pruning with uncertainty regularization for lifelong sentiment classification. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '1, pp. 1229-1238, New York, NY, USA, 2021. Association for Computing
Machinery.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733- 2743, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 1/D19-11/ [tt.tttss:://e/t http://caaaatth/o// ogy. wwwp:/gggg//122200
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3651-3657 Florence, Italy, July 2019. Association for Computational Linguis- tics. doi: TO.1 10.18653/1//19.
htt.//ttts:://catthhto hww.athhoooogoooo//111111113
10

---

Published as a conference paper at ICLR 2023
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang- Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging cor- pora. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human I Language Technologies, pp. 4764-4780 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v 12022. naacl-main. 351. URL httphhss//c..aaaoloooooooo org org/ 2022 naacl main. 351
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I; Long Papers), pp. 1601- 1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/ v1/P17-1147. htLhttps::/ //cp:/acllaa aocccannhollggo ott.ooogo....111/////.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-35 -3566, 2017.
D. Kraft. A software package for sequential quadratic programming. Technical report, DLR German Aerospace Center - Institute for Flight Mechanics, Koln, Germany, 1988.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeece pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line, July 2020. Association for Computational Linguistics. URL https: btttps://cclanthologg org 20w./220...o acl-main. 703
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Lin- guistic knowledge and transferability of contextual representations.
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 1073-1094, Minncapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-11 oog.aaaooooooooooo///11111-111
URL
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Lewis Mike, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs//197.11166 2019b. URL Utth:ttp:/aaxxxvvooo https///w..orr//////9/.... 11692
abs/1907
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:646776476, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7765- 7773, 2018.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24(C):109- 165, 1989.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical inves- tigation of the role of pre-training in lifelong learning. CoRR, abs/2112.09153,, 2021. URL
https://arxix>: orgo abs wttp://b///..../eeeeee
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint, 2020.
11

---

Published as a conference paper at ICLR 2023
Yujia Qin, Jiajie Zhang. Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. ELLE: Ef- ficient lifelong pre- pre-training for emerging data. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2789-28 2810, Dublin, Ireland, May 2022. Association for Computa- tional Linguistics. doi: 10. 10.18663/11/0022.om org/2022, f findi ins.innnnnnnnn... 220
http:///////2222.nnnnnn//e/22// URL
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 10.18653/ 1264. URL sttps://aclest
org/D16-1263
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 506-516, Long Beach, California, USA, 2017.
Mark Bishop Ring. Continual Learning in Reinforcement Environments. PhD thesis, the University of Texas Austin, 1994.
M. T. Rosenstein, Z. Marx. L. P. Kaclbling, and T. G. Dietterich. To transfer or not transfer. In Proceedings of the 5th International Conference on Neural Information Processing Systems, 2005.
Pan-Keng Sun, Cheng- Hao Ho, and Hung- Yi Lee. LAMAL: LAnguage modeling is all you need for lifelong language learning. In International Conference on Learning Representations, 2020.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019. URL id=8322gnRccX
net /fottm?
Sebastian Thrun. Lifelong learning algorithms. In S. Thrun and L. Pratt (eds.), Learning To Learn, pp. 181 - 209. Kluwer Academic Publishers, May 1998.
Tulia Ture, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, abs/1908.08962, 2019. URL /ttp:://axxii...aat./p org/ abs/1908. 08962]
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Lingxiao Wang. Jing Huang, Kevin Huang. Ziniu Hu, Guangtao Wang, and Quanquan Gu. Improv- ing neural language generation with spectrum control. In International Conference on Learning Representations, 2020a.
Wei Wang, Ming Yan, and Chen Wu. Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering. arXiv preprint arXiv: 1811. 1194,, 2018
Zirui Wang, Sanket Vaibhav Mehta, Barnabas Poczos, and Jaime G Carbonell. Efficient meta lifelong-lcarning with limited memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 535-548, 2020b.
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haf- fari. Pretrained language model in continual learning: A comparative study. In International Conference on Learning Representations, 2022. URLhttps:/ http:::::////peeeeeeeeee www.we.w.ettteeemme.e id=figzpGMrdD
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.
XLNet: Generalized Autoregressive Pretraining for Language Understanding. Curran Associates Inc., Red Hook, NY, USA, 2019.
12

---

Published as a conference paper at ICLR 2023
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings Machine Learning Research, 70:3987-3995, 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classi- fication. In Advances Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
A DATASETS AND ORDERS
For task-in mask-incrrmentll classification, we use the following orders train our models, which are the same as d' Autume et al. (2019) and Wang et al. (0220b)
Yelp
Yahoo.
2. DBPedia
3.
4. Y
For task-incremental question answering. we use the following orders to train our models, which are also the same as d' Autume et al. (2019) and Wang et al. (2020b):
2. SQuAD--Trivinne (Weh)->TriviaQ)) (Wiki)-
QuAv--TrivvaQA
(Web).
3. TriviaQA (Web)-+TriviaQ2 (irikii-SSQAA-QQQue
4. TriviaQA (Wiki) -aa---uAAC--iviir (Web)
Here, the Web part and the Wikipedia part of TriviaQA (Joshi et al. 2017) are treated as two separate datasets in the orders.
B PROBING ACCURACY SCORES OF ALL ORDERS FOR TEXT CLASSIFICATION
In this section, illustrate the probing results of all four orders in Figure Following the main body, background is colored by yellow when and after training on corresponding tasks. And spe- cially, since Amazon and Yelp share the same labels, color their background by light-yellow once the model is trained on the other.
C ANALYSIS FOR QUESTION ANSWERING TASKS
Similar to the analysis of text classification, also train models on 4 question answering (QA) tasks in designated orders. To verify whether BERT has a potential to keep knowledge in a long term in QA tasks, random sample 240K examples from each task (by repeated sampling), where their sizes are two or three times more than the original datasets. We set batch size as 16 and learning 10-5 without decay. Additionally, we do NOT use any memory module, which means the models are trained sequentially without memory replay.
rate
We save checkpoints every 1,250 steps, and then re-finetune the decoders 4 tasks respectively, with the parameters of BERT encoders fixed. Since, here, QA is formulated as a sequence-to- sequence task, there may be more than one golden answer span for a question. Therefore, use Fl score to evaluate the performance of models. All results are illustrated in Figure 6
The results imply BERT still has a durability to keep previously learned knowledge in a long term in more complex tasks like question answering. In QA, the model employs unified span position decoders for all tasks. therefore, the original F1 scores (before refintuning, red dashed lines) for previous tasks will not decrease to zero, which is different from text classification. Although the catastrophic forgetting problem is not too severe in QA, the models still achieve much better scores after re-finetuning their decoders, considering the gaps between blue lines and red dashed lines. the meantime, we find there is only a limited drop blue lines after the models finish learning from corresponding tasks. It means that BERT has satisfactory potential keep previous knowledge, even without any memory replay. Our conclusions in Section 5 can also apply to question answering tasks.
13

---

Published as a conference paper at ICLR 2023
AGNews
Amazon
DBPedia
Yahoo
Yelp
1.0
1.0
1.0
A20130
0.2
0.2
0.3
0.2
80
Yap
AChoes
DRPPaa
Amaren 1al0
Y:4
ACaloos
DiPeed
Amurac Ta00
AGNioe
1907 LiPPlt
Ammano
Falso
0.0
AGNos
Talon
14l
AGW
190 AiPPad
SBQ
REPLAY(USU
(a) Results of Order 1.
ACINews
DBPedie
Yahno
Accimyy
0
0.6
0.5
0.5
01.4
DisPidoo
Volos
AtNews
Anutia
Dinids
Attome
Amaunt
DtPPads
Vahio
ADNen
Amunnt
Disuudo
Amsangg
Der
Action
REPLAY(IO)
REPLAY 2000
REPLAYI0R)
REPLAY(LLK
(h) Results of Order
AGNews
Amazon
DBPedia
Yahoo
Yelp
1.0
1.05
1.0
1.5
1.3
0.5
Acamsey
100
Amtron
Uniiala
AGNevs
Palio
Amaraa
DEPada
AGiion
Villoo
Amaron
AONow
Velp
Talno
Amsunt
AcHoon
Yol
Vidls
Amarr
Difeda
SBQ
REPLAY(KK)
REPLAY(S03
REPLAY
(c) Results of Order 3.
AGNoww
Amazum
DBPedis
Yahoo
Yelp
Actmmyy
0.2
0.2
0.2
0.2
ADNe
Amssue
Valus
Amaro
Yahoo
DiFeaaa
Amsrron
Yabu
DR0
0.0
AGttee
Ammano
Yahoo
DRPP
ADNiess cp
Alarns
Ydno
REPLAY(M0)
REPLAY(UUU)
(d) Results of Order
Figure 5: Probing results of five text classification tasks training by each order. each row, we illustrate the results for 5 tasks separately, where the leftmost is AGNews, followed by Amazon, DBPedia, Yahoo, and Yelp.
14

---

Published as a conference paper at ICLR 2023
TrivisQA (Wik))
TrivisQA (Web)
QuAC
SQuAD
1.0
1.0
1.0
1.0
11 11.0m
6.6
0.5
04
6.4
0.7
0.2
0.7
Q6AC
T:WWo THNWI SQUAD
6.0
QUAC Tr- T-WKi SQUAD
0.0
QUAC
Te
Tr Wii SQUAD
8:0
Q6AC
Tr-web
To-33 SQUAD
7art
Yaak
Task
(a) Results of Order 1.
1.0
1.0
1.0
1.0
11 11.00000
0.6
0.6
2.5
0.2
0.2
0.2
0.2
SQuAD
Q6CC
To-Web
SQUAD QUAC
9QUAD
QAAC
SQUAAD QUAC
786k
(a) Results of Order 2.
1.0
3.0
0.5
08-
0.5
31 1130000
0.6
0.6-
0:5
64
0.2
0.2
0.2
5.2
SQUAD QuAC
Te-Ws SQUAD
To-Web T--ile
To-Wih
SQUAD
(b) Results of Order 3.
3.8
0.5
05-
3.8
11.5000
0.4
0.4
0.4
0.2
Te-
SQUAR
Wik
Trak 5.0
Te-Wob
Th-Wit
QuAC
To SQUAD
Lack
Takk
(c) Results Order 4.
Figure 6: Fl scores on four tasks trained by 4 different orders. each row, we plot the results for four tasks separately, where the lefumost is TriviaQA (Wiki), followed by TriviaQA (Web), QuAC, and SQuAD. The scores after re-finetuning decoders represented by blue lines, and as a com- parison, we draw Fl scores before re- re-finetuning decoders by red dashed lines. We color the back- ground into yellow since the model is trained on corresponding task. Specially, TriviaQA (Wiki) and TriviaQA (Web) are actually subsets of one task, therefore, color their background into light-yellow once the model trained on the other task.
15

---

Published as a conference paper at ICLR 2023
Table 2: Probing results of various PLMs
X
PLM
AGNews
Amazon
DBPedia
Yahoo
Yelp
BERT-tiny
92.46
55.91
98.70
71.67
57.45
BERT-mim
93.71
58.50
99.01
72.62
60.24
BHRT-small
92.01
54.05
99.09
73.41
61.17
BFRT-med
60.45
99.21
73.76
61.42
Upper
BERT-base
94.50
62.41
99.32
75.08
62.76
BERT-large
93.93
62.89
99.17
71.58
63.96
RoBERTa
94.49
63.21
99.24
74.76
64.75
ELECTRA
94.74
63.50
99.24
75.34
64.57
BART
94.50
62.50
75.05
64.04
GPT-2
94.34
61.17
99.14
74.28
63.04
XLNet-base
94.30
62.84
99.16
74.58
64.34
BERT-tiny
81.28/11.18
32.00/233.1
85.66/-13.04
49 49.42/22..25
36.08/21.37
BERT-mini
82.20/111.51
37.33/-21.17
94.43/ -4.58
54.70/-17995
41.82/-18.42
BERT-small
86.05/ -5.96
43.11/-10.95
97.39/ -1.70
61.00/-12.41
46.87/-14.30
BERT-med
85.78/ -8.36
43.61/-16.84
97,47/ -1.74
59.74/-14.03
46.92/-14.50
Lower
BFRT-base
80.03/-14.47
42.29/-20.1
86.61/-12.71
51.17/23.91
43.42/ 19.34
BERT-large
65.43//28.50
35.28/ 27.22
77. 77.84/221.3
29.20/44..30
35.24/ 28.72
RoBERTa
84.54/ -9.95
42.25/-20.96
88.34/-10.89
56.38/188.38
19.95
ELECTRA
72.66/-22.08
48.01/-15.49
80.63/-18.61
42.17/33.17
49.26/-15.30
BART
78.13/-16.37
43.07/-19.43
83.64/-15.59
47 67.688-27
45.25/-18.79
GPT-2
89.55/ -4.79
47.70/-13.4
95.11/ 44.04
66 28/ -8.00
47.74/-15.30
XLNet-base
88,50/ -5.80
50.75/-12.09
94.91/ -4.25
66.57/ -8.01
51.64/-12.70
BERT-tiny
87,80/ -4.66
40.32/-15.59
94.24/ -4.46
69.14/ -2.53
45.29/-12.16
BERT-mini
88.87/ -4.84
42.88/-15
15.662
97.12/ -1.99
71.99
-0.63
44.86/-15.38
BERT-small
90.95/ -1.07
50.91/ -3.14
98.53/ -0.57
73.45/ +0.04
$1,92/ -9.25
BERT-med
91.21/ -2.92
52.01/
-8.43
98.78/ -0.43
74.24/ 40.47
53.13/ -8.29
Ozde:
BERT-base
92.00/ -2.50
56.79/
-5.62
99.12/ -0.20
75.16/ +0.08
56.4M -6.33
BERT-large
92.43/ -1.50
59.51/
-3.38
98.84/ -0.33
75.831 44
59.01/ 4.95
RoBERTa
92.82/ -1.67
60.071
-3.14
98.70/ -0.54
75.45 +0.68
60.18/ -4.57
ELECTRA
91.50/ -3.24
54.79/
-8.71
97.57/ -1.67
76.01/ +0.67
55.18/ .9.88
BART
93.66/ -0.84
60.82)
-1.68
98.78/
75.64/ +0.59
61.03/ -3.01
GPT-2
92.54/ -1.80
57.11/ -4.07
98.82/ -0.33
74.37/ +0.09
$7.36/ -5.68
XL.Net-aas
92.97/ -1.33
-1.82 61.03/
98.38/ -0.78
75.33/ +0.75
61.86/ -2.49
BERT-tin
88.14/ -4.32
52.29/
-3.62
15.59/11..1
53.72/17.95
55,62/ -1.83
BERT-min
93.66/
54.17/-1..4
59.18/ -1.05
BERT-small
90.61/ -1.41
58.36
14.30
98.33/ -0.76
64.22/ -9.18
61.00V 40,17
BFKT-med
91.28/ -2.86
59.89/ -0.55
98.50/ -0.71
64.49/ -9.28
61.89/ +0.47
0rder_2
BERT-base
91.54/ -2.96
61.75/ -0.66
99.01/ -0,30
64.38/-10.70
63.00/+0.22
BERT-large
92.39/ -1.54
62.09/ -0.80
97.80V -1.37
68.61/ -2.97
64.68/+0.72
RoBERTa
93.34/ -1.14
63.12) -0.09
98.08/ -1.16
69.71/ -5.05
64.88/40.J
ELECTRA
92.36/ -2.38
62.95/ -0.55
97.11/ -2.13
60.43/-14.91
65.09/ 40.53
BART
93.26/
62.72/ +0.22
-1.18
69.55/ -5.50
64.53/ 40.49
GPT-2
92.71/ -1.63
60.88/
-0.29
98.42/ -0.72
70.51/
-3.76
63.61/ 40.57
XI. Netehhh
92.91/ -1.39
-0.24 62.61/
98.51/ -0.64
+3.24 71.34/
65.34/ +1.00
BERT-tiny
91.39/ -1.07
16.46
94.46/ -4.24
60.59/-11008
45.57/-11 88
BiRT-mini
92.80/ -0.91
16.70
96.61/
-2.41
48.61/-11.62
BERT-small
93.68/
54.83/
+0.78
98.59/
68.11/
-5.30
55.66/ -5.51
BERT-med
93.67/ -0.46
55.97)
.4.47
98. -1.05
68.71/ -5.05
55.50/ -5.92
Ordez.3
BERT-base
94.28/ -0.22
59.09/
-3.32
98.66/ -0.66
67.46/ -7.62
56,97/
-5.79
BERT-large
94.49/ +0.55
58.03/
-4.87
97.17/ -2,00
68.78/ -2.80
55.41/ -8.55
RoBERTa
94.87/ +0.38
60.78/
-2.43
98.91/ -0.33
70.92/ -3.84
60,92/ -3.83
ELECTRA
94.47/ -0.26
59,70/
+3.80
97.82/ -1.42
65 66/ -9.68
59.80/ 4.76
BART
94.53/ +0.03
-1.05
98.
-0.45
-1.97
62.03/
-2.01
GPT-2
94,25/ -0.09
98.87/ -0.28
72.41/ -1.87
S8.67/ -4.37
XLNet-base
94.83/ +0.53
62.09/
-0.75
98.58/ -0.58
73.18/ -1.39
61.30/ -3.04
BERT-tny
84.42/ -8.04
35.75/20016
98.07/ -0.63
66
26.26/
-5.41
42, 15.
BART-mini
85.43/ -8.28
39.89/-18.
98.82/ -0.20
68.83/
-3.79
45.42/-15.42
BERT-small
88.84/ -3.17
+0.04
70.96/
-2.45
53.67/ -7.50
BERT-med
90.25/ -3.88
55.32/
99,207 -0.01
72.38/
-1.38
55.551
-5.87
Order
BERT-base
90.91/ -3.59
59.62)
-2.79
99.33/ +0.01
73.78/
-1.30
60.17/
-2.59
BERT-large
90,50/ -3.43
59.25/
-3.64
99.33/ 40.16
74.07/
+2.49
61.50V -2.46
RoBERTa
91.42/ -3.07
+3.47 59.74)
99.41/ +0.17
73.41/
-1.36
59.63/
-5.12
ELECTRA
89,99/ -4.75
13.47/-10.03
99.241 -0.00
73.12/
-2.22
58.91/
-5.6
BART
92.39/ -2.11
60.53/ -1.97
99.29/ +0.05
75.16/
+0.11
60.82/ -3.22
GPT-2
91.84/
+0.01
-0.88 73.39/
58.54/ 4.50
XLNct-base
92.58/ -1.72
61.68/ -1.16
99.18/ +0.03
-0.30 74.28/
62.53/ -182
16

---

Published as a conference paper at ICLR 2023
D MORE PROBING STUDY ON OTHER PRE-TRAINED LANGUAGE MODELS
Our discussions in the main body are conducted almost exclusively on the ability of BERT BASE to keep knowledge. BERT (Devlin et al., 2019) is a representative of the PLM family, and widely used in various NLP tasks. We choose BERT in our study, since its transformer based architecture influences many other PLMs. However, it does not mean BERT is the particular PLM with the intrinsic ability to generate high-quality representations for previous tasks in a long term. In this section, we further investigate various other PLMs with different model scales, different pre-training procedures, or different attention mechanisms. For the pre-trained language models with different attention mechanisms or different pre-training strategies, we investigate RoBERTa-base (Liu et al.) 2019b), BART- (Lewis et al. 2020). ELECTRA LLECCTRA-baaeCClartt
base/Clark et al. 2020). XLNet-base Yang al. [2019) and GPT-2 (Radford et al. (2199) For the pre-trained language models with different scales, investigate BERT-tiny, BERT -iini BERT-small, BERT-medium, which are distilled versions
from Ture et al. (2019), and BERT-large from Devlin et al. (2019). Our probing experiments are detailed as below.
To reduce redundant calculations and to provide a concise quantitative analysis, we no longer track the encoding ability of a PLM at every checkpoint. Here, we only measure the encoding ability of a PLM which has learned all tasks sequentially without any memory replay. All the models employ a single-layer network as decoder, which is the same as Section [3.2] And we also train models with various PLMs with identical settings to former experiments. After sequentially training on the five text classification tasks, we save the parameter weights of PLM encoder and evaluate by probe-based method proposed in Section 3.1
We place emphasis on that di different PL should have different performances on task, even if they are trained under single-task task supervised paradigm. Therefore, we provide some results of control tasks (Hewitt& & Liang 2019) as a comparison. Specifically, we train every PLM on every dataset separately, where all parameters of encoder and decoder can be updated. These full-supervised re- sults on single task can be consider as the upper bounds. To check whether a PL.M itself can handle these text classification tasks well without downstream fine-tuning we also present some zero-shot probing results as the lower bounds. We download the weights of various PLMs without any fine- tuning from open-source platform. Then, we train decoders for every task separately, while keeping the original PLM weights fixed (actually probing study under zero-shot scenarios). Comparing with the results of control tasks, we can examine whether other PLMs can retain knowledge of previous tasks like BERT, after learning sequence of tasks.
We list all results (including the upper bounds and the lower bounds) in Table 2 From them, we can find that although these PLMs have various attention mechanisms and various scales, they have a similar intrinsic ability to keep previously learned knowledge. Although trained without Episodic Replay, these PLMs can all gain much better probing results than the lower bounds, without regard to training orders.
Comparing the results of BERT with different scales, we can find that, without Episodic Replay, the encoders with more parameters (e.g., BERT-base and BERT-large) have a little better abili- ties to maintain old-task knowledge than those with fewer parameters (e.g.. BERT-tiny and BERT- mini). However, among the encoders with similar scales but different architectures, including BERT- base (Devlin et al. 2 GPT-2 (base) (Radford et al.) 2019). BART (Lewis al., 2020). XLNet- base (Yang et al. 2019), they also have a similar ability to maintain old-task knowledge. Therefore, we guess this intrinsic ability to refrain from forgetting partly comes from the scale of model, while differences of model architectures (e.g., Transfoomer-Fnnoode v.s. Transformer-Decode make no obvious contributions.
E STRUCTURE OF REPRESENTATION SPACE
As Gao et al. 2019 and Wang et al. (222aa) mention in their work, large pre-trained language model will embedding all words in a narrow cone when trained with a decoder like that in Sec- tion 2.1 Following their opinions, we conjecture the pre-trained language model can also generate sentence-level representation vectors of the same label in narrow cone. To verify this consideration, we can check the cosine of any arbitrary two vectors produced by BERT. We select AGNews (Zhang et al. 2015), which has four classes, for investigation. We train model with BERT and linear
17

---

Published as a conference paper at ICLR 2023
0.03
0.02
0.02
0.03
0.01
CLASS
CLASS
CLASS
CLASS
Figure 7: Cosine distribution of vectors pairs from classes of AGNews, with axes aligned.
decoder on AGNews for one pass, and then store the representation vectors of training set by class respectively. For the i-th and the j-th class (1 < i S j S 4), randomly sample vector from each of them for IM times. And then, we can approximate the cosine distribution of two vectors from two classes, which illustrated in Figure
From the results, is obvious that two vectors sampled from the same class have near directions (co- sine between them almost 1). while two sample from different classes have visible discrete direc- tions. It implies the representation sub-spaces are anisotropic, therefore, we can describe them by using convex cones.
ADDITIONAL VISUALIZATION RESULTS
Figure 8: Additional visualization results of the representation space during lifelong learning, with points of AGNews and Amazon colored by yellow and blue respectively. Specially, we color the mixed area green, whose size should be smaller when the model has better ability to distinguish different tasks. From left to right, these columns are corresponding to the time of just finishing learned from AGNews, and the first, the second, the third replay. The top row is results before replay, while the bottom after replay.
In this section, visualize the change of representation space before and after memory replay during lifelong learning. Following the experiment setting in Section we first train the model on AGNews, and then Amazon with replaying three times. We save all representation vectors after
18

---

Published as a conference paper at ICLR 2023
learning AGNews, and every time before after replay. Then adopt t-SNE (van der Maaten & Hinton 2008) to draw all vectors the plane. Concerning the mixed areas of both classes, we can conclude that memory replay plays significant role to mitigate inter-task forgetting. Every time after replay, the model have a stronger ability to distinguish instances from different tasks, which is characterized by decrease of green area in Figure Also, comparing the results among columns, can confirm although it brings little confusion among tasks when learning one task continuously without break, sparse memory replay can eliminate the confusion effectively. Therefore, BERT model enhanced by memory replay can resist not only intra-task but also inter-task forgetting.
19


Reviews:
[{"review_id": "xuU2kOc2e8", "reviewer": "Reviewer_t1W5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work analyzes catastrophic forgetting during different multi-task learning strategies, specifically sequence training and replay. The authors test BERT on a suite of classification tasks and question answering, and use linear probes to test the ability of the model on each task before training, after training and after replay. They find that the sequential model does as well as the replay models however, t-SNE plots reveal that the sequential model often confuses representations across tasks. Next, they look at the angle between an exemplar and the mean of the class to characterize how the topological ordering changes due to replay. Finally, they look at the effect of replay in both the current dataset and the one being replayed.", "summary_of_the_review": "Overall, the paper presents an interesting set of tools to analyze the problem of catastrophic forgetting in neural LMs. However, the results and claims seem contradictory and it is not clear to me if the authors are claiming that purely sequential learning *suffers* from forgetting or it does not! To that end, I believe that these tools can be used to more systematically understand how the replay and sequence strategies differ in the geometry of the representations given that they both do well for the tasks described here.", "strength_and_weaknesses": "I appreciate the authors' detailed response. I've read through the comments and have updated my score accordingly.\n\n##################################\n\nStrengths:\nThe paper presents an interesting problem and novel use of the solutions prescribed to analyze it.\n\nWeaknesses/Questions:\n- On one hand the authors say that the task decoders are united in that the denominator for $P(\\hat{y}=\\alpha|x_i)$ is over all $y \\in Y$. But for question answering it seems that the decoders for start and end are disjoint, and the softmax is over all tokens in the context not all tasks. Could the authors clarify? This also plays into the fact that catastrophic forgetting is evaluated by training new probes on the frozen encoder. If the decoder is separate for each class, could we not use that directly? One problem would be that perhaps the model has changed how it stores information after seeing a new task. This could be verified by testing if the original decoders work as well, in comparison to the newly trained decoders.\n- Section 3 introduction: Please add standard errors here and everywhere. Just to be clear, the macro-averaged accuracy scores is based on SEQ and not REPLAY right?\n- Figure 1: Nit: Put the replay order backward! Also indicate what the dashed lines mean.\n- Figure 2: Where is the dashed red line for SQuaD? Nit: Order subpanels based on order of training tasks.\n- Section 3.2: “However, the probing results (blue lines) are still much higher than the original scores measured before re-training de- coders (red dashed lines). Comparing the obvious gap between them4, we can find that BERT still keeps most of knowledge of previous tasks when learning new ones.” Yes but the information might be re-organized which is why the blue line does a bit better. (as the authors also note in the 4th footnote)\n- Table 1 analysis: Just to be clear, the representation vectors for a particular dataset, say Amazon, is when one does inference on the Amazon training instances before training on it correct? This is done to define the cone axis and find narrowest angle? And the process is repeated after training on Amazon? I understand the method and motivation but this doesn’t test for inter-class separability no? Even if the topological order within a class might be high, the cone axes or the cones itself could be overlapping leading to reduced separability. This brings me to the point that we have no way of ascertaining that a given correlation is high or low without looking at what the value would be for two pairs sampled randomly from different classes. This would tell us about the ordering of the space entirely. Also, what are the variances for each entry in table 1? How much does this value vary for different examples?\n    - I would also be curious to see how this ordering varies for different subsets of classes. Do some classes, for example, get closer/farther depending on the distribution of the most recent training set?\n    - It would also be more interesting to see how this metric varies for the AGNews dataset before training, after training and then after incremental training.\n- What is the expected result for the inter-task forgetting experiment and the interpretation?  If the value is positive, it means that the new angle is smaller than the old. So as you replay more, the new and old angles become almost equal and since the old angle is fixed, equality would mean that the new angle became bigger? On the contrary, a negative value means that the old angle is smaller than the new. So as you replay more, the difference between them becomes small and since the old angle is fixed, equality would mean that the new angle became smaller? If this is correct, I understand “reveal that memory replay obliges the vectors of previous tasks rotating to their corresponding column-vectors in decoder efficiently, while dragging those of current task to deviate from optimal position.” but it wasn’t clearly written in the text.\n- I am confused by the contradiction between statements and results. For example, the first result (Fig. 1) suggests that the sequential task scheme does as well as memory replay but the authors go on to suggest it doesn’t for the rest of manuscript and do subsequent analyses based on this assumption. For example, the t-SNE plot suggests that under SEQ and not REPLAY, there is a high degree of inter-class overlap. How can we explain figure 1 results then? Additionally, aren’t the results in table 1 for SEQ? If this is the case, what does the high correlation tell us about purported catastrophical forgetting in SEQ? This is not clear from section 4.2 but section 4.3 leads me to believe it is replay. Please clarify.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gxtHCUwoqy", "reviewer": "Reviewer_Lykc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors conduct a two-part study to investigate phenomenon of \"forgetting\", i.e. degraded prediction performance on past data, when a language model is trained on a sequence of tasks. First, they show evidence via a probing study that a BERT encoder base in fact largely retains representations capable of high performance on a given task, even after subsequent training on other tasks. Second, they propose a sequence of analyses based on the idea that same-class embedding vectors fall in a relatively narrow cone, showing that (1) embedding vectors from the same class for a given task have similar nearest-neighbors after training on another task, and (2) replay re-aligns cone axes with the relevant decoder column for past tasks while inducing some misalignment for the current task. ", "summary_of_the_review": "This paper presents a coherent, creative sequence of empirical analyses that illuminate how the phenomena of sequential task forgetting and recovery via replay are related to the representational capacity and within-class structure of embedding vectors from a BERT encoder. The authors could further improve on their contributions by clarifying some details of a novel analysis in $\\S 4.2$ and releasing their code.", "strength_and_weaknesses": "Strengths:\n- The overall presentation and analysis are of good quality and rest on a sequence of interesting and informative experiments. These are unified under the general theme of understanding how the representations in the BERT encoding layer evolve during training. \n- The probing study is simple and effective in showing that \"forgetting\" is not a matter of representational degradation in the encoding layer, as decoder \"probes\" achieve similar accuracies both during and after training on a given task. This raises subsequent questions that the authors answer with a novel perspective.\n- The authors expand on previous observations in the literature to argue that same-class sentence-level embeddings lie within a cone, then use this representation to propose and execute novel analyses relevant for the topic of task forgetting and replay. These methods represent potentially new and useful ways to study the evolution of learned representations over the course of training. The results offer new insights as to how learned representations change (or don't change) when a new task is learned or after replay.\n\nWeaknesses:\n- There is some lack of clarity in the presentation of the novel methods, particularly in $\\S 4.2$. The authors should define precisely what it means for the \"rotating process\" to be \"topological(ly) ordered\". It is also unclear what it means to evaluate the \"correlation between the relative positions of $v_{y,i}^{(1)}$ and $(v_{y,k}^{(1)}), k \\in N_{y,i}$\" or why this is \"estimated by the Pearson correlation coefficient between $\\cos(c_{i}^{(1)}, v_{y,i}^{(1)})$ and $\\sum_{k \\in N_{y,i}} \\cos(c_{i}^{(1)}, v_{y,k}^{(1)})$\" (in the paper the quantity $v_{y,i}^{(1)}$ appears inside the sum but I assume this is a typo). Overall, it is somewhat unclear why this specific approach was taken if the goal is just to establish that same-class embedding vectors have the same or similar nearest neighbors after training on some subsequent task.\n- The authors spend relatively little time contextualizing the sequence of (interesting) results that they generate. How do these support or contradict existing hypotheses in the literature regarding the mechanism of task forgetting and replay? What hypotheses are the studies in $\\S 4$ designed to evaluate, and what are the directions for future analysis?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CqVTA5QBvt", "reviewer": "Reviewer_zUuP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors investigate whether or not pretrained language models like BERT have the ability to maintain previously learned knowledge in the long term. To do this, they track the encoding ability of BERT for specific tasks before, during, and after learning new tasks. They find that BERT can actually refrain from forgetting when learning a sequence of tasks, contrary to existing studies about catastrophic forgetting. The authors believe this is due to the fact that BERT has a strong potential to produce high-quality representations for previous tasks even without memory replay. They further investigate the topological structure of the learned representation sub-space within and among different tasks and find that forgetting can be interpreted as intra-task forgetting (forgetting what has been learned within a task) and inter-task forgetting (forgetting what has been learned across tasks).", "summary_of_the_review": "I like the paper and found the results to be clear and easy to understand. However, my current view is that much of what was presented in this paper had already been answered in another paper last year which the authors may be unaware of (as it hasn't been cited in this work). The main difference between the two papers, in the my opinion, is in Section 4. I recommend that the authors emphasize Section 4 as the main contribution of the paper.", "strength_and_weaknesses": "I like this paper, the results are neat, the exposition is clear (Section 4.2 could use some work as described later), and (almost) everything is easy to understand. But as I was reading this paper, I kept coming back to one paper [1] that I read last year which had done everything this paper is offering. They answered the same questions, they did an even more extensive set of experiments, and the only difference was in how they viewed why pretrained models maintain good representations even if the decoder needs to be retrained. I spent the last day reading both papers side by side and in my view, up until Section 4, there's hardly any difference between the two works. It appears that the authors may not be aware of [1] as it hasn't been cited in this paper so I invite the authors to read [1] and share if they disagree with my assessment. But in light of this, so far, I see the marginal value of this paper lying in Section 4 and would recommend the authors to amplify that as the core contribution of the paper (which is currently listed as their third contribution in Introduction), as the rest of their contributions (claimed contributions 1 and 2) have already been offered to the community previously. Additionally, please add standard errors in every result.\n\n[1] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. \"An empirical investigation of the role of pre-training in lifelong learning.\" arXiv preprint arXiv:2112.09153 (2021).\n\nErrata:\n- Introduction, first paragraph: \"learning the xxx one\". No idea what you meant to say here.\n- Introduction, first paragraph: \"learning a sequecne of tasks\" ---> \"learning a sequence of tasks\"\n- Section 3.2, second paragraph: Kingma and Ba (2015) should be \\citep not \\citet\n- Section 4.1 \"decoders,pre-trained\" ---> \"decoders, pre-trained\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster