Regularization Shortcomings for Continual Learning
Authors: Timothee LESORT
Andrei Stoian
Venue:  (2021)
Keywords: Continual Learning
Regularization

Abstract: In most machine learning algorithms, training data is assumed to be independent and identically distributed (iid).  
When it is not the case, the performances of the algorithms are challenged, leading to the famous phenomenon of \textit{catastrophic forgetting}. Algorithms dealing with it are gathered in the \textit{Continual Learning} research field. In this paper, we study the \textit{regularization} based approaches to continual learning and show that those approaches can not learn to discriminate classes from different tasks in an elemental continual benchmark, the class-incremental setting.
We make theoretical reasoning to prove this shortcoming and illustrate it with experiments.
Moreover, we show that it can have some important consequences on multi-tasks reinforcement learning or in pre-trained models used for continual learning.
We believe this paper to be the first to propose a theoretical description of regularization shortcomings for continual learning. 

Under review as a conference paper at ICLR 2021
REGULARIZATION S FOR CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In most machine learning algorithms, training data is assumed to be indepen- dent and identically distributed (iid). When is not the case, the algorithms performances are challenged, leading to the famous phenomenon of catastrophic forgetting. Algorithms dealing with it are gathered the "Continual Learning" research field. In this paper, we study the regularization based approaches to con- tinual learning and show that those approaches can not learn to discriminate classes from different tasks in elemental continual benchmark, the dass-incremental setting. We make theoretical reasoning to prove this shortcoming and illustrate it with experiments. Moreover, we show that it can have some important conse- quences on multi-task tasks reinforcement learning or in pre- trained models used for continual learning. We believe this paper to be the first to propose theoretical description regularization shortcomings for continual learning.
INTRODUCTION
Continual Learning a sub-field of machine learning dealing with non-iid (identically and indepen- dently distributed) data French (1999); Lesort et al. (2019c). Its goal is to learn the global optima to an optimization problem where the data distribution changes through time. This typically the case in databases that get regularly augmented with new data or when data is streamed to the algorithms with limited storage possibilities.
Continual learning (CL) looks for alternative methods to the iid training to avoid the complete retraining with data each time new data available. CL. algorithms propose different memory storage approaches to collect information from past learning experiences and learning algorithms to continue to learn with this memory and new data.
In this paper, we propose study the class-incremente setting with regularization based methods. The class incremeactt setting consists of learning sets of classes incrementally. Each task is composed of new classes. As the training ends, the model should classify data from all classes correctly. Without task labels for inferences, the model needs to both learn the discrimination of intra-task classes and the trans-task classes discrimination (i.e. distinctions between classes from different tasks). On the contrary, if the task label is available for inferences, only the discrimination of intra-task classes needs to be learned. The discrimination upon different tasks given by the task label. Learning without access to task labels at test time is then much more complex since it needs to discriminate data that are not available the same time in the data stream.
In such setting, we would like to demonstrate that regularization does not help to learn the discrimi- nation between tasks. For example, first task is discriminate white cats vs black cats and the second is the same with dogs, a regularization based method does not provide the learning criteria to learn features to distinguish white dogs from white cats.
We consider as regularization methods, those who aim at protecting important weights learned from past tasks without using a buffer of old data or any replay process. Those methods are widely used for continual learning Kirkpatrick et al. (2017); Zenke et al. (2017); Ritter et al. (2018); Schwarz et al. (2018). In this paper, we show that in the classical setting of insseennermmttall tasks, this approach has theoretical limitations and not be used alone. Indeed, it does not provide any learning criteria to distinguish classes from different tasks. Therefore, in practice, regularization algorithms need external information make an inference in class-incremental settings. It is provided by the task

---

Under review as a conference paper at ICLR 2021
label at test time. However, relying on the task label to make inferences is important limitation for algorithms' autonomy. i.c. its capacity to run without external information, in most application scenarios.
We believe this paper presents important results for a better understanding of CL which will help practitioners choose the appropriate approach for practical settings.
2 RELATED WORKS
In continual learning, algorithms protect knowledge from catastrophic forgetting French (1999) by saving them into memory. The memory should be able incorporate new knowledge and protect existing knowledge from modification. In continual learning, we distinguish four types of memorization categories dynamic architecture Rusu al. (2016); Li & Hoiem (2017), rehearsal Chaudhry et al. (2019); Aljundi et al. (2019); Belouadah & Popescu (2018); Wu et al. (2019); Hou al. (2019); Caccia et al. (2019). generative replay Shin al. (2017); Lesort al. (2019a): Wu et al. (2018) and regularization Kirkpatrick et al. (2017); Zenke et al. (2017); Ritter et al. (2018); Schwarz et al. (2018).
In this paper, we are interested in the capacity of making inferences without task labels at test time (test task label). The task label (typically a simple integer) is an abstract representation built to help continual algorithms to learn. It is designed to index the current task and notify if the task changes Lesort et al. (2019c). Dynamic architecture a well-known method that needs the task label at test time for inference. Indeed, since the inference path is different for different tasks, the task test label is needed to use the right path through the neural network Rusu et al. (2016); Li & Hoiem (2017). Rehearsal and Generative Replay methods generally need the task label at training time but not for inferences Lesort et al. (201 9a;b.. Finally, Regularization methods are often assumed as methods that need task labels only training time. In this article, we show that in class-incremental settings, it is also necessary at test time.
Test task labels have been used in many continual learning approaches, in particular in those referred to as "multi-headed" Lange et al. (2019). However, the need for task labels for inferences makes algorithms unable make autonomous predictions and therefore we believe that this requirement is not in the spirit of continual learning. Continual learning is about creating autonomous algorithms that can learn in dynamic environments Lesort et (2019c).
REGULARIZATION APPROACH
In this section, we present the formalism we use and we present the class-incremental learning problem with a regularization based approach.
3.1 FORMALISM
In this paper, we assume that the data stream is composed of N disjoint tasks learned sequentially one by one (with N >= 2). Task t is noted T; and ID; is the associated dataset. The task label i is simple integer indicating the task index. We refer to the full sequence tasks as the continuum, noted CN. The dataset combining all data until task t is noted C, While learning task Tú, the algorithm has access to data from D, only.
We study a disjoint set of classification tasks where classes of each task only appear in this task and never again. We assume at least two classes per task (otherwise a classifier cannot learn..
Let f be function parametrized by 0 that implement the neural network's 'ss model. At each task t the model learn an optimal set of parameters 0; optimizing the task loss (.). Since we are in a continual learning setting, 0 should also be optima for all tasks Te, Vt' E [].
We consider the mlas-incremmnnal setting with no test task label. It means that an optima 0 for Ti is set of parameters which at test time will, for any data point from Dy U D1, classify correctly without knowing if x comes from To or T,. Therefore, in our continual learning setting, the loss to optimize when learning given task is augmented with remembering loss:
2

---

Under review as a conference paper at ICLR 2021
AL((L-1) where (.) is the continual loss, fD. (.) is the current task loss, 12(C2-1) is the remembering loss with C--1 represents past tasks, À is the importance parameter.
(1)
3.2 PROBLEM
In continual learning, the regularization approach is define (2(-) as a regularization term maintain knowledge from Ct-1 in the parameters 0 such while learning new task Tú, fz: 0 ~ f(; 0), Vz C.-1. In other words, it aims to keep & (f; 0), y) low Vo C.-1 while learning Tç.
The regularization term Q4- act memory of 0 This memory term depends on the learned parameters 1 on f the loss computed on T and the current parameters 0. Ω4-1 memorizes the optimal state of the model at T and generally the importance of each parameter with regard to the loss c We note ftc,,, the regularization term memorizing past tasks optimal parameters.
When learning the task Tr, the loss optimize is then:
lc, (x;; 0). y) (J 0), Xlc. 107-1. 1,
Eq. 2 is similar to eq. 1 but in this case the function Q?-)) is regularization term depending on past optimal parameters 07 loss on previous tasks fc,_, and the vector of current model parameters only. It could be, for example, a matrix pondering weights importance in previous tasks Kirkpatrick al. (2017); Ritter et al. (2018); Zenke al. (2017).
4 PROPOSITIONS
In this section, present the proposition concerning the shortcomings of regularization methods in class-incremental settings. We first present definitions and lemmas to prepare for the proposition.
4.1 PRELIMINARY DEFINITION/ LEMMA
Definition 1. Linear separability
Let S and S' be two sets of points in an n-dimensional Euclidean space. S and S' are linearly separable if there exists n + 1 real numbers W1,W2, w2,,W,,,,,e w.. k such that Vo E S, >n wjz; k and Vxx S¹, Em W;3; < k
where x; the ith component of x. This means that two classes are linearly separable in an embedded space there exists hyperplane separating both classes of data points.
This property can also be written, Væ E and Ya' E S', (q x + (0) (q x + qo) < 0. With q
4/2 (12:42 .... w., and qo k respectively the normal vector and position vector of hyperplane Q. In the case of learning a binary classification with linear model, the model is a hyperplane separating two dataset. As soon as this equation can be solved, then it possible to define function fxx 0) and a loss E(.) to learn a hyperplane that will separate S and S' perfectly.
Definition 2. Interferences
In machine learning, interferences are conflicts between two (or more) objective functions leading to prediction errors. There are interferences when optimizing one objective function degrades the optimization of, at least, another one.
As such, optimizing one of the objective function increases the error on the other one. In continual learning, interferences happen often after drift the data distribution. The loss on previous data is increased with the optimization of the loss for the new data leading to interferences and catastrophic forgetting.
Lemma 4.1. S'S bounded set of discrete points in R and linearly separable by hyperplane Q. For any algorithm, it is impossible to assess Q as separation hyperplane without access to S' set.
The proof is in appendix B, but in an insightful way, for any bounded set of points S, there is a infinite number of linearly separable set of points. Thus, there exists infinite number of potential
3

---

Under review as a conference paper at ICLR 2021
separating hyperplanes. If the second set of points S' is not known, then it is not possible to choose among the infinite number of potential separating hyperplane which one is a correct one. And even one is chosen, there is no way to tell if it is better or not than another.
In the context of machine learning, without an assessment criterion for a classification problem, is not possible learn a viable solution. Hence, can not optimize the parameters. For binary classification, the Lemma 4.1 can be interpreted as: "The decision boundary between two classes can not be assessed nor learned if there is no access data from both simultaneously".
Lemma 4.2. V(S, S') two bounded datasets not linearly separable. For any algorithm, it is impossi- ble assess a function g() as a projection of S and S' into a space were they are linearly separable without access to S' set.
The proof is in appendix C, but in insightful way, for any bounded set of points, there is an infinite number of projections of the initial set of point in a space where it could be linearly separable from another set of points. Then, If you don't 't know the second set of points S' you can not choose among the infinite number of potential projections which one is a good one. And if you ever choose one, you have way tell if it is better not than another. In the context of binary classification, the previous lemma can be interpreted as: "Two classes representation cannot be disentangled if there is no access to data from both simultaneously".
In those lemma, the concept of "not having access to" a certain dataset can both be applicable to not being able to sample data point from the distributions and to not have a model of the dataset. It can be generalized to not having access to any representative data distribution of a dataset.
4.2 SHORTCOMINGS IN
TASKS
We now prove that in incremental-clas tasks, is not possible to discriminate classes from different tasks using only regularization based memory. The main point is that, to correctly learn to discrimi- nate classes over different tasks the model needs access to both data distributions simultancously.
In regularization methods, the memory only characterizes the model and the important parameters as explained in Section 3.2. This memorization gives insight on some past data characteristics but it is not model of their distributions. If we take the cat dog example, model that needs to discriminate white cats from black cats will learn to discriminate black features from white features. This "knowledge" can be saved in Z but Z will not save the full characteristics of cat because the model never has to learn it. We bring then the following proposition:
Proposition 4.3. While learning sequence of disjoint classification tasks, if the memory Ω of the past tasks is only dependent on trained weights and learning criterion of previous task and does not model the past distribution, it is not possible for deep neural networks to learn new tasks without interference.
Proof. The proof organized in the following way: first, we present material necessary for the demonstration, then in a second part, we demonstrate that at any moment the classification task can be reduced to a binary classification task and third part we will show that we can not learn to solve this binary classification correctly.
First part: In the context of learning with a deep neural network, we can decom- pose the model into a non-linear feature extractor g(-) and an output layer to predict a class y = argmma((softmax A g(x) + b)). With A and b, respectively the matrix of projection and the bias of the linear layer. softmax(() is the output function that for given class in a logits output ett
gives softmax(2:) Σyoo c-5 The softmax(.) function does not change the argmax result and
only regularize the output values and the gradient for later back propagation. We can thus remove for our demonstration purposes.
The non-linear projection g(-) should, therefore, disentangle classes and the linear output layer learns to predict the good class. The output layer allows for all classes i to learn hyperplanes Al; i] with bias bli] such as: Vi [1,
V(x, v) Ce, (A]: b(1)
(3)
with h = g(x).
4

---

Under review as a conference paper at ICLR 2021
Second part For the sake of the demonstration, would like reduce the multi- classification problem into a binary classification problem. Hence, we can artificially split classes into two groups: classes from the past c._, and current classes Yr.
We can then V(z, y) E C, compute which class _c.., upon the past classes Yc_,, is the most probable and compute which class fr, upon the current classes Yr, is the most probable.
cC argmax A|: i/4 b(i)) kEYO,_1
and in = (Ag b(i])
(4)
KeYn,
Hence, the equation 3 can be rewritten into binary operation:
v(a, E Ct. argmax (Al:, + b[i))
(5)
ve(âc, )
argmax (A[: ic._.l h + blū Al: Dr.1 h bl0r..)
(6)
= argmax (0, [A fr.1 Al: (c...)) h + bl0r.1 1 blūc._.l)
Equation can directly be rewritten into the linear separability equation from definition 1. To make proper decision, should have V(z,y) E Ct with g(x) h and y cc.., and V(r', D,, with g(z') = h and y = UT.
(q + qu) (q h + qu) <
Then, by identification, the classes ûc, and jT, need to be separated by the hyperplane Q defined by normal vector Al; Ur.] ( and position vector do = -b(ūr, b(üc,,,)).
This binary classification description highlight that it is essential to able to discriminate any class JC,_, from the past from any class ir, from the present for accurate predictions.
Third part: In this part, to prove proposition 4.3, we show that the model cannot learn the hyperplane Q from eq. 7.
To learn new tasks T, for 0 < < N, there are two different cases: first 9(-) already a good projection for Ct tasks, i.c. classes already disentangled in the embedded space. We assume that classes are already disentangled, only the output layer has to be trained to solve C2 tasks. Secondly, g(-) needs be adapted, i.e. classes are not yet disentangled in the embedded space and new features need to be learned by g(-) to fix it. We refer as features, intrinsic characteristics of data that a model needs to detect to distinguish class from another. We will show that is possible learn to discriminate correctly the classes ÿc, from WT, from previous part.
First case: Classes are disentangled Since we are in regularization setting, at task Tr, we have access to Ω which contains classifica- tion information from previous tasks (C,-1 tasks). However, by hypothesis, Ω,- does not model the data distribution from C2-1 and therefore it does not model data distribution from Ct-1 classes
Following from the second part of the proof, Va E C2 tasks, to make an accurate prediction, need the right hyperplane Q that distinguish the most probable class from Ct-1, 9c.., and the most probable class from Tú, 9T.
ic.., and dr. classes images are a bounded set of points and 9c.., points are, by definition, not accessible, consequently following Lemma 4. it impossible to assess boundary between ýr, and ic.., even if by hypothesis this boundary exists. Therefore, we can not learn the hyperplane that discriminate úc, from fi, and ensure an accurate prediction.
Second case: g(-) needs to be updated with new features.
Let 8,-1 be the set features already learned by 9t-1(.) the feature extractor from previous task. Ω4-1 should keep 8, unchanged while learning T,. The goal is to make Ic.,, and or, linearly separable Vx E C. Then, either 8,- already solve the problem and we are in first case, or new set of features 6, needs to be learned while learning Tr. In the second case, the set & contains features solve Te but features 6,-1:t that distinguish classes from T to classes from T; should also be learned. Then two cases raise, 8 it & 8, or 6,-1.x C &
if 5 & 8, then supplementary features 8,-1: need to be learned. $c-., and Gr. classes images bounded set of points not linearly separable and since Ω does not give access C--1 data
5

---

Under review as a conference paper at ICLR 2021
020000000
Task
(b) Task
(c) Task
Figure 1: The three tasks of the MNIST-Fellowship dataset.
points, from Lemma 4.2 we can not assess a projection that put images from CT, and ic, into linearly separable space, i.c. we can not learn the set of features d discriminate Ic,-, images from dr, images and solve the continual problem.
61-1:t 5g is possible, however, since data from Ct-1 are not available anymore, there is no way to project them in the new latent space with 6, features. Therefore, without access of classes from both Ct-1 and T₁ tasks at time we can not identify -;1 features which are ó, features. also impossible to know if d C 5; In other words, this case is not detectable and even if detected the features 6, can not used without data from T (which by definition prohibited).
In these two cases, there will be in any way conflic between losses leading to interference in the decision boundaries either because classes are not linearly separable or because a separation hyperplane cannot be found. In other words, the regularization methods can not discriminate classes from different tasks and they are then not suited to caassinncremenall settings.
We can note that proposition 4.3, still holds if tasks are only partially disjoint, i.c. only some classes appear only once in the continual curriculum.
Indeed, in partially disjoint settings, several classes pairs are never in the same task. If define two set of disjoint classes Y and Y", that will never be in the same task, the demonstration of proposition 4.3 can be applied on Y and Y'. Then, classes Y and Y' will suffer from interference showing a shortcoming of regularization methods for this case too.
Therefore, there class-incremental setting hidden into another setting, the regularization approach will not be able to solve it perfectly either. We could note that in many applications there are latent class-incremental problem to address in the learning curriculum. We mention some applications in Section 6.
simple trick used in some regularization approaches to compensate their shortcomings to use the task label for inferences, it gives simple way to distinguish tasks from each other. However, it assumes the algorithms rely on supervision signal for inferences. In the next section, we show that regularization shortcoming is easily highlighted with simple experiments.
EXPERIMENTS
To support the limitations presented earlier, we experiment with the "MMIST-Fellowship dataset proposed in Lesort (2020). This dataset is composed of three datasets (Fig. I): MNIST LeCun Cortes (2010), FaSHion-MNIST Xiao et al. (2017) and KMNIST Clanuwat al. (2018), 8), each composed of 10 classes, which should be learned sequentially one by one. We choose this dataset because it gathers three easy datasets for prototyping machine learning algorithms but solving those three quite different datasets is still harder than solving only one.
Our goal is to illustrate the limitation of regularization based methods disjoint settings. In particular that they not distinguish classes from different tasks. We would like also to show that the shortcoming happens both in the output layer and the feature extractor. Thus, we propose three dataset.
different settings with the MNIST-Fellowssd
1. Disjoint setting: all tasks have different classes (i.c. from 0 to 9, 10 to 19 and 20 to 29, the output size is 30). This setting highlights shortcomings of regularization methods without test task labels.

---

Under review as a conference paper at ICLR 2021
Ld: DNL
20
Epods
Operrs
Figure 2: Experiment on disjoint classes without vs with test task label. Left, the mean accuracy of all tasks, vertical lines are task transitions. Right, accuracy on the first task. Legends with 'Lab indicate experiments with task labels for test. The expert model is trained with i.i.d. data from all task and the baseline model is finetuned on each new task without any continual process.
100
Sauck
Aks
Bocins
Equal
Figure 3: Experiments with joint classes. Left, mean accuracy of all 3 tasks, vertical lines are task transitions. Right, accuracy on the first task.
2. Joint setting: all tasks have the same classes ( i.e. from 0 to 9 for each task and the output size 10) but different data. This scenario is designed as an instance incremental scenario Lesort et al. (2019c). This setting shows that they are interferences even when only the data instances change and not the class labels. Theoretically, this setting requires only the feature extractor to be adapted while task.
learning
Disjoint setting with test task label: All tasks have different classes but at inference time, we know from which task data-point is coming from. The output this setting is a multi-head output with one head of size 10 for each task. This setting shows that regularization methods work when they have access the test task labels.
With those settings, we present two experiments, the first one (Fig. 2) compares disjoint setting with and without label for inferences. The goal is bring light that regularization fails in disjoint settings if the task label is not provided. Secondly, we experiment with the joint setting (Fig. 3), to show that even the feature extractor only needs be learned the approach still struggles to learn continually and forget.
We present EWC results with diagonal Fisher Matrix Kirkpatrick al. (2017) and with Kronecker Factorization of the Fisher matrix Ritter et al. (2018). We add an expert model which learned all the datasets at once and a baseline model who learn without any memorization process. trained with stochastic gradient descent with learning rate of 0.01 and a momentum of 0.9. Even if continual learning does not support a- a-posteriori the parameter lambda has been tuned. The best lambda upon [0.1; 1; 5; 10; 20; 100; 1000] is selected for each model. Then the model is trained on 5 different seeds.
models are
selection, for fairness in comparison,
The first experiment (Fig. 2), exhibits that regularization methods performances significantly reduced when there no test task label in the disjoint settings. The experiment also shows that without labels for inferences, the model forgets almost instantaneously the first task when switching the

---

Under review as a conference paper at ICLR 2021
second one. Those results support the proposition 4.3. Indeed, the low performance of regularization methods without test task labels in disjoint settings illustrates the output layer shortcomings in continual learning (task separability problem example in appendix A).
In Experiment (Fig. 3), since the classes are the same in all tasks, only the feature extractor needs to be learned continually. The low performance of the proposed models illustrates the shortcomings in the continual learning of the feature extractor (the latent features problem example in appendix A).
These two experiments show that learning continually with regularization is only efficient in the setting with task labels and maintains performance on task 0. The two other settings seem to either have interference in the output layer and in the feature extractor.
6 APPLICATIONS
In this section, we point out supplementary shortcomings of regularization in other types of learning situations, namely a classification task with one only class and multi-task continual reinforcement learning. We also use proposition 4.3 for the case of pre-trained models.
Leaanning from one class only: classification task with only one class might look foolish, however, in sequence of tasks with varying number of classes, it makes more sense and it seems evident that a CL algorithm should be able to handle this situation. Nevertheless, a classification neural network needs at least two classes to learn discriminative parameters. Hence, in one-class task, the model learns no useful parameters, a regularization term can then a fortiori not protect any knowledge. As noted in Lesort al. (2019h), the regularization method is not suited for such setting. It is worth noting that in a real-life settings it is mandatory to be able to learn only one concept at a time.
Multi-task Continual Reinforcement Learning: Results from section 4.2 can also be generalized to continual multi-tasks reinforcement learning settings Traoré et al. (2019). In this setting, a model has to learn several policies sequentially solve different tasks. At test time with no task label, the model needs to both be able to run the policies correctly but also to infer which policy to run. class
However, since policies are learned separately inferring which one to run equivalent incremental task. Therefore, following proposition 4.3, the regularization based method will not be able to learn the implicit classification correctly. Hence, in continual multi-tasks RI. a regularization method alone will fail if task label is not enabled at test time.
Using pre-trained models for continual learning: We showed in Section 4.2 that, in a class incremental classification scenario, regularization methods are not sufficient to learn continually. In the case of a pre-trained classification model on N classes that we want to train on new classes without forgetting, if the training data not available for some reasons, then we don't 't even have a regularization term to protect some. Following the proposition 4.3 and a fortiori without the regularization term, the model will forget past knowledge while learning new classes. Using pre- trained models can be useful to converge faster to a new task solution but will undoubtedly forget what it has learn previously.
7 DISCUSSION AND CONCLUSION
Regularization is widespread method for continual learning. However, we prove that for class- incremental classification, no regularization method can continually learn properly the decision boundaries. At test time, this shortcoming makes them dependant on the task label for prediction.
The daas-incremmntal scenarios are benchmarks measuring the ability of algorithms to learn sequen- tially different classes. However, being unable to deal with this setting implies that in a more complex learning environment, all sub-tasks assimilable to a class-incremental task will be failed.
It is fundamental for continual learning to produce algorithms that can autonomously learn and be deployed, Algorithms that rely on the test task label are not autonomous and therefore should be avoided. This paper shows that in any setting where class-incremental setting may be hidden, regularization methods alone should be avoided. A fortiori, in continual learning the future is unknown, therefore future tasks might add new classes or not. Then, in order to deal with tasks whatever their nature, regularization methods alone should always be avoided on continual learning applications.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 11849- 11860. Curran Associates. Inc., 2019. URL pdf.
nips cc/paper/
Eden Belouadah and Adrian Popescu. Deesil: Deep-shallow incremental learning. CoRR, abs/1808..e 06396, 2018. URL http:://axii/ org/ abs 1808..18...0 06396.
Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. Online Learned Continual Compression with Stacked Quantization Module. arXiv e-prints, art. arXiv:1911.08019, Nov 2019. URL pnttts://arxive
abs 911. 08019.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In ICLR, 2019.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. CoRR, 2018.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, (4):128-135, 1999. ISSN 13646613. doi: 10.1016/S1364 (100666813344-66111999091244
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified clas- sifier incrementally via rebalancing. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proc. of the national academy of sciences, 2017.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks, 2019. URL https:
08383.333....
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. public, 2010. URL http: //yann. lecun. exdm//xx////////ooo//o/// mn msstt/
Sang- Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Lung-Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems, pp. 4652-4662, 2017.
Timothée Lesort. Continual learning data former, January 2020. URL hts.httss///oo.rrg///2 5281 3605202.
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Sean-François Goudou, and David Filliat. Generative Models from the perspective of Continual Learning. IJCNN International Joint Conference Neural Networks, Budapest, Hungary, July 2019a. URL https: /hal. archives-ouvertes. wwt.eee/hhaaalle1eeee
Timothée Lesort, Alexander Gepperth, Andrei Stoian, and David Filliat. Marginal replay vs conditional replay for continual learning. In Artificial Neural Networks and Machine Learn- ing ICANN 2019: 28th International Conference Artificial Neural Networks, pp. 466- 480, 2019b. doi: 10.1007/978-3 0-17788---033300 30484-318343330 978- 978-3-020-3048--2220
URL https: / //tp..oog/...100.//7

---

Under review as a conference paper at ICLR 2021
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodie Rodriguzz Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges. working paper preprint, November 2019c. URL https: //hal, arch ves ves- ooversssveeeeeeeeeee fr/hall htt/hhll023811.1 02381343
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2017.
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408-2417, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035, Curran Associates, Inc., 2019. URL htttt::/ppperrs.eeeee neurips cc/ppper///..... 9015 Sar---yytccc-h-- an imperative style hhihhhiigghhheeee performance- deep- learning library. pdf.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 3738-3748. Curran Associates, Inc., 2018. URL http: wwwpppapppppppappppeppeeaeaeeeep caappers..ipsso cc wwwppp/ppp/ppepecceeccceeeeeeepe 7631-onl onneinniinnneee wwwllinn-stuuuuuree structured structuree-laplacee laplace approximations -ttprp/eeeeeeepaaaaapppoooooaiio wwwerooooooooooooooooooooooooooo ott.rromoiiinattttttttoooooooooo pdf.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive Neural Networks. ArXiv e-prints, June 2016.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska- Barnkka-Brwiiska,, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Muchine Learning Research, pp. 4528-4537, Stockholmsmässann Stockholm Sweden, 10-15 Jul 2018. PMLR. URL hhtt:: //proceedings mlr. sagess/v800 / scwwarz18 html.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
René Traoré, Hugo Caselles- Dupré, Timothée Lesort, Te Sun, Guanghang Cai, Natalia Díaz Ro- dríguez, and David Filliat. Discorl: Continual reinforcement learning via policy distillation. CoRR, 0bs/1907.05855, 2019, URL http: / /aaaaxivvv. org/ abs /ttb//9900 1977 05855.
Chenshen Wu, Luis Herranz, Xialei Liu, yaxing wang, Joost van de Weijer, and Bogdan Raducanu. Memory replay gans: Learning to generate new categories without forget- ting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 5962- 5972. Curran Associates, Inc., 2018. URL http:/ /ttp://pppers.... wctps.cc/paper/ m0.6-memorr-reeley--eo pdf.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu, Large scale incremental learning. CoRR, abs/1905.132600 2019. URL http:: httptp:/aaxxv..og/g/// 1905. 13260.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1708. 2017.
Friedeman Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on
10

---

Under review as a conference paper at ICLR 2021
Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3987-3995, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http: wttp:eceeediiaa.....e.....e.eeee
/ /pproceedings... mlr wttppp/ees/e//pppp//epeeppp 7a 1a.mkll1l.......

---

Under review as a conference paper at ICLR 2021
A PRACTICAL EXAMPLES
To illustrate the proposition from section 4.2, we present two insightful examples of regularization limitations,
The Task Separability Problem:
In the first case of proposition 4.3 proof, we already have a perfect feature extractor. Classes are already linearly separable and only the output layer needs to be learned continually.
we have only two classes in the first task, the model will learn one hyperplane 2o separating the instances of these two classes (See Figure 4). For the second task, we have two new classes and a regularization protecting Qo- Then, we can learn hyperplane Q1 that separates our two new classes. In the end, we have learned the hyperplanes Co and Q1 to distinguish classes from To and classes from Ti But none of those hyperplanes helps to discriminate To classes from Ti classes, as illustrated Figure 4. This will lead to error in the neural networks predictions.
Figure 4: Simple case of continual learning classification in multi-task setting. Left, the task TO: learning a hyperplane splitting two classes (red and blue dots). Right, the task T1: learning a line splitting two classes (yellow and green squares) while remembering To models without remembering To data (pale red and blue dots).
The Latent Features Problem:
In the second case of Proposition 4.3 proof, the feature extractor needs to be updated to learn new features extractors.
we have only two classes the first task, the model will learn to separate classes instances into two groups with the features extractor du and one hyper-plan Qo separating the two classes instances (See Figure 5).
For the second task, we have learn features extractor to disentangle new class instances in the latent space and hyperplane 2, that separates them. In the end, have two hyperplanes Qo and Q to distinguish classes from To and classes from Ti. But we can not disentangle To classes from Ti classes and none of the learned hyperplanes helps discriminate To classes from Ti classes (See Fig. 6). It leads to errors in the neural network predictions. At test time, it will not be possible for the model discriminate between classes correctly.
classes and regularization protecting Qo and 9o- Then,
disentangle classes from To and classes from Tj and
However, with the task label for inferences, we could potentially perfectly use 90- 91, Qo and Q1 to make correct predictions. Nevertheless, assuming that the task label is available for prediction is strong assumption in continual learning and involves a need of supervision at test time.
Figure 5: Do feature space before learning To (Left), Do feature space after learning To with possible decision boundary (Right). Data points are shown by blue and red dots. The line (right part) the model learned separate data into the feature space.
12

---

Under review as a conference paper at ICLR 2021
Figure 6: Case of representation overlapping while continual learning classification in multi-task setting. At task Ti feature space of DD, before learning Ti (Left), Feature space D after learning Ti with possible decision boundary (Right). New data are plotted yellow and green squares and old data that are not available anymore to learn are shown with pale red and blue dots.
B PROOF OF LEMMA 4.1
lemma. V(S bounded set of discrete points in R and linearly separable by a hyperplane Q. For any algorithm, it is impossible assess Q separation hyperplane without S' set.
Proof. Let S and S' be two bounded and linearly separable set of discrete points in R Let Q be potential linear separation between S and S'. The hyperplane can not be assessed linear separation between S and S' there exists at least one hyperplane indistinguishable from Q and normal
which is a separation boundary between S and S Let P be hyperplane, defined vector and position vector is a separation boundary between S and S' if all the point of S are on one side of and all point of S are on the other side. It can be formalized as follows:
Va E & Va E S':
(p po) (p x' po)
Where < is the sealar product.
Without the access of 5", eq. 8 can not be evaluated. However, we can evaluate it, if all the point of S are on the same side of P
Eq. 8, verify that S and S' are each entirely different side of the P. By definition if all the point of S are above P then:
Va E S
(p- x Po) >
(9)
If all the point are under P then:
(p x + po) < 0
(10)
And if neither eq. 9 nor eq. 10 verified then all the points of S are not the same side of P.
Finally, we can merge both 9 and eq. 10 and verify only:
Vx S
sign(p x + po) constant
(11)
Where s.gm(.) is the function which returns the sign of any real value.
The Lemma 4.1 is proven 3 P such as eq. 11 is true but not eq. 8, because P would not be a linear separation of S and S' and would not be distinguishable from Q without access to S'.
Now, we will build an hyper-plan P that is unquestionably respect eq. 11 and not eq.
We know that S is bounded, then it has both upper and lower bounds in all the direction of R" If eq. 11 is respected, then Q is a bound of S in the direction its normal vector q. we move Q along the direction of (i.e. if we change do the position vector), respecting eq. 11: the opposing bound of 5 along the direction q-
find at least one other plane P
Since, P and Q are opposing bounds of S in the same direction q, then:
13

---

Under review as a conference paper at ICLR 2021
dr E S
sign(p- x + po) * (q a qu)
(12)
If Q is lowerbound of S in the direction and an upperbound of S' in the same direction then, lowerbound of S' in the direction is lowerbound of S in the same direction and upperbound of S in the direction is upperbound of S' in the same direction. (We leave the demonstration to the reader).
Therefore, Q and p are both upperbounds or both lowerbounds of S' in the direction of q. Vx' E S":
sign(p + po) = sign(q x' + qo)
(13)
Then with 12 and eq. 13:
x 6) (p x >
(14)
Consequently, from eq 11 and eq 14, a hyperplane which respects eq. 11 and not eq P indistinguishable from Q and is not separation boundary between S and S*.
C PROOF OF LEMMA 4.2
lemma. S") two bounded datasets not linearly separable. For any algorithmm, it is impossible to assess a function g(.) as a projection of S and S' into a space were they are linearly separable without S' set.
Proof. g() is projection of S and S" into a space where they are linearly separable means:
Vx & Val S', then g(x) and g(x') respect eq. 8.
Without access to S" this condition can not be verified. However, we can verify eq. 11 with g(x).
The Lemma 4.2 proven if Va S & Var E S", 3 projection that respect eq. 11 with f(x: but not eq. 8 with f(x) and f(z',, because then f and g are indistinguishable without access to S".
Let be the identity function, Vz R f(z) = z. We define Sy and S'y. the of point S and S' after projection by f. Since f the identity function, S and S' respectively identical to Sy and S Since S is bounded, S is also bounded. Hence there exists hyperplane P that verify eq. 11 with f(x) Væ E S. By hypothesis, S and S' are not linearly separable so Sy and Sy, is also not linearly separable. Then 3! hyperplane P which respect eq. 8 with fzr and f(x').
Thus, exists and therefore it impossible assess any function as a projection of S and S' into space were they are linearly separable without S" set.
D REGULARIZATION METHODS
To illustrate the previous section, we present several famous regularization methods in our formalism.
Elastic Weight Consolidation (EWC) Kirkpatrick al. (2017) is one of the most famous regular- ization approaches for continual learning. The loss augmented with a regularization term is at task
lc, (0) = lo (f(, + (0-1(0;_,-0))
(15)
with (.)2 the element-wise square function.
We can then by identification, extract our function P. ( 0)
12 lc._, 0) = * F F. ( 0)2
(16)
14

---

Under review as a conference paper at ICLR 2021
F/ is tensor of size card(0)², specific to task characterizing the importance of each parameter 0y- Fe is computed at the end of each task and will protect important parameters to learn without forgetting. In EWC, the F, tensor is implemented diagonal upproximation of the Fisher Information Matrix:
(( (olog p(ù) 00
2
F. = w)
(17)
where y P(f(z;0) 0)). The diagonal approximation allows to save only card(0) values in Fi.
K-FAC Fisher approximation Ritter et al. (2018) is very similar to EWC but approximates the Fisher matrices with Kronecker factorization (K-FAC) Martens & Grosse (2015) improve the expressiveness of the posterior over the diagonal approximation. However, the Kronecker factorization saves values than the diagonal approximation.
Incremental Moment Matching (IMM) Lee et al. (2017) proposes two regularization approaches for continual learning which differ in the computation of the mean lo. and the variance do. of the parameters on all tasks.
The idea is to regularize parameters such that the moments of their posterior distributions are matched in an incremental way. It means that each parameter is approximated as a normal distribution and their mean standard deviation should match from task to another. This regularization, on the parameters' low-order moments, helps to protect the model from forgetting.
Mean based Incremental Moment Matching (mean-IMM)
0o:
Σ 0 and Ooe = 1=0
ag + (07 - -o:z)3
(18)
2=0
ag importance hyper- pappeetrrsssterss to balance past task weight into the loss function. They up to one.
Mode based Incremental Moment Matching (mode-IMM)
Ou: = -u:: (a,o 0;)
and Ooa =
a;o; 1)-1
(19)
i=0
i=0
0; is computed as the Fisher matrix (eq. 17) at task i.
Then at task f, with fo.--1 and Joa-1 we can compute:
1
(0, 0) == 00.-- 1 0)2
(20)
Synaptic Intelligence: (SI) Zenke et al. (2017) The original idea is to imitate synapse biological activity. Therefore, each synapse accumulates task relevant information over time, and exploits this information to rapidly store memories without forgetting old ones. In this approach, we can identify Ω4 as:
0,(0", fc, 0) = M, 0)²
(21)
Mc a tensor of size card(0) specific to task t characterizing the importance of each parameter 0x over the all past tasks such as:
M, Σ
m; A
(22)
+
OCTICt
Mj is the sum over my which characterizes the importance of each parameter on task i, with A = 0 00:1. & is supplementary parameter to avoid null discriminator.
m,
(ode
(23)
JT-1
With (t) the parameter update at time step
15

---

Under review as a conference paper at ICLR 2021
E IMPLEMENTATION DETAILS
E. DATA PREPROCESSING
All data points were preprocessed to be between 0 and by division by 255.
E.2 DATASETS SPLITTING
For all datasets used, selected randomly 20% of the train set for validation and used the original split test/train of the datasets for the test sets and the train sets.
E.3 COMPUTING INFRASTRUCTURE
The experiments were run with GPU GeForce GTX 1080 Ti with CPU Intel Core 17-7700K @ 4.2 GHZ x 8.
E.4 NUMBER OF EVALUATION RUNS
single evaluation run have been executed after trainning.
F MODEL ARCHITECTURE
Table 1: Model architecture, convolution have 5*5 kernel size, maxpool have 2*2 kernel size. Parameters not mentioned are default parameters in Pytorch library Paszke et al. (2019) (in torch.nn). BS is for batch size, which 64, All layers are initialized with Xavier init method Glorot & Bengio (2010).
Layer Name
Layer Type PoL
Input Size
Output Size BS 12
Convl
8$$1#28*28
Conv2
BS 1$$10
2$*20*4#4
Linearl
ReLu((inear(input)
BS#320
BS*50
Linear2
soncioonaloogg_sttoxxiiiiiiii softmax((inear((nnut))
BS*50
BS*10
16


Reviews:
[{"review_id": "bgFomwYNFhD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a theoretical analysis of regularization based approaches to the problem of continually learning a sequence of tasks. The point of the paper is to demonstrate shortcomings of these kinds of approaches, in the context of class-incremental learning where classes are observed once and one after another. The authors argue that these kinds of methods require task labels at test time to correctly distinguish classes from different tasks. \n\nI would like to mention as strengths of the paper: \n-\tThe paper is a valuable attempt at examining properties of existing approaches to the problem of continual learning. I appreciate the rigorous approach to the evaluation of the shortcomings of these kinds of methods. \n- The paper is in general well-written and easy to follow. There are no major errors or typos. \n\nThe weaknesses I see in this paper are: \n-\tAlthough there is a clear and formal explanation of why it is not possible to discriminate among classes from different task when there is no access to data from those previous classes, I am not fully convinced that the set of parameters kept from previous classes, and used in regularization-based approaches, do not represent to some extent this data. In particular, there is no clear argument for the claim on page 5: “However, by hypothesis, \\omega_{t-1} does not model the data distribution from C_{t-1} and therefore it does not model data distribution from C_{t-1}  classes.”. I would like to see some discussion regarding how fairly a set of parameters \\theta_{t-1} would represent the S’ set.\n-\tIn terms of the experiments, I consider the number of tasks quite limited. To be convinced I would like to see several tasks (at least 10) and sequential results in terms of tasks learned rather than epochs. \n\nQuestions for authors: \n- Please address my comments on the weaknesses above.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "X2_aLMp5Le8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper argues that regularization-based approaches to continual learning fail to distinguish between classes from different tasks in the class-incremental setting (without relying on “task labels” to simplify the problem). The paper uses a theoretical argument showing it is not possible to assess the discriminability between two classes from different tasks without access to the data (or a model of the data distribution), and also demonstrates a difference in performance when conditioning on task labels.\n\nI think the thesis of the paper is plausible, but unfortunately, I don’t think this is demonstrated in the submission in its current form, and think there are a few flaws in the argument that need to be addressed.\n\nI do think this is an interesting and important line of inquiry, and could lead to valuable insights (eg. proving comprehensively that regularization-based approaches need to do a better job of capturing the data distribution, or showing they can be improved by doing so).\n\nI encourage the authors to address the comments below in lieu of this.\n\nMain comments:\n1) The argument relies heavily on the assumption that regularization-based approaches to supervised continual learning do not model the data distribution, eg. “However by hypothesis, [the regularization-based model] does not model the data distribution ….”\nFirst, this seems to conflate regularization-based methods and discriminative models, as regularization-based methods can be applied to generative modelling tasks (eg. VCL [1]), with a classifier used on learned representations - this scenario is not considered here. Second, there is work showing that discriminative models have some generative capabilities (eg. [2]), and it may be possible to extract this information to better model the data distribution and understand where previously learned decision boundaries are valid.\n2) The theoretical reasoning essentially demonstrates the simple intuition that we cannot assess whether the samples from two classes are linearly separable without having access to both sets of data or a model of the data from the class from the previous task. This insight makes sense, but it’s not clear that it is non-trivial or important: it says nothing about how well (and under what conditions) previously learned decision boundaries may generalise/transfer to a new task, and ignores my point (1) above that discriminative models do carry information about the data distribution.\n3) The empirical results primarily show that performance with some regularization methods is much worse when not using the task labels, in different situations (eg. with disjoint/joint classes in each task). This is known from previous work, since regularization-based methods get much poorer performance in class-incremental learning (ie. single head, no task label) compared to task-incremental learning (multi-head, task label given) - see for example [3][4].\nUnfortunately, these results alone cannot attribute the poor performance to the reasons argued in the paper (ie. the inability to capture previous data distributions, leading to the inability to discriminate classes across tasks), and it would be nice to see experiments that actually measure whether the poor performance is due to the arguments made. The paper also makes a broader claim about regularization-based methods being limited, but only evaluates EWC and K-FAC. Other regularization-based approaches like VCL (without coreset)[1] and BGD [5] could be explored too.\n4) The writing is quite difficult to follow at times: I’d recommend another thorough proofread to address grammar /spelling and clarity issues. See a list of some examples below, but there are many others.\n5) There are a number of papers in continual learning that have been missed - both older regularization-based approaches, and some newer ones (see refs below)\n\nSome writing issues:\n- “inferences” should be “inference” in most places in the text. . eg. “for inference”, “during inference”, etc.\n- Most citations seem to not have parentheses, so they blend into the text. (Eg. “for continual learning Kirkpatrick et al. (2017); …” on page 1). Please use \\citep instead of \\citet when the citation doesn’t flow from the text.\n- Section 3.2: “The regularization term omega act…” should be acts.\n- Also Section 3.2: “a matrix pondering weights importance” - not sure what ‘pondering’ should mean here.\n- Definition 1, “...separating two dataset…”, should be “datasets”\n- I think “disentangled” should be “discriminable” in the Third Part section of Section 4.2.\n\nReferences:\n[1] Nguyen, Cuong V., et al. \"Variational continual learning.\" arXiv preprint arXiv:1710.10628 (2017).\n[2] Grathwohl, Will, et al. \"Your classifier is secretly an energy based model and you should treat it like one.\" arXiv preprint arXiv:1912.03263 (2019).\n[3] Hsu, Yen-Chang, et al. \"Re-evaluating continual learning scenarios: A categorization and case for strong baselines.\" arXiv preprint arXiv:1810.12488 (2018).\n[4] van de Ven, Gido M., and Andreas S. Tolias. \"Three scenarios for continual learning.\" arXiv preprint arXiv:1904.07734 (2019).\n[5] Zeno, Chen, et al. \"Task agnostic continual learning using online variational bayes.\" arXiv preprint arXiv:1803.10123 (2018).\n[6] Aljundi, Rahaf, et al. \"Memory aware synapses: Learning what (not) to forget.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n[7] Aljundi, R., Lin, M., Goujaud, B., & Bengio, Y. (2019). Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems (pp. 11816-11825).\n[8] Rao, D., Visin, F., Rusu, A., Pascanu, R., Teh, Y. W., & Hadsell, R. (2019). Continual unsupervised representation learning. In Advances in Neural Information Processing Systems (pp. 7647-7657).\n[9] Chaudhry, Arslan, et al. \"Using hindsight to anchor past knowledge in continual learning.\" arXiv preprint arXiv:2002.08165 (2020).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Pjn-Nq1S91V", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an interesting conclusion that kinds of regularizations cannot learn to discriminate classes from different learning tasks. Both theoretical and experimental analysis is provided. Two regularization methods are tested.\n\nPro:\n1.\tThe conclusion is interesting and it reveals that regularizations fail when the task label isn’t given. This conclusion questions those proposed regularizations for continual learning.\n2.\tTheoretical analysis of the shortcomings of regularizations in continual learning is presented. Besides, the paper also pointed out supplementary shortcomings of regularization in other types of learning situations.\n\nCons:\n1.\tThe related work is not sufficient. It lacks the introduction and discussion of popular regularization methods in continual learning. \n2.\tThe experiments are implemented on 3 tiny datasets (MNIST, Fashion-MNIST, KMNIST). It’s not sufficient to show the importance of the proposed conclusion. The popular CIFAR100 dataset should be tested. \n3.\tOnly two regularization methods are tested. The popular knowledge distillation based regularization [a] is not tested. \n4.\tAlthough regularizations cannot learn to discriminate classes from different tasks, they can avoid the bias to new training data, which is also important for improving CL performance. The paper would be stronger if the analysis why regularizations can improve CL performance is presented.\n5.\tCan you explain why there is a sharp promotion/decrease of accuracy at the final steps in each learning task (Figure 2, 3)?\n\n[a] Learning without Forgetting, Li et al, 2018.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CupEGj8TnTp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "TL;DR: The paper presents an intuition as to why continual learning with regularization will be unable to discern subsequent tasks. While I believe the intuitions to be correct, the formal laguage used to present this intuition is not sufficiently rigorous to warrant publication.\n\nThe paper sets out to build on the surprisingly sparse literature on the theory of continual learning. The key point the paper wishes to make is that regularization-based Continual Learning approaches are unable to discern tasks without additional information such as task labels at training time. This is attempted in the form of three propositions.\n\nFirst of, I should say that I believe the authors are on the right track & have the correct intuitions: I believe that the claim they want to make is correct, and I believe that classification is a promising setting to prove their claim via separating hyperplanes. In other words, I want to like the paper but its current state does not allow me to recommend its publication. Let me also be transparent: I think this paper needs more than a bandaid, and will in my opinion not be publishable without a ‘major revision’/‘revise and resubmit’ order of magnitude in treatment (to use journal jargon). In essence, my problem is that I do not think the current version of the paper manages to deliver a convincing rigorously mathematical proof of the developed intuitions. Since this is what the paper sets out to do, I thus recommend that its current form not be published.\n\nThe main hindrance throughout is a serious lack in precision. This begins with the definitions: For instance, Definition 2 defines 'interferences' as follows: \n\"In machine learning, interferences are conflicts between two (or more) objective functions leading to prediction errors. There are interferences when optimizing one objective function degrades the optimization of, at least, another one.\" \nWhile this definition captures the intuition of interference, it is not precise enough (e.g., which objective functions, what kind of prediction error, … ?) to be used within a mathematical result as is done in Proposition 4.3, which states that \n\"While learning a sequence of disjoint classification tasks, if the memory $\\Omega$ of the past tasks is only dependent on trained weights and learning criterion of previous task and does not model the past distribution, it is not possible for deep neural networks to learn new tasks without interference.\" \nFor example, it is unclear which 'prediction error' (which is part of the interference definition from Def. 2) or which objective functions (again, they are part of the definition 2) is/are the relevant one/ones for Proposition 4.3. While both answer can be intuited to a certain degree if the reader is eager enough, a rigorous result requires the reader to exactly know all relevant objects in question. What \"model[ling] the past distribution\" is also not precisely defined, but used to state this result.\n\nThis kind of imprecision plagues the paper throughout. E.g., while Lemma 4.1 refers to $(S, S')$ as \"bounded set of discrete points in R^n\" (which is very clear), Lemma 4.2 suddenly calls the same tuple \"two bounded datasets\" without previously defining datasets as bounded sets of discrete points in R^n. While this particular imprecision can be intuited, this sort of imprecision does not help me trusting the result.\n\nSimilar problems plague the proof of Proposition 4.3 — my criticism is no that it is false. Instead, my criticism is that it is written in such vague terms that it becomes impossible to assess if it is correct or incorrect.  In fact, I believe that most of the intuitions I can follow are correct. But this is the problem: the proofs are written in an imprecise way, forcing me to believe rather than convince myself: As the definitions and wording throughout the paper, the statements in the proof lack rigour and are much more informal than what would be required for a watertight proof.  As I understand it, the main thrust of the proof is the following idea: It is possible to rewrite the binary classification problem as the problem of finding a hyperplane between classes. One can further rewrite the continual learning problem as the attempt of finding a hyperplane between a new class and previous classes. By Lemmas 4.1/4.2, this tells us that the problem is not solvable unless we store everything we observed previously in the regulariser. I actually believe that this idea is very elegant, simple, and could be shown rigorously. But I don’t believe the current paper does that. I think comparing the current paper’s approach with the related literature [1,2,3] will benefit the authors, and allow them to clearly define the relevant objects of interest & rigorously prove their interesting result.\n\nRegarding the lemmas, I will also note the following: While both results are clearly correct (in the sense that without the second set of points, the separating hyperplane problem is not even defined), the purpose of Lemmas 4.1/4.2 is doubtful. It is fairly clear that you will not be able to find the separating hyperplane between two collections of points if you are given only one of these collections. Accordingly, the proof of these lemmas feels  like an attempt at proving that you won't know what real number x+y is equal to if you are only given x=3, but y remains unknown. What I am trying to say is that the very problem of finding a separating hyperplane (or of finding the real number x+y) is not defined if one of the two collection of points (or y) is not given.\n\nThough the following is less important than my main points, I also would have enjoyed a more thorough review of the (very sparse) related literature. Since there are very few papers discussing the theory of continual learning, this part should be an easy fix. To the best of my knowledge, there are essentially 2, maybe 3 relevant papers.  \n[1] https://arxiv.org/abs/2006.11942 [unpublished, deriving generalization bounds for Continual Learning]\n[2] https://arxiv.org/abs/2006.05188 [ICML 2020, probably most related to the current paper as they basically show a different ‘impossibility’ result than Proposition 4.3 which I think holds more generally]\n[3] https://arxiv.org/abs/1610.08628 [JMLR 2017, probably least related but still worth discussing, derives regret bounds]\n\n\t\t\t ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers recommend rejecting the paper. However there is agreement that this is an interesting line of research, and the AC agrees. Reviewers provided extensive and well educated feedback. The authors did not respond to the raised concerns.
Decision: Reject