Optimizing Spca-based Continual Learning: A Theoretical Approach
Authors: Chunchun Yang
Malik Tiomoko
Zengfu Wang
Venue: ICLR 2023 poster (2023)
Keywords: continual learning
high dimensional statistics
machine learning theory

Abstract: Catastrophic forgetting and the stability-plasticity dilemma are two major obstacles to continual learning. In this paper we first propose a theoretical analysis of a SPCA-based continual learning algorithm using high dimensional statistics. Second, we design OSCL  (Optimized Spca-based Continual Learning) which builds on a flexible task optimization based on the theory. By optimizing a single task, catastrophic forgetting can be prevented theoretically. While optimizing multi-tasks, the trade-off between integrating knowledge from the new task and retaining previous knowledge of the old task can be achieved by assigning appropriate weights to corresponding tasks in compliance with the objectives. Experimental results confirm that the various theoretical conclusions are robust to a wide range of data distributions. Besides, several applications on synthetic and real data show that the proposed method while being computationally efficient, achieves comparable results with some state of the art.

Published as a conference paper at ICLR 2023
OPTIMIZING SPCA- CONTINUAL LEARNING: A THEORETICAL APPROACH
Chunchun Yang
Malik Tiomoko
University of Science and Technology of China
Huawei Noah's 's Ark Lab
Huawei Noah's Ark Lab
Paris, France
yangchunchun-40 @ hhwawei............e com
ma 1 ikk t www.ommeoahweee...ooooooo com
Zengfu Wang
University of Science and Technology of China
Hefei Institutes of Physical Science, Chinese Academy of Sciences, China z of fwwag@ust.c.e. edu.. en
ABSTRACT
Catastrophic forgetting and the stability- plaatiiiiyy dilemma are two major obsta- cles to continual learning. In this paper, we first propose a theoretical analysis of a SPCA-based continual learning algorithm using high-dimensional statistics. Sec- ond, we design OSCL (Optimized Spca-based Continual Learning) which builds on a flexible task optimization based on the theory. By optimizing single task, catastrophic forgetting can be prevented theoretically. While optimizing multi- tasks, the trade-off between integrating knowledge from the new task and retain- ing previous knowledge of the old tasks can be achieved by assigning appropriate weights to corresponding tasks in compliance with the objectives. Experimental results confirm that the various theoretical conclusions are robust to a wide range of data distributions. Besides, several applications synthetic and real data show that the proposed method while being computationally efficient, achieves compa- rable results with some state of the art.
INTRODUCTION
Continual learning paradigm. Machine learning methods generally learn from samples of data randomly drawn from stationary distribution. However, this scenario is rare in reality. Continual learning (CL) is particular machine learning paradigm in which data continuously arrive in pos- sibly non i.i.d. way and knowledge is accumulated over time (Schlimmer Fisher 1986 Ebrahimi et al. 2019 Lee et al. 2020 De Lange et al., 202I). For designing real-world machine learning systems that mimic humans, continual learning is essential. On the one hand, humans continue to acquire knowledge and solve new problems throughout their lifetimes. The goal of continual learn- ing is to mimic the capacity of humans to learn from a non-stationary data stream without forgetting catastrophically the learned knowledge (Titsias et al. 2019 Lee al. (2020). On the other hand, when deploying a trained model in real applications, the distribution of data will consistently drift over time. Therefore, the machine learning algorithm must be able adapt continuously these
changes KKrkkptrrick et al. 2017 Lesort et al. 2020).
Challenges in continual learning. One of the major challenges of continual learning to avoid catastrophic forgetting (McCloskey & Cohen 1989, Chen & Liu 2018, Aljundi] 2019). This oc- curs when the performance of previous tasks is severely degraded during the learning process. To take into account both the current task and the previous tasks, the suaility-plastiic dilemma was introduced (Nguyen et al. 2017 Rajasegaran et al.) 2019). More specifically, plasticity refers to the ability of integrating new knowledge, and stability to the capacity of retaining previous knowledge (which is related to catastrophic forgetting). Note that the term catastrophic forgetting, although strongly referenced in the literature in deep neural network models, is fairly general concept that can occur in any machine learning algorithm as it has been noted in shadow single-layer models, such self-organizing feature maps (Richardson & Thomas 2008 Chen & Liu 2018).

---

Published as a conference paper at ICLR 2023
State of the art. Most works on CL have focused on purely supervised tasks which is the focus of the present work (Kirkpatrick et al. 2017 Nguyen et al. 2017]] Lee et al.) 2020). Supervised CL can be classified into three main categories based on how knowledge and data are updated and stored: replay, regularization, and dynamic architecture-based methods (De I Lange et al. 2021). Replay- based approaches address the catastrophic forgetting by saving and reusing the previously seen data while learning new task (Lopez-Paz & Ranzato [017]] Isele& & Cosgun 2018 2018 Titsias et al. 2019)) Regularization-basee methods penalize the update of crucial weights to alleviate forgetting by troducing an extra regularization term in the loss function (Kirkpatrick et al. 2017 Ebrahimi et al., 2019). Dynamic architecture-bait
methods flexibly update the learning model as new tasks are added based on the task complexity and the relation between the tasks (Rajasegaran et al. [2019] Lee] al., 2020). Although successful cases of supervised continual learning were reported in articles, the algorithms tend to feature unpredictable behavior, generally requiring a host of additional in- convenient hyperparameters and requiring a high computational cost. More importantly supervised continual learning methods lack theoretical guarantees of avoiding catastrophic forgetting.
Contributions of the paper. In this paper, we introduce a novel CL method based on Supervised Principal Component Analysis (SPCA). We provide theoretical analysis of the proposed method using high-dimensional statistics. This analysis allows us to predict in advance the performance of the algorithm. Furthermore, we develop a label optimization scheme based the theory that avoids catastrophic forgetting theoretically. As a result, we obtain a simple and efficient continual learning algorithm free of hyperparameters, allows us to weight the tasks during the learning process according to the user's 's preferences and priorities. This is in line with the resolution of the stability-plastiiity dilemma. As such, the main contributions of the paper can be summarized as follows. We propose a simple continual learning al- gorithm, computationally inexpensive based on SPCA, and provide a theoretical analysis (to obtain exact classification error rather than bounds) using high dimensional statistics. Using the theoret- ical results, we develop a label optimization scheme that provably prevent catastrophic forgetting and also allows for a weighting of the tasks during learning. Several applications are presented to corroborate the practical usefulness of the approach in terms of efficiency and flexibility.
a low computational cost named OSCL. Moreover, the theory
Outlines. The remainder of the paper is organized as follows. Section 2 discusses several works in the continual learning literature and highlights the differences and contributions of the present paper. Section 3l introduces and formalizes the continual learning framework, and furthermore proposes simple continual learning algorithm based on SPCA. Section&AIrrpov algorithm as well as flexible optimization tools to benefit from all tasks. Section 5 provides several applications to corroborate the different conclusions of the paper.
a theoretical analysis of this
Notations. Matrices will be represented in bold capital letters (e.g., matrix A). Vectors will be represented in bold lowercase letters (e.g., vector v) and scalars will be represented without bold letters (e.g., variable a). The canonical vector of size is denoted by e!n E R", I m I n, where the i-th element is if m, and 0 otherwise. The diagonal matrix with diagonal x and O elsewhere is denoted by Dz Generally, the subscript i refers to the task number, and the superscript j to the class index. As an example, æ/g denotes the ê-th sample of class j for task t. In, E R" is the vector of all ones and the matrix E. E m==x denotes the covariance matrix of the random vector v. [n] denotes the set (1...., n). 3-) n, denotes the number of samples in task t, my is the number of samples in class j of task l.
2 RELATED WORKS
2.1 CONTINUAL LEARNING
As mentioned in the introduction, we focus on the state of the art of supervised CL., which is divided into three main concepts that we develop in detail in this section. The appendix provides a more detailed picture for interested readers.
Rased methods. Replay-based approaches aim to retain a certain amount of historical ex- amples, extracted features, or generated examples to reduce forgetting when training the model with new data. Three challenges need to be solved. The first problem is selecting appropriate previous samples. In this context, Isele & Cosgun| (2018) proposed four strategies for choosing which data will be stored. Aljundi et al. (2019) proposed sample selection constraint reduction problem.
2

---

Published as a conference paper at ICLR 2023
The second challenge is to find suitable format to store data from past tasks. In this direction, Shin et al. (2017); Ayub & Wagner (2020) used a deep generative model to sample pseudo-data points to replay the past experience while Farajtabbr et al. (2020) stored a set of gradients of each task to minimize catastrophic forgetting. The last challenge is to find a better way to use historical data.
Lopez-Paz & Ranzato (2017); Chaudhry et al. (2018); Farajtabar al. (2020) avoided interference between tasks by projecting the new gradients in the feasible region constrained by the gradient(s) of previous tasks. [Wang et al. (2022) achieved better stability-plastiiity trade-off using an attention- based framework while Hersche et al. (2022) composed a frozen feature extractor, trainable fixed size fully connected layer, and a re-writable dynamically growing EM (Explicit Memory) that stores as many vectors as the number of classes encountered. It is important to note that our method does not use any samples from previous tasks and only stores statistical data. Therefore our method is different from the aforementioned algorithms.
Regularization-aase based methods. Regularization- based approaches generally impose a regulariza- tion constraint the objective function to maintain a balance between stability and plasticity. The literature focuses on two main issues. In the first place, it is necessary to evaluate the importance of parameters in the previous model and, in the second place, to limit the adjustment of critical weights. With Kirkpatrick et al.] (2017). such importance is computed from the diagonal of the Fisher information matrix and important weights constrained to change slowly while learning new tasks. [Nguyen et al. (2017) extended online variational inference to handle more general con- tinual learning tasks. Ebrahimi al. (2019); Zeno et al.] (2018) used uncertainty as the importance of the weights and subsequently, adapted the learning rate whereas Jung et al. (2020) considers the average activation value the importance of a node. Our algorithm does not use any regularization and therefore differs from regularization-bee based approaches.
Dynamic architecture- methods. Dynamic architecture- -ased approaches adjust the model architecture according to the complexity of the tasks, and the relation between them. Two questions are critical in this approach: the timing to adjust the model architecture and the procedure to do so. In Fernando et al. (2017) the changing procedure uses evolutionary strategies to select pathways through the network and freeze older pathways. In contrast. Yoon et al. (2018) and Lee et al. (2019) don't modify systematically the architecture given new tasks but take into account the difficulty of the tasks and the relation between them. More specifically, Yoon et al. 2018) used group sparse reg- ularization to dynamically expand the architecture when the accumul ated knowledge alone cannot sufficiently explain the new task. Similarly, Lee et al.] (2019) expands the number of experts under a Bayesian non-parametric framework, which determines the complexity of the model based on the data. Our method keeps the same model from one task to another and does not fit into approaches based on dynamic architectures.
Although the idea of label optimization has been introduced in a transfer learning framework (Tiomoko et al. 2020), this paper the first to propose a line of label optimization in a CL context. As a result, our study is not directly part of the aforementioned CL classification methods but can be considered as a new research line for CL.
2.2 SUPERVISED PRINCIPAL COMPONENT ANALYSIS
Our continual learning approach is based on SPCA. Barshan et al. (011aa) is one of the first to pro- pose SPCA, which is generalization of principal component analysis (PCA) and aims at finding the principal components with maximum dependence on the response variables. Ritchie et al. (2019) presented manifold optimization approach SPCA that simultaneously solves the prediction and dimension reduction problems, which is general enough for both regression and classification set- tings. For a more detailed review, we refer to Ghojogh & Crowley (2019) which explains PCA, SPCA, kernel PCA, and kernel SPCA. The advantage of SPCA in a CL context is its ease of trans- ferring knowledge from previous tasks to the next in addition to its simplicity of theoretical analysis, which is a major advantage over more complicated algorithms such as SVM, LDA, logistic regres- sion, or neural networks.
2.3 ON LARGE DIMENSIONAL ANALYSIS OF MACHINE LEARNING ALGORITHMS
The field of theoretical analysis of machine learning algorithms has become increasingly large with a significant impact on the understanding and optimization of algorithms. Different fields of research
3

---

Published as a conference paper at ICLR 2023
explored, among which we find approaches using the theory of random matrix (Mai & Couil-
let [021 Couillet & Benaych-Grorgee 2016: Liao & Couillet 2019 Seddik et al. [020; Tiomoko
et a |
Niyazi et al.
2021). approaches based physical statistics (Cocco et al. 2018 Zde-
borová & Krzakala|
Malzahn & Opper 2005 Agliari et al. 2020 Carleo et al. 2019: Agliari
al. 2020 Taheeii et al.
202T and other approaches (Convex Gaussian Min Max theorem
lably (Thamppoliidis et al., 2014; Deng et al. (0200). These approaches generally have the same
goal: to predict in advance the classification error or predict the mean square error in a regression framework to anticipate the behavior of the algorithm and improve it. Our study is part of this wave of asymptotic analyses and is more specifically related to the approaches of the theory of random matrices.
Furthermore, using Bayesian and information theoretic arguments, Pentina Lampert 2014 2015); Doan et al. )202T); Alquier et al. (2017); Denevi et al. (2019a b) established bounds for the clas- sification error continual learning framework. Even though these works provide loose bounds and orders of magnitude, they do not provide satisfying and accurate evaluation of performance, despite their convenience for deciding whether an objective is possible. In this paper, we provide a precise and exact theoretical classification error for SPCA-based continual learning.
PRELIMINARIES
3.1 CONTINUAL. LEARNING FRAMEWORK
Let's consider m-class classification problem in d-dimensional space with training data of task t denoted X, [ X Rdx., where X [=/1.-.. ................................ of class j E (1...., m). We denote in short y, (h Vin.. _ E I the vector of all labels of task L.
..mn)) ....., Raxmu are the mej vectors .. To each d/e E IRd is attached a corresponding "label" (or score) vie E IR.
In continual learning setting, when dealing with task I, the user only has access to the data of task 4, i.e,, X, and the associated label yr. and needs predict label for a test data E RR coming from an already seen task 7 E [t]. If the task identity of the test data is known a priori, we refer the setting to the Task Incremental Learning (TIL). The unknown task label during the inference scenario is divided into the Domain Incremental Learning (DIL) where all tasks share the same label space and Class Incremental Learning (CIL) when it changes. van de Ven & Tolias 2019 details more deeply the relationships between the different settings.
3.2 SPCA-BASED CONTINUAL LEARNING ALGORITIM
For simplicity, we will limit ourselves to the binary case for now. Multi-class learning is handled by using binary-based approaches such as one-versus-one and one-versus-all which are explained in Section Jand fully discussed in Appendix SPCA (Barshan et al. (0011b) extracts the principal components of the data that are most dependent on the target variable.
More specifically, in a binary classification context, let the data matrix X = [X X,] E Rdx=, with Σ nly be the collection of the training data of the tasks already seen and their [.]] IT E RR", the decision function of SPCA is given for associated labels or scores = [uT....,vT new test data x by
g(x) = 1 - -
v. v.X.a.
Since (1)
Lin; are i.i.d. data vectors, we impose equal scores Vel within every class j of task t. As such, we may reduce the score vector E R." under the form
(t)
(1) = Vin, denoted Uts
(2)
[ 2 3 1T
for ÿ - M0 Di E 2 This consists in assigning to each class of each task a unique and common label. In classical way, class of all tasks will be assigned score of -1. while the other class will be given a score of 1, i.e., ÿ = [-1, -1, -1, 1]_ The class allocation is therefore performed based on the sign of the decision function g(z)). We will see that the classical
4

---

consider the regime where d Published as a conference paper at ICLR 2023
choice of labels for SPCA based continual learning leads catastrophic forgetting and does not allow us to solve the stability-plastiick dilemma. For this reason, we leave a free choice for the labels scores and we will optimize them using the theory similarly done in Tiomoko et al. (2020) This optimization of labels can be intuitively seen as a weight of different classes with respect their importance in the knowledge preservation.
This being said, Equation I
can then be rewritten a more convenient manner as
2
(
Thi o
je x,
1 nyy" nlj
Therefore, when dealing with task 6, the user just needs have access to the empirical means pil and the label 8e for т S t, which allows an important memory saving compared to sending the raw data.
CL has the important challenge of building flexible optimization method that solves the catas- Copphifoorgetii and the stability-plaxticiy dilemma. To achieve this goal, we propose theoretical analysis of the SPCA-based continual learning algorithm. Precisely, we will characterize the theo- retical classification error of each task such that a flexible optimization scheme can be performed based on the theoretical analysis.
To that end, we will make use of the following assumption on the data distribution.
Assumption (Concentration of X) For class Cj, j {1, 2 of task t, we assume that all vector 2 F......, amny C2 are i.i.d. such that Cov(x Σ and E|æ,1] = nd. Moreover, we assume that there exist two constants C, (independent of n. d) such that, for any I-Lipschitzffunction f: RR R.
PeND(X) (1((z)) - m)((a) 2 1) S Ce- (t/o)²
14 > 0,
where mz is median of the random variable Z.
Assumption I notably encompasses the following scenarios: the columns are (a) indepen- dent Gaussian random vectors with identity covariance, (b) independent random vectors uniformly distributed on the R2 sphere of radius vd, and, most importantly, (c) any Lipschitz continuous transformation thereof, such as GAN as it has been recently theoretically shown in (eddik et al. 2020). An intuitive explanation of Assumption i that the transformed random variable f(x) for any f: IRd + R Lipschitz has a variance of order (1). particular, it implies that it does not de- pend on the initial dimension d. Furthermore, we place ourselves in the following large-dimensional regime.
Assumption (lligh-dimensiona) asymptotics) As
(n) and assume d/n - co 0. Furthermore, we assume that nej/n + cl.
Section 4 proposes theoretical analysis of the classification error of SPCA-based continual learning algorithm.
Under Assumptions 4
4 THEORETICAL ANALYSIS AND OPTIMIZATION FRAMEWORK
4.1 THEORETICAL ANALYSIS OF THE SPCA-BASED CONTINUAL LEARNING
Theorem Under Assumptions 2 for test data vector of class in task t E [t], following Assumption i with E|xx] me and Σn Ej
g(x) G., 0 G ~ NN(m,, o4?)
in distribution where.
m 8J TT
MT Me.21 +j
-²-- - D.M
where [2t] [2]]
2x a one-hot vector, with the value at position x T + j is one. s = lc].... d). D. and Da are the diagonal ma-
[st...,,sf 3f] E 2t with sf 1 E.)))))
trix with and as diagonal elements and M [ 1 E Rdx2
5

---

Published as a conference paper at ICLR 2023
It is interesting to note that the theoretical high dimenssonn performance of SPCA-based continual learning depends only on small data statistics. This is in line with many analyses generally conducted in the field of aaggediiensiionl analysis. More specifically, in our these sufficient statistics are: 1) the correlation between the task means through M M, 2) the correlation between the covariance matrices through the vector and 3) the interaction between the means and covariance through M M. Note that these quantities incorporate the interaction between the tasks and the classes. We would like to mention that these statistics are easily estimated using empirical means and covariance matrices as explained in Remark [ of the Appendix B More importantly, performance depends on labels ÿ we have chosen to leave as changeable parameters. Next section will discuss the optimization of these labels in order to avoid forgeling and resolve the stubility-plaxtiiity dilemma. Section illustrates this experimentally.
Since g(r)) has Gaussian limit centered around m/. the asymptotic standard decision for x to be allocated to class 1 or class 2 for task T is obtained by
g(z) $
Y
for some threshold ?. To minimize the classification enror, the optimal threshold y" for separating two Gaussians lies the intersection between these two Gaussians and can be determined numeri- cally as function of m² and o... The resulting classification error rate for task is given as
e, 5 = (mt!
m --)= + +Q
m
(4)
oZ
with m² and of defined Theorem and where Q(z) 22n S /2 is the cumulative distribution function of standard normal distribution. As already mentioned, we defer the estimation of m² and of to Remark I in the Appendix
4.2 OPTIMIZATION FRAMEWORK
Having access priori to the theoretical classification error depending on the label ÿ. one can mini- mize the theoretical classification error as function of y. Interestingly. for the specific case where the weights of the tasks are w = et (which refers to optimizing the single task r), the optimal label has closed form solution (see details of the derivation in Appendix B
म = (D.D. D.M MMM,,,MD
E,,D). D...++DDDD.... D.M Me [2d]
2
where d! denotes the optimal label for test task r when t tasks are seen.
Theorem ( On catastrophic forgetting) Under Assumptions 1 Vt and T [t). (xl(tall" I (g(!"))
where e, ('!!!' is the classification error rate of task with optimal label after the model learned
on task
0,,0J E R2(++),, the vector أ!!' to which we add two-zeros for both classes of task + 1, have: e, ( (5l(+!)) 2 e, (,ff+11)" The equality is due to the fact that the addition of label to both classes of a new task doesn't contribute
Proof-sketch. If we denote by 97 [[++1] !n!t"
to the decision function (see equation 2). The inequality comes from that -!(+1]* is the optimal vector to minimize the classification eтror of task after seen task t + 1. Therefore, Theorem 2 ensures theoretically that catastrophic forgetting prevented when adding more tasks. This will be verified empirically in Section 5
In the case of a generic weight vector w (to optimize the multi-tasks), the optimization of ğ performed using Scipy optimization toolbox (Bressert (212)) with the cost function given as L(ÿ)
Σi= W+C e, (ū), where the weights w, depending the importance that user wants to put task T and (ÿ) is given by equation| 4 The optimization framework becomes flexible, allowing the stability and plasticity problems to be traded off and this will be illustrated in the Section

---

Published as a conference paper at ICLR 2023
4.3 ON THE MULTI-CLASS EXTENSION
Having an optimized framework for binary classifier, our approach to multi class setting aims to resort on one-versus-one or alternatively one-versus-all approach. In one-versusoone scheme, =m(m 1) binary classifiers are trained (one for each pair of class j and class j', solving a binary classification problem). For each test sample, each binary classifier decides on or votes for the more relevant class. The test sample is then attributed to the class having the majority of votes. Each binary classifier is optimized by choosing appropriate labels and decision thresholds, thereby largely improving over the basal classifier performance. The one- one-versus-all approach applied to m classes consists to build m binary classifiers where for classifier j [m]] the first class class j and all other classes j' * are gathered into one common class. The final decision is obtained by taking the argmin over the m classifiers. An inherent problem of one-versus-all is the scales and alignments of the binary classifiers to be compared which are different. We solve this problem thanks to the theoretical statistics of g(x:) per classifier in addition to the optimization of the basal classifier. This procedure is thoroughly discussed in the Appendix C Our algorithm (one-versus- one) is summarized as Algorithm Appendix and the code available online
APPLICATIONS
5.1
DATASETS DESCRIPTION
Through the experimental part, we will use 5 data sets (Synthetic, Permuted MNIST de- noted PMNIST, Split MNIST denoted SMNIST, Rotated MNIST denoted RMNIST, Split Fashion MNIST denoted as SFMNIS7). See more details in the Appendix D
5.2 THEORETICAL VERSUS EMPIRICAL PREDICTION
The purpose of this section is twofold. Our first goal is to demonstrate the robustness of the theo- retical analysis when applied to synthetic and real data by comparing the theoretical and empirical predictions. Secondly, we would like to highlight the significant improvement brought by the op- timization of labels and illustrate that catastrophic forgetting is avoided. Figure [ represents the empirical histogram of the decision function with the theoretical prediction for classical and opti- mized schemes. It is noticeable that the concentration random vector assumption is quite realistic since the theoretical and empirical predictions are close to each other.
Next we illustrate how catastrophic forgetting is avoided. To do so, we represent 3 examples in Figure 2 In the synthetic case we compare the average theoretical and empirical classification accuracy both for the classical and optimized scheme function of an increasing number of tasks. Here, optimization assumes equal weight for all tasks encountered. We can see a close fit between theory and practice while remarking that the performance of OSCL never decreases opposed of its counterpart non optimized (SPCA) which forgets knowledge as new tasks are encountered. In the second and third picture (preformed on PMNIST and RMNIST), we represent the trajectory of the accuracy for each seen task as a function of the task number we observed so far. Table [ shows the average forgetting measure where the values are all negative. We remark that by adding new tasks, we always improve the accuracy previously obtained, thus avoiding catastrophic forgetting.
Table Average forgetting of each task on PermutedMNIIT and RotatedMNIST (TIL 4).
Task
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
PMNIST
-1.019
-0.915
-0.473
-0.493
-0.150
-0.272
-0.206
-0.149
-0.221
RMNIST
-1.63
-1.840
-0.660
-0.480
-0.444
-0.330
-0.110
-0.219
-0.129
5.3 ON THE FLEXIBILITY OF THE FRAMEWORK
In this section, we investigate the flexibility of the optimization framework by assigning different weights to different tasks. Considering a two-task binary classification problem, Figure] shows
'Code is released at http:///iicccooaacccmhhooooeeeee
and

---

Published as a conference paper at ICLR 2023
<ixi of SPCA
01X of SPCA
QU0 of SPCA
C, (Emi))
XEC
2.0
C (Er6))
Cy C2-tmm))
XEG
X C, (T
Densty
VEG
21.5
XEG TN
XEC
3..05
XEC
8 81.0
X
0.5
0.0
0.00
0.0
-0.5
1.0
g(4]
9Xx]
90X1
(a) Synthetic
(b) PMNIST (4-9)
(c) RMNIST (4-9)
g0xt OSCL
gixt of OSCL
g(x) of OSCL
AEG: Seai
(Empl
0.8
xCC, (Empp)
Em
C2 (Erp
XEC, (Emi)
0.3
XEC (hh
Dansty
XEC, CTR
x C; [Th)
0.2
XEC (Th)
CE TTN
0.4
CE (Th
0.0
-2.5
0.0
2.5
91x)
g(x)
gbx)
(d) Synthetic
(c) PMNIST (4-9)
(f) RMNIST (4-9)
Figure 1: Empirical histogram versus theoretical prediction for the decision function g(x) of task 2 in two-task binary classification problem for the classical label (denoted SPCA) and for the optimized label (the optimization is performed with equal weight for both tasks). Close between theoretical and empirical predictions and better separation of the two classes with the optimized scheme in the bottom (denoted OSCL).
Syrthetic eccuracy OSCL)
accurary (OSCL)
Emiist arcuracy (OSCI)
0.94
RDC 30C_2
0.51
ESC 2eRi3
9.50
0.00
OSC) (TN) (Srrpt
ESC I0RCS
EDC SEMCA
accuracy
800
RECOME
0.56
SPCA1Emo)
0.67
0.83
0.06
0.82
20
30
tadt num
tasc num
task num
(a) Synthetic
(b) PMNIST
(c) RMNIST
Figure 2: (Left) Empirical and theoretical classification accuracy of task w.t.t task number for the classical label (SPCA) and optimized label (OSCL). (Middle-Right) The accuracy trajectory of all seen tasks as function of task number with the optimized label. In the learning process, the accuracy of tasks already seen increases over time.
the classification error of task 1 compared with task 2. Each coordinate of the point represents the classification error of task and along with the weight (wi, w2) next to the point. We showed the and a goes from 0.8 to performance of the weighting scheme with (Wi, w2) = 'Va2+ va2+32
0.2 and f goes from 0.2 to 0.8 with step size of 0.1 in Figure 3 In the same figure, a red cross represents the classification error of the classical label. A task that is given more weight is prioritized by the algorithm over a task that is given less weight. As a result of this flexibility, certain tasks can be favored at the user's discretion.
5.4 COMPARISON WITH SOME STATE OF THE ART ALGORITHMS
In this section, we compare our method to a number of state-of-the-art approaches to show the performance and the low computational cost of our algorithm. Four datasets and three scenarios (TIL, DIL and CIL) are used to fully validate the properties of our method.
For SMNIST and PMNIST, the experimental protocols following KJ & N Balasubramanian 2020 and the task number of RMNIST and SFMNIST are 10 and 5, respectively. More details about ex- perimental setting will be discussed in Appendix D From Table 2 and Table
3 as we can see, OSCL.
8

---

Published as a conference paper at ICLR 2023
weighding (synthetic)
weighting (mmnist)
0.13
0.18
0.12
0.16
task
10.92, 0.29)
0.10
(0.53.0.55)
SPCA
050L(ter1-err2
erar
0.12
0.14
0.291
10.82 0..9)
OSCL (em1-em2) em)
SPCA
0.09
0.00
(0./1, 0.73)
10 00.55.000 0.81)
0.97)
0.10
10.71, 0.713
10.55, 0.831
20.19 0.92)
(0.24.0,00)
10.39.6 92)
0.08
(a) Synthetic
(b) RMNIST
Figure 3: Classification error of task 1 versus task 2. For OSCL, different weights are used to trade off the performance of tasks and 2. Weights are formatted in (W1, w2). with wa and W2 representing the weight of task and task 2, respectively. In general, the induced error is smaller when the task weight is greater.
Table 2: Average accuracy of the five baseline methods and the proposed one across datatests.
Dataset
SMNIST
PMNIST
RMNIST
SFMNIST
scenario
CII.
TI).
DIL.
DIL (t)
CII. (f)
EWC Kirkpatrick et al. 2017
45.1
13.64=022
7.19
78.35 + 0.73
19.90
+ 0.10
GEM (Lopez- Paz & Ranzato 2017
36.7
97.68 0.32
82.5
8..45
77.57
1 0.20
iCaRL (ebbuh et al. 2017
0.99
98.30
80.70
1 1.29
SI (Zenke al. 2017
45.2 0.15
96.56
0.58
74.5 1
2.50
71.91
+ 5.83
20.00
1 1.50
MERLIN (KJ & N Bulasubramanian 2020
90.7 = 0.80
0.80
9.34
85.5
OSCL (ours)
91.2:10.10
98.0
77.4110.81
72.7:10.42
90.0:10.22
Table 3: Running time on SMNIST.
Methods
EWC
GEM
iCaRL
SI
MERLIN
OSCL
Running time (s) (L) 177.242.92
263.0.1..1 16 407 92.21..2200
44.0+1126
2218.96-1474.00
9..35-1.36
achieves competitive results with some baseline methods while being simpler in terms complexity. Which means our method is more useful for low computational resource hardware devices. Since we use a linear classifier we have extracted HOG features (Dalal Triggs 2005) for some images to fairly compare with deep neural network based approaches. We also validate the effectiveness of using a deep neural network as a feature extractor and our method as a classifier. With the feature extracted by a pretrained convolutional neural networks, the performance will improve from 1.1% to 6.5% on different dataset compare with using HOG features, as detailed in the Appendix D
CONCLUDING REMARKS
In this paper, SPCA-basee continual learning algorithm is analyzed from high-dimensional per- spective. Using the theoretical analysis, we propose an optimization the labels that avoids catas- trophic forgetting. In addition, the label optimization flexibly weight tasks during learning, thereby resolving the - dilemma. The method shows competitive results with the state of the art while being computationally inexpensive.
This work can be extended to the analysis of continual learning algorithm based on SVM and the integration of nonlinear kernels, well the combination with the neural networks.

---

Published as a conference paper at ICLR 2023
REFERENCES
Elena Agliari, Adriano Barra, Peter Sollich, and Lenka Zdeborova. Machine learning and statistical physics: theory, inspiration, application. Journal of Physics Special, 2020.
Rahaf Aljundi. Continual learning in neural networks. arXiv preprint
2019.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.
Pierre Alquier, Massimiliano Pontil, et al. Regret bounds for lifelong learning. In Artificial Intelli- gence and Statistics, pp. 261-269. PMLR, 2017.
Ali Ayub and Alan Wagner. Eec: Learning to encode and regenerate images for continual learning. In International Conference on Learning Representations, 2020.
Elnaz Barshan, Ali Ghodsi, Zohreh Azimifar, and Mansoor Zolghadri Jahromi. Supervised principal component analysis: Visualization, classification and regression on subspaces and submanifolds. Pattern Recognition, 44(7):1357-1371) la.
Elnaz Barshan, Ali Ghodsi, Zohreh Azimifar, and Mansoor Zolghadri Jahromi. Supervised principal component analysis: Visualization, classification and regression subspaces and submanifolds. Puttern Recognition, 44(7):1357-1371, 2011b. ISSN 0031-3203, doi: https:///oii.o orw.:/....///.00. 1016/j.
2atcoo.2210.12..0 12.015. URL https:////// www. sci ecciecedi rect com/ wtt.cccccccc..eeocccc hamm//ccennce//rrie artienneeeariieeeeeeeeeee
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Eli Bressert. Scipy and numpy: an overview for developers. 2012.
Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Re- views of Modern Physics, 91(4):045002, 2019.
Arslan Chaudhry, MarcAurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference Learning Representations, 2018
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1-207) 2018.
Simona Cocco, Rémi Monasson, Lorenzo Posani, Sophie Rosay, and Jérôme Tubiana. Statistical physics and representations real and artificial neural networks. Physica A: Statistical Mechanics and its Applications, 504:45-76, 2018.
Romain Couillet and Florent Seenaych-Georgee Kernel spectral clustering of large dimensional data. Electronic Journal of Statisties, 10(1):1393 1et11133-145551 1454,, 2016.
Navneet Dalal and Bill Triggs. Histograms oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05) (05), volume 1, pp. 886-893. Ieee, 2005.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-338, 2021.
Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-lean stochastic gradient descent with biased regularization. In International Conference on Machine Learning, pp. 1566-1575. PMLR, 2019a.
Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online- -intn- meta-learning. Advances in Neural Information Processing Systems, 32, 2019b.
10

---

Published as a conference paper at ICLR 2023
Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A model of double descent for high- dimensional logistic regression. In ICASSP 2020-2020 JEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp- 4267-4271. IEEE, 2020.
Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Stutistccs, pp. 1072-1080. PMIR, 2021.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guide continual learning with bayesian neural networks. arXiv preprint DRKi:vI906.02422 2019.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773.
PMLR, 2020.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 2017.
Bruno Fleury, Olivier Guédon, and Grigoris Paouris. A stability result for mean width of ceneroooi bodies. Advances in Mathematics, 2 214(2):865-877, 2007.
Benyamin Ghojogh and Mark Crowley. Unsupervised and supervised principal component analysis: Tutorial. arXiv preprint /901:3/990003//// 2019.
Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Ab- bas Rahimi. Constrained few-shot class itaas-incrementl learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9057-9061 2022.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node- importance based adaptive group sparse regularization. Advances in Neural Information Process- ing Systems, 33:3647-3653 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnkka-aaaiikaa,,.aa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences,
114(13):321-3326, 2017.
Joseph KJ and Vineeth Balasubramanian. Meta-consolidation for continual learning. Advances in Neural Information Processing Systems, 33:14374-114535 2020.
Bo' az Klartag. A central limit theorem for convex sets. Inventiones mathematicae, 168(1)911-131, 2007.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In International Conference on Learning Representationss 2019.
Soochan Lec, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-fred continual learning. arXiv preprint arXiv:2001.00668 2020.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez. Continual learning for robotics: Definition, framework, learning strategies, op- portunities and challenges. Information fusion, 58:52-68, 2020.
Ren-Cang Li. Rayleigh quotient based optimization methods for eigenvalue problems. In Matrix Functions and Matrix Equations, pp. 76-108. World Scientific, 2015.
Zhenyu Liao and Romain Couillet. A large dimensional analysis least squares support vector machines. IEFE Transactions Signal Processing, 1064-440611 2019.

---

Published as a conference paper at ICLR 2023
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Xiaoyi Mai and Romain Couillet. Consistent semi- supervise graph regularization for high dimen- sional data. Mach. Learn. Res., 22:94:1-94:48, 2021. URL htt.nttt:://mm...rr//pppeer
v22 / 9- 081. html
Dörthe Malzahn and Manfred Opper. A statistical physics approach for the analysis of machine learning algorithms on real data. Journal of Statistical Mechanics: Theory and Experiment, 2005 (11):P11001, 2005.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and molivation, volume 24, pp. 109-165. Elsevier, 1989.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint 1710:11100. 2017.
Lama B Niyazi, Abla Kammoun, Hayssam Dahrouj, Mohamed-Sling Alouini, and Tareq Al- Naffouri. Weight vector tuning and asymptotic analysis of binary linear classifiers. arXiv preprint arXiv:21 10.00567, 2021.
Anastasia Pentina and Christoph Lampert. pac-bayesian bound for lifelong learning. In Interna- tional Conference on Machine Learning, pp. 991-999. PMLR, 2014.
Anastasia Pentina and Christoph H Lampert. Lifelong learning with non-iid tasks. Advances in Neural Information Processing Systems, 28, 2015.
Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Ran- dom path selection for incremental learning. Advances in Neural Information Processing Systems, 3, 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Fiona M Richardson and Michael SC Thomas. Critical periods and catastrophic interference ef- fects in the development of self-organizing feature maps. Developmental science, 11(3):371-389 2008.
Alexander Ritchie, Clayton Scott, Laura Balzano, Daniel Kessler, and Chandra S Sripada. Su- pervised principal component analysis via manifold optimization. In 2019 IEEE Data Science Workshop (DSW), pp. 6-10. IEEE, 2019.
Jeffrey C Schlimmer and Douglas Fisher. A case study of incremental concept induction. In AAAI, volume 86, pp. 496-501, 1986.
Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti, and Romain Couillet. Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures. In International Conference on Machine Learning, pp. 8573-8582, PMLR, 2020.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Ghada AZN Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Learning invariant rep- resentation for continual learning. In 35th Conference Artificial Intelligence, AAAI 2021, 2020.
Hossein Taheri, Ramtin Pedarsani, and Christos Thrampoulidis. Fundamental limits of ridge- regularized empirical risk minimization in high dimensions. In International Conference on Arti- ficial Intelligence and Statistics, pp. 2773-2781. PMLR, 2021.
Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. The gaussian min-max theorem in the presence of convexity. arXiv preprint arXiv: 1408 4837, 2014.
12

---

Published as a conference paper at ICLR 2023
Malik Tiomoko, Hafiz Tiomoko Ali, and Romain Couillet. Deciphering and optimizing multi-task learning: a random matrix approach. In International Conference on Learning Representations, 2020.
Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. arXiv preprint arXiv:/901. 13356, 2019.
Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv e-prints, pp. arXiv-1904, 2019.
Zhen Wang, Liu Liu, Yiqun Duan, Yajing Kong, and Dacheng Tao. Continual learning with lifelong vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 171-181, 2022.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
Lenka Zdeborová and Florent Krzakala. Statistical physies of inference: Thresholds and algorithms. Advances in Physics, 65(5):453-55) 2016.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995, PMIR, 2017.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv: 1803. 10123, 2018.
ABSTRACT
The appendix contains the main technical arguments omitted in the core of the article due to space limitations, and is organized as follows.
Section 2 complements the related works with diagram illustrating some other CL methods. Section B derives the asymptotic classification error of the SPCA- based CL algorithm. To this end, Section B.1 proves the Gaussian distribution of the classification score under concentrated random vector assumptions. Sec- tion B.2 and B.3 computes respectively the first and second order moment of the classification score.
Section C complements the explanation of the multi-class extensions more pre- cisely the one-versus-dl approach. Section D provides additional experiments and additional insights into the experiments derived the main paper.
A COMPLEMENTARY SECTION TO RELATED WORKS
To better summarize the section the state of the art, provide rather didactic diagram in Figure 4 that we have constructed to classify the different methods of continual learning. Our method could be classified in the theoretical part but with important algorithmic implications of efficiency, flexibility and computation time.
B DERIVATION OF THE TECHNICAL RESULTS
In this section, we perform a theoretical analysis of SPCA-based continual learning. order to derive the theoretical classification error, need to understand the statistical behavior of the deci- sion function g(x) = y x T x in particular its distribution and compute its first and second order statistics.
To that end we recall the assumption about the data (Assumption 3 and the growth rate of the timension(Assumngnee
13

---

Published as a conference paper at ICLR 2023
Continual Learning
Supervised
repley
Dynamic architectures
Regularization
Raw samples Beueraciinn semph CTh
gradient
expansion
mask/parhviiaree
Sayesan
non boyesian
MI
sew
class-incre
0uD MMM
Proerssive IV seuisn u Ush Ema
OC <4OM
em lot kabhharnn Restl.
Ceneralied
IMM SCD OTS
tue
IAAS
469 Ct
Cereenti
Fil scon PXL
CPR
learning
opdmization
sequence//ii
GNN CM. SCAV
otheess
28-a unt Eu inttil Inlepiin jeaks
Prpe wivors
Recorillgmmm--eeenng and learting
Lauriing trddacconiie
KAAL 1910
ANC dord
Semi-supervised
Self-supervised CL
Continual clustering
Continual domsin adaptation
Ulldang Samkis hedllaurnng
How Waitbe superssed fra Forfuere sath steooiing cata
Ukving spoctal Chitainng
MA5S
Theoretical weaeootiicalexperiniitee eeppermmettellllll analysis of CL
Undertancing Role Traitin Regines Continal
dgstinal Contreual saaning has perfect kimmony andis
Figure 4: Summary of related works
Assumption (Concentration of X) For class Cj. [1, of task we assume that all vectors pi....., Tmny E Cj are i.i.d. such that Cov(z)1 = Σ and E[ = ud. Moreover, we assume that there exist two constants C, > (independent of d) such that, for any fisischitz,function f: RRdd R
Px)D(x) (If(x) - m(a)) 2 (t/o6))
Vt > 0,
where mz is median of the random variable Z.
Assumption (Hiig-dimennsionll asymptoties) As n - DG, we consider the regime where d = (n) and assume d/n - Co > 0. Furthermore, we assume that ntj/n + oe and we will denote ......,.
ON THE DISTRIBUTION OF THE DECISION SCORE.
In this section, the goal is study the asymptotic behavior of g(x))a e Cy.
Since conditionally the training data X, y X x expressed as the projection of the determinis- tic vector X on the Gaussian random vector x, it follows that y' X x is asymptotically Gaussian. The proof under the concentrated random vector unfolds from the following proposition.
Theorem (CLT for concentrated random vectors (Klartag 2007 Fleury et al., 2007) If x Rdd is a concentrated random vector with E[x] 0, E a= order (l) and a be the uniform measure the sphere Sd-1 Rd of radius 1, then for any integer k. small compared to d, there exist two constants C, e and set O c (Sd-13k C RRack such that
La with an observable diameter of
② @ o ( 2 1- 11--ddo VdCe-oVi and for all f = (0I...... 0x) E e,
P 10 2 Fa, < C Va e R
sup
EER
where F0,1 is the cunulative distribution function of the standard normal distribution (O, 1).
Since g(x) is asymptotically Gaussian, we will focus on computing its first and second order mo- ments.
14

---

Published as a conference paper at ICLR 2023
B.2 FIRST ORDER MOMENT OF THE DECISION SCORE.
Due to the independence between the training and test dataset X and æ, the mean for task / reads as
1
m
Blg(a) cy]
1 Ely x xc x E Cy]
n
1 y 'E E [ x C₁]
72
1 J JM Mei mT]
n
22 where De stands for the diagonal matrix containing on its diagonal elements the elements of vector [m.1/n..... T22: n
DDM Me,
B.3 SECOND ORDER MOMENT OF THE DECISION SCORE.
The variance of the decision score g(x) reads as
Varlg(æ)) = (lg(x)² [1)
1
1
Ely T X xxx T Xy)
Ely X 1 X
n2
n
1
Ely T X T
Cx Xy]
-
Ely X < Xy
n2
Moreover have that
E3 x C2 = =
CeEaa i#j
B Ca = tr (C,Cx)
Therefore
1 Flyy Dry
1 + n²
Var(g(>))
Ely JM EMM'M'J'
Tr(En1 Ex)
0
0
0
0
where Dy =
E RRxx
Tr Tr(Em' >.)
We deduce the final expression of the variance as Var(g(x)) = (D.D. E, M.M)
with [s]...., {2] E 2 with sf = Ex)
Remark (Estimation of m and σạ) All the quantities defined in Theorem are known a priori except the scalar products M M, E. and the vector Note that ( Ex,) is consis- tent estimator for s² with E) and En, the empirical covariance matrices. For large Gaussian data, M M can be effectively estimated as Tiomoko et (2020)
1rld:1 3 1T.
X 11i: my + (1)
MM M |l
m.
X X In + o1(
with = 2((T 1) + j and g' = 2(( - -1) + different and X,j = [X X Xryaa an even-sized division of X
15

---

Published as a conference paper at ICLR 2023
B.4 DERIVATION OF THE OPTIMAL LABEL
In the case of identical covariance for both classes i.e. Σ;- = Σx, the optimal threshold y is given
2 (m) + m?)
The associated classification error is given by
m m
6 =
20,
Therefore
m m
أاا² arg min Q
20; ,
(m? m
arg max
20?
g
ÿ D.. Mejj [20] [21]
M MD.9
tj
arg max
2ÿ (D,D. + D MD.)
The previous optimization is known as the optimization of Rayleigh quotient convex and the optimal solution is given
2015 which is
G! 11" = (D,Do + DOM'S, D Me
22
C ON THE MULTI-CLASS EXTENSION
The literature Bishop & Nasrabadi (2006) describes broad groups of approaches to deal with classi- fication with 2 classes. We focus here on the most common method, namely the one-versus-all approach. The complete optimization of one versus-all being theoretically heavy to handle and demanding prior knowledge on the decision output statistics, the method inherently suffers from sometimes severe practical limitations; these are partly tackled here exploiting the large dimen- sional analysis performed this article. In one-versu-oone setting, fmm(m -1) binary classifiers are trained (one for each pair j' of classes, solving binary classification). For each test sample, each binary classifier decides on - or "votes" for - the more relevant class. The test sample is then attributed to the class having the majority of votes. Although the number of binary classifiers is greater than in the one- one-versus-all all approach, the training process for each classifier is faster since the training database is much smaller for each binary classifier. Besides, the method is more robust to class imbalances (since only pairwise comparisons are made) but suffers from an undecidability limitation in the of equal numbers of majority votes for two more classes.
In the one-versus-al method, focusing on Task f, m individual binary classifiers, indexed by & = I,..., m, are trained, each of them separating Class Cie from the other m classes Ce., L' of l. Each test sample is then allocated to the class index corresponding to the classifier reaching the highest of the m classifier scores. Although quite used in practice, the approach first suffers a severe unbalanced data bias when using binary (+1) labels as the set of negative labels in each binary classification on average
times larger than the of positive labels, and also suffers center-scale issue when ultimately comparing the outputs of the m decision functions, the average locations and ranges of which may greatly differ; these issues lead to undesirable effects, as reported in (Bishop & Nasrabadi 2006 section 7.1.3)).
These problems are here simultancously addressed: specifically, having access to the large dimen- sional statistics of the classification scores allows us to appropriately center and scale the scores. Each centered-scaled binary classifier is then further optimized by appropriately selecting the class labels (different from +1) to minimize the resulting classification error. See Figure 5 a venient illustration of the improvement induced by this centering-scaling and label optimization approach.
16

---

Published as a conference paper at ICLR 2023
C.I ONE ALL MULTI-CLASS OPTIMIZATION
For each target task I, in a one-to-all approach, m SPCA-based CL binary classifications are solved with the target class C+s (renamed "class C, "), against all other Ct22 classes (combined into single "C,2 class"). Calling ( the output of the classifier for new datum x, the class allocation decision (1)
(m²) is traditionally based the largest among all scores fx -x,t- However, this presumes that C2. etc., more less have the same statistical mean and variance. This is not the case in general, as depicted in the first column of Figure 5) where data from class C₁ more likely be allocated to class C3 (compare the red curves).
the distribution of the scores 9x (1) when x C1, 9x
(2) when
By providing an accurate estimate of the distribution of the scores 8x for all l's and all genuine classes x, Theorem of the main article allows us to predict the various positions of the Gaussian curves in Figure In particular, it is possible, for each binary classifier to center and scale ( when x Cg. This operation averts the centering and scaling biases depicted in the first column of Figure 5 the result of the center-scale operation appears in the second column of Figure
This first improvement step simplifies the algorithm which now boils down determining the index of the largest ( m l (1.....m) while limiting the risks induced by the center-scale biases.
This being said, our theoretical analysis further allows to adapt the input labels g(() in such a way optimize the expected output. Ideally, assuming genuinely belongs class Ct, one may aim to increase the distance between the output score gx
(E') (2) and the other output scores Jx for l +
This is performed by maximizing the distance between the output score gx² for E C. and the scores 9x for x & C.. By "mechanically" pushing away all wrong decisions, this ensures that, when E Cr. gxx is greater than g% (e") for E' + l. This is visually seen in the third column of Figure where the distances between the rightmost Gaussians and the other two is increased when compared to the second column, and retrieve the desired behavior. Algorithm I and C.1 summarizes the OSCL for one-versus-one and one-versus-al schemes
Algorithm Proposed OSCL algorithm (one-versus-one).
Input: Training samples of current task t i.e., X, [X] X.") X E Rox",t, test data and empirical means and covariance matrices of previous tasks p Σ/ for T Output: Estimated class i E {I.....m) of x.
for j = to do for j' E [1..... m) {J'} do Estimate: M.M M M and according Remark
Create optimal scores ÿ (J', j) = arg min, Σx= w.C,(
Compute classification scores according to equation deduce the predicted class c(j.j') = j or c(j. j') j' based on the decision rule in equation
Update the mean matrix M and Σ by sending the empirical means p and covariance for using in the next task
matrices Σ end for end for
Output: j= mode
{
j' J'
Remark 2 (On the complexity of OSCL) The complexity of OSCL depends on the multi-class ex- tension chosen. OSCL may be sequentially described as performing m - times (for the one- versus-one) and m. times (for the one-versus-all the following procedure in computing the deci- sion score g(x) as per equation 3 2) estimation of M M, M E E. and s 3) optimizaaiio of ÿ. The complexity of computing (<) is matrix-vector multiplication, yielding complexity O(n2).
Thhe mode of set of indices is defined the most frequent value. When multiple indices occur equally frequently, the smallest of those indices is considered by convention.
17

---

Published as a conference paper at ICLR 2023
Classical
Scaled scores
Optimized labels
2 6
04
0.
133
02
0.1
01
03-
03-
0.1
0.1
-0.5
03-
03-
0.6
0.1
0.1
1.5
11
0.5
0.5
Figure 5: Test score distribution in 2-task and 3 classes-per-task setting, using one-versus-all multi-class classification. Every graph in row E depicts the limiting distributions of gx for in dif- ferent classes. Column (Classical) is the standard implementation of the one-versus-all approach. Column 2 (Scaled scores) is the output for centered and scaled g for C.. Column 3 (Optimized labels) is the same as Column 2 but with optimized input scores (labels) (a(()) Under "classical" approach, data from C1 (red curves) will often be misclassified as C2. With "optimized labels", the discrimination of scores for in either class C2 C3 is improved (blue curve in 2nd row further away from blue curve 1st row; and similarly for green curve in 3rd versus 1st row).
(2)
Algorithm Proposed OSCL. algorithm (one-versus-all).
Input: Training samples of current task t i.e., X, [X| X."). X E Roxx-t, test data and empirical means and covariance matrices of previous tasks p Σ/ for Output: Estimated class E {l.....m) of x. for j 1 to do Estimate: M, M E.M and s according to Remark | Create optimal scores ÿ ('( = arg min, E (.))(ū).
Compute classification scores g E) according to equation
Compute the theoretical statistics m² (f)
and of (*)
as per Theorem center and scale the deci-
sion function as g(x.e) m
Update the mean matrix M and the covariance matrices Σ by sending the empirical means ji and covariance matrices Σ for use in the next task) end for Output: -= arg max;e(1.....)) g(x; f).
18

---

Published as a conference paper at ICLR 2023
The computation of M M (as well as M M and the vector s) is of complexity ((dn+dd) (es- timation + product). The optimization of the labels being performed using the BFGS algorithm is of complexity ( t being the number of tasks. The overall complexity is O(m²T3) + O for the one-versus-one scheme and OmT)² + On²mm) for the one-versus-all. Note that competitors (using for most of them deep neural networks) can difficulty computes algorithmic complexity since depends on the architecture chosen and depends on the internal optimization of the deep neural networks. But as confirmed with the running time (both for training and inference), our method achieves a lower computational cost.
D EXPERIMENTAL PART
D.1 THE DESCRIPTION OF DATA SET AND EXPERIMENTAL SET UP
The Synthetic: Gaussian mixture model with two classes with means μrj (-1)² with Mr = 3, + v1 p- for 3, relatedness parameter and with e p e/p
The Permuted MNIST dataset has 70,000 images of handwritten digits from 9, with 60,000 being used for training and 10,000 for testing. Different from the original MNIST dataset, this dataset uses random permutations of pixels as inputs for each the ten tasks.
The Spilt MNIST is based on 5 subsets of consecutive digits within the MNIST training data set. Learning to distinguish between two consecutive digits from 0 to 10 is the goal of the five tasks.
In the Rotated MNIST, the digits were rotated by angle generated uniformly between 0 and 180 degrees and data for different tasks have different rotation angles. Based on this, the rotation angle and the factors of variation already included in MNIST are the factors of variation.
The Fashion MNIST dataset has the same size as MNIST, but is based on different (and more challenging) 10 classes. Similar to the Split MNIST, we will split the Fashion MNIST into 5 tasks, and the five binary classification tasks are: T-shirt/Trosser, Pullover/Dress, Coat/Sandals, Shirt/Sneaker, and Bag/Ankle boots.
Across all MNIST variants, 1000 samples were used as training data. For Rotated MNIST and Pe Permuted MNIST. 10 tasks were generated. And regarding Sp1 it MNIST and Split Fash- ion MNIST, 5 tasks are generated, as in KJ & N Balasubramanian 2020)) Sokar et al. 2222)). Each experiment was run five times randomly that we are using linear model, we used HOG (Dalal & Triggs) 2005 to extract features for Rotated MNIST, Split MNIST, and Split Fashion MNIST, and raw data for Permuted MNIST (without any feature extraction).
obtain the results. For a fair comparison, given
D.2 EVALUATION METRICS
We evaluate our algorithm by average accuracy across tasks. A 1 Er= Ac, where A, Σ ar.r and 01, is the accuracy of task T after the model is trained on tth task. We also use the metries F. 1-1 E}= mar
T
1E(1...!-1) 0t.j at,5) to measure the average forgetting of our approach
on task / (Table (J & N Balasubramanian, 2020 2020)
D.3 SUPPLEMENTARY EXPERIMENTS
Table 4: Accuracy of each task SplitMNIST and Split FashionMNIST.
Task
Tl
T2
T3
T4
T5
SMNIST
98.34 + 0.73 96.22 + 0.14
92.86 + 0.05 90.27 + 0.28
85.99 + 0.12
SFashionMNIST 99.39 + 0.20 89.70 + 0.21 89.12 + 0.13 86.78 4 0.10 85.02 + 0.22
19

---

Published as a conference paper at ICLR 2023
Table Accuracy of each task on PermutedMNIST and RotatedMNNST.
Task
T1
T4
T5
T6
T8
T10
RMNIST
85.37 0.07
81.01 0.22
80,5110009
78.17L 0.1 74.91
0.7
71.36 0.11 68.0110.155 62.60 0.21 61.84 10.14 60.87
D.3.1
COMBINATION WITH NEURAL NETWORK
We use an ordinary CNN model as pre-trained model, and has five layers (2 convolution layers 3 fully connected layers). The detail of model structure is follows.
PashionMMnistModel(
(layerl): Sequential(
(0): Conv2d(1, 32, (5, stride=(1, 1), padding=(1, 1))
(I): BuchhNorm2((322gpss-lco0)) momentum=0.1, affine-Tree)
(2): ReLUO
(3): Max stride=2, padding=0, dilation=1)
(layer2): Sequential(
(0): Conv2d(32, 32, (33 3), stride=(1, 1))
(1): BatchNorm22((22eppsss eps-le005, momentum=0.1, affine=Treee
(2): ReI.U)
(3): Max/ool22(kkermmelss=..222 stridde-- padding=0, dilation=1)
(fcl); Linear(in=800, out=600, bias=True)
(drop): Dropout2d(p=0.25, inllac
(fc2); Linear(in=600, out=120, bias=True)
(fc3): Linear(in=120, out=10, bias=True)
Table summarizzs the result obtained by using the aforementioned pretrained convolutional neural network together with the proposed method. The experimental settings and remarks are as follows:
Table Average accuracy on three scenarios (TIL, DIL and CIL) with the HOG features and with the pretrained features.
Task
SMNIST (TIL
RMNIST (DIL SMNIST (CIL
IIOG optimized SPCA
98.0
72.7
91.2 0.10 95.3110.44
CNN feature optimized SPCA
99.1 0.40
79.2 0.11
Considering we want to use the pre-trained model for MNIST data, we use ordinary CNN model convolution layers + fully connected layers, more details about the model's structure see Appendix in main paper) and pre-train it on FashionMNIST data (60000 train data).
FashionMNIST data is similar the MNIST data. Both of them in grayscale with the same shape. But the high-level semantic informations of these two datasets are totally different (handwritten digits clothing and personal adornment).
20

---

Published as a conference paper at ICLR 2023
We assume (or observe) that FashionMNIST and MNIST share similar semantic informa- tion at low level. So an extracted feature is derived from the flattened output of the second convolution layer.
From Table we can see significant improvement after we use a neural network as feature extractor. We believe that the improvement is caused by better representation of the pre-train neural network. is important to notice that extracted features must be chosen carefully, and several failures have been remarked in selecting high-level representations as features in our experiments.
21


Reviews:
[{"review_id": "uJLrGmsHagB", "reviewer": "Reviewer_62oS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a theoretical analysis using SPCA, which predicts the performance of algorithm in advance. In addition, it propose a label optimization scheme to prevent the catastrophic forgetting with theoretical support (under the strong assumption of X are independent Gaussian random vectors with identity covariance).", "summary_of_the_review": "Though the analysis and the proposed method is simple and limited in applicability, it presents an exact analysis on catastrophic forgetting and a novel label optimization formulation (though it's simple). The reviewer believe that this contribution is not trivial and meaningful to the community.", "strength_and_weaknesses": "**Strengths**\n- S1: (Even with strong assumptions) The theoretical analysis and a novel label optimization formulation seems novel.\n\n**Weaknesses**\n- W1: **Empirical validation is weak** -- Used benchmarks are all MNIST variants, thus in a small scale and relatively simple. Since the method is based on the strong assumption (See my summary), the empirical validation is quite weak.\n- W2: **No discussion on the empirical comparison** -- Why the proposed method does not outperform the GEM in RMNIST.\n- W3: **Presentation is somewhat unclear\" -- Missing citation for the compared methods. Though the method of EWC, GEM, iCaRL and etc are quite well known, they should be properly referred either in the text or in the table.\n- W4: **Recent replay-based methods (published in 2021-2022 -- see the list below) are largely ignored in review** - Some of them could be compared with but not all as they are evaluated in different set-ups.\n  - Wang et al., \"Continual Learning with Lifelong Vision Transformer\", CVPR 2022\n  - Hersche et al., \"Constrained Few-shot Class-incremental Learning\", CVPR 2022\n  - Wan et al., \"Continual Learning for Visual Search with Backward Consistent Feature Embedding\", CVPR 2022\n  - Tang et al., \"Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data\", CVPR 2022\n  - Bang et al., \"Rainbow memory: Continual learning with a memory of diverse samples\", CVPR 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "r-5xtIXGcfN", "reviewer": "Reviewer_6kJ9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to use supervised PCA to tackle catastrophic forgetting in continual learning. They provide a theoretical analysis of the proposed method. They develop a label optimization scheme. They validate the method on four datasets and three scenarios (task incremental learning, domain incremental learning and class incremental learning).", "summary_of_the_review": "I don't think the paper is ready to publish yet. \n\nPost-rebuttal:\nThe original form of the paper was unclear and confusing. The authors answered all my questions well and these issues can be addressed in the camera-ready version. Hence I have increased my score.  ", "strength_and_weaknesses": "Strength: \n1) using PCA and SPCA (or other dimensionality reduction methods) in continual learning can be beneficial, though it is not well motivated in this paper.  \n2) experimental results suggest performance improvement. \n\nWeakness:\n1) It is unclear why SPCA can prevent catastrophic forgetting in both the motivation and theoretical analysis.  \n2) There is a view (e.g. see https://arxiv.org/pdf/2011.05309.pdf) that SPCA can be simply replaced with a task loss plus the traditional PCA loss producing even superior result than Barshan’s method. Task loss is readily given in continual learning. Thus this may suggest a traditional PCA loss added to continual learning may do a better job that the proposed method. \n3) Gradients Orthogonal Projection method is relevant to the proposed method, but is not analysis/compared sufficiently nor empirically.  \n4) I don't understand why labels need to be optimised. Labels are given by the tasks/datasets. The distribution of the labels are, in general, inaccessible, and the dataset is a sample of the distribution. \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "koFFJU5nwv7", "reviewer": "Reviewer_CZ28", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel CL method based on Supervised Principal Component Analysis (SPCA). The main contributions can be fo summarized as:\n\n(1) Proposing a simple continual learning algorithm, which is based on supervised principal component analysis.\n\n(2) Providing a a theoretical analysis (exact classification error rather than bounds) using high dimensional statistics.\n\n(3) Developing a develop a label optimization scheme.\n\n(4) Several applications are presented", "summary_of_the_review": "I am not very good at supervised principal component analysis. I recommend marginally above the acceptance threshold. I will change my score according to other reviewer's comments.", "strength_and_weaknesses": "\nStrengths:\n\n(a) This paper provides extensive theoretical analysis.\n\n(b) The proposed method seems to be effective in the provided experiments.\n\nWeaknesses:\nI am not very good at supervised principal component analysis. Therefore, I talk about my feelings of this work.\n\n(a) A more detailed description of supervised principal component analysis (SPCA) is necessary. What is SPCA? How is SPCA connected with CL? \n\n(b) The description of proposed method is confusing. In the main file, no description of proposed method can be found, while they are placed in appendix. Clear description of proposed method is necessary.\n\n(c) What does assumption 2 mean? What can we infer from it? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CQZQWH0wCd", "reviewer": "Reviewer_w222", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose a theoretical analysis of a SPCA-based continual learning algorithm using high dimensional statistics. Based on the theoretical analysis, they propose a label optimization process. Empirical evaluations highlight the effectiveness of the method.", "summary_of_the_review": "Overall, this paper is interesting, since it provide a novel theoretical analysis of catastrophic forgetting. However, some important points (see above) need to clarify. The current version can not meet the requriement for publishing in ICLR.", "strength_and_weaknesses": "The theoretical result is sound and significant. While there exist some concerns as follows:\n- Some mathematical notations seems inconsistent across the paper, making it hard to understand. E.g., the subscript order of c = [n11/n, …, n2t/n] in the appendix does not seem to be consistent with the previous description. Ds and Dc is not explained for readers.\n- The formula of Q(z) is wrong. In the cumulative distribution function of standard normal distribution, The exponent of e should be -t^2/2. Whether this error will have an impact on subsequent theoretical proofs and experimental results.\n- For the conclusion of Theorem 1, when the number of tasks t increases indefinitely, can the comparison of the information that SPCA-based methods and other continuous learning methods need to preserve be illustrated?\n- Regarding the weight setting seems to be arbitrary, it is recommended to explain the rules for weight setting.\n- The analysis of the experimental results is weak, e.g, in Table 1, why in DIL setting, OSCL substantially underperforms compared methods. In addition, since it is theoretically proven that the OSCL method avoids catastrophic forgetting, to illustrate this, it is recommended to include the average forgetting measure when presenting the results.\n- Whether this method can be combined with a neural network?\n- How to update the statistics in the online task environment？In my understanding, the dimension is increase when the number of tasks increase. How to use previous statistics to obtain current statistics?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster