GACL: Exemplar-Free Generalized Analytic Continual Learning
Authors: Huiping Zhuang
Yizhu Chen
Di Fang
Run He
Kai Tong
Hongxin Wei
Ziqian Zeng
Cen Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Class incremental learning
closed-form solution
exemplar-free
continual learning
online continual learning
generalized continual learning
Primary area: optimization_for_deep_networks

Abstract: Class incremental learning (CIL) trains a network on sequential tasks with separated categories in each task but suffers from catastrophic forgetting, where models quickly lose previously learned knowledge when acquiring new tasks. The generalized CIL (GCIL) aims to address the CIL problem in a more real-world scenario, where incoming data have mixed data categories and unknown sample size distribution. Existing attempts for the GCIL either have poor performance or invade data privacy by saving exemplars. In this paper, we propose a new exemplar-free GCIL technique named generalized analytic continual learning (GACL). The GACL adopts analytic learning (a gradient-free training technique) and delivers an analytical  (i.e., closed-form) solution to the GCIL scenario. This solution is derived via decomposing the incoming data into exposed and unexposed classes, thereby attaining a weight-invariant property, a rare yet valuable property supporting an equivalence between incremental learning and its joint training. Such an equivalence is crucial in GCIL settings as data distributions among different tasks no longer pose challenges to adopting our GACL. Theoretically, this equivalence property is validated through matrix analysis tools. Empirically, we conduct extensive experiments where, compared with existing GCIL methods, our GACL exhibits a consistently leading performance across various datasets and GCIL settings. Source code is available at https://github.com/CHEN-YIZHU/GACL.

GACL: Exemplar-Free Generalized Analytic Continual Learning
Huiping Zhuang!* Yizhu Chen' Di Fang' Run He
Kai Tong'
Hongxin Wei
Ziqian Zeng't Cen Chen 13444
Sooth China University of Technology, China
Southern University of Science and Technology, China Shenzhen Institute, Hunan University, China *Pazhou Lab, China
Abstract
Class incremental learning (CIL) trains a network on sequential tasks with sepa- rated categories in each task but suffers from catastrophic forgetting, where models quickly lose previously learned knowledge when acquiring new tasks. The general- ized CII (GCIL) aims to address the CIL problem more real-world scenario, where incoming data have mixed data categories and unknown sample size distri- bution. Existing attempts for the GCIL either have poor performance or invade data privacy by saving exemplars. In this paper, propose new exemplar- free GCIL technique named generalized analytic continual learning (GACL). The GACL adopts analytic learning (a gradient-free training technique) and delivers an analytical (i.e., (losed-ffrm) solution to the GCIL scenario. This solution is derived via decomposing the incoming data into exposed and unexposed classes, thereby attaining a weight-invariant property, a rare yet valuable property supporting an equivalence between incremental learning and its joint training. Such equiv- alence is crucial in GCIL settings as data distributions among different tasks no longer pose challenges to adopting GACL. Theoretically, this equivalence prop- erty is validated through matrix analysis tools. Empirically, we conduct extensive experiments where, compared with existing GCIL methods, our GACL exhibits a consistently leading performance across various datasets and GCIL settings. Source code is available atthtps: //tp::////ttbbb com www/CHEN////.
1 Introduction
Class incremental learning (CIL.) (1, important form of continual learning, aims effectively tune an off-the-shelf network on incoming new datasets, with data excluding various categories from its previous states. The CIL has gained significant traction due its ability to refine learned models for new and unfamiliar data classes, eliminating the need to start the training process from scratch. This elimination of retraining saves valuable computational resources, which especially important in the era of pre-trained models that have absorbed a massive amount of data.
One significant challenge in CIL is catastrophic forgetting 22 3], which causes trained models to lose existing knowledge when gaining new information quickly. This can be attributed to the fundamental property of gradient-based based iterative algorithms that impose task-recency bias, i... predictions favor recently updated categories [4]. To the authors' knowledge, no solutions exist for these gradient- trained CIL. models fully tackle catastrophic forgetting.
'These authors contribute equally.
1 Corresponding authors: Ziqian Zeng syveng@scut.cdu.cm and Cen Chen (chencen @ @cut.edu.cc...
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

On the other hand, traditional CIL assumes that the number of samples in each task is fixed and that new tasks are entirely disjoint from previous ones. This paradigm does not align with real-world scenarios, where training data may include both new and previously encountered categories, and the number of data points often exhibits arbitrariness each task. This extended CII. setting is referred to as generalized CIL (GCIL) 5 6). Such an uneven task-wise distribution of training samples and data categories further complicates the forgetting issue. For instance, GCIL may lead to the neglect of minority samples within a batch, thereby undermining representation during the training process.
To mitigate catastrophic forgetting, a simple but effective approach is to replay historical samples. Replay-based baalay CIL [ 4 mitigates forgetting by storing a small number of samples from historical categories for the model review while learning new information. However, this replay mechanism poses risks to data privacy. Thus, the exemplar-free frre CIL (EFCIL) without saving old exemplars gains prominence due to the increasing concern for privacy. However, many EFCIL methods perform poorly due to the task-recency bias caused by the nature of gradient-based algorithms [.. Recently, this dilemma has been alleviated by the analytic continual learning (ACI.) 8 an emerging EFCIL branch that first achieves comparable or even more competitive performance over the replay-based CIL. This improvement occurs because, for the first time, ACL achieves a near "complete non- forgetting" by allowing an equivalence between the incremental learning and its joint training (i.e., the weight-invariant property).
The ACL provides powerful toolbox for traditional EFCIL scenarios where data categories among training tasks are mutually exclusive. However, an apparent gap exists between the existing ACL techniques and the more desired and real-world GCIL scenario. Exploring the possibility of incor- porating the weight-invariant motivation, as it has the potential to enhance overall performance. To achieve this, we propose a generalized analytic continual learning (GACL), a new and compensated ACL member, offering a weight-invariant property solution to the GCIL. The key contributions are summarized as follows.
property into the GCIL, framework is both a significant and natural
We present the GACL, an exemplar-fi free technique that achieves the equivalence between the GCIL (with split incoming data) and its joint training (with data centralized in single task).
We theoretically establish the GACL's weight-invan veight-invariant property. It is achieved and proved by separating the incoming data into exposed and unexposed components and aligning them structurally with matrix decomposition techniques.
We isolate the distinctive component of the GACL, namely the exposed class label gain (ECLG). from the existing ACL. This module explains the feasibility of achieving GCIL's analytic learning. offering high interpretability in the GCIL realm.
Experiments on various benchmark datasets are presented, showing that the GACL outperforms the existing EFCIL by large margin. It also exceeds most tfte-of-the-ane replay-based methods.
2
Related Works
This section reviews existing methods for CIL and its more real-world counterpart, i.e., GCIL.
2.1
CIL Techniques
Existing CIL methods can be roughly divided into three categories: replay-based methods, regularization--ased methods, and prototype-based methods.
The replay-based CIL methods such as iCaRL. (. LUCIR 4. PODNet 9 AANets I0] FOSTER 11 and OHO 12], retain past training samples as exemplars and utilize them during the learning of new ones. However, storing original training samples presents a significant challenge, particularly in scenarios with strict data privacy requirements.
The regularization- CIL aims to design loss function that prevents the change of activations or important weights. Methods such as the fers-forgg learning [3] and the LwF [14] introduce knowledge distillation [15] into their loss function to prevent the forgetting caused by activation drift. EWC [6], EWC++, RWalk [77), and Rotate your Networks [8], introduce regularization that slows down learning on the weights important for old tasks by calculating the Fisher information matrix.
The prototype-buse CII. maintains distinct prototypes for each category, which prevents overlapping representations of new and old categories. For example, the PASS [19] distinguishes prior categories

---

by augmenting feature prototypes. The SSRE technique [20 enhances the dissimilarity between old and new categories via selecting prototypes to incorporate with new samples into a distillation process. The FeTriL. 21
uses new representations to generate pseudo-features of old categories.
2.2 Analytic Continual Learning
The ACL is a recently developed EFCIL branch inspired by the analytic learning 22 23 2 where the training of neural networks yields closed-form solution using least squares. The ACIL 7
first converts a continual learning problem to a batch recursive least-squares problem, eliminating
the need to store samples by preserving the correlation matrix, and the RanPAC (25] applies this
trick to pre-trained models. The GKEAL [8] focuses on the few-shot CIL scenarios by leveraging a
Gaussian kernel projection. The DS-AL [26] introduces an additional linear classifier to learn the
residue of the ACIL to enhance the plasticity, while the REAL [27] introduces the representation enhancing distillation to improve the backbone's generalization capabilities. The AFL [28] extends the ACL to federated learning, transitioning from temporal increment to spatial increment, and similar techniques are applied to the reinforcement learning [29). The ACL is emerging competitive CIL branch with a closed-fore solution that leads to a valuable weight-invarinn equivalence between CIL and its joint learning. However, existing ACL methods are designed for the CIL scenario in which the categories of samples in each task must be entirely distinct. This restricts their applicability in real-world scenarios.
property, securing the
2.3 The Generalized Class Incremental Learning
The GCIL simulates rorl-world incremental learning, as distributions of data category and size could be unknown in one task. The GCIL arouses problems such as intra- and inter-task forgettings and the class imbalance problem [30 The key GCII. properties can be summarized as follows: (i) the number of classes across different tasks is not fixed; (ii) classes shown in prior tasks could reappear in later tasks; (iii) training samples are imbalanced across different classes in each task [6] (See Appendix B
There are several GCIL settings. In the BlurryM setting (5) a% the classes are disjoint between tasks, while the remaining classes appear in every task. The i-Blurry-N-M 31 setting has blurry task boundaries and requires the model to perform anytime inference. However, the i-Blurry scenario has fixed number of classes in each task with the same proportion of new and old classes. The Si-Blurry 30 is the most complex and realistic GCIL setting satisfying all three GCIL properties since has an ever-changing number of classes and is capable of effectively simulating newly emerging or disappearing data, highlighting the problem of uneven distribution in real-world scenarios.
To address the issue of the GCIL, gradient- based sample selection methods such as the GSS-IQP and the GSS-Greedy are proposed by (5). The RM [32] proposes a memory management strategy based on per-sample classification uncertainty and data augmentation, while the management in the CLIB 31 eliminates samples based on a per-sample importance score. The DualPrompt [33, as an EFCIL
method, introduces the prompt- based learning to the CIL problem, and the MVP 30 proposes an instance-wist wise logit masking and contrastive visual prompt tuning loss.
3 The Proposed Method
In this section, we deliver details the proposed GACL, We first define the learning problem. Then, we derive the GACL by employing matrix decomposition techniques. A corresponding theoretical analysis follows to indicate the interpretability of our work. An overview depicted in Figure]
3.1 Problem Definition
We denote the complete set of available data as D. When D is partitioned into sequence of GCIL, tasks, we assume that Draan ~ (Xtai y is the set of training samples that are present task k. The training dataset Di consists of labeled samples, where Xtrain E B Nexxwxh represents N₁ input image samples with shape of e w x h. Y!ain E R X represents Nt-skacked one-hot encoded label tensors with dy. classes that have been seen from task 1 to task k. Diest { y
3

---

(e) Gemoritiond Annlytc Clus Increnenial Learning
Finished flle
Dotaset
RE-1
Labele
Split
W
Task #k
W unexppose
Exposed Classes
Unexposed Claases
Exposed
update
Unroppard
W
Cluse
Outpui
Noca
Nura
R2
W WECLO
Stallithend
update
Exppoeed
Frosen Pre-trolaed Vit
Buffer Layer
(b) Rufiore Emnhehding Extraction
Figure 1:
An overview of our proposed GACL. (a) Labels of the exposed class and the unexposed
class are extracted each GCIL task (see definition in Section (.2p, respectively. (b) frozen pre-trained VIT and buffer layer are utilized to extract features from the inputs. (c) The key to the recursively updated formulation of the GACL contains two components. The ((( unexposed takes in the contribution of unexposed class data (see (II). The other is contributed by the ECLG module w ECLG (e.g., see T2) ) which reflects the gain of exposed class data on the seen categories. The recursive formulation flows aided by the autocorrelation memory matrix R throughout the GCIL.
the test dataset in task k. The goal of GCIL in task k is to train networks using Ditann and evaluate their performance on the test dataset Dist- Here, Disk denotes the joint dataset spanning tasks to
Exposed-unexposed Class Split
In each GCIL task, classes may not appear exclusively. Hence, in any GCIL task k. we refer to classes that have appeared in previous tasks to as the exposed classes of task while classes making their initial appearance are the unexposed classes of task k as shown in Figure (a). This distinction helps to characterize the evolving nature class occurrences throughout different GCIL tasks.
In a task-wise GCIL scenario, we can involve all class labels in set S. In task k, the set of the exposed class labels denoted Sexposed, S S, while the of unexposed class labels is marked by Sumexposed. C S, where Sexposed, & O Sumespused, & Ø. Note that Seaposed, & and Sunexposed, may evolve from task 1 to task k; that is
Sexposed k
S unexposed, k-1 Sexposed k-1 unoxpoosed k-1 U Sunexposed, k-2 ... Sunexposed, 1.
From the scope of exposed-unexposed classes, the dy, can be represented dy = |Sexposed, kl [Sunexposed, k d + Sunexposed, k/, where |-| denotes the cardinality of set.
In task k. given training dataset Ditain ~ {Xtriin Y!minn class labels Y!min can be partitioned due to the
split as follows:
Y
Y train
train
(I)
where Vrtrain E T Na is the exposed class label matrix and Y!aain Ne -dy, 1) is the unexposed class label matrix. They correspond to segments displaying the appearance exposed classes and unexposed classes.
3.3 Buffered Embedding Extraction
The power of pre-trained models allows the GACL to adopt frozen backbone from structures such as ViT 34 to extract the features of images shown in Figure (b). Let
X B
fbackbone
X, + backbone
(2)

---

be the features extracted by the backbone, where ② backbons indicates the backbone weight. Then, we use buffer layer to project features, i.e.,
X Soutter X (1)),
(3)
where foutter indicates the operation of the buffer layer. Several options for the buffer layer exist, including a randomly initialized linear mapping in the ACIL [7] or a kernel embedding projection in The selection the buffer layer
our focus. For convenience, we follow the ACIL, taking the random linear projection followed by non-linear activation function as the buffer layer, i.e. foutter X ( ReLU(X WB). where the elements of the buffer layer weight WB are randomly sampled from a normal distribution.
the GKEAL [8]
3.4 Generalized Analytic Class Incremental Learning
Here, we derive the GACL by partitioning training samples into unexposed and exposed categories, shown in Figure I (c). Let X and Y Y be the accumulated feature and label matrices in task k which can be extended from the accumulated matrices X 1:k-1
and Y in task & 1 as follows.
1:k-1
X total
Yooal 1ik-1 Y iirain
total 1:k:
I:k
Votaal
train
Subsequently. one could formulate the learning problem in task by fully connected network (FCN) as the classifier
argmin WKS
total 1:k
total 1:k
W (k)
FCN |
12
+
(r ||
FCN F
where ||-||r Frobenius-norm norm, 7 weight. The optimal solution to
0 is the regularization term and W (8)
FCN indicates the FCN layer
is
W
X
total
+ +
total y
FCN
The goal of the GACL is then to obtain (k) and
FCN recursively from V (k-1) 1:k )).
without directly involving
FCN
historical samples (e.g., X lsk
That is to solve
argmin w
Y Yotial
uuain
0 Yrain
X
lotal l
W
(k)
+7
(A) 1
FCN |F
(6)
by recursively updating the previous-task weight ( memory matrix follows.
To achieve this, we define an autocorrelation
FCN-
Rs Lolal
X total
Accordingly, we summarize the recursive formulation of the proposed GACL in Theorem 3.1
Theorem 3.1. Let (i(k) be the optimal estimation of (6) with all the training data from task to task k. Then Wi(k) FCN is equivalent its recursive form
PCN
((-1))
PCN
REX
(BIT
x
(((-") + X
FCN
(BIT uainn
RxX((B
(B)T
train
(8)
where
R = R R4
1
(B) (I X R R (B) 1 x R
(9)
Proof. See Appendix
As indicated in Theorem 3.1 the weight (k) (k)
FCN in task & recursively obtained using the previous-task
weight FCN
(K-k-) is identical its joint-learning counterpart formulated in (6). That is, the GACL. maintains the same weight-invariant property in the GCIL scenario as other ACL methods.
5

---

Algorithm The pseudo-code of GACL.
Input: GCIL tasks Diatn frozen weight O backboome
D with Di
{ y the pre- pre-trained backbone with
Initialization: Ro t -1, W
+ 0
FCN
for task& to K do
x f Shackbone X train θ
backbonne)
x fbutter X (B)
Decompose t into exposed and unexposed class components Y-train and ttaain
RR f- RE-1- Ra-1X
xx
(B)T
(I + +x+ R
R
W (k)
w R (B)T
X
x
(B)
W
(R-1)
X
11
unexposed
FCN
W
X (B)T Yarain
0
12
BCLG
W
W(k) (k)
+ ( (k)
FCN
unexpused
ECLG
end for
The pseudo-code of the GACL is listed in Algorithm I
Exemplar-free. The recursive formulation is aided by R& as indicated in Note that this autocorrelation memory matrix records the inverse of inner products among the historical embedding matrices as shown in 77. Hence, the embeddings (e.g.. x are not reversible. Saving R& instead of used samples is a safe alternative to preserve past knowledge. That is, our GACL is exemplar-free technique without the need to keep any historical samples.
To more properly explain our GACL, indicated in Figure be rewritten as the sum of the unexposed-class contributed weight (r(k) (A) ECLG. i.e.,
| (c), the recursive solution in can
unexposed and the ECLG weight
(R) W FCN
uncxposed + Pi
(10)
ECLG
where
W (4) unexposed
(i(k-1)
R
x x-
(((1) FCN
R R Pumin
(11)
FCN
W
(k) BCLG = [ (B)T ytrain 0]
(12)
Unexposed-class Contributed Weight. The unexposed-class contributed weight (k(k)
unexposed is
recursively updated by the data of the unexposed class only. Note that the unexposed class label Yrusin is applied on the concatenated weight along with new data x which is reasonable as historical information should not intervene with the weight update of unseen classes. On the other hand, new data (BB) could also affect historical knowledge. This marked by the gain of (B)
wi to the original weight (k-1) indicated in X TB)T x
(k-1)
Exposed-class Label Gain Weight. The ECLG module indicated in 12) captures knowledge from exposed-class labels. The supervision this weight component marked by Rd x Fuain is mainly contributed by the exposed-class labels (i.e., Yrrain). It important to note that when Y
empty (i.c., classes reappear in task k), this component does not contribute to the update of W This module also isolated to distinguish GACL's difference from the existing ACL methods in a mathematical analysis manner (indicated as follows).
FCN-
Difference from Existing ACL Methods. Overall, the GACL can be treated as a nontrivial gener- alization of ACIL 77) GKEAL 8 and various other ACL methods. For instance, in conventional CIL where no classes reappear new tasks (i.e., Yk, Vimin E R*x0), the classifier of the GACL
FCN unexposed which is equivalent to the recursive classifier of the ACIL. That is, the ACIL is a special case of our proposed GACL. The major difference lies in the ECLG module, corresponding to the exposed-class gain. This pattern makes sense as there must compensation on top of ACII. updates (specifically designed for traditional CIL) when exposed data (out of setting) participate.
(k)

---

Table 1: Comparison of AAUC, ANn. and A among the GACL and other methods under the Si-Blurry setting. Data in bold represent the best EFCIL results, and data underlined are the best among all settings. We run all experiments 5 times and show "mean staaduudd error
Mem
CIFAR-100 (%)
ImageNet-R (%)
Tiny- Mmn-IImeeNet (%)
Method
EFCIL
Size
AAuC
ANE
A
AAUC
ANa
ALa
AAUC
Aaux
ALax
EWC++ 16
X 21.0
20.95 1. 52.55 36.31 39.87 29.52 52.43 14.6 37.67
ER
35
X 56.17
53.80 55.60
39.31
23.0 32.09 4044
55.69
57.87=142 41.10
2000
RM 32
X 53.2221.2
$2.99
55.25
32.34_1as
36.464223
25.26
49.28
57.74_137 41.792034
MVP-R
30
X 60.62
57.58
64.30
47.16
50.36
42.05
61.15
62.41
51.12
EWC++
16
2 44.56
40.52
32.81
35.54
15.54:1160
g3:4
45.30
46.34
27.05
ER 65
X 51.59
48.03
24.09
35.96
39.01
21.54
26.14
48.95
20.44 29.97 2075
500
RM 32
X 41.072130
48.10
32.662a34
22.45_0s2
22.08
g.61
38.83_23x 18:23 36.66
MVP-R 80
X 56.20
53.61
55.35
43.28
45.74
35.60
55.28
55.45
40.1220.4
LwF [B
40.71=2.1 38.49
27.03
29.41gnas
31.95
19.67
39.88
41.35
24.93
L2P 36
42.68 39.89
28.59
30.21
32.21 21.75
18.01
41.67
42.53
24.78
DualPrompt 33
41.34_2250 38.59
2006
22.74
30.44_nass
32.54
16.07_1mm
39.16g1.11
39.81_30x 20.42
MVP 30
45.07 44.93
39.94
35.77=255
35.58
22.06
46.43gam
45.41 28.21 22,99
SLDA 37
43.00 22.7
61.79
33.11ga.17
33.78
1::0
49.17
47.93
g3.13
GACL (ours)
57.99 26.24
20.31
41.68gu07
47x30
63.14
gmos 69.32__aa7 62.6819.0
4 Experiments
4.1
Experimental Setup
In the section, we conduct experiments on various benchmark datasets and compare the GACI, with both EFCIL and replay-based state-of-ther oh-thh-art methods, including LwF [4], L2P (36). DualPrompt
33 ER 335, EWC++ [16], SLDA (37, RM [32], MVP [30,, and MVP-R (MVP with exemplars)
Datasets. We conduct experiments on three datasets: CIFAR- 100 38
ImageNet-R (39. and
Tiay-Ima [40]. We evaluate each method under the Si-Blurry setting
30
(the most complex
GCIL setting) with 5 independent seeds. For the Si-Blurry setting, we set the disjoint class ratio rD to 50% and the blurry sample ratio TB to 10%. More details about Si-Blurry are listed Appendix C
we pre-train the backbone on 611 ImageNet classes after excluding 389 classes that overlap with CIFAR and
Implementation Details. We utilize the DeiT-S/16 41 as our backbone. Following 42 43
to prevent data leakage. To ensure a fair comparison, all methods utilize a frozen backbone. All methods under comparison are implemented as specified in 30). The memory sizes of compared relay-based methods are set to 500 and 2000.
There two hyperparameters in the GACL, the regularization term 7 and the size of the buffer layer. Here, we adopt 7 = 100, which is determined by the grid search of {0. 10, 100, 500, 1000, 10000) on CIFAR-100 (by a 90%- 100% train-val split). As the regularization term 7 is not sensitive in proper range 77. we adopt this value for all datasets for convenience. We relocate its analysis to Appendix The size for the buffer layer WB B is set to 5000 for both the GACL and ACIL for convenience.
Evaluation Protocol. Three metrics are adopted to evaluate GCIL tasks. The real-time performance
is evaluated by the area under the curve of accuracy Asuc [31] i.e., AAUC = Σt= f(i An) An, where An is the number of samples observed between evaluation and f(-) is the curve in the accuracy- to- to-(number of training samples) plot, measuring anytime inference performance during training. A higher Asuc corresponds to method consistently maintaining high accuracy throughout the training. The overall performance is evaluated by the average incremental accuracy (or average accuracy) AAng
K41
kk=1 Ake where the task-wise accuracy A indicates the average test accuracy in task
K
'The results for MVP and MVP-R R are based on their official implementation, committed on October 26, 2024 (commit ID:
7

---

k by testing the network on D A higher AAve score is preferred when evaluating algorithms. The 1:k- last evaluation metric is the last-task accuracy Atas evaluating the network's 's last-task performance after completing all tasks.
4.2
Comparison with Sfate-of-the-arts
As shown in Figure we comprehensively compare the GACL with both EFCIL and replay-based methods.
CIFAR-100
ImageNet-R
Thy-ImageNet
80
EPCIL Methods
70
60
10
Replay-based baally-bbaeddde Methods
40
30
5
Task #
Task EWC++ (2000)
Task #
LWF
L2P
DualPrompt
ER (2000)
RM (2000)
MVP-R (2000)
MVP
SLDA
GACL (ours) EWC+ (500) ER (500) RM (500)
MVP-R (500)
Figure 2: The task-wise accuracy A of the GACL with EFCIL methods (top) and replay-based
methods (bottom) on benchmark datasets with the K = 5.
Compare with EFCIL Methods. EFCIL methods address privacy concerns and mitigate catas- trophic forgetting without exemplars. Among EFCIL methods, our GACL consistently exhibits superior performance across all three datasets, as illustrated in the lower panel of Table [
For instance, CIFAR-100, 6.15%, and 8.52% for AAUC, Anvg. and Arast respectively. On Tiny-ImageNet, the GACL achieves impressive results with Asuc, AAvg. and Ai reaching 63. 14%, 69.32%, and 62.68%, respectively, surpassing the previous best EFCIL by 13.97%, 21 21.39%, and 9.55%. Similar patterns are evident in the results of ImageNet-R, further confirming that the GACL is exceptional tool for GCII.
method surpasses the second- best method SLDA, by 4.99%.
Owing to the weight-invarien property, the GACL exhibits more accurate and stable evolutions as k increases observed in Figure 2 (a). All compared EFCIL methods exhibit sharp declines in accuracy, while the GACL delivers nearly non-declining curves. In particular, on CIFAR- 100, the GACL shows unnatural improvement of task-wiss accuracy throughout the learning tasks, with the GACL initially lagging behind other EFCIL methods. This is because the Si-Blurry samples more than 70% of the CIFAR-I00 categories in the first two tasks (see Appendix[F, constructing scenario where gradient-based algorithms could largely avoid the forgetting issue. Moreover, our method produces more stable predictions across diverse scenarios, indicated by much smaller standard errors (colored shades in Figure (a)). In summary. the experimental results demonstrate that our proposed GACL exceedingly accurate and robust, exhibiting exceptional generalization ability.
Compare with Replay-based Methods. Replay-based methods are considerably competitive as they leverage historical samples. The memory size is key adjustment, as increasing typically leads to performance improvements by allowing more historical knowledge to be reviewed. For instance, the MVP-R achieves 4.42%, 3.97%, and 8.95% gains for AAUC. ANg. and Aras (see Table on CIFAR- 100 when increasing the memory size from 500 to 2000.

---

As an exemplar-free technique, our GACL avoids re-using the historical samples. However, as indicated T the GACL still outperforms most existing replay-based results. For instance. the GACI. achieves the best ALas results among all settings. The GACL'S AAUC and Awv results are also mostly superior, except that our performance is slightly weaker than that of MVP-R with a memory size of 2000 on CIFAR-I00 and ImageNet-R, Although increasing the number of exemplars can further improve the results of replay-based methods, this approach could lead to higher training and memory costs and, more importantly, more severe privacy invasion.
As indicated in Figure (b), heplay-based methods experience accuracy declines similar to those observed in the EFCIL case. This decline is due to an inherent limitation of gradient- based iterative algorithms, which tend favor recently trained categories and thus lead catastrophic forgetting. The GACL is iterative- perative-free and then not constrained by this forgetting issue, thereby achieving nearly no performance reduction as increases.
Why the GACL Gives Leading Performance. The above comparisons show that the proposed GACL is a powerful GCIL technique. Its competitive performance can be explained as follows. (i) Weight-invarien property. As shown in Theorem 3.1. the weight obtained recursively is equal to its joint- soint-learingg counterpart, indicating that the GACL is "completely non-forgetting" technique (under the condition of a frozen backbone). (ii) Analytical solution. Existing GCIL techniques are gradient- iterative algorithms prone to catastrophic forgetting by nature. The GACL is new member of the ACI and inherits its non-iterativy gradient-free essence with analytical solution, thereby avoiding the task-recency bias to address forgetting.
4.3 Ablation Study on the ECLG Module
The ECLG module is a core component that allows the GACL to obtain the weight-invarian property in the GCIL scenario. Here, we conduct an ablation study justify the ECLG's contributions under various blurry sample ratios TB with FD = 50%. Larger т3 indicates more complex data distributions the Si-Blurry setting, As shown in Table 2 the GACL without ECLG exhibits poor performance with visible gap for Asuc, AAvg. and ALast For instance, on CIFAR-100 with TB 10%, the ECLG contributes 23.01 % ALast gain the GACL.
Table 2: Ablation study on the ECLG module of our GACL.
With ECLG
Without ECLG
Dataset
AAux
AAv
Ala((%)
AAuc (%)
Any
Ausa
CIFAR-100
57.99
56.24
70.31 10,06
45.68
42.04
47.30
10%
ImageNet-R
41.68
47.30
42.22
40.29
46.95
41.67 10:36
Ting-NmageNet
0..14
69.32
62.68 0.008
60.21 1.86
65.80
60. 60.13
CIFAR-100
1..3
58.74
69, 20000
42.53
42.26
45.49
30%
ImageNet-R
42.19
47.82
42.90
42.01 20:26
46.95
41.67 10.56
Tiny-ImageNet
60.73
67.31
59.73
60.63
57.03
60.13
CIFAR-100
56.74.1.14
58.29 1:95
70.02.:0.05
40.91 1:3.7
47.25
58.61 1:262
50%
ImageNet-R
41.33
46.42
42.92.
40.44
42.50
39.05
Tiny-ImageNet
60.96
66.28
62.24
60.32+440
60.70
56.97 1.88
As claimed in Theorem 3.1 the classifier without the ECLG module fails to absorb knowledge from joint classes in each task (i.e., classes that reappear), leading to substantial information loss under the GCIL setting. The GACL, equipped with the ECLG module, demonstrates competence in handling overlapping classes realistic scenarios.
4.4 Robustness Analysis in Si-Blurry Setting
Here, we conduct robust analysis by varying the disjoint class ratio rD and the blurry sample ratio TB- The comparison happens among the GACL, the second-best EFCIL method SLDA, and the top-performing replay-based method MVP-R with memory size of 500.

---

We evaluate our method under various rD. including extreme cases where each task shares classes (rD 0%) and traditional CIL scenarios (TD = 100%). dillstailustrttes that our GACL consistently outperforms the compared methods (e.g., leads the SLDA by 2%-10% and produces near-identical A values with varying TD. This shows the accurate and robust traits of the GACI.
We also evaluate our method using various PB values, as shown in Table 4 Similar patterns observed here align with those in Table 3 further demonstrating the robustness of the proposed GACL, which delivers exceptional performance across different GCIL settings.
Table 3: The performance at different TD with Table 4: The performance at different TB with
TB 10% on CIFAR-100.
TD 50% on CIFAR-100.
Method
(x)(
(w6
(
Method
Awu((%)
Awy (%) A (%)
SLDA 337
55,51 +iMs
53.94
67. +026
SLDA
37
53.004385
50.09
61.79
0%
MVP-R 30
4..49
50.7340.7
60.54
10%
MVP-R 30
56.20_147
S3.61
55.35
GACL (ours)
49.
49.961061
50.5620.49
69.94
GACL (ours)
57.99
56,24
70.31 +0000
SLDA 37
53.00
50.09
61.79
SLDA 37
54.55
54.06=241
63.04
50%
MVP-R 30
26.20
53.61
55.35
30% MVP-R 30
59.65
58.31
18.1641.w
GACL (ours)
57.99
56.24
70.31
GACL (ours)
57.33
58.74
69.90=.0o
SLDA 337
65.464479
67.29 #5.28
63.564266
SLDA 37
53.81
13443
52.93
63.45+2212
100% MVP-R 30
28.43
68. 18.44
53. 14.14
50%
MVP-R 30
59.10
S7.34
54.81 +0.21
GACL (ours)
70.72+
77.57 69.97
GACL (ours)
56.74
58.29
70.02+
4.5
Limitation and Future Work
Overall, the GACL exhibits various good characteristics as an exemplar-free GCIL technique. The major limitation here is the need for well-trained backbone because the GACL does not update backbone weights. This could motivate the exploration of adjustable backbones to continuously improve their feature extraction abilities, thereby further enhancing GACL's performance.
5 Conclusion
In this paper, we introduce the exemplar-fres generalized analytic class incremental learning (GACL) approach to address the GCIL problem. Building upon analytic learning, the GACL delivers closed- form solutions to GCIL through the decomposition of GCIL data into exposed and unexposed classes. The GACL achieves the weight-invariant property that provides identical solutions for GCIL to its joint learning counterpart. We theoretically validate this property and provide high interpretability through the matrix analysis tool. Various experiments are conducted under the Si-Blurry setting, demonstrating that our proposed GACL achieves remarkable performance with high robustness compared to state-of-N the-art EFCIL and haplay-based methods.
Acknowledgments and Disclosure of Funding
This research was supported by the National Natural Science Foundation of China (623061 17), the Guangzhou Basic and Applied Basic Research Foundation (2024A0413681, 2023A04J1687), Technology Innovation Fund, the Fundamen-
the South China University of Technology-Ter tal Research Funds for the Central Universities (2023ZYGXZR023, 2024ZY the Guangdong Basic and Applied Basic Research Foundation 1924A15150102220 the CAAI- MindSpore Open Fund developed on Openl Community, the Shenzhen Fundamental Research Program (CCY1202308070018000 JCY120010322123802000 10090000, and Shenzhen Science and Technology Plan (Grant No.
10

---

References
[1] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. Incremetal classifier and representation learning. In Proceedings the IEEE Confer- ence on Computer Vision and Puttern Recognition (CVPR), July 2017.
[2] Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks:
The sequential learning problem. In Gordon H. Bower, editor, Psychology of Learning and Motivation, volume 24, pages 109-165. Academic Press, 1989.
[3] lan J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks, 2013.
[4] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang. and Dahua Lin. Learning unified classifier incrementally via rebalancing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[5] Rahaf Aljundi. Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. "'lché-Buue E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[6] Fei Mi, Lingjing Kong, Tao Lin, Kaicheng Yu, and Boi Faltings. Generalized class incremental learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[7] Huiping Zhuang, Zhenyu Weng, Hongxin Wei, RENCHUNZI XIE, Kar-Ann Toh, and Zhiping Lin. ACIL: Analytic class-incremental learning with absolute memorization and privacy protection, In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 11602-11614 Curran Associates, Inc., 2022.
[8] Huiping Zhuang. Zhenyu Weng, Run He, Zhiping Lin, and Ziqian Zeng. GKEAL: Gaussian kernel embedded analytic learning for few-shot class incremental task. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 7746-7755, June 2023.
[9] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. PODNet: Pooled outputs distillation for small-tasks incremental learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, pages 86-102, Cham, 2020. Springer International Publishing.
[10] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class- incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2544-2553, June 2021.
[11] Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. FOSTER: Feature boosting and compression for dass-incremennal learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022, pages 398 4 14, Cham, 2022. Springer Nature Switzerland.
[12] Yaoyao Liu, Yingying Li, Bernt Schiele, and Qianru Sun. Online hyperparameter optimization for class-incremental learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(7):8906-89) 8913, Jun. 2023.
[13] Heechul Jung. Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetting learning in deep neural networks, 2016.
[14] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-294) 2018.
[15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.

---

[16] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grrbska-BBrwinnss Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell, Overcoming catas- trophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 11 114(13):321-3522 2017.
[17] Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Rieman- nian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[18] Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M. López, and Andrew D. Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2262-226220 2268, 2018.
[19] Fei Zhu, Xu- Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5871 15880, June 2021.
[20] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self-sustaining representation expansion for non-exemplar fass-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9296-9305, June 2022.
[21] Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. FeTrIL: Feature translation for exemplar-free cass-incrremenal learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 391 1-3920, January 2023.
[22] Ping Guo and Michael R. Lyu. A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data. Neuro- computing, 56: 101-1,,, 2004.
[23] Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Blockwise recursive moore- pooore-penrose inverse for network learning. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52(5):3237- 3250, 2022.
[24] Huiping Zhuang, Zhiping Lin, Yimin Yang, and Kar-Ann Toh. Analytic learning of convolu- tional neural network for pattern recognition, 2022.
[25] Mark D. McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton den Hengel. RanPAC: Random projections and pre-trained models for continual learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 12022-12053. Curran Associates, Inc., 2023.
[26] Huiping Zhuang, Run Hc, Kai Tong. Ziqian Zeng, Cen Chen, and Zhiping Lin. DS-AL: A dual-stream analytic learning for exemplar- free class incremental learning. Proceedings of the Mar. 2024.
AAAI Conference on Artificial Intelligence,
[27] Run He, Huiping Zhuang, Di Fang, Yizhu Chen, Kai Tong, and Cen Chen. REAL: Representa- tion enhanced analytic learning for exemplar-free free class- incrementtt learning, 2024.
[28) Huiping Zhuang, Run He, Kai Tong, Di Fang, Han Sun, Haoran Li, Tianyi Chen, and Ziqian Zeng. Analytic federated learning, 2024.
[29] Zichen Liu, Chao Du, Wee Sun Lee, and Min Lin. Locality sensitive sparse encoding for learning world models online. The Twelfth International Conference on Learning Representations, 2024.
[30] Jun- Yeo-Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 11731-11741, October 2023.
12

---

[31] Hyunseo Koh, Dahyun Kim, Jung- Woo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. In International Conference on Learning Representations, 2022.
[32] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition (CVPR), pages 8218 8227, June 2021.
[33] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. DualPrompt: Complementary prompting for rehearsal-free continual learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022, pages 631-648, Cham, 2022. Springer Nature Switzerland.
[34] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[35] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experi- ence replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F, d'Alché-Buc, E. Fox, and R. Gameet,, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[36] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 139-149, June 2022.
[37] Tyler L. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[38] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Toronto, ON, Canada, 2009.
[39] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV), pages 8340-8349, October 2021.
[40] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[41] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR, 18-24 Jul 2021.
[42] Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, and Bing Liu. Learnability and algorithm for continual learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 16877-16896. PMLR, 23-29 Jul 2023.
[43] Gyuhak Kim, Bing Liu, and Zixuan Ke. A multi-head model for continual learning via out-of- distribution replay. In Sarath Chandar, Razvan Pascanu, and Doina Precup, editors, Proceedings of The 1st Conference on Lifelong Learning Agents, volume 199 of Proceedings of Machine Learning Research, pages 448-563. PMLR, 22 22-24 Aug 2022.
[44] Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Correlation projection for analytic learning of a classification network. Neural Processing Letters, 53(6):3893-39) 3893-3914, 14, dec 2021.
13

---

A Proof of Theorem 3.1
Proof. in task ks I, have
(k-1) (X X 1:6-2
X
(B)
1ik-1 + +
X Irtooal
x( Yr
X M
1:k-2 1:2-2
k-1
(13)
Hence, in task k, we have
(b) FCN
(X Total X
1:k- 1 1:k-1
X (BIT
x +-1)=
X Vioaal
x
y
x I
1:k-1
(14)
We have defined the autocorrelation memory matrix R in the paper via
R
X lotal
X lik- + X [B)T
X +
(15)
To facilitate subsequent calculations, here we also define a cross-correlation matrix Qk-1, i.e.,
X
Iotal Y Vioial lik-
lik-2 + x Virain
X x
uain
(16)
1--1
Thus we can rewrite 13 as
W(k-1) RR--QQ--1
(17)
Therefore, in task & we have
w(lk) (k) FCN
R+Qu
(18)
From 15 we can recursively calculate R2 from R
R [ X x
(19)
According to the Woodbury matrix identity, we have
(A+UCV)- A A +VA -V-1
Let A R U (BBTT ,C I, and V x(() in 19 we have
R = R X
(B)T (I + X (B) R
x R
(20)
Hence, Rs can be recursively updated using its last-task counterpart R&-1 and data from the current task (i.e., x(8)) This proves the recursive calculation of the autocorrelation memory matrix.
Next, we derive the recursive formulation of W (k) To this end, also recurse the cross-correlation matrix Qx in task k. i.e.,
X
TotalT Vitall
x (B)T Ytrain
x (B)T uuain = Qr
X TB) rrain
X
(B)T Tranin
k:
l:k- 1:6-1
(21)
where
1 Qm-1,
dy. > dy dy. dy
Qh
(22)
Note that the concatenation in (22) is due to the assumption that Y in task l contains more data classes (hence more columns) than Yuni possible that there are no new classes appear in task k. then traiin should be 0.
14

---

Similar to what 22 does,
W
1y FCN
FCN
x -ma-
dun > dye
(23)
(k-1-
dyn dye-1
We have
W PCN
1
(24)
Hence, W RCN can be rewritten as
( RsQk FCN
RR + [x( P
X y
R + X( [raain
y
(25)
By substituting (0) into R4 we have
R R&QQ RA-1Q Q'
1 X (I x Rk- R 1BBT xx( Rk-1Q
W (k-11)) 131 R xx- (I x R (B) ) x x_1x x (k-1
(BIT
(26)
FCN
FCN
To simplify this equation, let K₂ = (I ( R X (B)T )-1 Since
I= K = K + X
1B) k
R X (B)T
we have K I K BB) R
1
(BBT Therefore,
(I+ x
R 1x-1
1 (B) K;
R (B)
X x Rs R 1
(Rk-1 R R
(BiT K X Rs 1) (B)T
x
(BB)
X (B)T
(27)
Substituting 27
into (6). Rg Q'- can be written as
R2 Q
W (k-1
X (BIT
x w(w'"!
(28)
FCN
Substituting (28 into
implies that
W (k) PCN
(k(k- 1yr
R&X (B)T X (B) (B
XBBTT
+ R2 xxx(1) (B)T [Ymmin
Vrimin
FCN
FCN Rs X (B)T x (i(k=)) + R X (B)T Pa
(B)T
R x (B) Y main
(29)
which completes the proof.
15

---

B GCIL Properties
The GCIL scenario is recent CIL focus. Given task-wise learning tasks, we can involve all class labels in a set S with the number of classes N. The sample sizes, such as the numbers of input images of different classes appearing in task k, are modeled as random vector C₁ R. Each entry Ca.i is random variable denoting the sample size of class in task k. In the generalized form, is sampled from task-dependent distribution. The GCIL scenario can summarized as the following three key properties.
Property B.1. The number of classes in a task is not fixed. Suppose my is the number of classes in task have:
Mr M= M C,,i 00 Mk;
(30)
where Mr is a lask-dependent distribution.
Property B.2. Classes appearing in different tasks could overlap. For two tasks k and k,, k + W, we have:
P(ck O cv #0)0
(31)
where 3 denotes element-wise multiplication of two vectors and P(.) is the probability
Property B.3. Sample sizes of different classes at the same task could be different. That for task k. we have
i,j 1,,81 j, # Ck. Ck.i + Ck. +0)
(32)
In short, the number of classes and samples could vary throughout the continual learning.
C Si-Blurry Setting
Train Dataset
Sample Sixe
Blurry Classes
Disjoint Classes
Division
YB
TP
F2
T
TR
7
T
-
T
Blurry Sample Division
7f
T2
7
100
7
D+D
7
7
7
Si- Bi-Blurry Tasks
Figure 3: configuration example of Si-B Si-Blurry setting.
The Si-Blurry setting 30) satisfies all the three properties of GCIL mentioned in Appendix B and can be treated as its good realization. As shown in Figure 3 for K-task learning, the Si-Blurry first randomly partitions all classes into two groups: disjoint classes that cannot overlap between tasks and blurry classes that might reappear. The ratio of partition is controlled by the disjoint class ratio TD. which is defined as the ratio of the number of disjoint classes to the number of all classes. Then disjoint classes and blurry classes are randomly assigned disjoint tasks (I") and blurry tasks (I") respectively. Next, each blurry task further conducts the blurry sample division by randomly extracting part samples to assign other blurry tasks based on blurry sample ratio TB. which defined as the ratio of the extracted sample within samples in all blurry tasks. Finally, each Si-Blurry task TBD with a stochastic blurry task boundary consists disjoint and blurry task. We adopt Si- Si-Blurry with different combinations of TD and rg for reliable empirical validations.
16

---

D Compute Resources
GPU Usage. We conduct experiments in PyTorch on one Nvidia Geforce RTX 4090 GPU with batch size of 64 for training and 128 for inference. Figure4sshow shows that the GACL uses minimal GPU memory. Our GACL significantly reduces GPU memory usage since it requires no back-pree pack-propggttion thereby detaching gradients from tensors during calculations. This characteristic allows our approach to be applied with a larger batch size without memory leaks.
GPU Memory Usage (GB)
10-
7,98
8-
5.21
5.29
5.39
4-
2-
1.63
1.69
1.96
1.96
CACL SLDA
LwP
ER
RM
DualPrompt
MVP-R EWC++
Figure 4: GPU memory consumption in GB with batch size of 64 where replay-based methods are with 2000 memory size.
Training Time. Table 5 further illustrates the GACL's training time compared to others on one Nvidia Geforee RTX 4090 GPU, highlighting its efficiency. The GACL is faster than any other baselines except SLDA on three datasets because only the classifier and autocorrelation memory matrix R are updated, leading to small numbers of trainable parameters compared to those baselines in back- paok-propagation manner.
Table 5: Average Training time of 5 independent seeds in seconds (s) where replay-based methods are with 2000 memory size.
Method
EFCIL
CIFAR-100 (s)
ImageNet-R (s)
Tiny-Imagenn (s)
RM
32
X
>2 days
>2 days
>2 days
MVP-R 30
X
717
527
1.597
ER 35
X
369
330
715
EWC++
16
X
650
391
1356
LwF [1]
334
229
862
L2P (36
651
285
1246
DualPrompt
33
656
332
1294
MVP 30
628
300
1345
SLDA
3
401
284
915
GACL (ours)
611
321
1246
E
Hyperparameter Analysis for Regularization Term
Table 6: AAUC, Argg and ALas of the GACL all benchmark datasets with various values of the regularization term 7.
CIFAR-100 (%)
Iet-e (%)
-mag-et (56)
AAUI
Aaiz
Auss
AAtiC
Awa
Auao
AAue
Awx
Aua
8.87
9,83
8.65 25.47
2.03
2.85
0.71
4.38
6. 6.14_401
0.62
57.574238
55.97
70.45
38.65
44.38
41.96.am0
62.742064
69.244a79
62.73
100
57.99
56.24
70.31gu06
41 41.68
47.30
42.221am0
63.14
69.32
62.68_am
500
56.984261
55 55.46.3.23
70.00
42.92
49.01
42.70
62.90=os7
68.9 68.95
62.41
1000
56.034270
54.764331
69.61
42.69
48. 48.90
42.67
61.962067
68.484483
62. 10.10
51.01
50.92
66.38
38.55
16.162044
40.10ga19
57.54
66.21
59.55
17

---

The regularization term y plays crucial role and demonstrates robust behavior throughout our experiments. We assess the impact of the regularization term y Table bldd visualize the real-time accuracy of the GACI as it learns from training samples in Figure 5 reveals the GACL's consistent performance across a broad range of 7 values, spanning from 10 10000. This highlights the versatility and robustness of our proposed GACL. However, as indicated in Figure S of 10000 leads to slightly poorer performance because the ACL is prone to underfitting due to simple linear regression 44
CIFAR-100
ImageNet-R
Tiny-ImageNe
50
Real-time Accuracy(%o
60
Real-time Accuracy(%)
40
Real-time Accuracyy(S)
60
30
40
20-
20
1k
10k
20k
30k 40k
1k 5k
10k
15k
20k
1k
20k 40k
60k 80k
Training Sample
Training Sample
Training Sample
7=0 (w/o regulanization)
7=10
7
100
7= 500
7= 1000
y 18600
Figure 5: Real-time accuracy of the GACL on all benchmark datasets with various values of the
regularization term ?,.
Notably, both Table 6 and Figure demonstrate that the absence of regularization results in significant decline in performance. This underscores the crucial importance of incorporating in the model. As indicated in (7) if we eliminate regularization by setting 7 to 0, the initial autocorrelation memory matrix Ro becomes zero. Subsequently, the computation of the autocorrelation memory matrix in task 1, denoted as R1, expressed as:
R1 = (Xlaal x (X (B)T (B)
However, it's crucial to emphasize that B non-invertible. This potential singularity introduces error in caleulating R1, leading to a decrease
x (B) might result singular matrix, rendering
in accuracy.
Analysis of task-wise Accuracy Trends of the GACL
As depicted in Figure 2 (a), the task-wise accuracy of the GACL CIFAR-100 demonstrates an increase. Notably, in the initial two tasks, the accuracy is lower compared to other EFCIL methods. However, on the other datasets, the GACL remains relatively stable. Upon more detailed examination of the dataset split, we infer that the observed variations in trends are attributed to the specific dataset settings.
For dataset with N classes, the class number ratio ra after training on i-th samples is defined as Ta d/N, where d, is the number of classes that have been observed at that point. As Figure indicates, by examining the real-time accuracy and the class number ratio Te across the three sets of figures, notable observation is made: when the sample size is small, the class number ratio re on CIFAR-100 always surpasses that of the other two datasets on 5 seeds. This suggests that tasks on CIFAR-100 notably more complex and intricate, resembling few-shot learning scenario.
Consequently, the GACL exhibits lower task-wise accuracy compared to other gradient- based EFCIL methods, particularly in the initial stages. However, more training samples are acquired, its accuracy progressively improves.
18

---

70
Accurncy (%
80
50
60-
-00
00
63
40
40
30
30a Training Sample
Training Sample
Sample
Sample
Sample
ImageNet-Ri Seed
ImageNet-R Seed
ImageNet-FR Seed
ImagoNeet-R
ImageNet-Rt
50
1.00
100
60-1
Accuracy (86
50
45
00
50
80
-00
60
00
50
30
-60
4.0
35
30
35
40
25
-Training Sample
Training Sample
"Training Sample
Training Sample
Training Sample
Tiny- Nit-ImmanNNet Sood
Tiny-ImagaNe Soed
2
Tiny- Tiny-ImageNet Seed
Tiny-EmaguNNot Seed 4
Tiny-ImageNee Seed
100
1.00
100
100
Accuricy (%)
-00
an
100
50
-00
66
50
50-
-20
20 50
50
1k 20h 40% d0k 6ikk 190k. Training Sample
20% 4th 600 ath 2002
1k 20a 40a. 6ok 62k 1006.
201. G0a 60u 90% 1008 # Training Sample
201 40k 0ok 80a 1000
Training Sample
* Training Sample
# "Training Sample
Accuracy (%)
Figure 6: Real-time accuracy and class number ratio Te on 5 independent random seeds.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims clearly clarified, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? [Yes]
Answer:
Justification:
In Section 4.5 a discussion of the limitations and the future work of the GACL is
conducted.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
19

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs, In general. empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The proof of the Theorem 3.1 is listed in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross rooss-refereneed.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Question:
Answer: [Yes]
Justification: Our paper comprehensively outlines both the experimental implementation and algorithmic details, ensuring transparency in our method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
If the contribution
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
20

---

(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
All datasets are publicly accessible, and we have provided the source code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipps.cc/pbblic for more details.
guides http://odeeumiiisiieieeeeeeeeeee
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: httt:::://iisscc/pplte
guides/Coddeuubii ssi ssisnnolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have reported all the necessary details of our experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that
necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Results in this paper are reported by the average of 5 different seeds with standard erтor.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). It should be clear whether the error bar is the standard deviation or standard error of the mean.
It is OK to report I-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification:
The information on the computer resources for our GACL is listed in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS http:///sw...ooceeccacaauuetaaaa
Code of Ethics
Answer:
[Yes]
Justification: The paper fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed point out that a generic algorithm for optimizing neural networks
could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses addition to attacks, mechanisms for moni- toring misuse, mechanisms monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The models and benchmark datasets mentioned in the paper are all openly accessible with no personally identifiable information or offensive content.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have verified that this paper cites all the datasets and models we used. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets. wwpprswwttttteee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [Yes]
Justification: We have made source code publicly available at nttpa://ggitue com/ CHEN-YIZHU/GACO
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
24


Reviews:
[{"review_id": "ze41RwddE9", "reviewer": "Reviewer_vKCe", "summary": "This paper deals with the generalized CIL (GCIL) problem where incoming data have mixed data categories and unknown sample size distribution. The author proposes generalized analytic continual learning (GACL) which adopts a pre-trained and fixed backbone and uses least squares to get a closed-form solution. Experiments verified that the GACL achieves better performance compared to other baselines.", "questions": "* How and why the dimensionality of the random projection affects the performance.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The generalized class incremental learning is an important and valuable problem.\n* This paper is generally clear and easy to follow.\n* Experiments show that GACL is effective in addressing forgetting in GCIL.", "weaknesses": "* Main concern. Most of the content on page 4 and page 5 (e.g., Theorem 3.1) in Section 3 is overly similar to existing ACL works [7, 8]. The main difference claimed by the authors, i.e., the ECLG module that corresponds to the exposed-class gain, is a trivial extension of the original ACL [7].\n* Experiments. This paper adopts the pre-trained backbone developed in [40, 41]. However, many baselines (Table 2 in [40]) are missing in experiments. The GACL method is similar to RanPAC [25]. Experiments are needed to compare the performance of GACL with RanPAC.\n* Comparison of the memory cost of weights in the buffer layer and other replay-based method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Replies to Reviewer vKCe\nThank you for your constructive and detailed feedback. We provide detailed responses to your concerns\nbelow.\n\n### W1. Most of the content on page 4 and page 5 (e.g., Theorem 3.1) in Section 3 is overly similar to existing ACL works. The main difference claimed by the authors, i.e., the ECLG module that corresponds to the exposed-class gain, is a trivial extension of the original ACL.\"  \n\n**Response to W_1**: Thank you for your valuable comment! We ackownledge that there are certain simlarities between ACIL and the GACL, mostly in the begnining part. However, this part is to **generate embedding** (e.g., Eq. 2, Eq. 3) and the **joint-learning objective function formulation** (e.g., Eq.4, Eq. 5), both of which are **NOT** the contributions of this paper. There are more like a **Preliminary** contents (e.g., like introducing what is loss function before designing a new one). Our key contributions start from Eq. 6 and Eq. 8 that separate the derivation into an exposed-unexposed pair. The derivation holds the key difference from ACL techniques (see lines 457-484). We apologize that appendixing these items have led to such confusion, and will try to highlight the difference by moving some of them in the main context.\n\nRegarding the concern of the ECLG module being a trivial extension, we respectfully disagree. According to the derivation in lines 457-484, the solution **CAN NOT** be trivially extended from ACIL or its variants. We **deliberately extracted the ECLG** to show the connection to the existing ACL techniques, to **better illustrate** the weight-invariant property (which has been recognied in ACIL, GKEAL). We are very sorry to receive the \"trivial extension\" impression, and will mark additional highlight in this part to avoid such an impression.\n\n### W2. Experiments are needed to compare the performance of GACL with RanPAC.\n\n**Response to W_2**: Thank you for pointing out this important reference as they both adopt closed-form solutions! The RanPAC emphasizes on random project while our GACL focuses on recursive formulation. As RanPAC's codes are for CIL tasks, and our settings (e.g., SI-Blurry) are very different, and they utilize different backbones, we are working on reproducing its results while facing several technical issues. We are sorry that we may miss the rebuttal deadline before providing the comparison. We will try to provide them during the disucssion period!\n\n\n### W3. Comparison of the memory cost of weights in the buffer layer and other replay-based method. \n\n**Response to W_3**: GACL operates without the need for prior knowledge regarding the total number of classes. We denote the size of features acquired after the feature extractor and buffer layer as \"feature_size,\" and the number of classes learned at phase $k$ as \"n_class.\" During the continual learning process, the classifier can progressively increase the number of classes in the head and simultaneously augment the dimension of $W_k$ (feature_size $\\times$ n_class), while maintaining the size of $R_k$ as **feature_size $\\times$ feature_size**.\n\nCompared with other replay-based methods, such as MVP [1] with a memory size of 500, the memory requirement is 500 $\\times$ feature_size $\\times$ feature_size $\\times$ n_class, which **far exceeds the storage needed by GACL**. This is **without even accounting for model parameters and prompts**.\n\n[1] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, Gyeong-Moon Park; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11731-11741\n\n### W4. How and why the dimensionality of the random projection affects the performance.  \n\n**Response to Q_1**: Based on Cover's theorem [2], a promising approach to enhancing the separability of features from different domains is to project the features extracted by the pre-trained model into a higher-dimensional space using a non-linear projection. This non-linear, higher-dimensional projection can improve performance by increasing the separability of the features.\n\n[2] T. M. Cover, \"Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition,\" in IEEE Transactions on Electronic Computers, vol. EC-14, no. 3, pp. 326-334, June 1965, doi: 10.1109/PGEC.1965.264137.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZOSFjkb66q", "reviewer": "Reviewer_Xb3R", "summary": "This paper proposes a new exempler-free generalized continual learning (GCIL), named generalized analytic continual learning (GACL) technique. It does not depend on gradient-based tranining, which avoids the task-recency bias leading to the forgetting issue. It also delivers an closed-form solution to the GCIL scenario that provides identical solutions to its joint training. Entensive experiments demonstrate that the proposed GACL achieves consistently leading performance.", "questions": "- I believe that the proposed gradient-free training scheme is beneficial also in the perspective of training speed. Does GACL run more fast than other (ACL) methods?", "limitations": "- As described, GACL cannot update backbone weights, which can be a problem. However, as new backbones emerge frequently, showing that GACL can work well not only with the used DeiT-S/16 but also others makes the arguments more convincing, instead.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper proposes very effective method that completely avoids forgetting in the gradient-free training technique.\n- Extensive experiments are included explaining that GACL works good overall in the three datasets compared with many studies.\n- Ablation studies are coducted that clarifies the differences among datasets and the reason why GACL is slightly worse than other baselines at early tasks. They also analyzed the contributions of ECLG module and its robustness.\n- The paper is well written and easy to read.", "weaknesses": "- Missing baselines. As described in the Section 2, many ACL methods have been proposed. Comparing GACL with those methods (especially RanPAC) derives more solid results.\n- Comparison to the joint-traning is not included. Although it is argued that GACL can bridge the gap between continual learning and joint-training, such result is not provided in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Replies to Reviewer Xb3R\nThank you for your constructive and detailed feedback. We provide detailed responses to your concerns\nbelow.\n\n\n### 1. Comparing GACL with those methods (especially RanPAC)derives more solid results.\" \n**Response to W_1**: Thank you for pointing out this important reference as they both adopt closed-form solutions! The RanPAC emphasizes on random project while our GACL focuses on recursive formulation. As RanPAC's codes are for CIL tasks, and our settings (e.g., SI-Blurry) are very different, and they utilize different backbones, we are working on reproducing its results while facing several technical issues. We are sorry that we may miss the rebuttal deadline before providing the comparison. We will try to provide them during the disucssion period. Thanks!\n\n### 2. Comparison to the joint-traning. Although it is argued that GACL can bridge the gap between continual learning and joint-training, such result is not provided in the experiments.\n**Response to W_2**:\n\nAs suggested, we have included the last-phase accuracy of GACL and its joint-training accuracy on the CIFAR-100 dataset with different buffer sizes. Results in the table below show that the accuracy of GACL is indeed the same as that of joint training (the mile differences are caused by quantization).\n\n| Buffer Size | Joint-training (%) | G-ACL (%) |\n| :---------: | :----------------: | :-------: |\n|    1000     |      $66.83\\pm0.14$      | $66.73\\pm0.16$ |\n|    2000     |      $69.22\\pm0.14$      | $69.22\\pm0.23$ |\n\n\n### 3. Does GACL run more fast than other (ACL) methods?\n**Response to Q_1**: In general, our GACL gives rather strong performance regarding speed. The table below further records the GACL's training time in seconds compared with EFCIL methods and replay-based methods with the memory size of 2000. \n\nThe GACL runs faster than many baselines except a few candidates such as SLDA on three dataset. For SLDA, since only the classifier and autocorrelation memory matrix $\\mathbf{R}$ are updated, leading to smaller numbers of trainable parameters compared with those baselines in back-propagation manner.\n| Methods      | EFCIL               | CIFAR-100 (s) | ImageNet-R (s) | Tiny-ImageNet (s)|\n|--------------|-------------|-----------|------------|---------------|\n| EM        | ❌     |  >2 days | >2 days        |    >2 days |\n| MVP-R        | ❌     |   717 | 527        |    1597 |\n| ER           | ❌       | 369       | 330        |     715   |\n| EWC++        | ❌       |  650  |     391   |      1356    |\n| LwF          | ✅        | 334       | 229        | 862           |\n| L2P          | ✅       |   651    |     285       |     1246     |\n| DualPrompt   | ✅       |    656    |    332  |     1294     |\n| MVP          | ✅         | 628       | 300        | 1345          |\n| SLDA         | ✅         | 401       | 284        | 915           |\n| GACL     | ✅       | 611   | 321    | 1246      |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FRHmt7B2hU", "reviewer": "Reviewer_qU32", "summary": "Class Incremental Learning (CIL) faces the problem of catastrophic forgetting when training a network, i.e., the model loses previous knowledge when learning a new task. Generalized CIL (GCIL) aims to address more realistic scenarios, but existing methods are either ineffective or violate data privacy. This paper propose a new exemplar-free GCIL technique called Generalized Analytic Continual Learning (GACL), which provides a closed-form solution through analytic learning. GACL achieves equivalence between incremental learning and co-training by disaggregating the input data to keep the weights constant. This approach is theoretically validated and performs well across multiple datasets and settings.", "questions": "1. What was the motivation for introducing ACL into GCIL? The claims in this paper seem to be an extended experiment of ACL in the GCIL scenario. It is difficult to generate further inspiration to the reader for solving the GCIL problem.\n2. The proposed method utilize the DeiT-S/16 as backbone but previous works typically use ViT-B/16. Regarding the comparison methodology, what backbone network was used to obtain the reported results?\n3. What are the technical improvements and contributions of this paper over previous ACL approaches such as ACIL [2] and DS-AL [3]?\n\n[2] Huiping Zhuang, Zhenyu Weng, Hongxin Wei, RENCHUNZI XIE, Kar-Ann Toh, and Zhiping Lin. ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. NeurIPS 2022\n[3] Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning. AAAI 2024.", "limitations": "It is hard to foresee any potential negative societal impact of this theoretical work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and easy to follow.\n* The proposed method outperforms other methods on several GCIL benchmarks.\n* This paper provides a detailed proof of the theorem.", "weaknesses": "* The experimental setup is not clear. Given that the most recent method MVP [1] compared in this paper is the GCIL method for online CIL, were the experiments in this paper also conducted in an online scenario?\n* The technical improvements on the previous work of ACL are somewhat incremental.\n\n[1] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Replies to Reviewer qU32\nThank you for your constructive and detailed feedback. We provide detailed responses to your concerns\nbelow.\n\n### W1. Were the experiments in this paper also conducted in an online scenario?\n**Response to W_1**: Yes, we follow the settings in Si-Blurry [1], which is a online setting with blurry task boundaries.\n\n[1] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, Gyeong-Moon Park; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11731-11741\n\n### W2. \"The technical improvements on the previous work of ACL are somewhat incremental.\" and \"What are the technical improvements and contributions of this paper over previous ACL approaches such as ACIL and DS-AL?\"\n**Response to W_2 and  Q_3**: We are very sorry that our delivery of GACL gives a \"trivial extension\" impression. In fact, the GACL is developed with nontrivial and sophisticated manupulations beyond existing ACL techniques. According to the derivation in lines 457-484, the solution **CAN NOT** be trivially extended from ACIL or its variants (the derivation takes **2 whole pages** in the appenix). We **deliberately** show the connection to the existing ACL techniques, to **better illustrate** the weight-invariant property (which has been recognied in ACIL, GKEAL). We will mark additional highlight in this part to avoid such an \"incremental\" impression.\n\n### Q1. the motivation for introducing ACL into GCIL.\n**Response to Q_1**: Many real-world datasets follow GCIL tasks, such as the autonomous driving dataset SODA10M [2], the IoT dataset described in [3], and the real-world e-commerce service dataset described by [4]. GCIL simulates real-world incremental learning, where the distributions of data categories and sizes can be unknown in a given task. ACL techniques emerging as a new CIL branch, have a very appealing weight-invariant property in the CIL community, which **works magically esepcially for large-phase CIL tasks**. GCIL tasks (e.g., Si-blurry setting) are usually online (very large-phase), where ACL techinique can thrive. Existing ACL methods are specifically designed for CIL tasks only, and **CAN NOT** process GCIL ones, and the extension is **NOT** trivial (see derivations in lines 457-484), hence the motivation of GACL.\n\n[2] Han, Jianhua, et al. \"SODA10M: A large-scale 2D self/semi-supervised object detection dataset for autonomous driving.\" *arXiv preprint arXiv:2106.11118* (2021).\n\n[3] Wen, Zhenyu, et al. \"Fog orchestration for IoT services: issues, challenges and directions.\" *IEEE Internet Computing* 21.2 (2017): 16-24.\n\n[4] Bang, Jihwan, et al. \"Rainbow memory: Continual learning with a memory of diverse samples.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2021.\n\n### Q2. What backbone network was used to obtain the reported results?\"\n**Response to Q_2**: As mentioned in Section 4.1, all experiments were conducted using the same pre-trained backbone. We utilize the DeiT-S/16 as our backbone and pre-train the backbone on 611 ImageNet classes after excluding 389 classes that overlap with CIFAR and Tiny-ImageNet to prevent data leakage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BnJmUg0RhM", "reviewer": "Reviewer_tsV2", "summary": "This paper proposes a new exemplar-free GCIL technique named generalized analytic continual learning (GACL). It adopts analytic learning (a gradient-free training technique), and delivers an analytical (i.e., closed-form) solution to the GCIL scenario, which is derived via decomposing the incoming data into exposed and unexposed classes. The GACL attains a weight-invariant property, supporting an equivalence between the incremental learning and its joint training. This method is both therectically and empircally validated.", "questions": "1. Continual learning has many variations (e.g., domain/task/class -incremental learning). Since the experiments only focus on generalized class-incremental learning, it would be better to adjust the title accordingly.\n2. Since generalized CIL is also a specific case of class-incremental learning, it would be better to discuss why the proposed method is specially designed for GCIL. For example, when the data stream only contains new classes, will GCIL still work?\n3. As the authors discussed in the main paper, the proposed method is a super case of ACIL and other analytical methods. As many readers would like to know, the authors are suggested to compare this method to other analytical methods to show how this method stands against other analytical works.", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This is very interesting technique that obtains an equivalence between the GCIL and the joint training. Although a pre-trained network is needed, the weight-invariant property is very valuable in this area.\n2. Another key contribution of GACL is that it is both accurate and exemplar-free, and this is powerful and significant. \n3. The approach is also well-motivated and clearly explained.\n4. The paper provides a comprehensive literature review, effectively contextualizing its contribution and facilitating the reader's understanding.\n5. It is very clear theoretically regarding how the algorithm is parted into W_unexposed and W_ECLG.", "weaknesses": "1. Could you explain why the avg accuracy on CIFAR-100 is much lower than the last accuracy?\n2. You need to explain more regarding \"weight-invariance\" in Theorem 3.1. It is not very easy to follow for those who are not in this area of research.\n3. In this paper, the authors assume that the pre-trained backbone is generalizable so that the model can be frozen throughout the learning process. However, there are also some cases where the pre-trained model cannot generalize to downstream tasks. It would also be interesting to see the scenario where the pre-trained backbone yields a large domain gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Replies to Reviewer tsV2\nThank you for your constructive and detailed feedback. We provide detailed responses to your concerns\nbelow.\n\n\n### W1. Why the avg accuracy on CIFAR-100 is much lower than the last accuracy?\n**Response to W_1**: As mentioned in Appendix G, tasks on CIFAR-100 are notably more complex and intricate, resembling a few-shot learning scenario. Consequently, the accuracy in the initial tasks tends to be lower, leading to a relatively low average accuracy.\n\n### W2. Explain more regarding \"weight-invariance\" in Theorem 3.1.\n**Response to W_2**: The weights of the classifier obtained by this recursive update are exactly the same as the weights obtained by training the classifier from scratch on the entire data set. This *weight-invariant property* achieves a near “complete non-forgetting” and makes the GACL outperforms all EFCIL methods and most replay-based methods.\n\n### W3. It would also be interesting to see the scenario where the pre-trained backbone yields a large domain gap.\n**Response to W_3**: As a demonstration, a 5-phase experiment is conducted in table below, with DeiT-S/16 backbone pretrained on ImageNet-1k, and CIL with DTD [1]. \n|                        | Buffer | DTD |         |          |\n|:----------------------:|:------:|:--------:|:-------:|:--------:|\n|                        |        | ACC_AUC  | ACC_AVG | ACC_LAST |\n| EWC++ (ECCV,2018)     |  2000  |        55.66       |      58.97       |    47.32          |\n| ER (ICML,2019)        |  2000  |       51.78        |      57.87         |    50.99           |\n| MVP (ICCV,2023)       |  2000  |       52.97       |      55.15        |    54.10         |\n| EWC++ (ECCV,2018)     |  500   |      54.28        |     57.82          |    45.44          |\n| ER (ICML,2019)        |  500   |      51.23        |     57.45         |     49.67          |\n| MVP (ICCV,2023)       |  500   |      52.76        |     55.19       |       53.72        |\n| LwF (ECCV,2016)       |   0    |      42.67         |    48.41          |     33.42          |\n| L2P (CVPR 2022)       |   0    |     43.11          |    51.25           |    39.61           |\n| DualPrompt (ECCV 2022)|   0    |     42.12        |     50.17        |        39.29       |\n| MVP (ICCV,2023)       |   0    |     46.88       |      49.98       |         44.41       |\n| SLDA (IEEE/CVF2020)   |   0    |       49.84       |        50.33      |      55.11         |\n| **GACL**                   | **0**      | **61.93**    | **66.16**   | **63.69**    | \n\n[1] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, Andrea Vedaldi; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 3606-3613.\n\n### Q1. Since the experiments only focus on generalized class-incremental learning, it would be better to adjust the title accordingly.\n**Response to Q_1**: Thank you for pointing out that. We will modify our title  into \"Exemplar-Free Generalized Analytic Class Incremental Learning\".\n\n### Q2. Why is the proposed method specifically designed for generalized class-incremental learning (GCIL)? \n**Response to Q_2**: As demonstrate in Section 3.2 and 3.3, we introduce the ECLG module that corresponds to the exposed-class gain since classes can reappear in GCIL setting.\n\n### Q3:Will GCIL still work if the data stream contains only new classes?\n**Response to Q_2**: Yes. We employ Si-Blurry[2] to generate GCIL tasks for our experiments. As demonstrated in Section 4.4, when the disjoint class ratio $r_{\\text{D}}$ is 0, it indicates that the data stream contatians only new classes.\n\n[2] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, Gyeong-Moon Park; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11731-11741", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We thank all the reviewers for their time, insightful suggestions and valuable comments. In summary, Reviewer vKCe, Reviewer Xb3R and Reviewer qU32 all appreciate that our writing is **clear** and **easy to follow**. Reviewer tsV2 appreciates that our method is **clear**, **well-motivated** and **valuable**.

We provide point-by-point responses to all reviewers’ comments and concerns. On the other hand, reviewers also point out that our delivery of GACL could resenble existing ACL techniques, giving an unnecessiry "trivial extension" impression. In this regard, we will marke the highlights of our GACL from formulation, derivation and contributions to avoid such an impression.

Meta-review: The paper addresses the problem of a generalized form of lifelong learning, where learning is pursued in a continuous manner, with the possibility of revisiting samples from previously learned concepts. The authors demonstrate that this problem can be efficiently addressed using the concept of analytical learning. Initially, the paper received mixed feedback: 1x3 (reject), 1x4 (borderline reject), 1x6 (weak accept), and 1x8 (strong accept). After the rebuttal and the author-reviewer discussion period, the scores improved to 1x5 (borderline accept), 1x6 (weak accept), 1x7 (accept), and 1x8 (strong accept), with all reviewers recommending acceptance.

The reviewers acknowledged the novelty of the idea within the context of the problem, and the AC agrees with this assessment, recommending the paper for acceptance. Congratulations.

The AC suggests including all the results discussed during the rebuttal and the author-reviewer discussions. Additionally, the formulation of the classification problem as a regression, as done in the paper, can be acknowledged as a limitation of the current work.

Congratulations again.
Decision: Accept (poster)