Title: GACL: Exemplar-Free Generalized Analytic Continual Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Class incremental learning

closed-form solution

exemplar-free

continual learning

online continual learning

generalized continual learning
Abstract: Class incremental learning (CIL) trains a network on sequential tasks with separated categories in each task but suffers from catastrophic forgetting, where models quickly lose previously learned knowledge when acquiring new tasks. The generalized CIL (GCIL) aims to address the CIL problem in a more real-world scenario, where incoming data have mixed data categories and unknown sample size distribution. Existing attempts for the GCIL either have poor performance or invade data privacy by saving exemplars. In this paper, we propose a new exemplar-free GCIL technique named generalized analytic continual learning (GACL). The GACL adopts analytic learning (a gradient-free training technique) and delivers an analytical  (i.e., closed-form) solution to the GCIL scenario. This solution is derived via decomposing the incoming data into exposed and unexposed classes, thereby attaining a weight-invariant property, a rare yet valuable property supporting an equivalence between incremental learning and its joint training. Such an equivalence is crucial in GCIL settings as data distributions among different tasks no longer pose challenges to adopting our GACL. Theoretically, this equivalence property is validated through matrix analysis tools. Empirically, we conduct extensive experiments where, compared with existing GCIL methods, our GACL exhibits a consistently leading performance across various datasets and GCIL settings. Source code is available at https://github.com/CHEN-YIZHU/GACL.

GACL: Exemplar-Free Generalized Analytic Continual Learning

Abstract

Class incremental learning (CIL) trains a network on sequential tasks with separated categories in each task but suffers from catastrophic forgetting, where models quickly lose previously learned knowledge when acquiring new tasks. The generalized CIL (GCIL) aims to address the CIL problem in a more real-world scenario, where incoming data have mixed data categories and unknown sample size distribution. Existing attempts for the GCIL either have poor performance or invade data privacy by saving exemplars. In this paper, we propose a new exemplarfree GCIL technique named generalized analytic continual learning (GACL). The GACL adopts analytic learning (a gradient-free training technique) and delivers an analytical (i.e., closed-form) solution to the GCIL scenario. This solution is derived via decomposing the incoming data into exposed and unexposed classes, thereby attaining a weight-invariant property, a rare yet valuable property supporting an equivalence between incremental learning and its joint training. Such an equivalence is crucial in GCIL settings as data distributions among different tasks no longer pose challenges to adopting our GACL. Theoretically, this equivalence property is validated through matrix analysis tools. Empirically, we conduct extensive experiments where, compared with existing GCIL methods, our GACL exhibits a consistently leading performance across various datasets and GCIL settings. Source code is available at https://github.com/CHEN-YIZHU/GACL .

1 Introduction

Class incremental learning (CIL) [1], an important form of continual learning, aims to effectively tune an off-the-shelf network on incoming new datasets, with data excluding various categories from its previous states. The CIL has gained significant traction due to its ability to refine learned models for new and unfamiliar data classes, eliminating the need to start the training process from scratch. This elimination of retraining saves valuable computational resources, which is especially important in the era of pre-trained models that have absorbed a massive amount of data.

One significant challenge in CIL is catastrophic forgetting [2, 3], which causes trained models to lose existing knowledge when gaining new information quickly. This can be attributed to the fundamental property of gradient-based iterative algorithms that impose a task-recency bias, i.e., predictions favor recently updated categories [4]. To the authors' knowledge, no solutions exist for these gradient-trained CIL models to fully tackle catastrophic forgetting.

On the other hand, traditional CIL assumes that the number of samples in each task is fixed and that new tasks are entirely disjoint from previous ones. This paradigm does not align with real-world scenarios, where training data may include both new and previously encountered categories, and the number of data points often exhibits arbitrariness in each task. This extended CIL setting is referred to as generalized CIL (GCIL) [5, 6]. Such an uneven task-wise distribution of training samples and data categories further complicates the forgetting issue. For instance, GCIL may lead to the neglect of minority samples within a batch, thereby undermining representation during the training process.

To mitigate catastrophic forgetting, a simple but effective approach is to replay historical samples. Replay-based CIL [1, 4] mitigates forgetting by storing a small number of samples from historical categories for the model to review while learning new information. However, this replay mechanism poses risks to data privacy. Thus, the exemplar-free CIL (EFCIL) without saving old exemplars gains prominence due to the increasing concern for privacy. However, many EFCIL methods perform poorly due to the task-recency bias caused by the nature of gradient-based algorithms [4]. Recently, this dilemma has been alleviated by the analytic continual learning (ACL) [7, 8], an emerging EFCIL branch that first achieves comparable or even more competitive performance over the replay-based CIL. This improvement occurs because, for the first time, ACL achieves a near 'complete nonforgetting' by allowing an equivalence between the incremental learning and its joint training (i.e., the weight-invariant property).

The ACL provides a powerful toolbox for traditional EFCIL scenarios where data categories among training tasks are mutually exclusive. However, an apparent gap exists between the existing ACL techniques and the more desired and real-world GCIL scenario. Exploring the possibility of incorporating the weight-invariant property into the GCIL framework is both a significant and natural motivation, as it has the potential to enhance overall performance. To achieve this, we propose a generalized analytic continual learning (GACL), a new and compensated ACL member, offering a weight-invariant property solution to the GCIL. The key contributions are summarized as follows.

We present the GACL, an exemplar-free technique that achieves the equivalence between the GCIL (with split incoming data) and its joint training (with data centralized in a single task).

We theoretically establish the GACL's weight-invariant property. It is achieved and proved by separating the incoming data into exposed and unexposed components and aligning them structurally with matrix decomposition techniques.

We isolate the distinctive component of the GACL, namely the exposed class label gain (ECLG), from the existing ACL. This module explains the feasibility of achieving GCIL's analytic learning, offering a high interpretability in the GCIL realm.

Experiments on various benchmark datasets are presented, showing that the GACL outperforms the existing EFCIL by a large margin. It also exceeds most state-of-the-art replay-based methods.

2 Related Works

This section reviews existing methods for CIL and its more real-world counterpart, i.e., GCIL.

2.1 CIL Techniques

Existing CIL methods can be roughly divided into three categories: replay-based methods, regularization-based methods, and prototype-based methods.

The replay-based CIL methods such as iCaRL [1], LUCIR [4], PODNet [9], AANets [10], FOSTER [11], and OHO [12], retain past training samples as exemplars and utilize them during the learning of new ones. However, storing original training samples presents a significant challenge, particularly in scenarios with strict data privacy requirements.

The regularization-based CIL aims to design a loss function that prevents the change of activations or important weights. Methods such as the Less-forgetting learning [13] and the LwF [14] introduce knowledge distillation [15] into their loss function to prevent the forgetting caused by activation drift. EWC [16], EWC++, RWalk [17], and Rotate your Networks [18], introduce regularization that slows down learning on the weights important for old tasks by calculating the Fisher information matrix.

The prototype-based CIL maintains distinct prototypes for each category, which prevents overlapping representations of new and old categories. For example, the PASS [19] distinguishes prior categories

by augmenting feature prototypes. The SSRE technique [20] enhances the dissimilarity between old and new categories via selecting prototypes to incorporate with new samples into a distillation process. The FeTrIL [21] uses new representations to generate pseudo-features of old categories.

2.2 Analytic Continual Learning

The ACL is a recently developed EFCIL branch inspired by the analytic learning [22, 23, 24] where the training of neural networks yields a closed-form solution using least squares. The ACIL [7] first converts a continual learning problem to a batch recursive least-squares problem, eliminating the need to store samples by preserving the correlation matrix, and the RanPAC [25] applies this trick to pre-trained models. The GKEAL [8] focuses on the few-shot CIL scenarios by leveraging a Gaussian kernel projection. The DS-AL [26] introduces an additional linear classifier to learn the residue of the ACIL to enhance the plasticity, while the REAL [27] introduces the representation enhancing distillation to improve the backbone's generalization capabilities. The AFL [28] extends the ACL to federated learning, transitioning from temporal increment to spatial increment, and similar techniques are applied to the reinforcement learning [29]. The ACL is an emerging competitive CIL branch with a closed-form solution that leads to a valuable weight-invariant property, securing the equivalence between CIL and its joint learning. However, existing ACL methods are designed for the CIL scenario in which the categories of samples in each task must be entirely distinct. This restricts their applicability in real-world scenarios.

2.3 The Generalized Class Incremental Learning

The GCIL simulates real-world incremental learning, as distributions of data category and size could be unknown in one task. The GCIL arouses problems such as intra- and inter-task forgettings and the class imbalance problem [30]. The key GCIL properties can be summarized as follows: (i) the number of classes across different tasks is not fixed; (ii) classes shown in prior tasks could reappear in later tasks; (iii) training samples are imbalanced across different classes in each task [6] (See Appendix B).

There are several GCIL settings. In the BlurryM setting [5], a % of the classes are disjoint between tasks, while the remaining classes appear in every task. The i-Blurry-N-M [31] setting has blurry task boundaries and requires the model to perform anytime inference. However, the i-Blurry scenario has a fixed number of classes in each task with the same proportion of new and old classes. The Si-Blurry [30] is the most complex and realistic GCIL setting satisfying all three GCIL properties since it has an ever-changing number of classes and is capable of effectively simulating newly emerging or disappearing data, highlighting the problem of uneven distribution in real-world scenarios.

To address the issue of the GCIL, gradient-based sample selection methods such as the GSS-IQP and the GSS-Greedy are proposed by [5]. The RM [32] proposes a memory management strategy based on per-sample classification uncertainty and data augmentation, while the management in the CLIB [31] eliminates samples based on a per-sample importance score. The DualPrompt [33], as an EFCIL method, introduces the prompt-based learning to the CIL problem, and the MVP [30] proposes an instance-wise logit masking and contrastive visual prompt tuning loss.

3 The Proposed Method

In this section, we deliver details of the proposed GACL. We first define the learning problem. Then, we derive the GACL by employing matrix decomposition techniques. A corresponding theoretical analysis follows to indicate the interpretability of our work. An overview is depicted in Figure 1.

3.1 Problem Definition

We denote the complete set of available data as D . When D is partitioned into a sequence of GCIL tasks, we assume that D train k ∼ { X train k , Y train k } is the set of training samples that are present in task k . The training dataset D train k consists of labeled samples, where X train k ∈ R N k × c × w × h represents N k input image samples with a shape of c × w × h . Y train k ∈ R N k × d y k represents N k -stacked one-hot encoded label tensors with d y k classes that have been seen from task 1 to task k . D test k ∼ { X test k , Y test k }

Split        into

GLYPH(cmap:d835)

GLYPH(cmap:d835)-GLYPH(cmap:d835)

(a) Exposed-unexposed Class Split

(c) Generalized Analytic Class Incremental Learning

GLYPH(cmap:d835)×

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)-GLYPH(cmap:d835)

Task #k

Finished Task #k-1

update

update

…

GCIL Dataset

Task #k

(b) Buffered Embedding Extraction

Norm

Norm

Multi-head

Attention

MLP

Exposed

Classes

Unexposed

Classes

GLYPH(cmap:d835)

GLYPH(cmap:d835)

…

…

…

…

0

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc04)GLYPH(cmap:d835)GLYPH(cmap:dc02)

Exposed

Class

Output

Exposed

Class

Output

Unexposed

Class

Output

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc2e)GLYPH(cmap:d835)GLYPH(cmap:dc27)GLYPH(cmap:d835)GLYPH(cmap:dc1e)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

…

…

is the test dataset in task k . The goal of GCIL in task k is to train networks using D train k and evaluate their performance on the test dataset D test 1: k . Here, D 1: k denotes the joint dataset spanning tasks 1 to k .

3.2 Exposed-unexposed Class Split

In each GCIL task, classes may not appear exclusively. Hence, in any GCIL task k , we refer to classes that have appeared in previous tasks 1 to k -1 as the exposed classes of task k , while classes making their initial appearance are the unexposed classes of task k as shown in Figure 1 (a). This distinction helps to characterize the evolving nature of class occurrences throughout different GCIL tasks.

In a task-wise GCIL scenario, we can involve all class labels in a set S . In task k , the set of the exposed class labels is denoted as S exposed, k ⊆ S , while the set of unexposed class labels is marked by S unexposed, k ⊆ S , where S exposed, k ∩ S unexposed, k = ∅ . Note that S exposed, k and S unexposed, k may evolve from task k -1 to task k , that is

From the scope of exposed-unexposed classes, the d y k can be represented as d y k = |S exposed, k | + |S unexposed, k | = d y k -1 + |S unexposed, k | , where |·| denotes the cardinality of a set.

In task k , given training dataset D train k ∼ { X train k , Y train k } , class labels Y train k can be partitioned due to the exposed-unexposed split as follows:

where ¯ Y train k ∈ R N k × d y k -1 is the exposed class label matrix and ˜ Y train k ∈ R N k × ( d y k -d y k -1 ) is the unexposed class label matrix . They correspond to segments displaying the appearance of exposed classes and unexposed classes.

3.3 Buffered Embedding Extraction

The power of pre-trained models allows the GACL to adopt a frozen backbone from structures such as ViT [34] to extract the features of images shown in Figure 1 (b). Let

be the features extracted by the backbone, where Θ backbone indicates the backbone weight. Then, we use a buffer layer to project features, i.e.,

where f buffer indicates the operation of the buffer layer. Several options for the buffer layer exist, including a randomly initialized linear mapping in the ACIL [7] or a kernel embedding projection in the GKEAL [8]. The selection of the buffer layer is not our focus. For convenience, we follow the ACIL, taking the random linear projection followed by a non-linear activation function as the buffer layer, i.e. f buffer ( X ( E ) ) = ReLU( X ( E ) W B ) , where the elements of the buffer layer weight W B are randomly sampled from a normal distribution.

3.4 Generalized Analytic Class Incremental Learning

Here, we derive the GACL by partitioning training samples into unexposed and exposed categories, as shown in Figure 1 (c). Let X total 1: k and Y total 1: k be the accumulated feature and label matrices in task k , which can be extended from the accumulated matrices X total 1: k -1 and Y total 1: k -1 in task k -1 as follows.

Subsequently, one could formulate the learning problem in task k by a fully connected network (FCN) as the classifier

where ∥·∥ F is Frobenius-norm, γ ≥ 0 is the regularization term and W ( k ) FCN indicates the FCN layer weight. The optimal solution to (4) is

The goal of the GACL is then to obtain ˆ W ( k ) FCN recursively from ˆ W ( k -1) FCN without directly involving historical samples (e.g., X total 1: k -1 and Y total 1: k -1 ). That is to solve

by recursively updating the previous-task weight ˆ W ( k ) FCN . To achieve this, we define an autocorrelation memory matrix as follows.

Accordingly, we summarize the recursive formulation of the proposed GACL in Theorem 3.1.

Theorem 3.1. Let ˆ W ( k ) FCN be the optimal estimation of (6) with all the training data from task 1 to task k . Then ˆ W ( k ) FCN is equivalent to its recursive form

where

As indicated in Theorem 3.1, the weight ˆ W ( k ) FCN in task k recursively obtained using the previous-task weight ˆ W ( k -1) FCN is identical to its joint-learning counterpart formulated in (6). That is, the GACL maintains the same weight-invariant property in the GCIL scenario as other ACL methods.

Algorithm 1 The pseudo-code of GACL.

Input: GCIL tasks D train 1 , . . . , D train K with D train k ∼ { X train k , Y train k } , the pre-trained backbone with frozen weight Θ backbone

Initialization:

for

(0)

FCN

Decompose Y train k into exposed and unexposed class components ¯ Y train k and ˜ Y train k

R

0

γ

I

,

W

0

R

R

X

k

W

R

R

k

-

(

k

)

unexposed

k

(B)

-

1

k

-

1

ˆ

W

X

(B)

⊤

k

(

k

-

FCN

[

k

-

1

(B)

k

(

I

+

X

(B)

k

W

(

k

-

1)

(B)

⊤

FCN

-

⊤

R

k

X

k

X

The pseudo-code of the GACL is listed in Algorithm 1.

Exemplar-free. The recursive formulation is aided by R k as indicated in (9). Note that this autocorrelation memory matrix records the inverse of inner products among the historical embedding matrices as shown in (7). Hence, the embeddings (e.g., X (B) k ) are not reversible. Saving R k instead of used samples is a safe alternative to preserve past knowledge. That is, our GACL is an exemplar-free technique without the need to keep any historical samples.

To more properly explain our GACL, as indicated in Figure 1 (c), the recursive solution in (8) can be rewritten as the sum of the unexposed-class contributed weight ˆ W ( k ) unexposed and the ECLG weight ˆ W ( k ) ECLG , i.e.,

Unexposed-class Contributed Weight. The unexposed-class contributed weight ˆ W ( k ) unexposed is recursively updated by the data of the unexposed class only. Note that the unexposed class label ˜ Y train k is applied on the concatenated weight along with new data X (B) ⊤ k , which is reasonable as historical information should not intervene with the weight update of unseen classes. On the other hand, new data X (B) ⊤ k could also affect historical knowledge. This is marked by the gain of -R k X (B) ⊤ k X (B) k ˆ W ( k -1) FCN to the original weight ˆ W ( k -1) FCN as indicated in (11).

Exposed-class Label Gain Weight. The ECLG module indicated in (12) captures knowledge from exposed-class labels. The supervision of this weight component marked by R k X (B) ⊤ k ¯ Y train k is mainly contributed by the exposed-class labels (i.e., ¯ Y train k ). It is important to note that when ¯ Y train k is empty (i.e., no classes reappear in task k ), this component does not contribute to the update of ˆ W ( k ) FCN . This module is also isolated to distinguish GACL's difference from the existing ACL methods in a mathematical analysis manner (indicated as follows).

Difference from Existing ACL Methods. Overall, the GACL can be treated as a nontrivial generalization of ACIL [7], GKEAL [8], and various other ACL methods. For instance, in conventional CIL where no classes reappear in new tasks (i.e., ∀ k, ¯ Y train k ∈ R ∗× 0 ), the classifier of the GACL ˆ W ( k ) FCN = ˆ W ( k ) unexposed , which is equivalent to the recursive classifier of the ACIL. That is, the ACIL is a special case of our proposed GACL. The major difference lies in the ECLG module, corresponding to the exposed-class gain. This pattern makes sense as there must be compensation on top of ACIL updates (specifically designed for traditional CIL) when exposed data (out of setting) participate.

1)

)

-

1

X

R

k

k

R

X

k

(B)

-

⊤

k

1

˜

Y

(9)

train

]

k

where

(11)

4 Experiments

4.1 Experimental Setup

In the section, we conduct experiments on various benchmark datasets and compare the GACL with both EFCIL and replay-based state-of-the-art methods, including LwF [14], L2P [36], DualPrompt [33], ER [35], EWC++ [16], SLDA [37], RM [32], MVP [30], and MVP-R (MVP with exemplars). 3

Datasets. We conduct experiments on three datasets: CIFAR-100 [38], ImageNet-R [39], and Tiny-ImageNet [40]. We evaluate each method under the Si-Blurry setting [30] (the most complex GCIL setting) with 5 independent seeds. For the Si-Blurry setting, we set the disjoint class ratio r D to 50% and the blurry sample ratio r B to 10% . More details about Si-Blurry are listed in Appendix C.

Implementation Details. We utilize the DeiT-S/16 [41] as our backbone. Following [42, 43], we pre-train the backbone on 611 ImageNet classes after excluding 389 classes that overlap with CIFAR and Tiny-ImageNet to prevent data leakage. To ensure a fair comparison, all methods utilize a frozen backbone. All methods under comparison are implemented as specified in [30]. The memory sizes of compared relay-based methods are set to 500 and 2000.

There are two hyperparameters in the GACL, the regularization term γ and the size of the buffer layer. Here, we adopt γ = 100 , which is determined by the grid search of {0, 10, 100, 500, 1000, 10000} on CIFAR-100 (by a 90%-10% train-val split). As the regularization term γ is not sensitive in a proper range [7], we adopt this value for all datasets for convenience. We relocate its analysis to Appendix E. The size for the buffer layer W B is set to 5000 for both the GACL and ACIL for convenience.

Evaluation Protocol. Three metrics are adopted to evaluate GCIL tasks. The real-time performance is evaluated by the area under the curve of accuracy A AUC [31], i.e., A AUC = ∑ k i =1 f ( i · △ n ) · △ n , where △ n is the number of samples observed between evaluation and f ( · ) is the curve in the accuracyto-{number of training samples} plot, measuring anytime inference performance during training. A higher A AUC corresponds to a method consistently maintaining high accuracy throughout the training. The overall performance is evaluated by the average incremental accuracy (or average accuracy) A Avg = 1 K +1 ∑ K k =1 A k , where the task-wise accuracy A k indicates the average test accuracy in task

k by testing the network on D test 1: k . A higher A Avg score is preferred when evaluating algorithms. The last evaluation metric is the last-task accuracy A Last evaluating the network's last-task performance after completing all tasks.

4.2 Comparison with State-of-the-arts

As shown in Figure 2, we comprehensively compare the GACL with both EFCIL and replay-based methods.

10

20

30

40

50

60

70

80

(a) EFCIL Methods

CIFAR-100

ImageNet-R

Tiny-ImageNet

1

2

3

4

5

Task #

10

20

30

40

50

60

70

80

(b) Replay-based Methods

1

2

3

4

5

Task #

1

2

3

4

5

Task #

LwF

MVP

L2P

SLDA

DualPrompt

GACL

(ours)

EWC++ (500)

ER (2000)

ER (500)

RM (2000)

RM (500)

MVP-R (2000)

MVP-R (500)

Compare with EFCIL Methods. EFCIL methods address privacy concerns and mitigate catastrophic forgetting without exemplars. Among EFCIL methods, our GACL consistently exhibits superior performance across all three datasets, as illustrated in the lower panel of Table 1.

For instance, on CIFAR-100, our method surpasses the second-best method SLDA, by 4.99 % , 6.15 % , and 8.52 % for A AUC, A Avg, and A Last, respectively. On Tiny-ImageNet, the GACL achieves impressive results with A AUC, A Avg, and A Last reaching 63.14 % , 69.32 % , and 62.68 % , respectively, surpassing the previous best EFCIL by 13.97 % , 21.39 % , and 9.55 % . Similar patterns are evident in the results of ImageNet-R, further confirming that the GACL is an exceptional tool for GCIL.

Owing to the weight-invariant property, the GACL exhibits more accurate and stable evolutions as k increases as observed in Figure 2 (a). All compared EFCIL methods exhibit sharp declines in accuracy, while the GACL delivers nearly non-declining curves. In particular, on CIFAR-100, the GACL shows an unnatural improvement of task-wise accuracy throughout the learning tasks, with the GACL initially lagging behind other EFCIL methods. This is because the Si-Blurry samples more than 70% of the CIFAR-100 categories in the first two tasks (see Appendix F), constructing a scenario where gradient-based algorithms could largely avoid the forgetting issue. Moreover, our method produces more stable predictions across diverse scenarios, as indicated by much smaller standard errors (colored shades in Figure 2 (a)). In summary, the experimental results demonstrate that our proposed GACL is exceedingly accurate and robust, exhibiting exceptional generalization ability.

Compare with Replay-based Methods. Replay-based methods are considerably competitive as they leverage historical samples. The memory size is a key adjustment, as increasing it typically leads to performance improvements by allowing more historical knowledge to be reviewed. For instance, the MVP-R achieves 4.42%, 3.97%, and 8.95% gains for A AUC, A Avg, and A Last (see Table 1) on CIFAR-100 when increasing the memory size from 500 to 2000.

As an exemplar-free technique, our GACL avoids re-using the historical samples. However, as indicated in Table 1, the GACL still outperforms most existing replay-based results. For instance, the GACL achieves the best A Last results among all settings. The GACL's A AUC and A Avg results are also mostly superior, except that our performance is slightly weaker than that of MVP-R with a memory size of 2000 on CIFAR-100 and ImageNet-R. Although increasing the number of exemplars can further improve the results of replay-based methods, this approach could lead to higher training and memory costs and, more importantly, more severe privacy invasion.

As indicated in Figure 2 (b), replay-based methods experience accuracy declines similar to those observed in the EFCIL case. This decline is due to an inherent limitation of gradient-based iterative algorithms, which tend to favor recently trained categories and thus lead to catastrophic forgetting. The GACL is iterative-free and then not constrained by this forgetting issue, thereby achieving nearly no performance reduction as K increases.

Why the GACL Gives Leading Performance. The above comparisons show that the proposed GACL is a powerful GCIL technique. Its competitive performance can be explained as follows. (i) Weight-invariant property. As shown in Theorem 3.1, the weight obtained recursively is equal to its joint-learning counterpart, indicating that the GACL is a 'completely non-forgetting' technique (under the condition of a frozen backbone). (ii) Analytical solution. Existing GCIL techniques are gradient-based iterative algorithms prone to catastrophic forgetting by nature. The GACL is a new member of the ACL and inherits its non-iterative gradient-free essence with an analytical solution, thereby avoiding the task-recency bias to address forgetting.

4.3 Ablation Study on the ECLG Module

The ECLG module is a core component that allows the GACL to obtain the weight-invariant property in the GCIL scenario. Here, we conduct an ablation study to justify the ECLG's contributions under various blurry sample ratios r B with r D = 50% . Larger r B indicates more complex data distributions in the Si-Blurry setting. As shown in Table 2, the GACL without ECLG exhibits poor performance with a visible gap for A AUC, A Avg, and A Last. For instance, on CIFAR-100 with r B = 10%, the ECLG contributes a 23.01% A Last gain to the GACL.

As claimed in Theorem 3.1, the classifier without the ECLG module fails to absorb knowledge from joint classes in each task (i.e., classes that reappear), leading to substantial information loss under the GCIL setting. The GACL, equipped with the ECLG module, demonstrates competence in handling overlapping classes in realistic scenarios.

4.4 Robustness Analysis in Si-Blurry Setting

Here, we conduct a robust analysis by varying the disjoint class ratio r D and the blurry sample ratio r B. The comparison happens among the GACL, the second-best EFCIL method SLDA, and the top-performing replay-based method MVP-R with a memory size of 500.

We evaluate our method under various r D, including extreme cases where each task shares classes ( r D = 0% ) and traditional CIL scenarios ( r D = 100% ). Table 3 illustrates that our GACL consistently outperforms the compared methods (e.g., leads the SLDA by 2%-10%) and produces near-identical A Last values with varying r D. This shows the accurate and robust traits of the GACL.

We also evaluate our method using various r B values, as shown in Table 4. Similar patterns observed here align with those in Table 3, further demonstrating the robustness of the proposed GACL, which delivers exceptional performance across different GCIL settings.

4.5 Limitation and Future Work

Overall, the GACL exhibits various good characteristics as an exemplar-free GCIL technique. The major limitation here is the need for a well-trained backbone because the GACL does not update backbone weights. This could motivate the exploration of adjustable backbones to continuously improve their feature extraction abilities, thereby further enhancing GACL's performance.

5 Conclusion

In this paper, we introduce the exemplar-free generalized analytic class incremental learning (GACL) approach to address the GCIL problem. Building upon analytic learning, the GACL delivers closedform solutions to GCIL through the decomposition of GCIL data into exposed and unexposed classes. The GACL achieves the weight-invariant property that provides identical solutions for GCIL to its joint learning counterpart. We theoretically validate this property and provide high interpretability through the matrix analysis tool. Various experiments are conducted under the Si-Blurry setting, demonstrating that our proposed GACL achieves remarkable performance with high robustness compared to state-of-the-art EFCIL and replay-based methods.

Acknowledgments and Disclosure of Funding

This research was supported by the National Natural Science Foundation of China (62306117), the Guangzhou Basic and Applied Basic Research Foundation (2024A04J3681, 2023A04J1687), the South China University of Technology-TCL Technology Innovation Fund, the Fundamental Research Funds for the Central Universities (2023ZYGXZR023, 2024ZYGXZR074), the Guangdong Basic and Applied Basic Research Foundation (2024A1515010220), the CAAIMindSpore Open Fund developed on Openl Community, the Shenzhen Fundamental Research Program (JCYJ20230807091809020), and Shenzhen Science and Technology Plan (Grant No. JCYJ20210324123802006).

References

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower, editor, Psychology of Learning and Motivation , volume 24, pages 109-165. Academic Press, 1989.

Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks, 2013.

Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Fei Mi, Lingjing Kong, Tao Lin, Kaicheng Yu, and Boi Faltings. Generalized class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2020.

Huiping Zhuang, Zhenyu Weng, Hongxin Wei, RENCHUNZI XIE, Kar-Ann Toh, and Zhiping Lin. ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 11602-11614. Curran Associates, Inc., 2022.

Huiping Zhuang, Zhenyu Weng, Run He, Zhiping Lin, and Ziqian Zeng. GKEAL: Gaussian kernel embedded analytic learning for few-shot class incremental task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7746-7755, June 2023.

Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. PODNet: Pooled outputs distillation for small-tasks incremental learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 , pages 86-102, Cham, 2020. Springer International Publishing.

Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for classincremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2544-2553, June 2021.

Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. FOSTER: Feature boosting and compression for class-incremental learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 , pages 398-414, Cham, 2022. Springer Nature Switzerland.

Yaoyao Liu, Yingying Li, Bernt Schiele, and Qianru Sun. Online hyperparameter optimization for class-incremental learning. Proceedings of the AAAI Conference on Artificial Intelligence , 37(7):8906-8913, Jun. 2023.

Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetting learning in deep neural networks, 2016.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(12):2935-2947, 2018.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114(13):3521-3526, 2017.

Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV) , September 2018.

Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M. López, and Andrew D. Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recognition (ICPR) , pages 2262-2268, 2018.

Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5871-5880, June 2021.

Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self-sustaining representation expansion for non-exemplar class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9296-9305, June 2022.

Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. FeTrIL: Feature translation for exemplar-free class-incremental learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 3911-3920, January 2023.

Ping Guo and Michael R. Lyu. A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data. Neurocomputing , 56:101-121, 2004.

Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Blockwise recursive moore-penrose inverse for network learning. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 52(5):32373250, 2022.

Huiping Zhuang, Zhiping Lin, Yimin Yang, and Kar-Ann Toh. Analytic learning of convolutional neural network for pattern recognition, 2022.

Mark D. McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. RanPAC: Random projections and pre-trained models for continual learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 12022-12053. Curran Associates, Inc., 2023.

Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning. Proceedings of the AAAI Conference on Artificial Intelligence , 38(15):17237-17244, Mar. 2024.

Run He, Huiping Zhuang, Di Fang, Yizhu Chen, Kai Tong, and Cen Chen. REAL: Representation enhanced analytic learning for exemplar-free class-incremental learning, 2024.

Huiping Zhuang, Run He, Kai Tong, Di Fang, Han Sun, Haoran Li, Tianyi Chen, and Ziqian Zeng. Analytic federated learning, 2024.

Zichen Liu, Chao Du, Wee Sun Lee, and Min Lin. Locality sensitive sparse encoding for learning world models online. In The Twelfth International Conference on Learning Representations , 2024.

Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 11731-11741, October 2023.

Hyunseo Koh, Dahyun Kim, Jung-Woo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. In International Conference on Learning Representations , 2022.

Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8218-8227, June 2021.

Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. DualPrompt: Complementary prompting for rehearsal-free continual learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 , pages 631-648, Cham, 2022. Springer Nature Switzerland.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 139-149, June 2022.

Tyler L. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2020.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Toronto, ON, Canada, 2009.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8340-8349, October 2021.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 10347-10357. PMLR, 18-24 Jul 2021.

Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, and Bing Liu. Learnability and algorithm for continual learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 16877-16896. PMLR, 23-29 Jul 2023.

Gyuhak Kim, Bing Liu, and Zixuan Ke. A multi-head model for continual learning via out-ofdistribution replay. In Sarath Chandar, Razvan Pascanu, and Doina Precup, editors, Proceedings of The 1st Conference on Lifelong Learning Agents , volume 199 of Proceedings of Machine Learning Research , pages 548-563. PMLR, 22-24 Aug 2022.

Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Correlation projection for analytic learning of a classification network. Neural Processing Letters , 53(6):3893-3914, dec 2021.

A Proof of Theorem 3.1

Proof. in task k -1 , we have

Hence, in task k , we have

We have defined the autocorrelation memory matrix R k -1 in the paper via

To facilitate subsequent calculations, here we also define a cross-correlation matrix Q k -1 , i.e.,

Thus we can rewrite (13) as

Therefore, in task k we have

From (15), we can recursively calculate R k from R k -1 , i.e.,

According to the Woodbury matrix identity, we have

Hence, R k can be recursively updated using its last-task counterpart R k -1 and data from the current task (i.e., X (B) k ). This proves the recursive calculation of the autocorrelation memory matrix.

Next, we derive the recursive formulation of ˆ W ( k ) FCN . To this end, we also recurse the cross-correlation matrix Q k in task k , i.e.,

where

Note that the concatenation in (22) is due to the assumption that Y train 1: k in task k contains more data classes (hence more columns) than Y train 1: k -1 . It is possible that there are no new classes appear in task k , then ˜ Y train k should be 0 .

Similar to what (22) does,

We have

Hence, ˆ W ( k ) FCN can be rewritten as

By substituting (20) into R k Q ′ k -1 , we have

To simplify this equation, let K k = ( I + X (B) k R k -1 X (B) ⊤ k ) -1 . Since

we have K k = I -K k X (B) k R k -1 X (B) ⊤ k . Therefore,

Substituting (27) into (26), R k Q ′ k -1 can be written as

Substituting (28) into (25) implies that

which completes the proof.

B GCIL Properties

The GCIL scenario [6] is a recent CIL focus. Given task-wise learning tasks, we can involve all class labels in a set S with the number of classes N . The sample sizes, such as the numbers of input images of different classes appearing in task k , are modeled as a random vector c k ∈ R N . Each entry c k,i is a random variable denoting the sample size of class i in task k . In the generalized form, c k is sampled from a task-dependent distribution. The GCIL scenario can be summarized as the following three key properties.

Property B.1. The number of classes in a task is not fixed. Suppose m k is the number of classes in task k , we have:

where M k is a task-dependent distribution.

Property B.2. Classes appearing in different tasks could overlap. For two tasks k and k ′ , k = k ′ , we have:

̸

̸

where ⊙ denotes element-wise multiplication of two vectors and P ( · ) is the probability.

Property B.3. Sample sizes of different classes at the same task could be different. That is, for task k , we have

̸

̸

̸

̸

In short, the number of classes and samples could vary throughout the continual learning.

C Si-Blurry Setting

Blurry Sample Division

Disjoint Classes

Sample Size

Train Dataset

𝑻𝑲

𝑫

…

𝑻𝑲

𝑩

𝑻𝟏

𝑩

𝑻𝟐

𝑩

𝑻𝟑

𝑩

Random Class Division

…

…

𝑻𝟏

𝑩+𝑫

𝑻𝟐

𝑩+𝑫

…

𝑻𝟑

𝑩+𝑫

…

𝑻𝑲

𝑩+𝑫

…

…

Si-Blurry Tasks

…

𝑻𝟏

𝑫

𝑻𝟐

𝑫

𝑻𝟑

𝑫

…

𝑻𝟏

𝑩

𝑻𝟐

B

…

𝑻𝟑

𝑩

…

𝑻𝑲

𝑩

…

…

𝒓𝑩

𝒓𝑫

…

…

The Si-Blurry setting [30] satisfies all the three properties of GCIL mentioned in Appendix B and can be treated as its good realization. As shown in Figure 3, for a K -task learning, the Si-Blurry first randomly partitions all classes into two groups: disjoint classes that cannot overlap between tasks and blurry classes that might reappear. The ratio of partition is controlled by the disjoint class ratio r D, which is defined as the ratio of the number of disjoint classes to the number of all classes. Then disjoint classes and blurry classes are randomly assigned to disjoint tasks ( T D ) and blurry tasks ( T B ) respectively. Next, each blurry task further conducts the blurry sample division by randomly extracting part of samples to assign to other blurry tasks based on blurry sample ratio r B, which is defined as the ratio of the extracted sample within samples in all blurry tasks. Finally, each Si-Blurry task T B+D with a stochastic blurry task boundary consists of a disjoint and blurry task. We adopt Si-Blurry with different combinations of r D and r B for reliable empirical validations.

D Compute Resources

GPU Usage . We conduct experiments in PyTorch on one Nvidia Geforce RTX 4090 GPU with a batch size of 64 for training and 128 for inference. Figure 4 shows that the GACL uses minimal GPU memory. Our GACL significantly reduces GPU memory usage since it requires no back-propagation, thereby detaching gradients from tensors during calculations. This characteristic allows our approach to be applied with a larger batch size without memory leaks.

GACL

(ours)

SLDA

LwF

ER

RM

MVP

DualPrompt

L2P

MVP-R

EWC++

0

2

4

6

8

10

1.58

1.63

1.69

1.96

1.96

5.21

5.29

5.39

7.98

9.87

Training Time . Table 5 further illustrates the GACL's training time compared to others on one Nvidia Geforce RTX 4090 GPU, highlighting its efficiency. The GACL is faster than any other baselines except SLDA on three datasets because only the classifier and autocorrelation memory matrix R are updated, leading to small numbers of trainable parameters compared to those baselines in a back-propagation manner.

E Hyperparameter Analysis for Regularization Term

The regularization term γ plays a crucial role and demonstrates robust behavior throughout our experiments. We assess the impact of the regularization term γ in Table 6 and visualize the real-time accuracy of the GACL as it learns from training samples in Figure 5. Table 6 reveals the GACL's consistent performance across a broad range of γ values, spanning from 10 to 10000. This highlights the versatility and robustness of our proposed GACL. However, as indicated in Figure 5, γ of 10000 leads to slightly poorer performance because the ACL is prone to underfitting due to simple linear regression [44].

1k

10k

20k

30k

40k

20

40

60

Real-time Accuracy(%)

CIFAR-100

1k

5k

10k

15k

20k

0

10

20

30

40

50

Real-time Accuracy(%)

ImageNet-R

1k

20k

40k

60k

80k

0

20

40

60

Real-time Accuracy(%)

Tiny-ImageNet

γ

=0

(w/o regularization)

γ

=10

γ

=100

γ

=500

γ

=1000

γ

=10000

Notably, both Table 6 and Figure 5 demonstrate that the absence of regularization results in a significant decline in performance. This underscores the crucial importance of incorporating γ in the model. As indicated in (7), if we eliminate regularization by setting γ to 0, the initial autocorrelation memory matrix R 0 becomes zero. Subsequently, the computation of the autocorrelation memory matrix in task 1, denoted as R 1 , is expressed as:

However, it's crucial to emphasize that X (B) ⊤ 1 X (B) 1 might result in a singular matrix, rendering it non-invertible. This potential singularity introduces an error in calculating R 1 , leading to a decrease in accuracy.

F Analysis of task-wise Accuracy Trends of the GACL

As depicted in Figure 2 (a), the task-wise accuracy of the GACL on CIFAR-100 demonstrates an increase. Notably, in the initial two tasks, the accuracy is lower compared to other EFCIL methods. However, on the other datasets, the GACL remains relatively stable. Upon a more detailed examination of the dataset split, we infer that the observed variations in trends are attributed to the specific dataset settings.

For a dataset with N classes, the class number ratio r c after training on i -th samples is defined as r c = d i /N , where d i is the number of classes that have been seen observed at that point. As Figure 6 indicates, by examining the real-time accuracy and the class number ratio r c across the three sets of figures, a notable observation is made: when the sample size is small, the class number ratio r c on CIFAR-100 always surpasses that of the other two datasets on 5 seeds. This suggests that tasks on CIFAR-100 are notably more complex and intricate, resembling a few-shot learning scenario.

Consequently, the GACL exhibits lower task-wise accuracy compared to other gradient-based EFCIL methods, particularly in the initial stages. However, as more training samples are acquired, its accuracy progressively improves.

1k

10k

20k

30k

40k

50k

30

40

50

60

70

Accuracy (%)

CIFAR-100 Seed 1

1k

10k

20k

30k

40k

50k

40

50

60

70

CIFAR-100 Seed 2

1k

10k

20k

30k

40k

50k

50

60

70

CIFAR-100 Seed 3

1k

10k

20k

30k

40k

50k

40

50

60

70

CIFAR-100 Seed 4

1k

10k

20k

30k

40k

50k

30

40

50

60

70

CIFAR-100 Seed 5

20

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

rc

(%)

1k

5k

10k

15k

20k

35

40

45

50

Accuracy (%)

ImageNet-R Seed 1

1k

5k

10k

15k

20k

25

30

35

40

45

50

ImageNet-R Seed 2

1k

5k

10k

15k

20k

30

35

40

45

ImageNet-R Seed 3

1k

5k

10k

15k

20k

25

30

35

40

45

50

ImageNet-R Seed 4

1k

5k

10k

15k

20k

40

50

60

ImageNet-R Seed 5

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

rc

(%)

1k

20k

40k

60k

80k 100k

50

60

70

Accuracy (%)

Tiny-ImageNet Seed 1

1k

20k

40k

60k

80k 100k

40

50

60

70

Tiny-ImageNet Seed 2

1k

20k

40k

60k

80k 100k

50

60

70

Tiny-ImageNet Seed 3

1k

20k

40k

60k

80k 100k

40

50

60

70

80

Tiny-ImageNet Seed 4

1k

20k

40k

60k

80k 100k

50

60

70

Tiny-ImageNet Seed 5

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

rc

(%)

Accuracy (%)

rc

(%)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All the claims are clearly clarified, including the contributions made in the paper and important assumptions and limitations.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 4.5, a discussion of the limitations and the future work of the GACL is conducted.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proof of the Theorem 3.1 is listed in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper comprehensively outlines both the experimental implementation and algorithmic details, ensuring transparency in our method.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All datasets are publicly accessible, and we have provided the source code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have reported all the necessary details of our experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Results in this paper are reported by the average of 5 different seeds with standard error.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information on the computer resources for our GACL is listed in Appendix D. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper fully complies with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The models and benchmark datasets mentioned in the paper are all openly accessible with no personally identifiable information or offensive content.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have verified that this paper cites all the datasets and models we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have made our source code publicly available at https://github.com/ CHEN-YIZHU/GACL .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new exemplar-free GCIL technique named generalized analytic continual learning (GACL). It adopts analytic learning (a gradient-free training technique), and delivers an analytical (i.e., closed-form) solution to the GCIL scenario, which is derived via decomposing the incoming data into exposed and unexposed classes. The GACL attains a weight-invariant property, supporting an equivalence between the incremental learning and its joint training. This method is both therectically and empircally validated.\n\nSTRENGTHS:\n1. This is very interesting technique that obtains an equivalence between the GCIL and the joint training. Although a pre-trained network is needed, the weight-invariant property is very valuable in this area.\n2. Another key contribution of GACL is that it is both accurate and exemplar-free, and this is powerful and significant. \n3. The approach is also well-motivated and clearly explained.\n4. The paper provides a comprehensive literature review, effectively contextualizing its contribution and facilitating the reader's understanding.\n5. It is very clear theoretically regarding how the algorithm is parted into W_unexposed and W_ECLG.\n\nWEAKNESSES:\n1. Could you explain why the avg accuracy on CIFAR-100 is much lower than the last accuracy?\n2. You need to explain more regarding \"weight-invariance\" in Theorem 3.1. It is not very easy to follow for those who are not in this area of research.\n3. In this paper, the authors assume that the pre-trained backbone is generalizable so that the model can be frozen throughout the learning process. However, there are also some cases where the pre-trained model cannot generalize to downstream tasks. It would also be interesting to see the scenario where the pre-trained backbone yields a large domain gap.\n\nQUESTIONS:\n1. Continual learning has many variations (e.g., domain/task/class -incremental learning). Since the experiments only focus on generalized class-incremental learning, it would be better to adjust the title accordingly.\n2. Since generalized CIL is also a specific case of class-incremental learning, it would be better to discuss why the proposed method is specially designed for GCIL. For example, when the data stream only contains new classes, will GCIL still work?\n3. As the authors discussed in the main paper, the proposed method is a super case of ACIL and other analytical methods. As many readers would like to know, the authors are suggested to compare this method to other analytical methods to show how this method stands against other analytical works.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nClass Incremental Learning (CIL) faces the problem of catastrophic forgetting when training a network, i.e., the model loses previous knowledge when learning a new task. Generalized CIL (GCIL) aims to address more realistic scenarios, but existing methods are either ineffective or violate data privacy. This paper propose a new exemplar-free GCIL technique called Generalized Analytic Continual Learning (GACL), which provides a closed-form solution through analytic learning. GACL achieves equivalence between incremental learning and co-training by disaggregating the input data to keep the weights constant. This approach is theoretically validated and performs well across multiple datasets and settings.\n\nSTRENGTHS:\n* The paper is well-written and easy to follow.\n* The proposed method outperforms other methods on several GCIL benchmarks.\n* This paper provides a detailed proof of the theorem.\n\nWEAKNESSES:\n* The experimental setup is not clear. Given that the most recent method MVP [1] compared in this paper is the GCIL method for online CIL, were the experiments in this paper also conducted in an online scenario?\n* The technical improvements on the previous work of ACL are somewhat incremental.\n\n[1] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. ICCV 2023.\n\nQUESTIONS:\n1. What was the motivation for introducing ACL into GCIL? The claims in this paper seem to be an extended experiment of ACL in the GCIL scenario. It is difficult to generate further inspiration to the reader for solving the GCIL problem.\n2. The proposed method utilize the DeiT-S/16 as backbone but previous works typically use ViT-B/16. Regarding the comparison methodology, what backbone network was used to obtain the reported results?\n3. What are the technical improvements and contributions of this paper over previous ACL approaches such as ACIL [2] and DS-AL [3]?\n\n[2] Huiping Zhuang, Zhenyu Weng, Hongxin Wei, RENCHUNZI XIE, Kar-Ann Toh, and Zhiping Lin. ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. NeurIPS 2022\n[3] Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning. AAAI 2024.\n\nLIMITATIONS:\nIt is hard to foresee any potential negative societal impact of this theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new exempler-free generalized continual learning (GCIL), named generalized analytic continual learning (GACL) technique. It does not depend on gradient-based tranining, which avoids the task-recency bias leading to the forgetting issue. It also delivers an closed-form solution to the GCIL scenario that provides identical solutions to its joint training. Entensive experiments demonstrate that the proposed GACL achieves consistently leading performance.\n\nSTRENGTHS:\n- The paper proposes very effective method that completely avoids forgetting in the gradient-free training technique.\n- Extensive experiments are included explaining that GACL works good overall in the three datasets compared with many studies.\n- Ablation studies are coducted that clarifies the differences among datasets and the reason why GACL is slightly worse than other baselines at early tasks. They also analyzed the contributions of ECLG module and its robustness.\n- The paper is well written and easy to read.\n\nWEAKNESSES:\n- Missing baselines. As described in the Section 2, many ACL methods have been proposed. Comparing GACL with those methods (especially RanPAC) derives more solid results.\n- Comparison to the joint-traning is not included. Although it is argued that GACL can bridge the gap between continual learning and joint-training, such result is not provided in the experiments.\n\nQUESTIONS:\n- I believe that the proposed gradient-free training scheme is beneficial also in the perspective of training speed. Does GACL run more fast than other (ACL) methods?\n\nLIMITATIONS:\n- As described, GACL cannot update backbone weights, which can be a problem. However, as new backbones emerge frequently, showing that GACL can work well not only with the used DeiT-S/16 but also others makes the arguments more convincing, instead.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper deals with the generalized CIL (GCIL) problem where incoming data have mixed data categories and unknown sample size distribution. The author proposes generalized analytic continual learning (GACL) which adopts a pre-trained and fixed backbone and uses least squares to get a closed-form solution. Experiments verified that the GACL achieves better performance compared to other baselines.

STRENGTHS:
* The generalized class incremental learning is an important and valuable problem.
* This paper is generally clear and easy to follow.
* Experiments show that GACL is effective in addressing forgetting in GCIL.

WEAKNESSES:
* Main concern. Most of the content on page 4 and page 5 (e.g., Theorem 3.1) in Section 3 is overly similar to existing ACL works [7, 8]. The main difference claimed by the authors, i.e., the ECLG module that corresponds to the exposed-class gain, is a trivial extension of the original ACL [7].
* Experiments. This paper adopts the pre-trained backbone developed in [40, 41]. However, many baselines (Table 2 in [40]) are missing in experiments. The GACL method is similar to RanPAC [25]. Experiments are needed to compare the performance of GACL with RanPAC.
* Comparison of the memory cost of weights in the buffer layer and other replay-based method.

QUESTIONS:
* How and why the dimensionality of the random projection affects the performance.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12