Confusion-Driven Self-Supervised Progressively Weighted Ensemble Learning for Non-Exemplar Class Incremental Learning
Authors: Kai Hu
Zhang Yu
Yuan Zhang
Zhineng Chen
Xieping Gao
Venue: NeurIPS 2025 poster ()
Keywords: Non-Exemplar Class Incremental Learning
Representation Learning
Self-Supervised Learning
Ensemble Model
Primary area: general_machine_learning

Abstract: Non-exemplar class incremental learning (NECIL) aims to continuously assimilate new knowledge while retaining previously acquired knowledge in scenarios where prior examples are unavailable. A prevalent strategy within NECIL mitigates knowledge forgetting by freezing the feature extractor after training on the initial task. However, this freezing mechanism does not provide explicit training to differentiate between new and old classes, resulting in overlapping feature representations. To address this challenge, we propose a **C**onfusion-driven se**L**f-supervised pr**O**gressi**V**ely weighted **E**nsemble lea**R**ning (*CLOVER*) framework for NECIL. Firstly, we introduce a confusion-driven self-supervised learning approach that enhances representation extraction by guiding the model to distinguish between highly confusable classes, thereby reducing class representation overlap. Secondly, we develop a progressively weighted ensemble learning method that gradually adjusts weights to integrate diverse knowledge more effectively, further minimizing representation overlap. Finally, extensive experiments demonstrate that our proposed method achieves state-of-the-art results on the CIFAR100, TinyImageNet, and ImageNet-Subset NECIL benchmarks.

Confusion-Driven Self-Supervised Progressively Weighted Ensemble Learning for Non-Exemplar Class Incremental Learning
Kai Hu Xiangtan University
Zhang Yu Xiangtan University
Yuan Zhang" Xiangtan University yuanz@xtu, edu, cn
kaihu@xtu. edu. cn
2022216330020smail xtu, edu. cn
Zhineng Chen Fudan University
Xieping Gao
Hunan Normal University xpgaoChunnu. edu. cn
zhinchen@fudan. edu.
Abstract
Non-exemplar class incremental learning (NECIL) aims to continuously assimi- late new knowledge while retaining previously acquired knowledge in scenarios where prior examples are unavailable. A prevalent strategy within NECIL miti- gates knowledge forgetting by freezing the feature extractor after training the initial task. However, this freezing mechanism does not provide explicit training to differentiate between new and old classes, resulting in overlapping feature represen- tations. To address this challenge, we propose a Confusion prOgressiV Vely weighted Ensemble lea leaRning (CLOVER) framework for NECIL. Firstly, we introduce a confusion-drive enhances representation extraction by guiding the model distinguish between highly confusable classes, thereby reducing class representation overlap. Secondly, develop progressively weighted ensemble learning method that gradually adjusts weights to integrate diverse knowledge more effectively, further minimizing representation overlap. Finally, extensive experiments demonstrate that our pro- posed method achieves state-of-the-arr results on the CIFAR 100, TinyImageNet, and ImageNet-Subset NECIL benchmarks.
seLf-supervised
self-supervised learning approach that
Introduction
Over the past few decades, deep neural networks have exhibited remarkable performance in various domains I 2 3 5) Given that the real world is both open and dynamic, it is crucial for models to continuously acquire and integrate new knowledge. However, directly fine-tuning a model on new data often results in catastrophic forgetting [6), where the performance previously encountered data deteriorates significantly. To address this challenge, several class incremental learning (CIL) methods g store a subset of previous exemplars and replay them during the learning of new tasks to preserve prior knowledge. Nevertheless, due to privacy concerns and storage limitations, non-exemplar class incremental learning (NECIL)) 9 which aims continuously adapt classes while preventing forgetting previously learned ones without retaining earlier samples, has gained increasing attention from researchers.
Existing NECIL methods can be broadly classified into two categories: prototype-based methods and frozen feature extractor-based methods, based on their training paradigms. Prototype-basee based
Corresponding author.
"Code is publicly available at: http://gghhh..comMMMMMP/....OO
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Raw class
90" class
180 class
GBR class
Noise class
(a) Representation distributions of raw and
(b) Representation distributions of raw and task-agnostic classes in CDSSL methods
task-agnostic classes in SSL methods
80
Ensemble[| | experts)
Accuracy (%)
75
Ensemble(3 expens)
Ensemble(5 expens)
70
Ensembler5 PXP
65
60
0
2
3
5
6
8
9
10
11
12
13
14
15
16
17
18
19
20
Incremental Phase
(c) Accuracy variation of ensemble models with different numbers of experts
Figure 1: (a) In self-supervised learning, substantial discrepancies exist in the representation distribu- tions between raw classes and task-agnostic classes (sharing the same color but differing shape). Training a model to differentiate between these classes does not effectively enhance its capability for representation extraction. (b) CDSSL introduces highly confusable task-agnostic classes, with GBR and noise classes generated through color channel swapping and noise injection, respectively. This approach aims to encourage the model to extract more discriminative representations. (c) The accuracy curves for the ensemble model with varying numbers of experts indicate that adding new expert (during phases and 3) consistently leads to performance degradation, which is effectively mitigated by the proposed PWP strategy.
methods 9 10 12 take advantage of the mean class representations as prototypes to maintain decision boundaries during model fine-tuning for incremental tasks. These prototypes serve substi- tutes for samples from previous classes, conveying distributional information that effectively mitigates catastrophic forgetting. However, the model undergoes continuous updates, these approaches inevitably experience prototype degradation 12], wherein the preserved prototypes of old classes diverge from their true distributions. In contrast, frozen feature extractor- based methods [ avoid degradation by freezing the feature extractor after training on the initial task, while maintaining the stability of representations and mitigating forgetting. Despite these ad advantages, notable draw- back of this method is its propensity to induce representation overlap between new and old classes. As the model is optimized solely for the initial class distribution, lacks the necessary adaptability to incorporate new class data effectively. Consequently, representations of new classes often exhibit dispersed distributions or overlap with those of existing classes within a fixed representation space, particularly when similarities exist along certain dimensions. This phenomenon hinders the establish- ment of clear decision boundaries and significantly impairs classification performance incremental learning tasks.
14 [5]
There are currently two primary approaches to addressing representation overlap: self-supervised learning (SSL) and ensemble learning. SSL 19. 16 [7 is utilized in the initial task to train the feature extractor. SSL applies transformations to raw classes and assigns novel task-agnostic labels, thereby constructing task-agnostic classes. Subsequently, by encouraging the model to discriminate among these classes, SSL enhances the model's 's generalization capability and effectively mitigates
representation overlap and confusion [18]. However, most studies [9] I8 19 20 21] integrate SSL with additional techniques aimed at improving representation extraction, thereby leaving much of SSL's inherent potential largely untapped. As shown in Fig. a the rotation transformation employed in SSL significantly alters the representational distribution of raw classes, enabling the model to distinguish these classes more easily even without SSL. As result, it fails to effectively enhance

---

the model's ability to extract discriminative representations. Conversely, ensemble learning [22] capitalizes on multiple expert models to extract discriminative representations for each class from diverse perspectives. This method effectively differentiates between new and old class representations while minimizing their overlap. Nonetheless, introducing a new expert can lead to performance degradation (see Fig. (c).. This issue stems from substantial heterogeneity in the predictive capabilities exhibited by the experts. The first expert, trained on the base task, exhibits stronger predictive capability and covers broader range of classes than experts trained on incremental tasks. This makes the first expert more reliable in the early training stages, while later experts remain unreliable for a period after their introduction. Thus, newly added expert, often with relatively limited predictive capability, can interfere with predictions made by more experienced experts.
23
To address the aforementioned challenges, we introduce novel NECIL framework, named Confusion- DoogressiVely weighted Ensemble leaRning (CLOVER). this framework, we propose a Confusion-Driven Self-Supervised Learning (CDSSL) method that enhances the model's capability for representation extraction by guiding to differentiate between highly confusing classes generated through color channel swapping and noise inception transformations (see Fig. (b)). This approach effectively alleviates the issue of representation overlap between new and old classes. Furthermore, we introduce Progressively Weighted Prediction (PWP) strategy aimed at mitigating the influence of unreliable experts within the ensemble model. Specifically, we assign an initially low weight to new experts and incrementally increase their weight task by task until it reaches predetermined maximum value. The final prediction is then computed as a weighted sum of the predictions from all experts. In conclusion, our main contributions are summarized follows:
driven sel.f-supervised
We propose a Confusion-driven -superieee ProgressiVelyy weighted Ensemble leaRning framework named CLOVER for NECIL. This framework improves the model's capability for representation extraction by learning task-agnostic classes and diverse ensemble predictions. These advancements strengthen the discriminative power for new classes, thereby mitigating the representation overlap with old classes.
We propose confusion-driven self-supervised learning method that enhances the ability to extract discriminative representations by guiding it to differentiate between highly confusing agnk-tgi classes generated through color channel swapping and noise injection. Fur- thermore, we introduce a progressively weighted prediction strategy that gradually increases the weight of new experts to mitigate performance degradation caused by unreliable new experts within the ensemble model.
Extensive experimental results on CIFARI00, TinylmageNet and Image have demonstrated the superior performance of our proposed CLOVER over other NECIL ap- proaches.
2 Related Work
2.1 Non-Exemplar Class Incremental Learning
In general, NECIL methods can be broadly categorized into prototype-base based methods 9 12
and
frozen feature extractor- based methods [14 15 25).
Prototype-based methods investigate strategies for leveraging prototypes to deliver more accurate insights regarding prior tasks, thereby effectively mitigating the phenomenon of knowledge forgetting. employed Gaussian noise augment prototypes, providing distributional information of old classes within the representation space to maintain decision boundaries of previous tasks. SSRE [10] identified the primary challenge in NECIL as the overlap between the representations of new and old classes and addressed this issue through a prototype selection mechanism, which used prototypes to preserve prior knowledge while simultaneously facilitating the learning of new tasks. NAPA-VQ [II learned the topological relationships of the feature space to identify confusable neighboring classes and used this information to generate representative prototypes for old classes, thereby facilitating the delineation of boundaries between new and old classes and reducing overlap. However, prototype-based methods inevitably suffer from representation shifts caused by the continual
For example, PASS [
updates to the feature extractor. To address this issue, a variety of methods [22 26 27 have been proposed to alleviate representation shift by reconstructing the prototypes of previously learned classes, thereby enhancing the accuracy and effectiveness of the information they convey about the old
3

---

class distributions. More recently. DCMI (28] proposed novel generation method for synthesizing images with both semantic and domain consistency, which effectively facilitated knowledge retention.
On the other hand, frozen feature axtractor-based methods keep the feature extractor fixed after training the initial task and continually update the classifier to incorporate new tasks. Prototypical networks [13) utilized frozen feature extractor to generate class prototypes, enabling classification by calculating the distances to the prototype representations of each class. FeTrIL [ generated pseudo-feature for old classes by applying a geometric shift to new class features based on the difference between new and old class prototypes. which enabled the joint training of the classification head with both new and old class features. FeCAM [14] emphasized the greater heterogeneity in the distribution of new classes compared to old ones and classified the features based on the anisotropic Mahalanobis distance to the prototypes after the first task. To bridge the significant distribution gap between the real and the recorded features, DiffER [8 constructed diffusion models to generate features that closely resembled the real features and reconstructed them to enhance the robustness of the classifier.
2.2 Sele-Supervieed Learning
In recent years, self-supervised learning (SSL) has proven to be an effective approach for learning general representations from unlabeled data. This is accomplished through predefined proxy tasks, such as rotation prediction [6], patch permutation [29]. image colorization [30.. inpainting [31. and
clustering [32] These tasks help the model acquire task-agnostic features, enabling it to generalize across various downstream tasks like few-shot learning [33] and semi-superrvsed learning [34), to
improve the model robustness [35), class imbalance [36], etc. Recently, SLA [17] demonstrated that
SSL can enhance supervised classification performance by augmenting the original labels through input transformations.
In NECIL, PASS 19] employed rotation self-supervision to learn generalizable and transferable features for future tasks. SASS [19] introduced auxiliary classifier after each ResNet block to perform self-supervised rotation classification, which facilitated the learning of more generalized representations and achieved better pability-plasticity trade-off. HRESN 20 used random positive samples to perform rotation prediction tasks, enabling the feature extractor to learn richer feature representations through complex rotation prediction tasks. DiffRR [18] integrated SSL with instance similarity constraints to train the feature extractor, reducing feature overlaps by applying class- and instance-level discrimination constraints. Our CDSSL is implemented in manner similar the SSL in PASS (9). Building upon this, we guide the model to learn task-agnostic classes that are highly ambiguous, thereby facilitating the extraction of more discriminative representations.
2.3 Growing Architectures and Ensemble Leaning
Architecture-basel methods [37 38 design task-specific parameters to adapt to new tasks while maintaining performance on previous ones. Although these approaches substantially enhance per- formance, the linear increase parameters as the number of tasks increases becomes unsustainable. Moreover, the effective integration of knowledge across tasks remains significant challenge. To address these issues, CoSCL (23) proposed an ensemble composed of a limited number of experts and penalized the discrepancies in their feature representation predictions to promote cooperation.
SEED (39] developed a selective training ensemble of experts to mitigate forgetting, while leveraging the diverse expertise of the specialists for joint prediction. However, the newly created experts in this method lead to a significant decline in performance, causing the combined predictive capability of the expert ensemble to be inferior to that of single model. Therefore, this paper explores solutions to address this issue.
3 Method
In this section, we first define the Non-Exemplar Class Incremental Learning (NECIL) scenario. Then, we outline the training and inference process of the proposed CLOVER. Finally, we provide detailed introduction of the proposed confusion-driven self-supervised learning (CDSSL.) method and the progressively weighted prediction (PWP) strategy.

---

Training
Tink
Ensemble
Chastlee lngits
Inference
3000OOC
Tiger
Liee
Tiger,
Angaase
Liger.
BBK
Note
&0
Liget. GaIR
Tiget. 270*
Enserrill
Tigur BRG
Noise
PW P
TeAe
(oopund
Tonn
Ingits
Léd
Rabit
Last
wanghhot maam
Aeamer
0000
BRG
Note
ex-i
gr.
Ribbi. GBR
RRbbl. 270""
Ribii BRG
orgines
Rebii, Nose
Tisk
Figure 2: Overview of our proposed CLOVER. For tasks < K 1, augmented to generate task-agnostic classes using transformations such rotation, color channel swapping, and noise injection. A new expert (f o et) is subsequently introduced and trained on this combined set of classes enhance representation extraction capabilities and mitigate representation overlap between new and old classes. During the inference phase, the input image is processed through the ensemble model to generate expert-specific logits. These logits are then combined using Eq. (2) to calculate the weighted mean logits, with the weights for each expert determined by the proposed PWP strategy. Finally, the class prediction obtained by performing the argmax operation on these logits.
1, -
the classes of the current task
3.1 Preliminaries
NECIL emphasizes the capability to sequentially acquire knowledge across a series of unrelated tasks without retaining revisiting previously encountered examples. The training data for task N: ) where N,, x and y E C2 represent t e {0, (,......... T} is defined as D, = {X,, Y?} {a/. s 1=1 the number of training images, training image and the corresponding label for task I, respectively. Here, C is the set of classes associated with task i, and the class sets for each task are disjoint, ie., ConCn M.C.nn..n n...nn, Cr Ø. The model is evaluated the combined test sets Zo-a Zo Zi U...U Zz, with the objective of accurately predicting both the classes of a new task and those previously learned tasks, where Co-t Co U U ...UUGG U...
3.2 Overall Framework
As illustrated in Fig. 2 CLOVER is an ensemble model comprising K experts, including a shared encoder f, K expert encoders e; and their corresponding linear classifier g; (i 0, 1,.... K 1). Notably, f frozen after training the first task. For task / < K - CDSSL is employed to train freeze e; and discard g Next, each existing
the expert i corresponding to task. After training, expert k < computes the representation of class E C2 and models it using a distinct Gaussian distribution G(uf. ) which is preserved and employed for classification in manner similar to that of a prototype.
The inference process is similar that of SEED 39 The input is passed through each expert to obtain the expert-specific representation ra ex (x))). We then compute the log-likelihool of r with respect to the distribution of each class within that expert:
(x) = ------ [n( Σ )) S1m(22) + (rx ME) 2 (rx - Pf)
(1)
where is the representation dimension. It is worth noting that, since the new expert h (where h 0) does not have access to data from previous classes and cannot compute their representation distributions, we set 1 (x) = -o for i Co-h-1 to indicate that predictions for these previous
classes are impossible. Subsequently, we apply the softmax function to these values 12 12. softmaa (1), ....... C 7) to obtain the predicted probabilities for each class within each expert, where C' is the set of all classes and represents the temperature. Finally, we calculate the weighted
5

---

sum of the predicted probabilities from all experts by:
cc
I(x) Wk x
==1
where iij
"kk and M, (d = 1,2,..../CI ICI) represents the number of experts capable of M.
my
predicting class (since each expert only retains the distributions of subsequent classes and performs predictions after completing its training). wa denotes the prediction weight calculated using the PWP method. Finally, class c argmar(((2)) with the highest probability is selected as the final prediction.
3.3 Confusion-Driven Self-Supervised Learning
Considering single expert, the feature extractor remains fixed following the initial training phase, and the distributions the newly introduced classes are directly derived from the mapping of the frozen feature extractor. However, the absence of discriminative training between new and old classes increases the likelihood of confusion, causing the model to forget previous knowledge. To reduce the overlap of representations, we investigate the training methodology of feature extractor and propose the CDSSL strategy. This strategy focuses on training the model to differentiate between highly confusable task-agnostic classes, thereby enhancing its representation learning capabilities and reducing the overlap between representations of new and old classes.
we apply several transformations to the given image Tq E DR. including rotation, color channel swapping, and noise inception:
Specifically, as illustrated in Fig. 2
Tis ralate(r,. x 90%), GBR(I,),
j
<
T7its
j =
(3)
(RG(Z.),
j=
2 + X noise 1),
j j=
and assign new label Vro+j = Tys + j. where ratate(zi.) refers to rotate v; by 90p, 180%, 270". G-) and RGG indicate swapping RGB channels GBR and BRG, respectively. moise(0, 1) represents value randomly sampled from a standard normal distribution. denotes the noise intensity, which is typically set to 0.5. generalizes the original N-way classification problem to 7N-way classification problem. Next, we use cross-entropy loss to train the model:
L= Le
Comparing the widely used 4N-way self-supervised tasks, demonstated in 18) generated new classes with higher levels of confusion by employing color channel swapping and noise inception. Thus, our proposed CDSSL encourages the model to learn more distinctive features, enabling it to more effectively differentiate between representations new and old classes and thereby reducing their overlap. It is noteworthy that these task-agnostic classes excluded during the inference phase and that their representation distributions across experts are neither computed nor stored.
3.4 Progressively Weighted Prediction
In NECIL, the base task typically constitutes half of the training data, with the remainder evenly distributed across incremental tasks. Consequently, due to the varying creation times of experts, the distributions of classes they cover differ significantly. Typically, newer experts encompass a narrower range of predictable classes. When such an expert encounters an old class outside its prediction scope, often misclassifies with high confidence. As shown in Fig. [ec, such errors can mislead the equally weighted joint prediction system, ultimately resulting in inferior performance of the ensemble model compared to that of the individual models.
Therefore, considering that the number of predictable classes and the predictive ability vary across experts, we propose the progressively weighted prediction method. Specifically, we assign a low weight to the new expert and progressively increase it by 3, with the maximum weight being k

---

Thus, the weight of each expert is defined as:
K- Σ wij
i i> 0
1=1
min (a + 8 (1 (1-1) ()
where denotes the current task. As shown in Eq. ( a weighted average of the predictions from each expert is then computed to derive the final outcome during joint prediction. In general, PWP accounts for the varying predictive capabilities of each expert by assigning distinct weights, thereby enhancing the joint prediction performance of the ensemble model in the early phase.
Table 1: Comparisons of the average accuracy and last accuracy (%) at different settings on CIFARI00. Tiny ImageNet, and ImageNet-Subbe Subset. CEAT 40] and FGKSR 4T are based on the VIT 42] structure, while all other methods are based on ResNet18 using the author's ss official codebase, denoted by the asterisk (*). The best result is highlighted in bold,
The result of SEED 39 obtained
whereas the second best method is indicated by underlining.
CIFARTON FOU
TinllnageNaa
ImageNettSubaag
Method
lasks
DO tasks
20 tasks
tasks
10 tasks
20
tasks
S tasks
10 tasks
20 tasks
Avg
Last
Avg
Last
Avg
Avg.
Last
Avg.
Last
Avg
Last
Avg
Avg
Avg
Last
MC
27.A
20.1
12.3
17.4
8.8
17,
PASS 9
63.5
55.7
61.8
19.0
58.1
48.5
19.6
41.6
47.3
39.9
12.1
32.8
63.1
52.6
61.8
50.4
55.2
46.1
SSRE 10
63.9
56.3
65.0
55.0
61.7
50.5
50.4
41.7
48.9
39.9
48.2
39.8
69.5
58.5
67.7
57.5
61.2
50.1
PRAKA
70.0
61.6
68.9
60.4
46.4
52.6
45.2
49.8
40.6
POLO 26
69.0
68.0
65.7
54.9
47.0
53.4
45.3
49.9
40.4
70.8
59.5
69.1
57.9
TASS 15
68.8
59.3
67.4
57.9
62.8
53.8
55.1
41.1
54.2
43.9
52.8
43.6
74.3
63.1
72.6
57.9
68.8
57.6
70.1
CEAT
71.1
70,0
66.1
58.3
50.4
57.4
49.4
56.8
48.0
76.9
67.4
75.9
66.3
71.5
60.1
SEED*
10
71.1
66.3
69.9
65.0
68.2
6144
51.7
50.6
54.5
50.0
53.9
18.9
75.0
70.3
73.6
68.1
71.1
63.8
KeCAM
70.9
62.1
70.8
62.1
69.4
5W.5
59.6
52.8
59.4
57.8
59.3
52.8
78.
70.9
70.
75.1
66.3
(Ours)
58.5
52,8
CIPAR100 mals)
CIFAR100 (10 usks)
CIFAR100 (20 eacis)
54100
100
TeylmngeNet tosks)
Tiny Net (10 naks)
Tim Imogg Net (20 tasks)
100
120
140
150
150
1200
140
160
100
120
140
150
200
Figure 3: Illustration of the classification accuracy changes as tasks being learned on CIFARIOR
and Tiny Imagee,,,,,, which contains the complete curves. Precise data of our method is presented in Appendix B
4
Experiments
4.1 Experimental Setup
Datasets. We conduct extensive experiments on three benchmark NECIL datasets: CIFAR100 [46], TinylmageNet [47], and ImageNet-Subset 48). The CIFAR100 dataset comprises 100 classes, each with 600 color images at resolution 32x32 pixels, including 500 images for training and 100 images for testing. TinyImageNet is subset of the ImageNet [48] designed for image classification, consisting of 200 classes, with each class containing 500 training images, 50 validation images, and 50 test images, all sized 64x64 pixels. The ImageNet-Subel dataset, comprising 100 classes
7

---

Table 2: Ablation study of each component of CLOVER on CIFARI00 and TinylmageNet. The reported metric is average accuracy (%)
Component
CIFAR100
TinylmageNet
Baseline CDSSL
PWP
10
20
5
10
20
69.9
69.7
67.7
56.3
55.3
54.5
72.1
71.7
70.5
59.9
59.4
58.1
70.5
70.2
68.2
56.5
55.8
54.9
72.7
72.3
71.0
60.2
59.9
58.5
selected from ImageNet, contains about 13,000 training images and 50 test images per class, all
standardized to a resolution of 224x224 pixels. We follow the configuration from PASS [9] and divide each dataset into three incremental settings: 5, 10, and 20 tasks.
Evaluation Metrics. We report two common metrics for CIL: the average accuracy and last accuracy. The former evaluates overall performance during incremental learning by calculating the average accuracy across all incremental phases. The latter assesses comprehensive performance upon completion of all tasks, emphasizing its validity in mitigating forgetting and integrating knowledge.
Implementation Details and Reproducibility. We use ResNet18 44 as the backbone (same as SEED (39] ). During the training phase, linear classification head is employed, while Bayesian classifier is used in the inference phase. The number of experts K is set to 5. During training, the batch size is set 128 and the model is optimized by the SGD optimizer with an initial learning rate 0.1 and weight decay le-4. The learning rate is multiplied by 0.1 at epochs 60, 120 and 160. All experiments are repeated three times and the average results are reported. The baseline is based on SEED [39], trained on the first 5 tasks, and shares the parameters of the first 5 layers across all expert models. Based the aforementioned settings, all algorithms can be trained on single NVIDIA A100 GPU.
4.2 Benchmark Comparison
Juble/coopant CLOVER with state-of-the-art NECIL methods on the CIFARIO0, TinylmageNet, and ImageNettSubbett datasets. On CIFAR100, our CLOVER significantly outperforms the state- of-the-art methods across 5, 10, and 20 tasks, achieving higher average accuracy of 1.6%, 1.5%, and 1.6%, well as improved last accuracy of 1.7%, 2.5%. and 3.5%, respectively. For 5 and 10 tasks on TinylmageNa CLOVER exhibits superior average and last accuracy compared to the leading competing method. However, when the number of tasks increases to 20, FeCAM achieves better average accuracy. A similar trend is observed across all configurations on the ImageNet- Subset. Nonetheless, CLOVVER achieves the highest last accuracy among all methods evaluated, underscoring its effectiveness in mitigating catastrophic forgetting while maintaining an optimal balance between plasticity and stability. Fig. 3 presents the accuracy variation curves on CIFARI00 and TinylmageNet. Although CLOVER shows comparatively lower performance in early tasks, it consistently outperforms other methods last accuracy across all task settings on all three datasets. This demonstrates CLOVER's s superior ability to preserve knowledge during incremental training, establishing highly competitive method even in long task settings.
4.3 Ablation Study
Ablation Study of Different Components. As presented Table we conduct ablation studies on CIFAR100 and TinylmageNet across the all settings (5, 10 and 20 tasks). The results indicate that CDSSL is a critical component, contributing to performance gains of 2.2%, 2.0% and 2.8% well as 3.6%, 4.1%, 3.6% on the TinylmageNet. Furthermore, incorporating PWP yields an additional performance enhancement of approximately 0.5%. PWP complements CDSSL by improving the collaborative inference ability of the multi-expert model, thereby offsetting imbalances between experts and enhancing the model's 's overall complementarity. Collectively, our method achieves overall improvements range from 2-3% CIFAR100 and 3-5% TinyImageNet compared to the baseline. Moreover, CDSSL exhibits significant performance advantages in longer task sequences, highlighting the effectiveness of approach in mitigating catastrophic forgetting.
on CIFAR100,
Ablation Study of CDSSL. To verify the effectiveness of each transformation CDSSL, an ablation experiment is conducted on CIFAR 100 and TinylmageNet in Table 3 The incorporation of rotation

---

Table Ablation study of each transformation in the CDSSL on the CIFARI00 and TinylmageNet. Each transformation is applied independently without combinations or overlaps. The reported metric average accuracy (%)
Method
CIFARI00
TinylmageNet
Baseline
Rot
Color
Noise
5
10
20
5
10
20
69.9
69.7
67.7
56.3
55.3
54.5
70.7
70.8
69.2
57.2
55.9
54.9
d
70.5
70.0
68.3
56,4
55.3
54.6
70.3
69.9
68.1
56.4
55.4
54.6
/
71.6
71.3
70.1
59.2
58.3
57.2
71.8
71.3
70.1
59.3
58.0
57.0
71.1
70.3
68.5
56.7
55.6
55.0
72.1
71.7
70.5
59.9
59.4
58.1
TinylmggeNer fisks--fint
TenylmageNet baks)-ffrstt
TiryEnggeNet faks)-ffntt
Daselinil CDSSL
TlavlmageNer new
Met new task
TiaylmsseNet tadis) n-eeww
Axxmaa (h)
20
ghase
20ace
phase
Figure 4: Accuracy of first/new task at each phase. The first row illustrates the variation in clas- sification accuracy for the first task throughout the training process. The second row presents the classification accuracy of new tasks at various stages of training. Our CDSSL method enables more effective learning of new tasks while minimizing the impact on existing knowledge.
transformations yields average accuracy improvements of 0.8%, 1.1%, and 1.5% on CIFARI00, and 0.9%, 0.6%, and 0.4% on Tiny ImnylntgeNet, reflecting the performance gains achieved through rotation- based SSL methods commonly employed existing studies. However, as illustrated Fig. aa), the rotational transformation significantly alters the distribution of class representations, making it difficult for the model to learn highly discriminative features from such a simple task. To address this limitation, we propose extension the rotation-baie based SSL method by incorporating color channel swapping and noise inception transformations within our CDSSL. This strategy generates new classes that closely resemble their original counterparts, allowing the model to improve its representation learning by distinguishing between these highly similar classes. Specifically, the integration of color channel swapping resulted in additional average accuracy gains of 0.9%, 0.5%, and 0.9% CIFAR100, alongside improvements of 2.0%, 2.4%, and 2.3% on TinyImageNer Moreover, further incorporating noise inception transformations contributed additional performance boost of least 0.4%. Overall, we demonstrate both the effectiveness and necessity of each transformation employed in CDSSL while highlighting its substantial superiority over conventional SSL methods.
4.4 Representation Quality and Generalization Performance Comparison with SSL
In Fig. 4 we compare the effects of CDSSL, SSL, and baseline on the accuracy of the first task and new tasks during incremental training. The results indicate that CDSSL outperforms the other methods, significantly improving the accuracy of the first task while maintaining high performance for new tasks. In contrast, although SSL also achieves comparable results, its performance is markedly inferior to that of CDSSL. These findings suggest that CDSSL offers superior representational capabilities in incremental learning tasks compared SSL., facilitating more effective category differentiation and more suitable for NECIL.

---

4.5 Visualization
Fig. 5 shows a 1-SNE visualization [9 of the dynamic changes in the representation space of the first expert during the early stages of training on CIFAR100. Specifically,
visualize the feature representations of visible classes at two distinct time points: 1) after training on 50 classes and adding 2 new classes, and 2) after training on 50 classes and adding new classes. The introduction of new classes leads to a slight overlap between the representation distributions of new and old classes within the Baseline. As more new classes are introduced, SSL struggles to effectively reduce the overlap between new and old classes, whereas CDSSL maintains clear separation. Furthermore, CDSSL enhances the cohesion within each class's distribution, indicating that the model has acquired more represeniative and discriminative features. This improvement reflects an enhanced capability for feature extraction, which is essential for addressing inter-class overlap.
Baseline
Baseline+SSL.
Baseline+ CDDSLL
Incremental phase
53
Figure 5: The visualization illustrates the distribution class representations following the application of SSL and CDSSL, respectively. Initially, the model was trained on the first task containing 50 classes (denoted by circles). Subsequently, new classes (denoted by triangles) were incrementally introduced at each step, with the progression depicted from top bottom.
Conclusion
In this paper, we develop CLOVER, an innovative framework for NECIL that aims to minimize the overlap between and old classes within the representation space. Within this framework, propose CDSSL approach that not only employs rotation transformations commonly utilized in traditional self- supersupeeeieed learning methods but also generates highly confusable task-agnostic classes through color channel swapping and noise injection. The model is subsequently trained to differentiate these classes, significantly enhancing its feature extraction capabilities and thereby mitigating the overlap between representations of new and old classes. Furthermore, we introduce an ensemble model improve the discrimination of unknown classes and then present a progressively weighted prediction strategy to address interference caused by newly introduced experts within the ensemble model. The quantitative and qualitative results on three widely used datasets have demonstrated that our method achieves state-of-the- performance, particularly excelling in improving last accuracy. worth noting that both plasticity and stability are crucial NECIL, and thus, developing method that can enhance both simultaneously is of substantial value. The proposed CDSSL introduces confusing classes increase the training difficulty, thereby improving the model's 's discriminative capability between representations of new and old classes. This enables the model to maintain strong performance across both unknown and previously learned classes. Such an approach by enhancing training difficulty through the introduction of confusing classes offers an effective and promising direction for advancing non-exemplar lass-incrrmennal learning, and it holds potential for further extension to more complex large-scale incremental learning scenarios in future research.
10

---

Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grants 62372170, 62272404, and 62502419, in part by the Natural Science Foundation of Hunan Province of China under Grant 2023J140638, and in part by the Research Foundation of Education Department of Hunan Province of China under Grant 23A0146.
References
[1] Kai Hu, Ye Xiao, Yuan Zhang, and Xieping Gao. Multi-view masked contrastive representation learning for endoscopie video analysis. Advances in Neural Information Processing Systems, 37:47987-48014, 2024.
[2] Jianqing Xu, Shen Li, Jiaying Wu, Miao Xiong, Ailin Deng, Jiazhen Ji, Yuge Huang, Guodong Mu, Wenjie Feng, Shouhong Ding, al. ID models for synthetic face recognition. Advances in Neural Information Processing Systems, 37:77777-77798,, 2024.
preserving-yet-divessee diffusion
[3] Yuan He, Moy Yuan, Jiaoyan Chen, and Ian Horrocks. Language models as hicrarchy encoders. Advances in Neural Information Processing Systems, 37: 17:14690-14771, 2024.
[4] Kai Hu, Jinhao Li, Yuan Zhang, Xiongjun Ye, and Xieping Gao. One-to-multiple A progres- sive style transfer unsupervised domain domain-adaptic adaptive framework for kidney tumor segmentation. Advances in Neural Information Processing Systems, 11-24496-245222 2024.
[5] Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng. and Qibin Hou. Storydiffusion: Consistent self-attention for long-range image and video generation. Advances in Neural Information Processing Systems, 37:1 10315-110340. 2024.
[6] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109-165. Elsevier, 1989.
[7] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2001-2010 2017.
[8] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in Neural Information Processing Systems, 33:15920-15930, 2020.
[9] Fei Zhu, Yu- Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5871-5880, 2021.
[10] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self-sustaining representation expansion for non-exemplar class-incremennal learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9296-9305, 2022.
[11] Tamasha Malepathirana, Damith Senanayake, and Saman Halgamuge. Napa-vq: Neighborhood- aware prototype augmentation with vector quantization for continual learning. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 1 11674-11684, 2023.
[12] Wuxuan Shi and Mang Ye. Prototype reminiscence and augmented asymmetric knowledge aggregation for non-exemplar class-incrementee learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1772-1781, 2023.
[13] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30, 2017.
[14] Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski, and Joost Van De Weijer. Fecam: Exploiting the heterogeneity of class distributions exemplar- free continual learning. Advances in Neural Information Processing Systems, 36:6582-6595. 2023.

---

[15] Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. Fetril: Feature translation for exemplar-free class-incremental learning. In Proceedings of the IEEE/CVF Winter Conference Applicattins of Computer Vision, pages 3911-3922 2023.
[16] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. International Conference on Learning Representations, 2018.
[17] Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Self-supervised label augmentation via input transformations. In International Conference on Machine Leurning, pages 5714-5724. PMLR,
2020.
[18] Jichuan Zhang, Yali Li, Xin Liu, and Shengjin Wang. Diffusion model meets non-exemplar class- -easeenncremetal learning and beyond. arXiv preprint arXiv:2408 2tti::2208.00988,,, 2024.
[19] Zhiling Fu, Zhe Wang, Xinlei Xu, Mengping Yang, Ziqiu Chi, and Weichao Ding. Semantic alignment with sus--superviion for class incremental learning. Knowledge-Bard Systems, 282:111114, 2023.
[20] Wenyi Feng. Zhe Wang. Qian Zhang, Jiayi Gong, Xinlei Xu, and Zhilin Fu. Hybrid rotation self-supervision and feature space normalization for class incremental learning, Information Sciences, 691:121618, 2025.
[21] Jayateja Kalla and Soma Biswas. S3c: Self-supervised stochastic classifiers for few-shot class-incremental learning. In European Conference on Computer Vision, pages 432-448, 2022.
[22] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
[23] Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong, Coscl: Cooperation of small continual learners is stronger than a big one. European Conference on Computer Vision, pages 254 271, 2022.
[24] Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23554-23564, 2024.
[25] Kai Hu, Yunjiang Wang, Yuan Zhang, and Xieping Gao. Progressive learning strategy for few-shot class-incremental learning. IEEE Transactions Cybernetics, pages 1210-1223, 2025.
[26] Shaokun Wang, Weiwei Shi, Yuhang He, Yifan Yu, and Yihong Gong. Non-exemplar class- incremental learning via adaptive old class reconstruction. In ACM International Conference on Multimedia, pages 4524 4534, 2023.
[27] Qiwei Li, Yuxin Peng, and Jiahuan Zhou. Fes: Feature calibration and separation for non- exemplar class incremental learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 28495-28504, 2024.
[28] Zihuan Qiu, Yi Xu, Fanman Meng, Hongliang Li, Linfeng Xu, and Qingbo Wu. Dual- consistency model inversion for non-exemplar class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24025-24035, 2024.
[29] Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69-84, 2016.
[30] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In European Conference on Computer Vision, pages 577-593, 2016.
[31] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting- Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2536-2544, 2016.
12

---

[32] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In European Conference on Computer Vision, pages 132-149, 2018.
[33] Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with self-supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8059-8066 2019.
[34] Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1476-1485, 2019.
[35] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in Neural Information Processing Systems, 32, 2019.
[36] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class- class-imbalanced learning. Advances in Neural Information Processing Systems, 13:19290-19301, 2020.
[37] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3023, 2021.
[38] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Trans- formers for continual learning with dynamic token expansion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[39] Grzegorz Rypeść, Sebastian Cygert, Valeriya Khan, Tomasz Trzciński, Bartosz Zieliński, and Bartlomiej Twardowski. Divide and not forget: Ensemble of selectively trained experts in continual learning. International Conference on Learning Representations, 2024.
[40] Songlin Dong, Xinyuan Gao, Yuhang He, Zhengdong Zhou, Alex C Kot, and Yihong Gong. Ceat: Continual expansion and absorption transformer for non-exemplar class-incremental learning. IEEE Transactions on Circuits and Systems for Video Technology, pages 3146-3159, 2024.
[41] Jiang- Jiang-Tian Zhai, Xialei Liu, Lu Yu, and Ming-Ming Cheng. Fine-grained knowledge selection and restoration for non-exemplar class incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6971-6978. 2024.
[42] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016
[44] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5138-5146, 2019.
[45] Xialei Liu, Jiang- Jiang-Tian Zhai, Andrew D Bagdanov, Ke Li, and Ming-Ming Cheng. Task-adaptive saliency guidance for exemplar- free class incremental learning. In Proceedings of IEEF/CV Conference on Computer Vision and Pattern Recognition, pages
2024.
[46] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[47] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)3, 2015.
[48] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
13

---

[49] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(1 1), 2008.
[50] Solomon Kullback and Richard Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86 1951.
[51] Simone Magistri, Tomaso Trinci, Albin Soutif- Cormerais, Joost de Weijer, and Andrew D Bagdanov. Elastic feature consolidation for cold start exemplar-free incremental learning. International Conference Learning Representations, 2024.
[52] Keinosuke Fukunaga. Introduction statistical pattern recognition. Elsevier, 2013.
[53] Anil Bhattacharyya, On measure of divergence between two statistical populations defined by their probability distribution. Bulletin of the Calcutta Mathematical Society, 55:99-110, 1943.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims stated in the abstract and introduction accurately reflect the contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We introduce our method in detail in Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results of the paper in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We open-source the code. We do not need to release any data, as we use publicly available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr ec/
public/guides/ Sowpess/oooeuuubsssssoooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nipscc//pie wwwppbblccgguiicuceccccc /eegguddssseCeebbbiniinn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed information about the experimental setup in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
All experiments are repeated three times and the average results are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information about the computing resources needed for the experiments in Section 4. and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in our paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential broader impacts of our work in Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all the relevant assets used in paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We use publicly available datasets, so this issue does not arise.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We use publicly available datasets, so this issue does not arise.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in our research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

Appendix
In the appendix of this paper, Section provides detailed introduction to the evaluation metrics. Subsequently, Section BB offers the detailed values of the accuracy curves on CIFAR 100 and Tinylm- ageNet datasets. Section C presents a more comprehensive and detailed ablation study. Section D comprchensively demonstrates the advantages of CDSSL over SSL from multiple perspectives, such as performance and visualization. Section conducts an analysis of the model's 's scalability and investigates the rationale behind CDSSL's adoption of three specific transformations: rotation, color channel permutation, and noise injection. Finally, Section F discusses the potential limitations and the broad impacts of our method.
A Evaluation Metries
We utilize two evaluation metrics commonly applied in NECIL to assess the performance different models under various settings. Each metric is defined formally below.
Average Accuracy. Aug is defined as the average accuracy across all incremental stages, providing comprehensive metric for fairly evaluating the overall incremental performance of different methods. Letting Un,m represents the accuracy of task n after the training of phase m. The average accuracy expressed as follows:
T
Aug T
an,m-
(6)
m== ==
Last Accuracy. Last evaluates the model's final performance on the validation sets of all previously encountered tasks, serving as a key metric for assessing ability to mitigate catastrophic forgetting, integrate knowledge across tasks, and maintain robust generalization. Last accuracy is defined as follows:
Last Σ
a,T
(7)
B Detailed Values of the Accuracy Curves
To facilitate comparisons in future research, we present the detailed values of the accuracy curves (as depicted in Fig. of the main text) in Tables 4 and 6
Table 4: Detailed values of classification accuracy under the setting of 5 tasks.
Phase
Dataset
4
5
CIFAR 81.70
74.68
72.6
70.49
68.70
67.99
TinylmageNe
65.82
61.47
60.60
59.48
57.56
56.02
Table 5: Detailed values of classification accuracy under the setting of 10 tasks.
Phase
Dataset
2
5
9
10
C0F
81.62
78.33
74.55
73.53
72.69
71.36
70.13
68.65
68.12
67.45
ImnyemageNet
66.62
66.40
62.27
60.94
61.00
60.47
59.46
58.13
56.51
55.39
54.12
22

---

Table 6: Detailed values of classification accuracy under the setting of 20 tasks.
Phase
Dataset
6
CIFAR100
83.25
79.33
77.96
77.4
74.69
73.07
72.24
71.21
70.56
70.51
TinylmageNet
65.72
64.36
63.47
62.24
61.03
60.35
59.70
59.78
59.73
59.16
Phase
Dataset
10
12
13
16
17
19
20
CIFARI00
70.37
69.89
69,42
68.57
67.76
67.39
66.78
65.51
65.23
64.90
TinylmageNet
58.64
58.21
57.74
57.30
56.65
55.84
55.17
54.71
54.05
53.25
52.80
C More Ablation Study
C.1
Components Ablation for ImageNet-Subset
To further verify the generalizability of CLOVER, we conduct components ablation on the ImageNet- Subset dataset, illustrated in Table 7
Table 7: Ablation study of each component of CLOVER on ImageNet-Subset. The reported metric is average accuracy (%).
Component
ImageNet-Subset
Baseline
CDSSL
PWP
10
75.1
73.9
71.1
77.7
76.8
74.3
/
/
77.8
77.1
74.5
C.2 Ablation Study of CDSSL for ImageNet-Subset
We conduct ablation studies on the various transformations of CDSSL on the ImageNet-Subset dataset. As presented Table g the diverse transformations within CDSSL yield substantial performance improvements, confirming the effectiveness of our CDSSL approach on large-scale datasets.
Table 8: Ablation study of each transformation in the CDSSL on ImageNet-Subset. The reported metric average accuracy (%).
Method
ImageNet-Subee
Baseline
Rot
Color
Noise
5
10
20
75.1
73.9
71.1
76.7
75.6
73.4
/
77.4
76.5
73.9
d
d
77.7
76.8
74.3
C.3 Number of Shared Layers
In Table 9 we compare the performance of CLOVER under different parameter configurations (adjusted by varying the number of shared layers f) with SEED [39], which is likewise ensemble- based model. The first expert, extensively trained on the initial task with abundant data, demonstrates strong representation extraction capabilities. By sharing certain layers with the first expert, the representation extraction capabilities of the other experts are enhanced, leading to an overall im- provement CLOVER's performance. Notably, the best results are achieved when layers are shared. Furthermore, CLOVER maintains robust performance even with reduced parameter counts, consistently outperforming SEED. These results strongly validate the effectiveness of the proposed method.
23

---

Table 9: Comparisons of the average accuracy and last accuracy (%) with various shared layers CIFAR100.
CIFARI00
Method
#Params
5 tasks
10 tasks
20 tasks
Avg
Last
Avg
Last
Avg
Last
S [39
56.0M
71.1
66.3
69.9
65.0
68.2
61.4
CLOVER(O shared)
56.0M
72.6
67.5
72.1
66.8
71.0
64.5
CLOVER(5 shared)
55.4M
72.7
68.0
72.3
67.5
71.0
64.9
CLOVER(9 shared)
53.3M
72.6
67.9
72.2
67.3
70.9
64.6
CLOVER(13 shared)
44.9M
72.4
67.5
72
66.9
70.4
64.1
CIFARTOO average sccuracy
CIFAR100 last
72
66
Accurcy
70
tasks 10
tasks
10 tasks
tasks
20 tasks
68
58
8
10
Number experts
Number of experts
Figure 6: The impact of the number of experts in CLOVER on both the average accuracy and last accuracy.
C.4 Number of Experts
In Fig. 6 we investigate the impact of the number of experts on the average accuracy and the last accuracy attained by CLOVER. As the number of experts increases from 4, CLOVER exhibits consistent improvements in both average accuracy and last accuracy on 5, 10, and 20 task settings. However, as the number of experts continues to increase, the average accuracy of CLOVER 5 and 10 tasks scenarios exhibits declining trend, with no significant enhancement observed in the last accuracy. While additional performance improvements can be achieved in 20 tasks setting, these enhancements are not uniformly effective across all configurations and result in substantial rise in parameter count. Consequently, we ultimately adopt CLOVER with five experts (i.e., K=5), following the methodology outlined by SEED 399. to ensure fair and equitable comparison.
C.5 The Sensitivity of Hyper-parameter a and /
In PWP, the parameters a and 3 play a critical role in determining weight allocation. To investigate their impact, conduct sensitivity analysis CIFAR100 and TinylmageNet, as presented in Table [IO] Given that the maximum weight for new expert is constrained to 0.2 (as K = 5), we vary a within the range 0.15, 0.10. 0.05 and allow the weight to reach its maximum either after one task or two tasks. Notably, a higher initial a yields better performance in short-task settings, while a lower initial is advantageous in long-task scenarios. This distinction arises from the fact that, in long-task settings, newly introduced experts have access to less training data, leading to less reliable predictions that can be effectively mitigated by assigning them lower weights. Overall, PWP demonstrates robustness to variations in and 3, consistently achieving significant improvements.
D Further Comparison with SSL
D.1 Representation Quality and Generalization Performance Comparison
To assess the effectiveness of CDSSL, we conduct comparative analysis of its impact the accuracy of both the first task and the new tasks, comparison with SSL. and the baseline, across the CIFAR 100 and ImageNet-Subbet datasets during incremental training, illustrated in Fig. 7 The proposed

---

CDSSL achieves higher accuracy on both the first task and new task across all evaluated datasets, highlighting the generalizability of improved representation extraction capabilities.
Table 10:
Robustness testing hyper-parameter a and on CIFARIO0 and TinylmageNet. The first row, where and 3 equal to represents the results obtained without PWP. The reported metric is average accuracy (%).
Parameters
CIFARTO0
Tiny KinyImageNet
3
5
10
20
5
10
20
72.1
71.7
70.5
59.9
59.4
58.1
0.15
0.05
72.7
72.0
70.9
60.2
59.9
58.5
0.025
72.7
72.1
70.9
60.1
59.9
58.5
0.10
0.10
72.7
72.1
70.9
60.1
59.9
58.5
0.05
72.7
72.3
71.0
59.9
59.9
58.6
0.05
0.15
72.7
72.2
71.0
59.8
59.9
58.6
0.075
72.6
72.2
71.1
59.7
59.9
58.6
CIFAR10012 sald)
CBRAR10012
tragc/oo-Sedect sabi--dar
Inaachiasdatee 12 aubi-ccer
Jecine
11 adoisini
Figure Accuracy of first/new task each phase on CIFAR100 and ImageNet-Subsee
D.2 Quantitative Comparison
To more clearly demonstrate the advantages CDSSL over traditional SSL, Table [1 compares the performance of the Baseline enhanced with CDSSL versus that enhanced with traditional SSL.. The experimental results show that CDSSL achieves superior performance across all settings. This improvement arises from the introduction of highly confusable task-agnostic classes in CDSSL, which enable the model to extract the most discriminative features for various class. Consequently, CDSSL significantly minimizes the overlap in representation between new and old classes, thereby improving the model's capacity for incremental learning.
D.3 The Similarity Between Various -Anostgi Classes and the Raw Classes
In Fig. g we compare the KL divergence (50] between the raw classes and task-agnostic classes, generated through rotation, color channel swapping, and noise injection, on CIFAR100. A smaller
Table 11: Quantitative comparison between CDSSL and SSL. The reported metric is average accuracy (%).
CIFAR100
ImnylmageNet
ImageNet-Suiee St--Stsett
Method
10
5
10
20
10
Baseline
69.9
69.7
67.7
56.3
55.3
54.5
75.1
73.9
71.1
Baseline+SSL
70.7
70.8
69.2
57.2
55.9
54.9
76.7
75.6
73.4
BaselineCCDSSL
72.1
71.7
70.5
59.9
59.4
58.1
77.7
76.8
74.3
25

---

KLL. divergence between task-agnostic and raw classes
30
25
KL divergence
20
15
10
5
0
90"
180
270
GBR
BRG
Noise
Task-agnostic classes
Figure 8: KL divergence between task-agnostic and raw classes on CIFARI00
Raw class
90 class
180
GBR
Noise class
L-SE on CIFARI00
(b) t-SNE TinyImageNet
Figure 9: t-SNE visualization of raw classes and task-agnostic classes on CIFAR-100 and Tinylma- geNet. Samples with similar colors correspond the same raw class, while different marker shapes represent task-agnostic classes derived by applying distinct transformations.
KI. divergence indicates greater similarity between the representation distributions, leading higher likelihood of confusion. Furthermore, Fig. provides intuitive visualization of the spatial relationships between the raw and task-agnostic classes. These results indicate that task-agnostic classes generated via color channel swapping and noise injection are more susceptible confusion with the raw classes.
E
More Analysis
E.1
Parameter Scalability Analysis
To better demonstrate scalability of CLOVER, Fig. 10 illustrates the variation in model parameters across incremental phases. Simply, throughout the training process from Task 0 to Task 4, the number of parameters increases by approximately 11.2M with each task that arrives. Beyond this stage, no additional experts are incorporated, and the number of parameters remains stable at 55.4M.
E.2 More Transformation Experiments
To validate the effectiveness and generalizability the transformations employed in CDSSI, Table 12 presents comparison of the performance improvements achieved by different transformations. The
26

---

Parameter count across incremental phases
70
55.4M
60
Parameter (M)
50
40
30
20
10
2
10
Incremental Phase
Figure 10: Variation of model parameter count across incremental phases.
Table 12: Comparison model performance on CIFARI00 and TinylmageNe datasets after applying various transformations. "Blurring" applys 5x5 Gaussian blur kernels with standard deviations of 1.0. "CenterCrop" crops a 224 x224 image to 192x 192 and then resizes it back to 224 x224. The reported metric is average accuracy (%)
CIFARION 100
TinylmageNet
Method
5
10
20
10
20
Baseline
69.9
69.7
67.7
56.3
55.3
54.5
Baseline+Rot
70.7
70.8
69.2
57.2
55.9
54.9
Baseline+Color
70.5
70.0
68.3
56.4
55.3
54.6
Baselime+Noise
70.3
69.9
68.1
56.4
55.4
54.6
Baseline+Blurrin
70.3
69.9
68.2
56.1
55.9
54.3
Baseline+CenterCrop
71.4
70.5
69.0
55.8
54.7
54.3
results indicate that not all transformations yield effective results, as some may perform well on CIFAR-100 while exhibiting poor performance on TinylmageN In contrast, the transformations in CDSSL (rotation, color channel swapping, and noise injection) consistently improve performance across datasets, demonstrating superior generalizability.
In addition, an excessive number of task-agnostic classes may lead to an overemphasis on task- agnostic knowledge, which can impede class recognition. To investigate this phenomenon, Table T3] demonstrates the results CDSSL. enhanced with blurring, cropping and more color channel swapping (i.e. swapping RGB channels to RBG, GRB and BGR) transformations and compares them with those produced by CDSSL alone. While these transformations improved performance on CIFAR- 100, they adversely affected TinylmageNet due to its greater number of categories and finer details requiring more effective feature extraction.
Table 13: Comparison of model performance on the CIFAR-I00 and TinyImageNet datasets after applying additional transformations beyond Baseline CDSSL. "MoreColor" refers to the creation of new classes by swapping the RGB channels into RBG, GRB, and BGR. The reported metric is average accuracy (%).
CIFART00
T'nylmageNNe
Method
5
10
20
5
10
20
Baselinc+CDSSL
72.1
71.7
70.5
59.9
59.4
58.1
Buseline+CDSSSL-lluiin
72.3
71.7
70.9
59.8
58.7
57.6
Buseine-CCDSSL-CCntCCCng
73.1
72.6
70.9
60.1
58.6
57.7
Basline-CDDSLLLMorCColl
72.4
71.9
70.9
59.7
58.5
58.0
27

---

Table 14: Comparison of the average accuracy (%) of different methods under the cass-balanced setting.
CIFAR 100
Method
5
10
20
EFC [51]
60.87
55.78
SEED
70.9
69.3
62.9
Baseline
71.1
66.7
60.3
Baseline+CDSSL
73.2
71.0
66.7
Table 15: Evolution of average Bhattacharyya distance during training under the 5-task setting CIFAR-100.
Method
Task
Baseline
SSL
CDSSL
0
2
4
7.62
6.75
6.19
5.76
5.38
5.05
10.17
9.34
8.85
8.41
8.06
7.74
12.12
11.02
10.34
9.77
9.31
8.89
E.3 Performance under the class-balanced setting
To further evaluate the effectiveness of CDSSL across different incremental learning configurations, conduct experiments under the class-balanced setting, where each task contains equal number of classes. As presented in Table 14, CLOVER exhibits strong performance and generalizability, with CDSSL achieving 6.4% improvement in average accuracy under the long-task setting and consistently surpassing existing methods across all settings. These results further attest to the robustness and adaptability of CDSSL in diverse incremental learning scenarios.
E.4 Quantitative Analysis of Inter-Class Distribution Confusion
To intuitively demonstrate the effectiveness of our method in enhancing inter-class separability, the Bhattacharyya distance [52 (33 is employed as metric to quantify the overlap between two probability distributions, where a greater overlap corresponds to a smaller distance, while lesser overlap yields larger one. This metric effectively captures the degree of separability and the clarity of class boundaries. The Bhattacharyya distance calculated as follows:
Σ
DB (11
122
2 In(
Σ
Σ2
where f1 denote the mean vectors of the two class distributions, Σ, Ee are the corresponding covariance matrices, Σ = 1 [Σ1 + Σg) represents the average covariance matrix, and | denotes the determinant.
Based on this metric, we track the average class-boundary clarity throughout the training process under the 5-task setting of CIFAR-100, summarized in Table 15 The results show that CDSSL effectively improves class separability by increasing the Bhattacharyya distance, thereby reducing confusion and overlap between class representation distributions.
F Limitations and Broader Impacts
Limitations. In this study, we propose a method for non-exemplar class incremental learning. However, the approach relies on base task with abundant training data and has not yet been validated in setting where all tasks contain equal amount of data. Moreover, the training cost of our method is relatively high. Specifically, we extend the original N-way classification task to 7N-way classification task, thereby increasing the training time. In addition, compared to single-model approaches, our method employs an ensemble model, which results in a larger number of parameters. To address the high training cost, future work will focus on developing strategies that reduce both training time and parameter size, aiming to achieve a more favorable balance between performance and computational efficiency.
28

---

Broader Impacts. Non- Non-exemplar class incremental learning, which enables the continuous accumu- lation of knowledge without retaining old samples, is of considerable significance in the current era of growing data privacy concerns. Our proposed CLOOVER framework fully leverages the advantage that frozen models are inherently resistant catastrophic forgetting during updates, focusing on mitigating representation overlap between old and new classes. CLOVER synergistically combines self-supervised learning and ensemble learning to effectively and non-conflictually address this challenge, offering novel pathway for future researchers to combat forgetting. Moreover, the idea of enhancing the representation extraction capabilities of frozen models to further alleviate representa- tion overlap, explored in this work, presents promising direction for continued investigation and may foster further advancements in the field. Overall, our study advances the development of machine learning and introduces innovative solution to the domain of non-exemplar class incremental learning.
29


Reviews:
[{"review_id": "APw7X8rp7x", "reviewer": "Reviewer_1GN5", "summary": "This paper proposes CLOVER, a novel framework for non-exemplar class incremental learning (NECIL), which aims to sequentially learn new classes without retaining data from previous tasks. To address the challenge of representation overlap between old and new classes, a common issue when the feature extractor is frozen, the authors introduce two key components. First, a confusion-driven self-supervised learning (CDSSL) module generates challenging task-agnostic classes using transformations such as color channel swapping and noise injection, encouraging the model to learn more discriminative features. Second, a progressively weighted prediction (PWP) strategy gradually increases the influence of newly added ensemble experts, mitigating performance degradation caused by their initial unreliability. Experimental results on CIFAR100, TinyImageNet, and ImageNet-Subset demonstrate that CLOVER achieves strong performance across various incremental learning settings.", "questions": "1. The actual spatial relationship between raw and transformed classes remains unclear. Maybe t-SNE visualizations that directly compare the embedding distributions of raw classes versus task-agnostic classes could explain this.\n2. From Table 3 and related ablations, the performance gain from rotation-based SSL appears greater than that of color channel permutation. This is counterintuitive if the argument is that color transformations produce more confusing task-agnostic classes.\n3. Why do some augmentations like CenterCrop in Table 12 perform well on CIFAR-100 but poorly on TinyImageNet?\n4. Please provide measurements of training time per task, inference cost (e.g., FLOPs or forward pass time), and GPU memory usage. How do these compare to baseline methods like SEED or FeCAM?", "limitations": "Yes, the authors include a dedicated section (Appendix F) where they discuss the limitations of their work, including the reliance on a strong base task, increased training cost, parameter growth.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The proposed Confusion-Driven Self-Supervised Learning (CDSSL) module introduces task-agnostic classes using color channel permutation and noise injection, enhancing representation discrimination and mitigating feature overlap between old and new classes.\n2. The Progressively Weighted Prediction (PWP) strategy effectively addresses a common issue in ensemble-based NECIL methods: performance degradation when newly added experts are weak. By gradually increasing expert weights across tasks, it improves ensemble reliability without requiring retraining.\n3. CLOVER achieves state-of-the-art performance across multiple NECIL benchmarks. Ablations and analyses support its claims.\n\nWeaknesses:\n1. The proposed two core modules are functionally independent and loosely coupled. CDSSL focuses on representation learning during training, while PWP controls prediction aggregation at inference. There is no interaction, joint optimization, or design dependency between them. This makes the method feel more like a stacked combination of two standalone techniques, rather than a coherent, integrated approach.\n2. The paper’s title “Confusion-Driven...” suggests that both components are driven by confusion modeling, but only CDSSL fulfills this. PWP, while effective, is a heuristic weighting scheme that neither exploits confusion nor adapts to inter-class uncertainty. This overclaims conceptual coherence and may mislead readers into overestimating the role of confusion modeling in the ensemble component.\n3. Confusion-based SSL and color transformation augmentations have already been explored in NECIL contexts [1-3]. These works also use perturbation-based or proxy-label-based augmentation to enhance fine-grained feature discrimination. CLOVER builds incrementally on these ideas, and the degree of novelty in CDSSL is therefore limited.\n4. Ensemble learning has been extensively used in NECIL to reduce forgetting[4-5], but a well-known challenge is the linear growth in model parameters and memory usage. CLOVER inherits this problem without proposing a mitigation strategy, where each task adds a new expert, leading to scalability concerns in long-task or high-capacity models.\n[1] Semantic Alignment With Self-Supervision for Class Incremental Learning, KBS 2023.\n[2] Non-Exemplar Class-Incremental Learning by Random Auxiliary Classes Augmentation and Mixed Features, TCSVT 2024.\n[3] Hybrid Rotation Self-Supervision and Feature Space Normalization for Class Incremental Learning, Information Science 2025.\n[4] CoSCL: Cooperation of Small Continual Learners is Stronger Than a Big One, ECCV 2022.\n[5] Expandable Subspace Ensemble for Pre-trained Model-based Class-Incremental Learning, CVPR 2024.", "was_revised": true, "final_justification": "I appreciate the authors' responses to my comments. The authors supplemented the additional experiments. Additionally, they addressed each of the raised weaknesses and questions individually and promised to make revisions in the revised version. Hence, I suggest changing the score to 5: Accept.", "rebuttal": "We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.\n\n**W1: There is no interaction, joint optimization, or design dependency between CDSSL and PWP.**\n\nCertainly, CDSSL and PWP operate in different areas, but they complement each other effectively. CDSSL enhances the predictive capabilities of individual experts. Nevertheless, due to the varying amounts of data utilized for training each expert, it inadvertently exacerbates the performance disparities among them. PWP addresses this issue by improving the collaborative inference capabilities of the multi-expert model, thereby alleviating such imbalances and enhancing the overall complementarity of the model. Experimental results presented in Table 2 of the paper further substantiate that the combined performance of CDSSL and PWP surpasses the sum of their individual contributions, underscoring their complementary and synergistic strengths.\n\n**W2: The paper’s title “Confusion-Driven...” suggests that both components are driven by confusion modeling, but only CDSSL fulfills this. PWP, while effective, is a heuristic weighting scheme that neither exploits confusion nor adapts to inter-class uncertainty. This overclaims conceptual coherence and may mislead readers into overestimating the role of confusion modeling in the ensemble component.**\n\nThank you for your valuable suggestion. The term “Confusion-Driven” in the title refers to CDSSL, a strategy specifically designed to address the overlap between representations of old and new classes. Specifically, this approach encourages the model to extract more discriminative representations by generating task-agnostic classes that are likely to be confused with existing ones. Consequently, this reduces the overlap in class representation distributions. On the other hand, PWP is a multi-expert balancing strategy built upon CDSSL. It aims to address performance degradation from training new experts and enhance collaborative prediction by leveraging the diverse knowledge of various experts more effectively. We will reconsider the title in future revisions to mitigate potential misunderstandings.\n\n**W3: The degree of novelty in CDSSL is therefore limited.**\n\nBoth SSL and CDSSL enhance training complexity by introducing task-agnostic classes, thereby prompting the model to extract discriminative features through differentiation. This strategy effectively alleviates the issue of representation overlap. However, in SSL, the task-agnostic classes generated via rotation exhibit considerable semantic divergence from the original classes (see Fig. 1a and Fig. 7), which limits their effectiveness. In contrast, CDSSL constructs task-agnostic classes that are highly confused to the raw classes. This approach further compels the model to extract more discriminative representations, thus addressing representation overlap more effectively. Overall, we conducted a comprehensive analysis of the fundamental mechanisms underlying performance improvements in SSL, identified the generalization limitations inherent in traditional SSL approaches, and proposed an innovative strategy to overcome these challenges, resulting in significant enhancements in performance. As demonstrated in Fig. 4 and Fig. 6 of the paper, CDSSL consistently outperforms SSL in accuracy on both new and old tasks, thereby validating its superiority.\n\n**W4: Ensemble learning has been extensively used in NECIL to reduce forgetting [4-5], but a well-known challenge is the linear growth in model parameters and memory usage. CLOVER inherits this problem without proposing a mitigation strategy, where each task adds a new expert, leading to scalability concerns in long-task or high-capacity models.**\n\nOur method utilizes a fixed number of experts instead of assigning a separate expert to each task, thereby effectively mitigating the issue of parameter explosion in lengthy task sequences. As presented in Table 9 of the paper, CLOVER achieves significantly superior performance compared to SEED, which also employs an ensemble learning paradigm, while maintaining a lower parameter count due to the sharing of parameters at shallower layers. Building upon ensemble learning principles, we further introduce the PWP strategy, which mitigates the problem of task data imbalance in NECIL by narrowing the experience gap among experts. This strategy dynamically adjusts expert weights based on both the number of predictable classes for each expert and its predictive ability, leading to enhanced overall performance. Moreover, PWP and CDSSL operate at distinct yet complementary levels within the ensemble framework: while CDSSL enhances individual expert training by fostering discriminative representation learning, PWP improves collaborative inference across experts. Together, these strategies empower the multi-expert model to more effectively confront the challenges inherent in NECIL.\n\n**Q1: The actual spatial relationship between raw and transformed classes remains unclear. Maybe t-SNE visualizations that directly compare the embedding distributions of raw classes versus task-agnostic classes could explain this.**\n\nThank you for your insightful comment. We previously generated t-SNE plots to compare the distributions of raw classes and task-agnostic classes. However, we omitted these visualizations due to space constraints. As suggested, we will incorporate t-SNE plots in future revisions to more effectively illustrate and compare the embedding distributions of both raw classes and task-agnostic classes.\n\n**Q2: From Table 3 and related ablations, the performance gain from rotation-based SSL appears greater than that of color channel permutation. This is counterintuitive if the argument is that color transformations produce more confusing task-agnostic classes.**\n\nWhen applied individually, rotation-based SSL indeed demonstrates greater performance gains compared to color channel permutation-based SSL. However, the combination of these two approaches leads to a performance enhancement that significantly surpasses the sum of their individual contributions. This phenomenon can be attributed to the role of confusable classes, which further intensify training difficulty on top of rotation-based SSL, thereby compelling the model to learn more discriminative representations. The experimental results presented in Table 3 of the paper provide additional support for this assertion: task-agnostic classes generated through color channel permutation and noise injection markedly improve SSL performance. Furthermore, the overall enhancement exceeds the cumulative effect observed when each technique is employed in isolation.\n\n**Q3: Why do some augmentations like CenterCrop in Table 12 perform well on CIFAR-100 but poorly on TinyImageNet?**\n\nMany classes in TinyImageNet are categorized under fine-grained classification (e.g., various dog breeds and bird species), where the images themselves possess inherent complexity, featuring small objects and indistinct boundaries. The application of cropping or blurring techniques to such images often undermines their discriminative features: center cropping may eliminate critical information (e.g., a bird’s beak or a dog’s eyes), while Gaussian blurring smooths texture boundaries, thereby diminishing the distinctiveness of local details. In these instances, the newly constructed task-agnostic classes cease to represent distinguishable variants and instead introduce additional label confusion. This form of SSL compromises the feature learning process by introducing label noise, ultimately resulting in degraded performance.\n\n**Q4: Please provide measurements of training time per task, inference cost (e.g., FLOPs or forward pass time), and GPU memory usage. How do these compare to baseline methods like SEED or FeCAM?**\n\nAs suggested, we have provided the training time per epoch on a single A100 GPU, the inference time for 128 samples, and the GPU memory usage during training for the three methods, as shown in Table 1. As highlighted in the Limitations section, the training cost of CLOVER is relatively high, but overall, the computational burden is still within an acceptable range compared to the performance improvement it brings.\n\n**Table 1. Comparison with SEED and FeCAM in terms of training time, inference cost, and GPU memory usage on CIFAR100.**\n\n|     Method    |     Training time per epoch    |     Avg forward pass time    |     GPU usage    |\n|---|---|---|---|\n|     CLOVER    |     103s    |     165ms    |     38,377MB    |\n|     SEED    |     23s    |     165ms    |     7,427MB    |\n|     FeCAM    |     17s    |     27ms    |     2,569MB    |"}, {"review_id": "AmlKSiemBB", "reviewer": "Reviewer_5czS", "summary": "This paper proposes a confusion-driven self-supervised progressively weighted ensemble learning framework for NECIL task. Self-supervised learning and ensemble learning are applied to significantly improve performance. Extensive experimental results on CIFAR100, TinyImageNet and ImageNet-Subset benchmarks demonstrate its superior performance.", "questions": "1. Here are many symbols in the figures whose meanings are confusing; the figures should be self-contained (e.g., different class names in Fig. 1).\n2. The meaning of \"weighted ensemble\" has not been specifically explained in the abstract or introduction.", "limitations": "yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The motivation is simple and efficient.\n2. The experimental results are outstanding.\n\nWeaknesses：\n1. Self-supervised learning and ensemble learning are general methods for improving representations, so why are they used to address the specific task of NECIL?\n2. As shown in Figure 3, the baseline performance of the proposed method is already much higher than that of other methods. Could this be the main reason for the overall superior incremental performance? Using the same baseline performance or comparable performance drop indicators could be considered to enable a fairer comparison.\n3. Compared with other incremental learning methods that incorporate self-supervised learning (e.g., [i]), what are the specific advantages of the self-supervised approach used in this paper?\n[i] Prototype Augmentation and Self-Supervision for Incremental Learning. CVPR2021.", "was_revised": true, "final_justification": "My concerns about the initial performance gap have not been addressed, so I will maintain my original score.", "rebuttal": "We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.\n\n**W1: Self-supervised learning and ensemble learning are general methods for improving representations, so why are they used to address the specific task of NECIL?**\n\nIn NECIL, a prominent approach involves freezing the feature extractor after it has been trained on the initial task. This frozen extractor serves as a fixed mapping function to compute and store the representation distribution for each class. During inference, classification is conducted by measuring the distance between the features of a given sample and the stored class distributions. \n\nHowever, since the model is optimized exclusively for the initial class distribution, it lacks the adaptability necessary to effectively accommodate new class data. Consequently, representations of new classes introduced in subsequent tasks often become dispersed or overlap with those of existing classes within the fixed representation space—particularly when similarities exist along certain dimensions. This overlap obstructs the formation of clear decision boundaries and significantly degrades classification performance in incremental learning scenarios. \n\nIn this context, enhancing the representational capacity of the feature extractor becomes crucial, as it determines how well class distributions can be separated. To improve this capacity, we integrate and build upon self-supervised learning and ensemble learning through our proposed methods: CDSSL and PWP. These two strategies complement each other. CDSSL enhances individual experts’ discriminative power while PWP improves collaborative inference among experts. Together, they elevate the overall representational quality of the model and effectively address issues related to representation overlap between old and new classes.\n\n**W2: As shown in Figure 3, the baseline performance of the proposed method is already much higher than that of other methods. Could this be the main reason for the overall superior incremental performance? Using the same baseline performance or comparable performance drop indicators could be considered to enable a fairer comparison.**\n\nAs shown in Table 1, compared to FeCAM, both our method and SEED employ a multi-expert architecture, which results in a higher parameter count compared to certain other models. However, our baseline is designed to be comparable to SEED. It essentially represents a variant of SEED with shared parameters across the first five layers, leading to a reduced parameter count and slightly diminished performance. This makes our baseline a fair comparison point. Furthermore, as presented in Table 2 of the paper, the proposed CDSSL and PWP achieve performance enhancements of 2–3% on CIFAR-100 and 3–5% on TinyImageNet, underscoring the effectiveness of our approach. Most importantly, our main comparison method is SEED. As shown in Table 9 of the paper, CLOVER not only achieves better performance but also requires fewer parameters than SEED, thereby highlighting its superior efficiency and effectiveness.\n\n**Table 1. Average accuracy comparison of baselines with SOTA methods. ‘–’ indicates that the information is not reported in the original paper.**\n| Method | CIFAR100 |  |  |\n|:---:|:---:|:---:|:---:|\n|  | 5 tasks | 10 tasks | 20 tasks |\n| CLOVER | 69.9 | 69.7 | 67.7 |\n| SEED | 71.1 | 69.9 | 68.2 |\n| FeCAM | 64.8 | - | - |\n\n**W3: Compared with other incremental learning methods that incorporate self-supervised learning (e.g., [i]), what are the specific advantages of the self-supervised approach used in this paper? [i] Prototype Augmentation and Self-Supervision for Incremental Learning. CVPR2021.**\n\nBoth SSL and CDSSL increase training difficulty by introducing task-agnostic classes, thereby encouraging the model to extract discriminative features by distinguishing them. This strategy effectively mitigates the issue of representation overlap. However, in SSL, the task-agnostic classes generated through rotation exhibit significant semantic differences from the raw classes (see Fig. 1a and Fig. 7), limiting their effectiveness. Unlike traditional SSL (e.g., [1~4]), CDSSL generates highly confusable task-agnostic classes, prompting the model to differentiate them from the raw classes. This method promotes more effective discriminative feature extraction and reduces representation overlap, particularly between old and new classes, as demonstrated in Fig. 4, Fig. 6, and Table 11 of the paper.\n\n[1] Prototype Augmentation and Self-Supervision for Incremental Learning. CVPR 2021.\n\n[2] Diffusion model meets non-exemplar class-incremental learning and beyond. Arxiv 2024.\n\n[3] Hybrid rotation self-supervision and feature space normalization for class incremental learning. Information Sciences 2025.\n\n[4] Semantic alignment with self-supervision for class incremental learning. Knowledge-Based Systems 2023.\n\n**Q1: Here are many symbols in the figures whose meanings are confusing; the figures should be self-contained (e.g., different class names in Fig. 1).**\n\nIn Figs. 1a, b of the paper, circles represent two original classes. Diamonds denote task-agnostic classes generated via 90° rotation, squares indicate those generated via 180° rotation, triangles correspond to task-agnostic classes created by permuting RGB channels to GBR, and stars represent classes derived from noise input. Symbols with the same color indicate that they originate from the same raw class and its associated task-agnostic variants generated through label augmentation. \n\nThank you for your comment. We will revise this figure in future versions to eliminate any potential confusion and will also review other figures to ensure clarity and avoid ambiguity.\n\n**Q2: The meaning of \"weighted ensemble\" has not been specifically explained in the abstract or introduction.**\n\nThank you for your valuable suggestion. The term “weighted ensemble” refers to our multi-expert model that incorporates the PWP strategy. The PWP effectively mitigates task data imbalance in NECIL by bridging the experience gap among experts. It dynamically adjusts expert weights based on both the number of predictable classes for each expert and its predictive ability, resulting in enhanced performance. Detailed implementation information regarding this weighted ensemble method can be found in Section 3.4. As suggested, we will integrate the concept of weighted ensemble into both the abstract and introduction in future revisions to improve clarity."}, {"review_id": "N2iYdeWBtx", "reviewer": "Reviewer_Sosb", "summary": "This paper presents CLOVER, which is a new framework for non-exemplar class incremental learning (NECIL for short) containing two key technical modules, namely confusion-driven self-supervised learning (CDSSL) and a progressively weighted ensemble scheme (PWP). Following the basic framework of SEED [39], CDSSL additionally applies more challenging augmentations like color channel swapping and noise injection, in order to improve representation discrimination. Also, PWP adjusts ensemble weights over time to mitigate prediction degradation due to the insufficient accuracy of newly added experts. In the standard benchmark datasets in NECIL, CLOVER shows clear performance gains.", "questions": "1. Can you provide a more principled characterization or metric that generalizes the notion \"confusable\" beyond the specific augmentations used?\n2. You claim that CDSSL reduces representation overlap between old and new classes, yet the evidence is limited to KL divergence trends and t-SNE plots. Have you considered using direct quantitative measures of inter-class distance or feature separation to strengthen this claim?\n3. The progressively weighted prediction strategy uses a linear increase in expert weights up to 1/K. What is the rationale behind this schedule? Have you considered adaptive alternatives (e.g., accuracy-based), or is there evidence that the linear schedule is near-optimal?\n4. Given that CDSSL uses cross-entropy loss over synthetic class labels, how do you distinguish its benefits from simply performing supervised learning on augmented data?\n5. Given that your method builds on the SEED framework, can you provide controlled comparisons isolating the gain from CDSSL versus changes to the ensemble structure itself?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "(Strengths)\n1. The proposed CDSSL introduces a novel use of confusable transformations to enhance representation separability. While the use of rotation in self-supervised learning is standard, incorporating RGB channel permutation and Gaussian noise as task-agnostic class generators is new in the NECIL context. This transformation-based framing offers a unique angle on augmenting discriminative power.\n2. Experimental results are consistently strong across benchmarks and task settings, except for several cases such as underperforming FeCAM in Table 1.\n3. The paper includes thorough ablations and supporting analysis.  The ablation study effectively isolates the contributions of each component, enabling a clear assessment of their individual impact on overall performance.\n\n(Weaknesses)\n1. The notion of confusion is only weakly formalized and lacks theoretical justification. While the term \"confusable classes\" is used repeatedly, there is no principled definition of what makes a class confusable, nor a formal link between confusion and representation quality beyond empirical KL divergence trends.\n2. The progressively weighted ensemble strategy is heuristic and lacks adaptivity. The PWP method linearly adjusts weights using fixed hyperparameters without consideration of expert reliability, prediction confidence, or class coverage. The ceiling at 1/K and linear increase lack theoretical support.\n3. Most contributions are extensions or recombinations of existing ideas.\nCDSSL builds directly on PASS-style self-supervision and SEED-style ensembles. The architectural structure and loss design are largely inherited, and the novelty lies primarily in reinterpreting these components under a new framing.\n4. Overlap reduction is asserted rather than directly measured.\nThe claim that CDSSL reduces representation overlap is supported indirectly (via KL divergence and t-SNE) but is not quantitatively measured through metrics such as inter-class distance or class boundary sharpness.", "was_revised": true, "final_justification": "Most concerns have been addressed, and the results are convincing. The remaining gap is a general, theoretically grounded notion of confusion/confusability, which is central to the method’s framing. I maintain a borderline rating but lean toward acceptance, and encourage the authors to add a formalization or generalization discussion in the camera-ready.", "rebuttal": "We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.\n\n**W1 & Q1: The notion of confusion is only weakly formalized and lacks theoretical justification. While the term \"confusable classes\" is used repeatedly, there is no principled definition of what makes a class confusable, nor a formal link between confusion and representation quality beyond empirical KL divergence trends. Can you provide a more principled characterization or metric that generalizes the notion \"confusable\" beyond the specific augmentations used?**\n\nWe observe that methods based on frozen feature extractors, which do not involve training on new classes, tend to yield dispersed representation distributions for these new classes. Consequently, these representations often overlap or become conflated with those of previously learned classes, thereby obscuring inter-class decision boundaries and ultimately impairing model performance. \n\nRepresentation quality refers to the model’s ability to extract class-specific and discriminative representation, that is, its capacity to distinguish between different classes within the representation space. In CDSSL, confusable classes are task-agnostic categories generated through transformations such as color channel permutation and noise injection. These transformations introduce relatively minor semantic alterations to the original images, resulting in representations that are susceptible to confusion with those of raw classes—hence the term “confusable.” \n\nIn summary, our method integrates these highly confusable task-agnostic classes into the training process, effectively increasing learning complexity. This approach encourages the model to extract more discriminative representations, thereby enhancing inter-class separability, sharpening decision boundaries, and ultimately improving the model’s representational capacity.\n\n**W2 & Q3: The progressively weighted ensemble strategy is heuristic and lacks adaptivity.**\n\nPWP is designed to compensate for disparities in the predictive capabilities of individual experts within an ensemble model, thereby fully harnessing the diversity of their knowledge. In fact, we previously investigated adaptive weighting strategies, such as those based on the number of covered classes and boundary clarity. However, we found that these approaches did not yield improved performance and introduced unnecessary computational complexity. In multi-expert ensemble prediction, key considerations include each expert’s predictive ability and the range of classes they can reliably identify. PWP achieves significant performance improvements by linearly adjusting expert weights in a straightforward yet effective manner, thus eliminating the need for more complex adaptive weighting schemes.\n\n**W3: Most contributions are extensions or recombinations of existing ideas. CDSSL builds directly on PASS-style self-supervision and SEED-style ensembles. The architectural structure and loss design are largely inherited, and the novelty lies primarily in reinterpreting these components under a new framing.**\n\nAlthough there are some works (e.g, PASS, SEED), we believe there are fundamental differences between our method and existing approaches.\n\n**First**, in traditional SSL methods such as PASS, the task-agnostic classes generated through rotation exhibit significant semantic differences from the raw classes (see Fig. 1a), which limits their effectiveness. In contrast, our CDSSL generates highly confusable task-agnostic classes, encouraging the model to distinguish them from the raw classes. This approach facilitates more effective discriminative feature extraction and reduces representation overlap, particularly between old and new classes, as demonstrated in Figs. 4 and 6, as well as Table 11 of the paper.\n\n**Second**, we propose PWP to compensate for disparities in the predictive capabilities of individual experts within an ensemble model, thereby fully leveraging the diversity of their knowledge. As illustrated in Fig. 1c, this approach effectively mitigates the influence of unreliable experts during the early training stages and significantly improves the model’s performance in the initial prediction phase, enabling CLOVER to maintain strong predictive ability throughout the entire training process.\n\n**Third**, PWP and CDSSL operate at distinct yet complementary levels within the ensemble framework. while CDSSL enhances the training of individual experts by fostering the development of discriminative representations, PWP improves collaborative inference among these experts. Experimental results presented in Table 2 of the paper further confirm that the combined performance of CDSSL and PWP surpasses that of their individual contributions alone, underscoring their complementary and synergistic strengths.\n\n**Finally**, extensive experiments conducted on three NECIL benchmarks have shown the superior performance of the proposed method over state-of-the-art approaches.\n\n**W4 & Q2: Overlap reduction is asserted rather than directly measured. The claim that CDSSL reduces representation overlap is supported indirectly (via KL divergence and t-SNE) but is not quantitatively measured through metrics such as inter-class distance or class boundary sharpness. Have you considered using direct quantitative measures of inter-class distance or feature separation to strengthen this claim?**\n\nThank you for your valuable comment. In the representation space, each class is modeled as a probability distribution utilized for classification. Consequently, the degree of confusion between classes cannot be assessed solely based on inter-class distances. Instead, it should be evaluated by examining the extent of overlap and indistinguishability among class distributions. To this end, we employ the **Bhattacharyya distance** as a metric to quantify the overlap between two probability distributions, where greater overlap corresponds to a smaller distance. Less overlap results in a larger distance. This metric effectively reflects the degree of separability or clarity of boundaries between classes. The Bhattacharyya distance is computed as follows:\n\n${D_B} = \\frac{1}{8}{({\\mu _1} - {\\mu _2})^T}{\\Sigma ^{ - 1}}({\\mu _1} - {\\mu _2}) + \\frac{1}{2}ln(\\frac{{|\\Sigma |}}{{\\sqrt {|{\\Sigma _1}||{\\Sigma _2}|} }})$,\n\nwhere $\\mu_1, \\mu_2$ denote the mean vectors of the two class distributions, $\\Sigma_1, \\Sigma_2$ are the corresponding covariance matrices, $\\Sigma = \\frac{1}{2}(\\Sigma_1 + \\Sigma_2)$ represents the average covariance matrix, and $|\\cdot|$ denotes the determinant.\n\nBased on this metric, we track the average class-boundary clarity throughout the training process under the 5-task setting of CIFAR-100, as summarized in Table 1. The results show that CDSSL effectively improves class separability by increasing the Bhattacharyya distance, thereby reducing confusion and overlap between class representation distributions.\n\n**Table 1. Evolution of average Bhattacharyya distance during training under the 5-task setting on CIFAR-100.**\n\n| Method |  |  | Task |  |  |  |  |  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Baseline | SSL | CDSSL | 0 | 1 | 2 | 3 | 4 | 5 |\n| ✔ | 　 | 　 | 7.62  | 6.75  | 6.19  | 5.76  | 5.38  | 5.05  |\n| ✔ | ✔ | 　 | 10.17  | 9.34  | 8.85  | 8.41  | 8.06  | 7.74  |\n| ✔ | 　 | ✔ | 12.12  | 11.02  | 10.34  | 9.77  | 9.31  | 8.89  |\n\n**Q4: Given that CDSSL uses cross-entropy loss over synthetic class labels, how do you distinguish its benefits from simply performing supervised learning on augmented data?**\n\nCDSSL employs a self-supervised learning strategy that utilizes label augmentation instead of conventional data augmentation. While traditional data augmentation enhances robustness against perturbations, label augmentation significantly improves the model’s representational capacity by encouraging it to distinguish between raw and augmented data instances. \n\nIn frozen feature extractor-based methods, representational capacity is of utmost importance. Specifically, the ability to extract the most discriminative representations for each class is crucial for enhancing incremental learning performance. \n\nConsequently, we adopt a self-supervised learning approach rather than relying solely on data augmentation. Furthermore, Table 2 presents a comparison of performance gains achieved under identical transformations when applied exclusively for data augmentation. The results clearly indicate that our label augmentation-based strategy yields markedly superior improvements.\n\n**Table 2. Average accuracy comparison of the effects of CDSSL and data augmentation on CIFAR-100.**\n\n| Method | CIFAR100 |  |  |\n|:---:|:---:|:---:|:---:|\n|  | 5 Tasks | 10 Tasks | 20 Tasks |\n| Baseline | 69.9 | 69.7 | 67.7 |\n| Baseline+CDSSL | 72.1 | 71.7 | 70.5 |\n| Baseline+DataAugment | 70.5 | 70 | 67.5 |\n\n**Q5: Given that your method builds on the SEED framework, can you provide controlled comparisons isolating the gain from CDSSL versus changes to the ensemble structure itself?**\n\nAs suggested, we have evaluated the performance of CDSSL in a single-model context to isolate its contribution without the influence of ensemble methods, as shown in Table 3. The results indicate that even without multi-expert collaboration, CDSSL alone can significantly enhance the model’s capacity to learn discriminative representations, leading to marked improvements in incremental learning performance.\n\n**Table 3. The effectiveness of CDSSL in the single-model settings.**\n\n| Method | CIFAR100 |  |  |  |  |  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|  | 5 Tasks |  | 10 Tasks |  | 20 Tasks |  |\n|  | Avg | Last | Avg | Last | Avg | Last |\n| SinguleBaseline | 67.7 | 59.8 | 68.1 | 59.9 | 66.5 | 56.9 |\n| SinguleBaseline+CDSSL | 70.2 | 62.4 | 70 | 62 | 68.4 | 59.4 |"}, {"review_id": "kByNwm9AQz", "reviewer": "Reviewer_KrMA", "summary": "This paper presents CLOVER, a framework for Non-Exemplar Class Incremental Learning (NECIL) that addresses the representation overlap problem. It introduces two main components: Confusion-Driven Self-Supervised Learning (CDSSL), which enhances representation robustness via challenging augmentations, and Progressively Weighted Prediction (PWP), which gradually adjusts the influence of expert models. A shared encoder is frozen after the first task, and each new class is modeled with Gaussian distributions by task-specific experts. During inference, predictions are made via a Bayesian ensemble. Experiments on CIFAR100, Tiny ImageNet, and ImageNet-Subset show state-of-the-art performance.", "questions": "1.What’s the difference between your proposed method and SEED?\n\n2.The training and inference process are demonstrated in Sec. 3.2, but the training process when t > K-1 wasn’t specified in the paper. Is it the same as SEED? Or stop training after learning the K-1 task? This should be clearly specified. \n\n3.There’s only one loss function employed during the training stage. Are there any other constraints or regularizations that are utilized to mitigate catastrophic forgetting? \n\n4.As shown in Table 1, CEAT and FGKSR are based on the ViT structure, while the other methods are based on ResNet18. Is this comparison fair? What is the comparison of the trainable parameters between CEAT, FGKSR and other ResNet18-based methods?\n\n5.The evaluation metric that was employed in the experiments should be clearly specified in the paper. Which accuracy metric was used in Table 2 and Figure 4?\n\n6.What about the performance of the proposed method on an equally split task setting?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1.This paper is generally well-written and well-motivated. It’s important and interesting to mitigate catastrophic forgetting from “representation overlap” perspective.\n\n2.This paper proposed a method in NECIL which combined with SSL and ensemble learning, achieving state-of-the-art performance in the proposed evaluation protocol.\n\n3.This paper offers extensive experimental analysis and results, demonstrating the effectiveness of the method and the significance of different components of the proposed method.\n\nWeakness:\n\n1.The method's reliance on a substantial amount of classes for the base task is a notable constraint, as its efficacy has not yet been validated in scenarios where all incremental tasks contain an equal amount of classes.\n\n2.The novelty of this paper is limited: self-supervised learning, ensemble learning, and frozen feature extractors are established techniques in CIL.\n\n3.This paper doesn’t provide a clear demonstration and detailed explanation of the learning process, especially the training stage.\n\n4.The training cost of the proposed method is relatively high, stemming from the expansion of the original N-way classification problem to a 7N-way problem.\n\n5.The comprehensiveness of the experimental comparisons could be further strengthened. Specifically, it would be beneficial to include empirical evaluations against several state-of-the-art NECIL approaches [a,b] which are published in recent years if possible.\n\n[a] Magistri, Simone, Tomaso Trinci, Albin Soutif, Joost van de Weijer, and Andrew D. Bagdanov. \"Elastic Feature Consolidation For Cold Start Exemplar-Free Incremental Learning.\" In The Twelfth International Conference on Learning Representations.\n\n[b] Rypeść, Grzegorz, Sebastian Cygert, Tomasz Trzciński, and Bartłomiej Twardowski. \"Task-recency bias strikes back: Adapting covariances in exemplar-free class incremental learning.\" Advances in Neural Information Processing Systems 37 (2024).", "was_revised": true, "final_justification": "Thank you for the detailed response from the authors. The feedback addresses some of my concerns. However, I remain unconvinced by several aspects. For instance, the performance drop with PSP in the equally split task setting is not well justified—the explanation that PSP is designed with a base task in mind is insufficient. Additionally, the novelty of the proposed approach appears limited, as similar ideas have been explored in previous work. Moreover, the training cost remains a significant concern, both in terms of time and GPU resources. Overall, I believe the paper still requires substantial improvement.", "rebuttal": "We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.\n\n**W1 & Q6: Performance validation on an equally split task setting.**\n\nIn NECIL, two primary data partitioning strategies are employed. The first strategy, which is utilized in our work as well as by many others, involves a base task that encompasses a substantial number of classes, followed by incremental tasks that each contain an equal number of classes. The second strategy distributes the total number of classes evenly across all tasks without designating a specific base task. These two setups emphasize different aspects of the model’s performance. The former prioritizes the stability of previously acquired knowledge during incremental updates, while the latter emphasizes the model’s plasticity in acquiring new knowledge. These differences reflect distinct evaluation priorities in incremental learning. \n\nTable 1 reports the performance of our method under the class-balanced setting, demonstrating the effectiveness and generalizability of the proposed CDSSL. Notably, CDSSL achieves a 6.4% improvement in average accuracy under the long-task setting. In this configuration, each expert is trained on a same number of classes, resulting in relatively balanced predictive capabilities across experts. However, our proposed PWP strategy is specifically designed for scenarios involving a base task, which results in suboptimal performance within class-balanced settings. Nonetheless, it is important to emphasize that our method, which relies exclusively on CDSSL, surpasses several robust baselines. This further validate the robustness and adaptability of CDSSL across various incremental learning scenarios.\n\n**Table 1. Comparison of the average accuracy (%) of different methods under the class-balanced setting.**\n\n|Method|CIFAR100|||\n|:---:|:---:|:---:|:---:|\n||5 Tasks|10 Tasks|20 Tasks|\n|EFC|-|60.87|55.78|\n|SEED|70.9|69.3|62.9|\n|Baseline|71.1|66.7|60.3|\n|Baseline+CDSSL|73.2|71|66.7|\n|Baseline+CDSSL+PWP|71.8|70.2|66.4 |\n\n**W2: The novelty of this paper is limited: self-supervised learning, ensemble learning, and frozen feature extractors are established techniques in CIL.**\n\nAlthough there are some works (e.g, PASS, SEED), we believe there are fundamental differences between our method and existing approaches.\n\n**First**, in traditional SSL methods such as PASS, the task-agnostic classes generated through rotation exhibit significant semantic differences from the raw classes (see Fig. 1a), which limits their effectiveness. In contrast, our CDSSL generates highly confusable task-agnostic classes, encouraging the model to distinguish them from the raw classes. This approach facilitates more effective discriminative feature extraction and reduces representation overlap, particularly between old and new classes, as demonstrated in Figs. 4 and 6, as well as Table 11 of the paper.\n\n**Second**, we propose PWP to compensate for disparities in the predictive capabilities of individual experts within an ensemble model, thereby fully leveraging the diversity of their knowledge. As illustrated in Fig. 1c, this approach effectively mitigates the influence of unreliable experts during the early training stages and significantly improves the model’s performance in the initial prediction phase, enabling CLOVER to maintain strong predictive ability throughout the entire training process.\n\n**Third**, PWP and CDSSL operate at distinct yet complementary levels within the ensemble framework. while CDSSL enhances the training of individual experts by fostering the development of discriminative representations, PWP improves collaborative inference among these experts. Experimental results presented in Table 2 of the paper further confirm that the combined performance of CDSSL and PWP surpasses that of their individual contributions alone, underscoring their complementary and synergistic strengths.\n\n**Finally**, extensive experiments conducted on three NECIL benchmarks have shown the superior performance of the proposed method over state-of-the-art approaches.\n\n**W3 & Q2: Provide a clear demonstration and detailed explanation of the learning process, especially the training stage.**\n\nAs suggested, we have provided detailed pseudocode to illustrate the training procedure of CLOVER (see Algorithm 1). We will incorporate and refine this pseudocode in subsequent versions.\n\n**Algorithm 1**: Training stage of CLOVER.\n\n```\nInput: The number of task T, maximum expert number K, class sets C_t, training samples D_t= {(x_i, y_i)}^t\nOutput: ensemble model θ(f·[e_0,e_1,...,e_(K – 1)]), Gaussian distribution {G(µ_c^e , Σ_c^e)}\n\n1: for t ∈ {0,1,2,...,T} do\n2:\tif t < K then\n3:\t\tCreate a new expert f·e_t and corrsponding linear classification head g_t\n4:\t\ttrain θ(f·e_t·g_t) by minimizing L from Eq.4\n5:\t\tfreezing θ(f·e_t) and discard g_t\n6:\tend if\n7:\tfor e∈{0,1,…,t} do\n8:\t\tCalculation and save Gaussian distribution G(µ_c^e , Σ_c^e) for c∈C_t\n9: \tend for\n10: end for\n11: return θ(f·[e_0,e_1,...,e_(K – 1)]) , {G(µ_c^e , Σ_c^e)}\n```\n\n**W4: The training cost of the proposed method.**\n\nTable 2 shows a detailed breakdown of the training overhead. As highlighted in the Limitations section, the training cost of CLOVER is relatively high, but overall, the computational burden is still within an acceptable range compared to the performance improvement it brings.\n\n**Table 2. Comparison with SEED and FeCAM in terms of training time, inference cost, and GPU memory usage on CIFAR100.**\n\n|Method|Training time per epoch|Avg forward pass time|GPU usage|\n|---|---|---|---|\n|CLOVER|103s|165ms|38,377MB|\n|SEED|23s|165ms|7,427MB|\n|FeCAM|17s|27ms|2,569MB|\n\n**W5: Include empirical evaluations against several state-of-the-art NECIL approaches (like EFC and AdaGauss) which are published in recent years if possible.**\n\nThese two papers employ a different data partitioning strategy than ours. We have already compared our method with state-of-the-art NECIL approaches under the class-balanced setting  in Table 1. Nevertheless, we will consider including a discussion of EFC, AdaGauss, and other leading NECIL methods in a subsequent version as necessary.\n\n**Q1: What’s the difference between your proposed method and SEED?**\n\nThe core strategy of SEED lies in its adaptive selection of the most suitable expert for fine-tuning during training. In contrast, we only draw inspiration from SEED’s ensemble learning framework, without adopting its adaptive fine-tuning mechanism. In our method, each expert is frozen once trained (see Algorithm 1 for training details), which constitutes a fundamental difference from SEED. \n\nMoreover, whereas SEED maintains fully independent parameters across its five experts, we choose to share a portion of parameters. This allows for the transfer of the robust representational capacity acquired by the first expert to the others. The effectiveness of this parameter-sharing strategy is empirically validated in Table 9 of our paper.\n\nBuilding upon this design, we further introduce CDSSL and PWP, which operate at distinct yet complementary levels within the ensemble framework. While CDSSL enhances the training of individual experts by fostering the development of discriminative representations, PWP improves collaborative inference among these experts. Experimental results in Table 2 of the paper demonstrate that the combined use of CDSSL and PWP leads to performance that surpasses either component in isolation, highlighting their complementary and synergistic contributions.\n\n**Q3: Are there any other constraints or regularizations to mitigate catastrophic forgetting?**\n\nWe do not adopt additional constraints or regularization techniques to mitigate catastrophic forgetting, as such methods are often sensitive to hyperparameters and yield inconsistent results. In contrast, our proposed CDSSL effectively alleviates catastrophic forgetting and achieves consistently strong performance across different datasets, as demonstrated in Figs. 4 and 6 of the paper. \n\nMoreover, our PWP strategy further addresses this issue by coordinating the predictive capabilities of multiple experts. By compensating for the limited knowledge of individual experts, PWP enables the model to leverage diverse knowledge sources for recognizing new classes while simultaneously preserving information about old ones from multiple perspectives, thereby providing an additional effective approach to mitigating catastrophic forgetting. \n\n**Q4: CEAT and FGKSR are based on the ViT structure, while the other methods are based on ResNet18. Is this comparison fair? What is the comparison of the trainable parameters between CEAT, FGKSR and other ResNet18-based methods?**\n\nThe number of trainable parameters in CEAT and FGKSR is 10.89MB and 9.3MB, respectively, whereas methods based on ResNet-18 typically require 11.2MB. Overall, the parameter scales are comparable, making the comparisons relatively fair. Most importantly, both CEAT and FGKSR achieve performance levels that are close to the state-of-the-art and are based on ViT architectures. We include them in our comparison primarily to ensure completeness and diversity in the set of baselines, rather than to create an unfair comparison.\n\n**Q5: The evaluation metric that was employed in the experiments should be clearly specified in the paper. Which accuracy metric was used in Table 2 and Figure 4?**\n\nThank you for your insightful comment. The evaluation metrics in Tables 2, 3, 7, 8, 10, 11, 12, and 13 are all based on average accuracy. The first row of Fig. 4 shows the accuracy variation curve of the model on the first task, while the second row presents the accuracy variation curve on the new task (current task). In subsequent versions, we will incorporate references within the main text to guide readers to the detailed formulas and explanations presented in Appendix A. Additionally, we will clearly specify the evaluation metrics in all figures and tables."}]

Author rebuttal:

Meta-review: This paper proposes CLOVER, a framework for non-exemplar class incremental learning (NECIL), introducing two main components: Confusion-Driven Self-Supervised Learning (CDSSL) to enhance feature discrimination through challenging augmentations, and a Progressively Weighted Prediction (PWP) strategy to balance expert contributions in ensemble inference. Its strengths include novel integration of highly confusable classes for representation learning and strong empirical results across standard NECIL benchmarks. Weaknesses involve limited theoretical grounding for "confusion," loose coupling between modules, and higher computational costs. During rebuttal, authors addressed concerns by adding quantitative inter-class distance metrics, clarifying baseline fairness, and providing cost analyses. While some conceptual gaps remain, the convincing empirical improvements and comprehensive responses justify acceptance.
Decision: Accept (poster)