Title: Confusion-Driven Self-Supervised Progressively Weighted Ensemble Learning for Non-Exemplar Class Incremental Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Non-Exemplar Class Incremental Learning

Representation Learning

Self-Supervised Learning

Ensemble Model
Abstract: Non-exemplar class incremental learning (NECIL) aims to continuously assimilate new knowledge while retaining previously acquired knowledge in scenarios where prior examples are unavailable. A prevalent strategy within NECIL mitigates knowledge forgetting by freezing the feature extractor after training on the initial task. However, this freezing mechanism does not provide explicit training to differentiate between new and old classes, resulting in overlapping feature representations. To address this challenge, we propose a **C**onfusion-driven se**L**f-supervised pr**O**gressi**V**ely weighted **E**nsemble lea**R**ning (*CLOVER*) framework for NECIL. Firstly, we introduce a confusion-driven self-supervised learning approach that enhances representation extraction by guiding the model to distinguish between highly confusable classes, thereby reducing class representation overlap. Secondly, we develop a progressively weighted ensemble learning method that gradually adjusts weights to integrate diverse knowledge more effectively, further minimizing representation overlap. Finally, extensive experiments demonstrate that our proposed method achieves state-of-the-art results on the CIFAR100, TinyImageNet, and ImageNet-Subset NECIL benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents CLOVER, a framework for Non-Exemplar Class Incremental Learning (NECIL) that addresses the representation overlap problem. It introduces two main components: Confusion-Driven Self-Supervised Learning (CDSSL), which enhances representation robustness via challenging augmentations, and Progressively Weighted Prediction (PWP), which gradually adjusts the influence of expert models. A shared encoder is frozen after the first task, and each new class is modeled with Gaussian distributions by task-specific experts. During inference, predictions are made via a Bayesian ensemble. Experiments on CIFAR100, Tiny ImageNet, and ImageNet-Subset show state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1.This paper is generally well-written and well-motivated. It’s important and interesting to mitigate catastrophic forgetting from “representation overlap” perspective.\n\n2.This paper proposed a method in NECIL which combined with SSL and ensemble learning, achieving state-of-the-art performance in the proposed evaluation protocol.\n\n3.This paper offers extensive experimental analysis and results, demonstrating the effectiveness of the method and the significance of different components of the proposed method.\n\nWeakness:\n\n1.The method's reliance on a substantial amount of classes for the base task is a notable constraint, as its efficacy has not yet been validated in scenarios where all incremental tasks contain an equal amount of classes.\n\n2.The novelty of this paper is limited: self-supervised learning, ensemble learning, and frozen feature extractors are established techniques in CIL.\n\n3.This paper doesn’t provide a clear demonstration and detailed explanation of the learning process, especially the training stage.\n\n4.The training cost of the proposed method is relatively high, stemming from the expansion of the original N-way classification problem to a 7N-way problem.\n\n5.The comprehensiveness of the experimental comparisons could be further strengthened. Specifically, it would be beneficial to include empirical evaluations against several state-of-the-art NECIL approaches [a,b] which are published in recent years if possible.\n\n[a] Magistri, Simone, Tomaso Trinci, Albin Soutif, Joost van de Weijer, and Andrew D. Bagdanov. \"Elastic Feature Consolidation For Cold Start Exemplar-Free Incremental Learning.\" In The Twelfth International Conference on Learning Representations.\n\n[b] Rypeść, Grzegorz, Sebastian Cygert, Tomasz Trzciński, and Bartłomiej Twardowski. \"Task-recency bias strikes back: Adapting covariances in exemplar-free class incremental learning.\" Advances in Neural Information Processing Systems 37 (2024).\n\nQUESTIONS:\n1.What’s the difference between your proposed method and SEED?\n\n2.The training and inference process are demonstrated in Sec. 3.2, but the training process when t > K-1 wasn’t specified in the paper. Is it the same as SEED? Or stop training after learning the K-1 task? This should be clearly specified. \n\n3.There’s only one loss function employed during the training stage. Are there any other constraints or regularizations that are utilized to mitigate catastrophic forgetting? \n\n4.As shown in Table 1, CEAT and FGKSR are based on the ViT structure, while the other methods are based on ResNet18. Is this comparison fair? What is the comparison of the trainable parameters between CEAT, FGKSR and other ResNet18-based methods?\n\n5.The evaluation metric that was employed in the experiments should be clearly specified in the paper. Which accuracy metric was used in Table 2 and Figure 4?\n\n6.What about the performance of the proposed method on an equally split task setting?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents CLOVER, which is a new framework for non-exemplar class incremental learning (NECIL for short) containing two key technical modules, namely confusion-driven self-supervised learning (CDSSL) and a progressively weighted ensemble scheme (PWP). Following the basic framework of SEED [39], CDSSL additionally applies more challenging augmentations like color channel swapping and noise injection, in order to improve representation discrimination. Also, PWP adjusts ensemble weights over time to mitigate prediction degradation due to the insufficient accuracy of newly added experts. In the standard benchmark datasets in NECIL, CLOVER shows clear performance gains.\n\nSTRENGTHS AND WEAKNESSES:\n(Strengths)\n1. The proposed CDSSL introduces a novel use of confusable transformations to enhance representation separability. While the use of rotation in self-supervised learning is standard, incorporating RGB channel permutation and Gaussian noise as task-agnostic class generators is new in the NECIL context. This transformation-based framing offers a unique angle on augmenting discriminative power.\n2. Experimental results are consistently strong across benchmarks and task settings, except for several cases such as underperforming FeCAM in Table 1.\n3. The paper includes thorough ablations and supporting analysis.  The ablation study effectively isolates the contributions of each component, enabling a clear assessment of their individual impact on overall performance.\n\n(Weaknesses)\n1. The notion of confusion is only weakly formalized and lacks theoretical justification. While the term \"confusable classes\" is used repeatedly, there is no principled definition of what makes a class confusable, nor a formal link between confusion and representation quality beyond empirical KL divergence trends.\n2. The progressively weighted ensemble strategy is heuristic and lacks adaptivity. The PWP method linearly adjusts weights using fixed hyperparameters without consideration of expert reliability, prediction confidence, or class coverage. The ceiling at 1/K and linear increase lack theoretical support.\n3. Most contributions are extensions or recombinations of existing ideas.\nCDSSL builds directly on PASS-style self-supervision and SEED-style ensembles. The architectural structure and loss design are largely inherited, and the novelty lies primarily in reinterpreting these components under a new framing.\n4. Overlap reduction is asserted rather than directly measured.\nThe claim that CDSSL reduces representation overlap is supported indirectly (via KL divergence and t-SNE) but is not quantitatively measured through metrics such as inter-class distance or class boundary sharpness.\n\nQUESTIONS:\n1. Can you provide a more principled characterization or metric that generalizes the notion \"confusable\" beyond the specific augmentations used?\n2. You claim that CDSSL reduces representation overlap between old and new classes, yet the evidence is limited to KL divergence trends and t-SNE plots. Have you considered using direct quantitative measures of inter-class distance or feature separation to strengthen this claim?\n3. The progressively weighted prediction strategy uses a linear increase in expert weights up to 1/K. What is the rationale behind this schedule? Have you considered adaptive alternatives (e.g., accuracy-based), or is there evidence that the linear schedule is near-optimal?\n4. Given that CDSSL uses cross-entropy loss over synthetic class labels, how do you distinguish its benefits from simply performing supervised learning on augmented data?\n5. Given that your method builds on the SEED framework, can you provide controlled comparisons isolating the gain from CDSSL versus changes to the ensemble structure itself?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes a confusion-driven self-supervised progressively weighted ensemble learning framework for NECIL task. Self-supervised learning and ensemble learning are applied to significantly improve performance. Extensive experimental results on CIFAR100, TinyImageNet and ImageNet-Subset benchmarks demonstrate its superior performance.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The motivation is simple and efficient.
2. The experimental results are outstanding.

Weaknesses：
1. Self-supervised learning and ensemble learning are general methods for improving representations, so why are they used to address the specific task of NECIL?
2. As shown in Figure 3, the baseline performance of the proposed method is already much higher than that of other methods. Could this be the main reason for the overall superior incremental performance? Using the same baseline performance or comparable performance drop indicators could be considered to enable a fairer comparison.
3. Compared with other incremental learning methods that incorporate self-supervised learning (e.g., [i]), what are the specific advantages of the self-supervised approach used in this paper?
[i] Prototype Augmentation and Self-Supervision for Incremental Learning. CVPR2021.

QUESTIONS:
1. Here are many symbols in the figures whose meanings are confusing; the figures should be self-contained (e.g., different class names in Fig. 1).
2. The meaning of "weighted ensemble" has not been specifically explained in the abstract or introduction.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CLOVER, a novel framework for non-exemplar class incremental learning (NECIL), which aims to sequentially learn new classes without retaining data from previous tasks. To address the challenge of representation overlap between old and new classes, a common issue when the feature extractor is frozen, the authors introduce two key components. First, a confusion-driven self-supervised learning (CDSSL) module generates challenging task-agnostic classes using transformations such as color channel swapping and noise injection, encouraging the model to learn more discriminative features. Second, a progressively weighted prediction (PWP) strategy gradually increases the influence of newly added ensemble experts, mitigating performance degradation caused by their initial unreliability. Experimental results on CIFAR100, TinyImageNet, and ImageNet-Subset demonstrate that CLOVER achieves strong performance across various incremental learning settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed Confusion-Driven Self-Supervised Learning (CDSSL) module introduces task-agnostic classes using color channel permutation and noise injection, enhancing representation discrimination and mitigating feature overlap between old and new classes.\n2. The Progressively Weighted Prediction (PWP) strategy effectively addresses a common issue in ensemble-based NECIL methods: performance degradation when newly added experts are weak. By gradually increasing expert weights across tasks, it improves ensemble reliability without requiring retraining.\n3. CLOVER achieves state-of-the-art performance across multiple NECIL benchmarks. Ablations and analyses support its claims.\n\nWeaknesses:\n1. The proposed two core modules are functionally independent and loosely coupled. CDSSL focuses on representation learning during training, while PWP controls prediction aggregation at inference. There is no interaction, joint optimization, or design dependency between them. This makes the method feel more like a stacked combination of two standalone techniques, rather than a coherent, integrated approach.\n2. The paper’s title “Confusion-Driven...” suggests that both components are driven by confusion modeling, but only CDSSL fulfills this. PWP, while effective, is a heuristic weighting scheme that neither exploits confusion nor adapts to inter-class uncertainty. This overclaims conceptual coherence and may mislead readers into overestimating the role of confusion modeling in the ensemble component.\n3. Confusion-based SSL and color transformation augmentations have already been explored in NECIL contexts [1-3]. These works also use perturbation-based or proxy-label-based augmentation to enhance fine-grained feature discrimination. CLOVER builds incrementally on these ideas, and the degree of novelty in CDSSL is therefore limited.\n4. Ensemble learning has been extensively used in NECIL to reduce forgetting[4-5], but a well-known challenge is the linear growth in model parameters and memory usage. CLOVER inherits this problem without proposing a mitigation strategy, where each task adds a new expert, leading to scalability concerns in long-task or high-capacity models.\n[1] Semantic Alignment With Self-Supervision for Class Incremental Learning, KBS 2023.\n[2] Non-Exemplar Class-Incremental Learning by Random Auxiliary Classes Augmentation and Mixed Features, TCSVT 2024.\n[3] Hybrid Rotation Self-Supervision and Feature Space Normalization for Class Incremental Learning, Information Science 2025.\n[4] CoSCL: Cooperation of Small Continual Learners is Stronger Than a Big One, ECCV 2022.\n[5] Expandable Subspace Ensemble for Pre-trained Model-based Class-Incremental Learning, CVPR 2024.\n\nQUESTIONS:\n1. The actual spatial relationship between raw and transformed classes remains unclear. Maybe t-SNE visualizations that directly compare the embedding distributions of raw classes versus task-agnostic classes could explain this.\n2. From Table 3 and related ablations, the performance gain from rotation-based SSL appears greater than that of color channel permutation. This is counterintuitive if the argument is that color transformations produce more confusing task-agnostic classes.\n3. Why do some augmentations like CenterCrop in Table 12 perform well on CIFAR-100 but poorly on TinyImageNet?\n4. Please provide measurements of training time per task, inference cost (e.g., FLOPs or forward pass time), and GPU memory usage. How do these compare to baseline methods like SEED or FeCAM?\n\nLIMITATIONS:\nYes, the authors include a dedicated section (Appendix F) where they discuss the limitations of their work, including the reliance on a strong base task, increased training cost, parameter growth.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}