Title: AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Continual Learning

Class-Incremental Learning

Contrastive Learning

Extreme Learning Machines
Abstract: This paper studies the problem of class-incremental learning (CIL), a core setting within continual learning where a model learns a sequence of tasks, each containing a distinct set of classes. Traditional CIL methods, which do not leverage pre-trained models (PTMs), suffer from catastrophic forgetting (CF) due to the need to incrementally learn both feature representations and the classifier. The integration of PTMs into CIL has recently led to efficient approaches that treat the PTM as a fixed feature extractor combined with analytic classifiers, achieving state-of-the-art performance. However, they still face a major limitation: the inability to continually adapt feature representations to best suit the CIL tasks, leading to suboptimal performance. To address this, we propose AnaCP (Analytic Contrastive Projection), a novel method that preserves the efficiency of analytic classifiers while enabling incremental feature adaptation without gradient-based training, thereby eliminating the CF caused by gradient updates. Our experiments show that AnaCP not only outperforms existing baselines but also achieves the accuracy level of joint training, which is regarded as the upper bound of CIL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAnalytical Contrastive Projection (ACP) tackles class-incremental continual learning by using a frozen pre-trained model (PTM) with an entirely closed-form feature-adaptation pipeline. It first projects PTM features into a higher-dimensional random space, then applies an analytical contrastive projection that pulls same-class embeddings together and pushes different-class embeddings apart in the closed form solutions type, without any gradient-based training of the PTM.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths – \n1. Contrastive projection layer that is trainable in a non-iterative closed loop form.\n2. Pseudo replay features to train final ELM classifier, requiring synthetic replay but only through closed-form updates.\n3. Extensive experiments conducted with different PTM backbones and datasets.\n\nWeakness – \n1. Through the strong generalization capabilities of PTM, why do they still require replay of samples.\n2. Banking heavily on a prior work RANPAC[1] and adding the ACP layer on top of the well performing RANPAC. \n3. Ablation studies should take into consideration of removing random projections and then comparing the performance. \n4. It is unclear how contrastive projections is different from negative-repulsion.\n[1] RanPAC: Random Projections and Pre-trained Models for Continual Learning (Mark McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, Anton van den Hengel)\n\nQUESTIONS:\nI will be happy to update the scores if the authors can effectively address the following questions. \n1.\tWhy are exemplars required despite the strong generalization capabilities of the pre-trained models?\n2.\tCan you compare the performance with and without random projections in the ablation studies?\n3.\tHow is negative repulsions different from the underlying mechanism of contrastive learning containing both positive and negative data? (SimCLR[1] for example)\n4.\tWhy isn’t forgetting performance compared? This gives better insight into the performance numbers, helping understand the stability-plasticity tradeoff. \n\n[1] A Simple Framework for Contrastive Learning of Visual Representations\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the class-incremental learning (CIL) and proposes ACP (Analytical Contrastive Projection) to enable incremental feature adaptation while maintaining computational efficiency through analytical classifiers. ACP is implemented in a gradient-free method and achieves strong performance on various CIL benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1.\tThis paper is well-written, and the motivation is clear.\n2.\tThis paper proposes a new way to improve the performance of analytical classifier methods, which are limited by the fixed pre-trained model and linear classifier.\n3.\tExperiments have demonstrated the effectiveness of the proposed ACP on various datasets and settings.\n\nWeakness\n1. Positive Alignment\n- Improvement in Method Description Clarity: While the mathematical formulation of the proposed method appears rigorous, the current presentation significantly increases the cost of understanding for readers. Additionally, redefining the left-hand side of Eq. 9 as $ H = Z^\\top P$ may improve clarity (only suggestion, depending on the authors).\n- Dependence on Pretrained Model: The strategy enhances the RP features by class means of the input layer, aiming to enhance the discriminative power of the analytical classifier. However, this approach is inherently dependent on the quality of the pretrained model (PTM), especially when dealing with cross-domain or fine-grained data. As shown in the experiments, the method primarily benefits from strong self-supervised models such as DINO-v2 and MoCo-v3. Briefly, this strategy does not fundamentally improve the separability of features but rather leverages already well-structured representations.\n\n2. Negative Repulsion\n- Lack of Intuitive Explanation for Lemma 4.1: The construction of $\\delta_i$ is not clearly explained. What specific information or property does this diagonal matrix encode?.\n- Need for Discussion on Relationship with Neural Collapse: It would also be valuable to discuss how the proposed negative repulsion mechanism relates to or differs from the phenomenon of *neural collapse*, which describes the tendency of features within each class to collapse to a fixed simplex structure during training.\n\nMinor:\nPlease carefully check the language and grammar. For example, in Line 3, “Traditional CIL methods does not use” should be “Traditional CIL methods do not use”.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles continual class-incremental learning with pretrained models\n\nSTRENGTHS AND WEAKNESSES:\nStrengths. The paper tackles a relevant problem of interest to the continual learning audience. The manuscript overall is clearly written with good coverage and clear discussion on the literature. The proposed method is clear (though could be better motivated, see below). The experimental results look strong and the paper has a nice set of ablation studies. \n\n\nWeaknesses. My first major concern is about the comparison regarding RanPAC, one of the most relevant approaches. There are a few implementation differences in the provided RanPAC implementation and its original implementation. RanPAC sets ridge parameter by cross validation and sets dimension $D$ to $10,000$, and uses ReLU as its activation. The paper sets smaller dimension $D=5000$, and  sets a fixed number 100 for both RanPAC and the proposed method, and the proposed method instead uses GELU. It would make more sense and make the paper more convincing if their setups are unified with the same activation (with experiments on both ReLU and GELU), and if RanPAC uses its original dimension and original cross validation approach.\n\nThe second major concern is on the technical development. Section 4 reads like a cookbook and the intuition or motivation behind the strategy proposed is often unclear at least to me. More specifically:\n- What is the motivation to regress a target prototype which is defined to be the class means? If regressing the class mean is a good goal, why don't we just output the class mean of that layer? Note that we can compute the class mean directly. We want to regress the one-hot label as there is no explicit map from features to labels, but we do have an explicit map from features to their means.\n- Why are whitening and de-whitening steps essential? I imagine that we can do SVD directly on C and correct its singular values. It is unclear to me why the whitening is applied on the left in Eq 11 and de-whitening is on the right at Eq 16. Could the authors explain this in more detail? \n- I am confused around Eq 13 and Lemma 4.1. What's that intuition behind Eq 13? Namely, why adding a diagonal shift would reduce cosine similarity? Further, how does this condition on $\\delta_i$ arise? How large $\\alpha$ is, and what is the rationale of choosing $\\alpha=1$? Can Lemma 4.1 be proved for $\\alpha=1$? In general I find a detailed explanation of Lemma 4.1 on its significance is lacking.\n\n\n\nFinally, it would be nice if the authors compare the following baselines either in text or in experiments:\n- HidePrompt, which seems to work very well among prompt based methods (https://proceedings.neurips.cc/paper_files/paper/2023/hash/d9f8b5abc8e0926539ecbb492af7b2f1-Abstract-Conference.html). \n- LoRanPAC, which seems to be a stronger alternative of RanPAC (https://openreview.net/forum?id=bqv7M0wc4x)\n- InfLoRA, which is a representative adapter-based method and the paper seems to compare no methods from this category (https://openaccess.thecvf.com/content/CVPR2024/html/Liang_InfLoRA_Interference-Free_Low-Rank_Adaptation_for_Continual_Learning_CVPR_2024_paper.html)\n\nQUESTIONS:\n- How to guarantee the covariance in Eq 10 is strictly positive definite? This is important as in Eq 11 (which has a typo), its inverse is needed to be computed.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for class-incremental learning (CIL), called Analytical Contrastive Projection (ACP), which bridges the gap between the efficiency and stability of analytical methods and the adaptability of feature learning in continual learning settings. ACP leverages a pre-trained model (PTM) as a frozen feature extractor and proposes an analytical, non-gradient-based mechanism for feature adaptation using a Contrastive Projection (CP) layer. The method analytically aligns samples from the same class while increasing inter-class separation via prototype regression and class mean repulsion and integrates an ELM classifier for final prediction. The authors demonstrate through comprehensive experiments on several benchmark datasets that ACP not only outperforms prior analytical CIL methods and rehearsal-free baselines but also reaches or surpasses the performance of joint training in some cases. Ablation studies and efficiency analyses further support the method’s design and scalability.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe paper addresses a highly relevant problem in CIL, while maintaining their computational advantages. The proposed ACP method is conceptually elegant and grounded in strong mathematical formulations, using closed-form solutions to approximate contrastive learning effects. The methodology is clearly described and logically structured. The experimental section is thorough, covering multiple datasets, PTMs (DINO-v2 and MoCo-v3), and a wide array of baselines including recent prompt-based and analytical methods. Results consistently show significant improvements in both average and final accuracy, with ACP performing on par with joint fine-tuning in many scenarios. The ablation studies are comprehensive and confirm the contributions of individual components like the CP layer, class mean repulsion, and the ELM classifier.\n\n## Weaknesses\n\nDespite its merits, a few limitations of the paper deserve further clarification. First, the analytical derivation of class mean repulsion using Lemma 4.1 is mathematically dense and lacks an intuitive geometric explanation or visual illustration, which may limit accessibility to a broader audience. Second, while the proposed pseudo-replay strategy is efficient and empirically effective, the justification for using a single shared covariance matrix across all classes is not fully addressed, especially considering that this shared covariance is used to generate pseudo samples. It remains unclear how the method distinguishes between samples from different tasks (e.g., task A vs. task B) during pseudo-replay, or whether such differentiation is even required under the proposed framework. A deeper discussion of why this approximation works and what its potential limitations are would be beneficial. Finally, although the methodology is described in detail, some parts of the pipeline remain difficult to follow without visual aid. In particular, the main figure (Figure 1) could be improved by clearly showing how the various matrices introduced in Section 4 (e.g., Gram matrices, projection matrices, and components of the pseudo-replay mechanism) are constructed and used during learning.\n\nQUESTIONS:\n- Could you provide a more intuitive or geometric explanation of Lemma 4.1? A visual illustration showing how the angular separation between class means is improved would significantly enhance accessibility.\n- Could you elaborate on why using a single shared covariance matrix across all classes works well for pseudo-replay? Have you observed any failure cases or dataset-specific limitations when using this approximation?\n- Since the shared covariance is also used during pseudo-sample generation and classification, how are labels assigned to the generated samples, and how robust is this process across tasks?\n- Would you consider updating or expanding Figure 1 to more clearly show how the matrices described in Section 4 are constructed and applied across tasks? For example, depicting the flow of Gram matrices, projection heads, and ELM updates could greatly help clarify the method.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}