Random Anchors with Low-rank Decorrelated Learning: A Minimalist Pipeline for Class-Incremental Medical Image Classification
Authors: Xinyao Wu
Zhe Xu
Raymond Kai-yu Tong
Venue: ICLR 2026 Poster (2026)
Keywords: Medical Image Classification; Feature Calibration; Continual Learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Random Anchors with Low-rank Decorrelated Learning (RA-LDL), a minimalist yet surprisingly powerful representation-based pipeline for class-incremental medical image classification.

Abstract: Class-incremental learning (CIL) in medical image-guided diagnosis requires models to preserve knowledge of historical disease classes while adapting to emerging categories. Pre-trained models (PTMs) with well-generalized features provide a strong foundation, yet most PTM-based CIL strategies, such as prompt tuning, task-specific adapters and model mixtures, rely on increasingly complex designs. While effective in general-domain benchmarks, these methods falter in medical imaging, where low intra-class variability and high inter-domain shifts (from scanners, protocols and institutions) make CIL particularly prone to representation collapse and domain misalignment. Under such conditions, we find that lightweight representation calibration strategies, often dismissed in general-domain CIL for their modest gains, can be remarkably effective for adapting PTMs in medical settings. To this end, we introduce Random Anchors with Low-rank Decorrelated Learning (RA-LDL), a minimalist representation-based framework that combines (a) PTM-based feature extraction with optional ViT-Adapter tuning, (b) feature calibration via frozen Random Anchor projection and a single-session-trained Low-Rank Projection (LRP), and (c) analytical closed-form decorrelated learning. The entire pipeline requires only one training session and minimal task-specific tuning, making it appealing for efficient deployment. Despite its simplicity, RA-LDL achieves consistent and substantial improvements across both general-domain and medical-specific PTMs, and outperforms recent state-of-the-art methods on four diverse medical imaging datasets. These results highlight that minimalist representation recalibration, rather than complex architectural modifications, can unlock the underexplored potential of PTMs in medical CIL. We hope this work establishes a practical and extensible foundation for future research in class-incremental image-guided diagnosis. Code will be made publicly available.

Published as a conference paper at ICLR 2026
RANDOM ANCHORS WITH LOW-RANK DECORRELATED LEARNING: A MINIMALIST PIPELINE FOR CLASS-
INCREMENTAL MEDICAL IMAGE CLASSIFICATION
Xinyao Wu!
Zhe Xu Raymond Kai-yu Tong"
l "Department of Biomedical Engineering, The Chinese University of Hong Kong Data Science Institute, Columbia University
xxinyacwu, jackxz) 8link. cuhk edu. bk; kytonggcunkk edu. hk
ABSTRACT
Class-incremental learning (CIL.) in medical image-guided diagnosis requires mod- els preserve knowledge of historical disease classes while adapting to emerging categories. Pre-trained models (PTMs) with well-generalized features provide a strong foundation, yet most PTM-based CIL strategies, such as prompt tuning, task-specific adapters and model mixtures, rely on increasingly complex designs. While effective in general-domain benchmarks, these methods falter in medical imaging, where low inter- and intra-class variability and high inter-domain shifts (from scanners, protocols and institutions) make CIL particularly prone to rep- resentation collapse and domain misalignment. Under such conditions, we find that lightweight representation calibration strategics, often dismissed in general- domain CIL. for their modest gains, be remarkably effective for adapting PTMs in medical settings. To this end, we introduce Random Anchors with Low-rank Decorrelated Learning (RA-LDL), minimalist representatiion-base framework that combines (a) PTM-based feature extraction with optional ViT-Adapter tuning, (b) feature calibration via frozen Random Anchor projection and single-session- trained Low-Rank Projection (LRP), and (c) analytical closed-fone form decorrelated learning. The entire pipeline requires gradient- optimization only in the first session; subsequent tasks rely solely on efficient analytic classifier updates based on recursively accumulated statisties, making appealing for efficient deployment. Despite its simplicity, RA-LDL achieves consistent and substantial improvements across both general-domat and medical-specific PTMs, and outperforms recent state-of-the thtt-o methods on four diverse medical imaging datasets. These results highlight that minimalist representation recalibration, rather than complex architec- tural modifications,
unlock the underexplored potential of PTMs in medical CIL. We hope this work establishes a practical and extensible foundation for future research in class-incremental image-guided diagnosis.
INTRODUCTION
In real-world medical settings, diagnostic models must continually accommodate newly emerging disease categories while preserving expertise on previously learned conditions. This evolving need highlights the importance of class-incremental learning (CIL) for scalable medical image-guides diagnosis. In the general domain, CIL has undergone a paradigm shift with the advent of pre- trained models (PTMs), leveraging the generalized feature representations learned from large-scale datasets to enable more efficient and effective incremental learning. Unlike conventional models trained sequentially from scratch, PTMs inherently possess generalized representations, allowing for improved knowledge retention and faster adaptation to new classes with minimal task task-specific tuning.
Existing paradigms in the PTM-based CIL can be categorized into three fashions as shown in Fig. [ prompt-based, representation-basel and model mixture-based methods. While these methods attempl
Equaal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
Protoggets
Prompt Selection L2P: QquryyKay Marchhig
ADAM
LAE, MDS
Irput
W
Input
Ture task aciapter
Input
Cross peoroogie-mspppnng
DusiPromytt Export Genorol From
X000 2 SAATE
PTM
PIM.
A
13 W
Online Macel
Offlnne Modal
Gomaie usi protstepe
Tume reeeii-petiii adapoers
Adapsivity
Gerverdizaaliityy
CodisPrompt Arnention ho Combination
SSSUT: Eatimate protodyte snat Sontuple generoood foonures
Ensembic
PTM
Rep buneed tradtrann
task sspeoiii adapners
Prediction
(4) Prompt-hased
(b) Repr based
Figure 1: Overview of recent paradigms in the PTM-based CIL. (a) Prompt-based methods (e.g., L2P (Wang et al. 2022c), DualPrompt (Wang et al., 2022b), CodaPrompt (Smith et al. 2023)) dynamically construct and insert eettacc-sppeiffee prompts to guide the frozen PTM. (b) Representation based methods (e.g., ADAM (Zhou et al. (2024a). EASE (Zhou et al. (024b), SSIAT (Tan et al. (024)) focus on adapter tuning and prototype-based classification or calibration. (c) Model Mixture-based
methods (e.g., LAE (Gao et al. (023), MOS (Sun et al. 2025)) combine different models to balance
adaptivity and generalizabilit.
to balance stability and adaptability, they often adopt increasingly complex designs. Prompt-based methods (Wang et al. 2022cc] by Smith et al. (023) aim to reuse frozen PT by introducing learnable context tokens that serve as dask-specific prompts. While elegant in theory, they face scalability challenges as the prompt pool grows over time. Managing prompt selection, updating old prompts, and ensuring compatibility between prompts and unseen classes require additional routing mechanisms, leading to increased memory and inference complexity. Moreover, prompts are often sensitive to distribution shifts, making them brittle in cross- medical scenarios. Representation based methods (Zhou et al. 2024b Tan et al. 2024) build the strong and conceptually simple bascline SimpleCII. (Zhou et al. 2024a).. and typically insert task-specific lightweight modules into the frozen PTM backbone do the representation decoration. Recent variants (Zhou al.) [022bb] Tan et al.) 2024; Sun et al. 2025) further increase complexity by introducing multiple task-specific adapters, requiring task-specific retrieval mechanisms at inference time, reminiscent of prompt-selection overhead. These designs also demand maintaining consistent mappings across evolving feature spaces, often using prototype reconstruction, which can become unstable under domain shift and difficult to scale. Model mixture-based methods (Sun et al. 2025 Gao et al. 2023) require additional mechanisms for effective model ensemble and large memory resources to store all historical models. Yet, this surge in architectural and algorithmic complexity presents a question: Do these sophisticated methods, successful in general benchmarks, generalize effectively to medical imaging domains?
Our study (Table 2) reveals that they often do not, Medical imaging poses unique challenges (dis- cussed in Appendix including low inter- and intra-class variability, where diseases share highly similar visual cues, and high inter-domain shifts arising from differences in scanners, protocols, and institutions. Under these conditions, complex mechanisms such as prompt routing, prototype reconstruction, adapter mixtures tend to collapse or misalign representations. Furthermore, the representation quality of PTMs is uneven. For example, we empirically found that Microsoft's BiomedCLIP (Zhang et al., (2023b) sometimes outperforms Google's ViT-B/16-IN2IK (Alexey (0200) in prototyee-based evaluations, yet other medical-specine PTMs such as UniMedCLIP (Khattak) al., 2024) and RAD-DINO (Pérez-García et al.) 2025) perform substantially worse, and even BiomedCLIP remains inconsistent across tasks (Table This reveals critical insight: domain specialization does not guarantee stronger medical representations. whereas well- well-generalized PTMs from the large-scale natural image domain often provide more reliable foundations. Accordingly, following prior
-omminn PTMM-based CII. studies (Zhou et al. 2024a| Tan et al., 2024] [Sun al. 2025), we still adopt VIT-B/16-IN2IK (Alexey) 2020) our primary PTM. Here, we wonder: Can well- rell-gennealizable PTMs be effectively adapted to medical CIL through minimalist representation recalibration strategies, rather than increasingly complex architectural and algorithmic designs?
To this end, we introduce Random Anchors with Low-rank Decorrelated Learning (RA-LDL), a minimalist yet powerful representation-bat based pipeline for tion. RA-LDL consists of three lightweight and synergistic steps (Fig. 2 each designed to resolve core challenge in PTM--based medical CIL:: (a) P'MM-based feature extraction with one-time
medical image classifica-
2

---

Published as a conference paper at ICLR 2026
adapter tuning: Medical images differ substantially from the distributions used in PTM pretraining, raising the challenge of how to adapt to downstream domains while preserving the generalizability of the foundational representation. To mitigate this domain gap, we apply one-time first-session ViT-Adapter update, which improves compatibility with medical images while retaining broad PTM representations. (b) Feature calibration via frozen random anchors and low-rank residual: Under domain shifts, PTM features often lack sufficient discriminability. This raises the key question: how can we make PTM-derived features more class-separable under domain shifts? We first apply a frozen Random Anchor (RA) projection to embed features into a randomized space, enhancing linear separability without additional training. Yet RA alone may be insufficient for severe domain-speciffe shifts. To address this, we introduce Low-Rank Projection (LRP), trained only in the first session, that provides a parameter-efficient residual correction. LRP explicitly cali- brates residual distortions from domain shift while complementing the frozen RA, yielding features that are both discriminative and robust. (c) Analytica closed-forme decorrelated learning: Even with calibrated features, class prototypes in medical CIL often suffer from strong inter-class corre- lation, especially under domain shifts. Prototype-base based classifiers such as SimpleCII. (Zhou et al.) 2024a) demonstrate the generalization strength of PTM features, but their reliance on raw feature prototypes leaves them prone to representation collapse (Table To overcome this, we introduce the analytical learning perspective (Zhuang al. 2022). framing classifier construction as closed-form solution rather than iterative backpropagation. Specifically, we estimate classifier weights via ridge regression, whose closed-form solution depends on the feature autocorrelation (Gram) matrix and thus explicitly leverages second-orde statistics. This reweighting behaves like a whitening operation: down-wen nown-weights dominant and redundant directions, captures intra-class variations, and reduces inter-class correlations. The resulting decorrelated prototypes maintain discriminative power across tasks, align more closely with test features, and improve robustness under domain-shifted medical scenarios. Note that while some components have surfaced in general-domain CIL, to our knowledge, this work provides the first systematic analysis of their synergy and effectiveness in medical CIL. The pipeline needs gradient- gradient-based optimization only in the first session; subsequent tasks rely solely on efficient analytic classifier updates based on recursively accumulated statistics, making it attractive for deployment.
gigher-dimenssonal
Our main contributions are as follows:
We empirically reveal that while general-domain CIL methods increasingly rely on complex designs, these strategies often fail to generalize in medical imaging, where low inter- and intra-class variability and high inter-domain shifts exacerbate representation collapse. In contrast, simple representation calibration, although yielding only modest gains on general- domain benchmarks, achieves strong and robust performance in medical CIL, highlighting the underappreciated potential of PTMs.
We introduce RA-LDI., minimalist pipeline for class-incremental medical image clas- sification. Built on three lightweight representation-bant based steps, it combines theoretical analysis and empirical validation to demonstrate improvements in class separability, feature a adaptability, and generalizability within single training session.
We evaluate RA-LDL on four diverse medical CIL benchmarks, demonstrating substan- tial and consistent gains over state-of-the-art methods. The framework approaches the joint-training upper bounds while requiring gradient-based optimization only in the first session. Moreover, RA-LDL achieves consistent improvements with both general-domain and medical-specift PTMs, further confirming its robustness across different backbone choices.
2 RELATED WORK
Class-incremental Learning. Conventional CIL methods rely on three strategies: replaying prior data (Castro et al. 2018 Hou et al. 2019; Rebufli et al. 2017). regularizing adversarial parameter updates (Li& & Hoiem] 2017 Kirkpatrick et al.] 2017: Wang et al., 2022a), and using adaptive modules to update models dynamically (Abati et al. 2020: Yoon et al. 2017). However, these methods generally require models training from scratch and necessitate extensive parameter tuning, which limits their scalability especially in practical deployment. Recent advances in PTMs have revitalized CIL research by leveraging their generalized feature representations (Wang et al. 2022c b.
3

---

Published as a conference paper at ICLR 2026
Smith al. 2023 Zhou et al.
2024a Tan et al. 2024 Gao et al. 2023 Zhou al. 2024b Tan
et al. 2024 Sun et al., 2025). However, as discussed above, the effectiveness of these methods often hinges on architectural and algorithmic complexity, which hinders scalability and struggles to generalize consistently in cross-domain tasks such as medical imaging. Interestingly, recent evidence (Zhuang et al. 2024 McDonnell et al.) (024) suggests that even lightweight feature calibration mechanisms for instance, mapping features into suitably structured spaces-coe can yield strong incremental performance with minimal adaptation. While more sophisticated designs may outperform these strategies, they impressively reveal the untapped potential of PTMM-xtrrcted features for incremental learning. Remarkably, in medical imaging. where inter- and intra-class variability is low and inter-domain shifts are pronounced, we found that simple yet well-crafted feature calibration can achieve competitive CIL performance, some cases approaching the joint-training upper bound.
Class-incremental Learning in Medical Image Analysis. Previous work in medical CII. has primarily focused on training models from scratch, often relying on replay-based techniques or Darameter-inefficcintt full-model regularization (Perkonigg et al. [022] Byun al. [0233 Bai et al.
Yeganch et al. 2023] [Sadati et al. [023 Wu et al. 2025). With the emergence of PTMs offering strong generalization, the challenge should
2023; Chee et al. 2023 Wu et al. 2023; Li &
shift to effectively balancing plasticity and stability during continual adaptation. Yet,
CIL, in the medical domain remains underexplored, and existing medical PTMs still tend to be modality-specific with poor transferability across medical sub-domains. This work presents an early exploration of PTM-based medical CIL, showing that minimalist representation- based incremental learning pipeline can achieve excitingly strong performance.
3 PRELIMINARIES
3.1 PROBLEM SETUP
Class-incremental learning (CIL) aims to learn from a continual data stream where new classes are incrementally introduced. Formally, training set that belongs the incremental session can be represented as D' = [ }r,)) where E {1, 2, T'} represents the session index among T total session, and each session contains n, instances, Mark that each session contains a unique set of classes Y,, with no overlap between sessions: Yn
Ø for t & f'. The goal of CIL is to construct a model f(x) X + Vr. where 8/+ Yi U ... Y, that can learn new classes incrementally without forgetting previously learned ones. Following prior replay-free PTM--ased CIL studies (Tan et al. 2024, /Zhng et al., 2023a| Zhoo et al., 2024a|b), b)). we consider a pre-trained Vision Transformer (ViT) model available for initializing f(x). We then decompose into feature extraction backbone Fer.. and a linear classification layer fe... The backbone Forme extracts features from the input images, serving as a feature embedding function s(-) mapping RDD + Ro, while the classifier layer fea. represented by a weight matrix W E Rddxx(Ya, projects the feature embeddings to class predictions. The model can then be formularized as f(x) = W ((x), where W = [W1, W2, wyl, with w, denoting the classifier weights for class j. We treat the embedded [CLS] token as ((x) for VIT.
3.2 ADAPTIVITY AND GENERALIZABILITY
In PTM--bsed CIL, a key challenge lies in balancing adaptivity (the model's ability to learn new classes efficiently) and generalizability (the capacity of PTMs to transfer to downstream tasks without learning). Parameter-efficient tuning (PET) methods, such as Scaling and Shifting Fine-tuning (SSF), ViT-adapters and Visual Prompt Tuning (VPT), are widely adopted to enhance adaptivity by modifying and tuning only a small subset of parameters in the PTMs (details appendix B). During training, the majority of pretrained weights remain frozen, with only the PET components and classifier head being optimized. The set of trainable parameters defined as: = 0werr U Aw- While these techniques enable efficient adaptation to downstream domains, they often compromise generalizability, leading to forgetting of prior knowledge. To address this, recent studies introduce increasingly complicated strategies upon the SimpleCIL baseline (Zhou et al. (2024a). such as replay- based unified training (Tan et al. 2024) and cross-task prototype mapping (Zhou et al. 2024b). to better balance the trade-off. However, such complexity could hinder scalability and interpretability. and has empirically shown inconsistent or even controversial performance in medical CIL. This motivates us introduce minimalist pipeline.
4

---

Published as a conference paper at ICLR 2026
Table Overview of medical image classification datasets.
Dataset
Classes
Training set
Test set
Task Num.
Size
Covid (CTX (Wane
2025
11
3,939
1,072
Blood Acevedo et al 2017
8
11,965
5.127
360 363
Skin8 Tectand 2013
8
3,555
705
4
[600. 1024]
MedM Tang ar
36
302,002
75,659
28 28
3.3 MATERIALS AND IMPLEMENTATION
Since we will progressively combine theoretical analysis and empirical evidence to construct the RA-LDL pipeline step by step, we first introduce the datasets and implementation protocol.
Datasets, We involve four medical image classification datasets, as summarized in Table COVID (CT & X-ray) (Wang et al. 2023) contains both CT and X-ray scans for respiratory disease diagnosis. Blood (Acevedo et al. 2019) comprises normal peripheral blood cell images obtained from blood smear slides. Skin8 (Tschandl et al. 2018) focuses on skin lesion classification from dermatoscopic images and presents high class imbalance. MedMNISTV2 (Yang et al., 2021 2023) provides standardized benchmark with 12 2D and 6 3D biomedical datasets covering multi-class, multi-label, and ordinal regression tasks. We follow (Zhang et al. (023c) and adopt four classification datasets, BloodMNIST, ArNISA PathMNIST and TisSUe referred to as MedMNIST-SIn We the same data splits in (Bayasi et al. 2024 Zhang et al. 2023c), and each image is resized to 224 x 224 pixels during training. As supplementary validation, we also evaluate on general-domant benchmarks, including CIFAR100 (Krizhevsky al. 2009), CUB200 (Wah al.) 2011), ImageNet-A (IN-A) (Hendrycks et al. 2021b), and ImageNet-R (IN-R) (Hendrycks et al. 2021a).
Implementation. Our experiment is implemented PyTorch primarily utilize ViT-B/I6-IN2IK (Alexey) (2200) as the PTM, since general-domain PTMs remain the mainstream choice whereas medical-specific PTMs are still relatively underexplored. However, in Sec. 5.2, we analyze the sensitivity and comparative advantages of RA-LDL when applied to both general-domain and medical-sppe sppecifie PTMs. All experiments use batch size of 48, with 20 training epochs for the initial session and 15 epochs for each subsequent session. We employ SGD with momentum and use cosine-annealed learning rate starting 0.01. We apply random flipping and rotation for weak data augmentation to enhance generalization. Following TTan et al.) 2024 ZZang ctt al. 2023a; Zhou et al. 2024a b). we report the sinal-session accuracy AccLast and the average accuracy across all sessions, defined as ACCAve T Σ Acc. As the ultimate goal of CIL is to maintain knowledge after all tasks, Accias is the most critical metric, while Accong reflects stability throughout training. All methods are reproduced using the same seed for fair comparison. Code is available at htltttss//githhi com www/CUHK BOOHH-BBEAII CIRGAI/RRARR
NVIDIA A100 GPU. We
4 METHODOLOGY
Our proposed RA-LDL pipeline operates in three steps, as shown in Fig. 2 In step (a), we extract features from a frozen pre-trained backbone. A lightweight ViT adapter is optionally inserted to support task-specific adaptation, which is only trainable during the first task (1 = 1) and kept frozen in subsequent tasks 1). In step (b), we adopt frozen random anchor and learnable low-rank projection (LRP) to complementarily calibrate representations. In step (c), we perform the analytical closed-form decorrelated classification based on ridge regression. This pipeline maintains minimal architectural and training overhead while enabling controllable balance between downstream adaptability and PTM generalizability in medical continual learning.
4.1 FIRST-STAGE ADAPTATION AND ANALYTICAL DECORRELATED CLASSIFIER
We build our pipeline on top of an optional ViT-adapter-basee first-stage adaptation, which has been observed to provide stable performance, together with the analytical decorrelated classifier (Zhuang et al.) 2022). Details of alternative PET variants and classifier formulations are provided in Appendix B and Appendix
common choice in CIL is the class- plass-prototype (CP) classifier (Zhou et al. 2024a). which computes class means and applies the nearest-class-mean (NCM) principle (Mensink al. 2013). This
5

---

Published as a conference paper at ICLR 2026
Random Anchor
Google VIT-B/16-IN21K
Microsoft BiomedCLIP
Rdoxd1
RiLL
Image Input
PTM
A
W adc
Logíts
Medical Imaging
(t,,, 17:)
W; Rd
Actraation
Wg E grads
Relu
W Wade- (G *C,
LRP
Task 1-T
(t=1:00 (>::8)
VIT-Adapter
(Adapted) PTM-based Feature Extraction
b. Random Anchor & LRP
c. Analytical Decorrelated Classifler
Figure 2: Overview of our minimalist RA-LDL framework. The pipeline requires only a single training session and consists of three lightweight components: (a) PTM-based Feature Extraction with optional ViT-Adapter-basst based adaptation. (b) Random Anchor & LRP to calibrate representations. (c) Analytical Decorrelated Classifier based on ridge regression.
approach implicitly assumes that class embeddings are isotropic and well separated. In PTM-based medical CIL, however, prototypes often show strong inter-class correlation due to domain shifts, resulting in representation collapse and reduced discriminability. Addressing this issue requires a classifier that not only estimates prototypes but also explicitly reduces correlations between them. Motivated by this, we adopt the analytical learning perspective (Zhuang et al. (002), where classifier construction is formulated
ridge regression problem instead of relying on iterative optimization. Specifically, we assume that we have projected feature h(x) E Rd for each input in general. To capture second-order relationships and variance structure, we compute the feature autocorrelation (Gram) matrix across sessions to t: = E, E he, E IRd xd, and the class-accumulated
feature matrix: Cp E. En n4,n E IRdix/Ml The analytical classifier weights obtained by solving the ridge regression problem:
arg min || - HW adde 1 + + Wadel Wade
where E Nxd is the stacked matrix all projected sample features; Y R.Vx contains the corresponding one-hot labels; N E N₂ is the total number of samples accumulated up to the current task; || || is the Frobenius norm; and is the ridge regularization parameter selected by cross-validation-bade optimization. Differentiating the ridge regression objective in Eq. yields the closed-form solution (Appendix D). which can be compactly represented using the accumulated feature statistics from sessions to E:
Wade (G+ Cp d
where / is the identity matrix. The logits for projected feature h be obtained as = hW ade h(G + -31 RIX.. This formulation shows that Wade corresponds to set of analytically derived and decorrelated prototypes. The inverse Gram matrix (G+ 31) reweights eigendirections of the feature space, acting similarly to whitening: it suppresses dominant shared components (the source of prototype correlation), captures intra-class variations, and reduces inter-class correlations. This yields prototypes that are more discriminative across tasks and more robust to medical -peccffcc challenges such as low inter- and intra-class variability and high inter-domain shifts. Appendix [E] discusses the -rrvcc-preeervingg nature of maintaining only aggregated statistics. Formal proofs of the decorrelation property are provided in Appendix [F] together with empirical results showing that the analytical decorrelated classifier consistently outperforms the standard prototype-base classifier.
4.2 RANDOM ANCHORS
Recent studies (Zhuang et al. 2024 McDonnell et al. 2024) have shown that nonlinear transformation improve linear separability among expected class representatives. As shown in Fig. (b).

---

Published as a conference paper at ICLR 2026
adopt lightweight random anchor to transform frozen features while preserving their intrinsic structure. Specifically, we project features into dimensoaame spaces in a characteristics- preserving manner. We define a frozen random matrix B e R randomly initialized from Gaussian distribution (0, d²), and apply the transformation:
(x) = ReLL o(x)),
where do is the feature embedding dimension and di the projected dimension. d(x) ) is the extracted feature from the frozen pretrained model, and hRA(X) is the projected feature. The matrix B is initialized once and frozen throughout all incremental learning stages. This projection introduces minimal perturbation to the embedding space while improving class separability.
Proof of Feature Characteristics Preservation and Enhancement. In Appendix H. we formally lemma effectively preserve
prove that random anchor projections guided by the original feature characteristics and, when used to increase dimensionality, can potentially improve the structural and statistical properties of features extracted from pre-trained models.
Impact of Different Random Matrix. Given the diversity of weight initialization techniques, we conducted study to determine the optimal strategy for initializing the random matrix. Our evaluation includes widely-adopted methods, including Kaiming uniform/normal and Xavier uniform/normal distributions. As shown in Appendix G.T the standard Gaussian initialization demonstrates the most stable and consistently strong performance. While other methods yield competitive results in some datasets, they exhibit greater variance across tasks and datasets. In contrast, the Gaussian initialization reliably provides high performance without requiring fine-tuned hyperparameters, aligning well with our pipeline in a minimalist perspective.
Impact of Projection Dimension. To better evaluate the randomness introduced by the frozen random anchor and how it helps improve class prototype separability, we analyze how the projection dimension di influences performance. As shown in Appendix G.2 increasing di generally improves linear separability by expanding the feature space, and thus yields some performance gains. However, the effect is not uniform across datasets: while some benefit steadily from larger projections, others show fluctuations or diminishing returns. This variability reflects the fact that RA, though effective preserving global geometry and enhancing separability, does not explicitly account for domain- specific distortions. These observations motivate the need for a complementary adaptation branch: rather than relying solely on random projection, we introduce parameter-efficient Low-Rank Projection (LRP) to provide a residual correction. For all subsequent experiments, we set dj = 5do. which we find provides a sufficiently large feature space.
4.3 LOW-RANK PROJECTION LAYER
Random Anchors (RA) effectively preserve global feature geometry and improve separability through higher-dimensional embedding. However, RA alone does not explicitly address domain-specific variations, which are particularly pronounced in medical imaging where a substantial distribution gap between pre-training and downstream domains often exists. this regard, we introduce a Low-Rank Projection (LRP) trained in the first session. This module provides residual parameter-efffcient correction to the frozen RA pathway. We define the composite output as:
h(x)= = hxA(X) hx),(x),
where (x) = ReLU(B (((x) is the frozen RA projection and (xnr(x) RcLU(Act(o(x))) WT) W2) is the trainable low-rank residual. Here, W, do W2 E Rexd1 and < min do, dz). The activation function Act(-) is set to GELU (Hendrycks & Gimpel 2016). This design aims to preserve the core structure learned by the PTM through RA, while the LRP corrects domain-specific distortions in compact and regularized fashion. The residual structure enables the model to bridge the domain gap with minimal parameters and computation.
Proof of LRP residuel-induced Variance Reduction and Margin Enlargement. We further provide theoretical analysis in Appendix[] I proving that the LRP residual reduces intra-class variance while enlarging inter-class margins. With the low-rank constraint (r & min (dg, ), the parameter count of the residual correction ((do + di) r) is significantly smaller than the full linear layer (dgd1), remaining parameter-effiiient to alleviate overfitting.
Impact of Rank r. Appendix G.3 presents a sensitivity analysis examining how varying the rank affects the performance of our low-rank projection module. The rank determines the trade-off
7

---

Published as a conference paper at ICLR 2026
between model capacity and parameter efficiency by controlling the number of trainable parameters and the dimensionality of the learned correction. As shown in Table increasing the rank from to 256 yields only marginal improvements in accuracy. This indicates that moderate rank (e.g.. 64) sufficient to capture the dominant domain-specific variations required for effective adaptation, while preserving the parameter efficiency and regularization benefits of the low-rank design.
EXPERIMENTAL ANALYSIS
5.1 COMPARISON WITH STATE-OF-THE- OFE------TEEEEE ART METHODS
After progressively designing and validating each component, we integrate them into unified three-step RA-LDL pipeline. Table 2 compares our approach with recent PTM-based CIL methods on four challenging medical imaging benchmarks. We also include traditional CIL methods (Rebuffi) et a 2017: Wang et al. 2022a Yan et al., 202T); while they perform competitively on the Blood dataset, their reliance on data replay raises privacy concerns in clinical scenarios. Excitingly, RA-LDL achieves striking performance using minimalist representation recalibration strategies, highlighting its effectiveness in handling cross-domain tasks with heterogeneous data without using complicated architectures heavy loss regularization. Fig- further illustrates the performance trajectories across incremental sessions, where RA-LDL sustains more robust accuracy as class numbers grow, while competing methods show sharp declines, particularly Skin8 and COVID. Appendix GeanddxXG..4hhe shows that RA-LDL achieves comparably competitive performance on general-domain benchmarks, validating the generalizability of our minimalist pipeline beyond the medical domain. Appendix G.5 further shows that RA-LDL achieves the best overall performance our cross cross-dataset continual learning benchmark, surpassing recent PTMM-based CIL methods in both average and final accuracy. The last four rows of Table 2 present the ablation results where we remove first-stage adaptation and LRP residual. The performance exhibits mixed outcomes. This highlights the insights: the first-stage adaptation is optional, whereas the combination of the random anchor and the LRP residual plays complementary and essential role in balancing generalizability and adaptivity. Together, they form a minimalist yet effective pipeline for recalibrating representations in incremental downstream tasks, requiring minimal training overhead and offering excellent scalability without complex architectural designs.
Table Experimental results compared recent PTM--bssed CIL methods on four medical classi- fication benchmarks. "1" denotes reliance on replayed data. The best and second-best results are marked in red and blue, respectively.
Method
Actin (5)
Sub
Arcu. (8%
Antuy (9)
Saind
Amaua (5)
Artto (5)
COVID (AA-rats)
Arci,
Blood
Accia (5)
73.61
Finetume
FOSTER Wang al
2022
58.24
31.46
55.00
39.01
71.63
55.50
87.70
90.24
KCRRL Rab a
2017
68.23
38.44
58.45
39:57
70.10
65.21
$1.41
$1.57
DEK!
27.05
87.10
1,2
Wing o F026oc
56.24
28.96
35.80
20:05
85.46
76.15
DullFicopp Wilh 0
5022
$4.92
26.31
52.32
27.38
43.62
19.68
76.62
66.27
CodsPhamet Ninte
61.12
78.65
50.17
32.48
36.57
30.85
72.64
61.57
LAB Can FuM
45.46
18.35
49.52
24.00
46.26
21.83
55.51
33.94
SUCA 22noee Bk141
58.39
16,42
53.91
$0.71
63.05
60.82
90.23
92.29
SimpleCII: 2200 202435
68.07
$1.63
56.61
38.31
75.84
$7.57
333
79.79
ADAM-Adtier
70.97
53.11
50.82
41.84
79.
61.10
09
83.52
SSIAT Tin
59.43
25.79
11.99
72.00
60.17
86.00
81.63
BASE (1
74.59
Ongini arb)).-DL)
Adapied
97.97
Original BBLLR (RALLP
86.30
20.29
80.85
08.05
Adapted W/ B&ERP (RA-LDD))
84.79
70.69
75.66
62.49
98.07
97.76
Bocd Sumberr of
Cora Sumber Ouees
MEgHAST Cut number
CDVO Eart Sumber Of
Figure 3: The performance curves across learning sessions on four medical CIL benchmarks.
8

---

Published as a conference paper at ICLR 2026
5.2 SENSITIVITY ANALYSIS
Robustness to General-domain and Mediccl-specific PTMs. While our main experiments primarily adopt general-domain PTMs such as ViT-B/I6-IN2IK also generalizes to medical-specific PTMs. To this end, as shown in Table 3 we further evaluate RA-LDL on recent medical PTMs, including BiomedCLIP (Zhang et al., 2023b), UniMedCLIP (Khattak et al. 2024), and RAD-DINO (Pérez-García et al., (025). Interestingly, we find that features from these medical PTMs do not consistently outperform those from general-domain PTMs, suggesting that domain specialization does not guarantee stronger representations across diverse medical subsets. Nevertheless, RA-LDL consistently yields stable improvements across both general-domain and medical-specific PTMs, confirming its robustness to backbone choice. Moreover, we observe complementary strengths: when applied to medical PTMs, RA-LDL can better leverage domain-specifii priors; when applied to reneral-domaii PTMs, its lightweight recalibration effectively mitigates domain shifts. These results suggest that RA-LDL can serve as unifying framework unlock the potential of both general-domain and medical-specine PTMs for medical CIL. It is important to note, however, that all strategies rest on the assumed strong generalizability of PTMs: efficient adaptation alone cannot fully compensate for weak representations. Ulimately, the achievable ceiling remains bounded by the quality of the underlying feature extractor. We therefore believe that advances in PTM quality and CIL strategies should evolve together, jointly driving progress in medical continual learning.
important question is whether RA-LDL
Table 3: Performance comparison using different PTMs (medical-specific vs, general-domain),
Methed
FTM
Artup (15)
MraINNGTTaal
Art... (1)
Artiag (16)
Skint
Almu (56)
Arcive (S)
COVID (CT&A-mas)
Artive (16
Actou (1)
Blood
Artuue (16)
Ximpix
Zoure 28 b3900
15.10
25.1
15.25
15.71
31:221
29:21
15.01
SinpleCIL
RAD DINO
48.62
38.0]
34.82
21.56
57,09
46.60
43.14
25.20
SinpleCIL
0936
53.21
57.43
41.95
33.51
65.30
85.86
25.17
StepkcIL
68.07
58.61
5527
11.8
Asger
BioneCLIIP
70.88
53.45
WI-1115-IN11K
3097
5511
1958
41.84
79.19
A1.00
85.09
MD8 San rinea
5/10
12
"60
MOS
RAD-DINO
22.95
8.35
34.09
5.53
21.28
11.01
31.68
9.08
MOS STOELEI Pule
BomeeCLIP
78.07
6396
67.48
51.91
90.52
73.97
00.35
39.58
MOS Sugpett p005
MTT:V/1-INEIK
74.99
$1.40
68.54
51.73
89.06
50.00
02.53
90.18
R...
750
3077
00.35
350
7336
65.55
84552
RA-UDL
KAD-DINO
75.29
15.04
18001
79.53
68.00
53.79
BionedCLIP
24.33
70:04
73.71
RA
IN
75,06
62.49
Note: SimpleCIL is featur-quality Sheck haseline presested in (2hoa et
20024
Daing a frozen PTM with mesrest-llassmmenn classitier.
We further evaluate RA-LDL under reversed task sequence (Table 4 On COVID, Skin8, and Blood, both Accavg and AccLast vary by less than 1.5%, confirming strong order robustness. On MedMNIST-Sub, AcCazz shows a moderate drop (-1 11),, reflecting the high heterogeneity of tasks, yet importantly Accuast still improves by ~3%. Since the final-session performance is the most critical mcasure in continual learning, these results demonstrate that RA- LDL remains competitive even under unfavorable task orders. Overall, RA-LDL exhibits desirable robustness to task sequencing, a key property for realistic scenarios where task order is uncontrollable.
Sensitivity to Task Order:
Table 4: The performance of our RA-LDL using a reverse task order.
Method
Accwu (9))
COVID (-Taa)
Aco (5)
Acesu (%)
Skink
Acc (%)
Accou (36)
MedMNisT-Sab
Acc (%)
Acco, (1)
Blood
Aca (%)
RA-LDL
93.09
20.14
15.16
62.49
$4.79
95007
91.76
(rovers)
94.58
36.76
74,20
62.30
73.77
73.51
99.04
97.37
Robustness to the Number of Tasks. We further assess the impact of task granularity by increasing the number of tasks on MedMNIST-Suu from 4 to 8, comparing RA-LDL with representative baselines. As shown in Appendix G.6 RA-LDL maintains stable performance: Accang only changes from 84.79% (4 tasks) to 86.71 % (8 Tasks), and Accuss from 70.60% to 70.80%. This demonstrates RA-LDL's scalability and strong long-term continual learning potential.
CONCLUSION
In this paper, we introduced RA-LDL. minimalist pipeline for class-incrementee learning (CIL) in medical image classification that serves as a practical alternative to existing complex PTM-based designs. RA-LDI applies frozen random anchor projection and peeele-session-triinne low-rank

---

Published as a conference paper at ICLR 2026
projection calibrate extracted features, followed by analytical bigeerrereesion-baaee classifier that produces decorrelated prototypes and maintains discriminative power over time. Without relying medical-specific pretraining repeated task-specifie tuning, RA-LDL delivers consistently strong performance across four diverse medical datasets. Extensive experiments show that not only surpasses recent state-of-the-ar methods but also reveals the underexplored potential of simple representation refinement strategies in medical CIL. Looking ahead, RA-LDL opens promising directions for future work. Its minimalist design positions it as lightweight adapter for continual updates in emerging foundation models, while extensions beyond classification to tasks such as segmentation and detection can further demonstrate its scalability and impact.
ACKNOWLEDGMENTS
This research was partly supported by Research Impact Fund (R5039-23) from Research Grants Council of Hong Kong.
ETHICS STATEMENT
Our study does not involve human subjects, personally identifiable information, or sensitive clinical data. All datasets used are publicly available medical imaging benchmarks (e.g.. MedMNIST, Skin8, COVID-CT/X-raa, and Blood), which have been properly anonymized by their providers. We have carefully considered potential risks and believe that our work poses no foreseeable harm to patients or institutions. This research strictly adheres to the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
All implementation details, including model architectures, training hyperparameters, and data pre- processing steps, are described in the main paper and Appendix. Additional ablation studies and sensitivity analyses are provided in the supplementary material, and complete proofs of theoretical results are included in the Appendix. The source code, along with documentation and scripts to reproduce our experiments, will be released upon paper acceptance.
REFERENCES
Davide Abati, Jakub Tomezak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pp. 3931-3940, 2020.
Andrea Acevedo, Santiago Alférez, Anna Merino, Laura Puigví, and José Rodellar. Recognition of peripheral blood cell images using convolutional neural networks. Computer Methods and Programs in Biomedicine, 180:105020, 2019.
Dosovitskiy Alexey. An image worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:
2020.
Long Bai, Mobarakol Islam, and Hongliang Ren. Revisiting Distillation for Continual Learning on Visual Question Localize-AAnswering in Robotic Surgery. arXiv preprint arXiv:2307.12045 2023.
Nourhan Bayasi, Ghassan Hamarneh, and Rafeef Garbi. Continual zoo: Leveraging zoo models for continual classification of medical images. In IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pp. 4128-4133 2024.
Y. Byun, S. Garg, S. V. Mehta, P. Singh, J. Kalpathy- Cramer, B. Wilder, and Z. C. Lipton. Conditional Diffusion Replay for Continual Learning in Medical Settings. ICML, 2023.
Francisco M Castro, Manuel Marín- Jiméée, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision, pp. 233-248. 2018.
10

---

Published as a conference paper at ICLR 2026
Evelyn Chee, Mong Li Lee, and Wynne Hsu. Leveraging Old Knowledge to Continually Learn New Classes in Medical Images. arXiv preprint arXiv:2303.13752. 2023.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Yue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In Advances Neural Information Processing Systems, 2022.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang A unified continual learning framework with general parameter-efficiet tuning. In JEEE/CVF International Conference on Computer Vision, pp. 1 11883-1180 11493, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint urXiv: 08415, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of dfist---iittitttion generalization. pp. 8340-8340 8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15277 15271, 2021b.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, 2022.
M.U. Khattak, S. Kunhimon, M. Naseer, S. Khan, and F.S. Khan. Unimed-clip: Towards a uni- fied image-text pretraining paradigm for diverse medical imaging modalities. In arXiv preprint :XXi:24422.10322 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-335 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Chia-Hao Li and Niraj K. Jha. DOCTOR: A Multi-Disease Detection Continual Learning Framework Based on Wearable Medical Sensors. arXiv preprint arXiv:2305.057338 2023.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Paltern Analysis and Machine Intelligence, 40(12): 2017.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new bascline for efficient model tuning. In Advances in Neural Information Processing Systems, 2022.
Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. RANPAC: Random projections and pre-trained models for continual learning. Advances Neural Information Processing Systems, 36, 2024.
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-basee based image classification: Generalizing to new classes at near-zero cost. IEEE transactions on pattern analysis and machine intelligence, 25366)7222-2637, 2013.
Matthias Perkonigg, Johannes Hofmanninger, Christian J. Herold, James A. Brink, Oleg Pianykh, Helmut Prosch, and Georg Langs. Dynamic memory alleviate catastrophic forgetting in continual learning with medical imaging. Nature Communication, 12, 2021.
F. PrrezaGareia, H. Sharma, S. Bond-Taylor, K. Bouzid, V. Salvatelli, M. Ilse, S. Bannur, Castro D.C., A. Schwaighofer, M.P. Lungren, and M.T. Wetscherek. Exploring scalable medical image encoders beyond text supervision. In Nature Machine Intelligence, pp. 1-12, 2025.

---

Published as a conference paper at ICLR 2026
Sylvestre-ANN Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCARL: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Ario Sadafi, Raheleh Salchi, Armin Gruber, Sayedali Shetab Boushehri, Pascal Giehr, Nassir Navab, and Carsten Marr. A Continual Learning Approach for Moss-Domain White Blood Cell Classifi- cation. arXiv preprint arXiv:2308. /2679, 2023.
J.S. Smith, L. Karlinsky, V. Gutta, P. Cascante Bonitll,, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira. Coda-prompt: Continual decomposed attention-baser prompting for rehearsal-free continual learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11909-11919, 2023.
Hai-Long Sun, Da- Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Ye. MOS: Model Surgery for Pre- Trained Model-Based Vlass-Inccemmenal Learning. In Proceedings of the AAAI Conference Artificial Intelligence, 2025.
Yuwen Tan, Qinhao Zhou, Xiang Xiang, Ke Wang, Yuchuan Wu, and Yongbin Li. Semantically- Shifted Incremental Adapter-Tuning A Continual ViTransformer. JEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1):1-9, 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Boyuan Wang, Du Zhang, and Zonggui Tian. Corotrans-cl: A novel transformer-bast based continual deep learning model for image recognition of coronavirus infections. Electronics, 12(4))866, 2023.
FY Wang, DW Zhou, HJ Ye, and DC Zhan Foster. Feature boosting and compression for class- incremental learning. ECCV, 2022a.
Z. Wang, Z. Zhang, S. Ebrahimi, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister. DualPrompt: Com- plementary prompting for rehearsal-fiee continual learning. European Conference on Computer Vision, pp. 631-648, 2022b.
Z. Wang, Z. Zhang, C.Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Plister. Learning to prompt for continual learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139- 149, 2022c.
Huisi Wu, Zhaoze Wang, Zebin Zhao, Cheng Chen, and Jing Qin. Continual nuclei segmentation via prototype-wis relation distillation and contrastive learning. IEEE Transactions Medical Imaging, 2023.
Xinyao Wu, Zhe Xu, Donghuan Lu, Jinghan Sun, Hong Liu, Sadia Shakil, Jiawei Ma, Yefeng Zheng, and Raymond Kai-yu Tong. Conservative-radica complementary learning for caassinncrementll medical image analysis with pre-trained foundation models. In International Conference Medical Image Computing and Computer Assisted Intervention, 2025.
Shipeng Yan, Jiangwei Xie, and Xuming He. DER: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014-3023, 2021.
Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: lightweight automl benchmark for medical image analysis. In 2021 IEEE 18th International Symposium Biomedical Imaging (ISBI), pp. 191-195. IEBE, 2021.
Jiancheng Yang. Rui Shi, Donglai Wei. Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data, 10(1):41, 2023.
12

---

Published as a conference paper at ICLR 2026
Yousef Yeganeh, Azade Farshad. Goktug Guevercin, Amr Abu-zer, Rui Xiao, Yongjian Tang. Ehsan Adeli, and Nassir Navab. SCOPE: Structural Continuity Preservation for Medical Image Segmen- tation. arXiv preprint arXhv:2304.14572 2023.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically 2017.
expandable networks. arXiv preprint
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. SLCA: Slow Learner with Classifier Alignment for Continual Learning on Pre-trained Model. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2023a.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint ar 15X1v:2303.00915, 2023b.
W. Zhang, Y. Huang, T. Zhang, Q. Zou, W.S. Zheng, and R. Wang. Adapter learning in pretrained feature extractor for continual learning of diseases. arXiv preprint arXiv:2304.. 09042, 2023c.
Da- Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu, Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, 2024a.
Da- Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based dasssincremmntal learning. CVPR, pp. 23554-23564, 2024b.
Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Am Toh, and Zhiping Lin. Acil: Analytic dlass-incrementa learning with absolute memorization and privacy protection. Advances in Neural Information Processing Systems, 35:1 1602-11614, 2022.
Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. Ds-al: A dual- stream analytic learning for exemplar-free free class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17237-17244, 2024.
13

---

Appendix
Why Medical CIL Difters from -omaiin CIL
15
Parameter-ellicient Tuning (PET) Methods
15
Three Types of Classifiess
16
D
Closed-form Solution Derivation of Analytical Decorrelated Classitter
17
Privacy-preserving Nature of Analytical Decorrelated Classifier
17
Proof of the Decorrelation Property
17
Additional Experiments
18
Proof of Random Anchors for Feature Characteristics Preservation and Enhancement
20
Proof of LRP benefits
23
Limitations and Societal Impacts
23
LLM Usage
24
14

---

Published as a conference paper at ICLR 2026
A WHY MEDICAL CIL DIFFERS FROM GENERAL-DOMAIN CIL
critical distinction between continual learning in medical imaging and general-domain tasks lies in the intrinsic data characteristics. Two properties in particular make medical CIL more challenging:
(1) Low inter- and intra-class variability. medical imaging, samples within the same class (e.g., benign lesions, specific cell types) often appear highly homogeneous, especially after standard preprocessing. While this reduces within-class spread, it also compresses class prototypes into overly compact regions of the feature space. As new classes are introduced, these compact prototypes become highly correlated, increasing the risk of representation collapse where different classes cannot be reliably distinguished.
(2) High inter-domain shifts. Medical data are acquired across diverse scanners, protocols, and institutions, leading to substantial distribution shifts in texture, resolution, and intensity. Such variability fragments class distributions and destabilizes the alignment of prototypes, resulting in domain misalignment. The same class may occupy multiple disjoint regions in feature space, undermining the assumption of stable class anchors that many CIL. methods rely on.
Why matters. natural image benchmarks such as CIFARIO0 or ImageNet, intra-class variability is typically high: dog class can include many breeds, poses, and backgrounds, which encourages feature learning that captures broad semantic diversity. Domain shifts also exist (e.g., between natural photos, sketches in ImageNet-R, adversarial corruptions in ImageNet-A), but these shifts tend to be more semantic or stylistic in nature rather than stemming from systematic acquisition differences. In contrast, medical imaging presents the opposite challenge. Intra-class variability is often low, as samples from the same condition (e.g., red blood cells, COVID-positive CT scans) appear visually homogeneous, compressing class distributions in feature space. At the same time, domain shifts are severe and systematic, caused by differences in scanners, acquisition protocols, and institutions, which fragment same-class features into disjoint regions. This interplay compact intra-clas clusters and strong inter-domain scatter magnifies representation collapse and domain misalignment, explaining why methods effective in natural images may falter in medical CIL.
B PARAMETER-BIFICIESE TUNING (PET) METHODS
We briefly summarize the formulations of three PET methods commonly used in existing PTM-based
CIL methods:
2022). VPT introduces a set of learnable prompts P E Roxdo prepended to the input embeddings X., forming the extended input [P.xe], The pre- trained VIT backbone is frozen, and only the prompts and classification head are updated:
Visual Prompt Tuning (VPT) (Jia et al.,
θ
0p U@w.
(5)
Scale and Shift (SSF) (Lan al. 2022).
SSF applies learnable scaling and shifting to the input
of each transformer block. For each input X; E R L du the output is computed as:
X.
(6)
where 8 E Rd and @ is the Hadamard product. Only the SSF parameters and classifier are optimized:
= OsEF U U@w-
ViT-Adapter CCen al. 2022). Adapter inserts a bottleneck module into the MLP blocks of VIT. The module consists of down-projcction W Wdown k a ReLU activation function a, and an
up-projection Wap k Following (Tan et al. 2024 Zhou et al. 2024b), we augment the ViT's multilayer perceptron (MLP) layers with adapters with projected dimension k equal to 64 (Chen Let Tin be the input of the MLP layer, the output of the adapter-equipped MLP is:
et al. 2022)
Fout MLP(
(Nin Waown ) W up:
(
where * denotes matrix multiplication. During training, the ViT backbone remains frozen, and the optimizable parameters be denoted as: θ Owam. Ow.,. U Ow.
15

---

Published as a conference paper at ICLR 2026
C THREE TYPES OF CLASSIFIESS
This section provides detailed formulations for three types of classifiers
Fully-connected (FC) classifier. First, given the extracted feature ((x),, the softmax-based logits for the standard FC classifier can be formulated as:
W o(x).
(9)
where W E R represents the trainable weight matrix for N classes learnt so far, and = R contains the unnormalized scores for each class. These logits are typically passed through softmax explay)
where Py denotes the predicted probability function to obtain class probabilities: Py Σm exp(=.)) for class y.
Class prototype (CP)-based classifier. prototypes as:
The CP-based classifier (Zhou et al. 2024a) computes class
DD' Σ I(y; = i)0(1)). Ps K
(10)
where A' denotes the number of samples belonging to class within the current incremental learning batch 1 and T (y, = i) is an indicator function that equals if the sample æ; belongs to class i, and O otherwise. Following the nearest class mean (NCM) principle (Mensink et al.] 2013), classifier assigns new samples to the class whose prototype is closest in the feature space during inference and measures the cosine similarity between the sample embedding and class prototype:
<(x) Pi
i
N,
(11)
[Ø(x) Ip:
where N is the total number of classes learned up the current session.
Analytical decorrelated prototype-based classifier. Analytical learning refers to approaches where classifier parameters are computed in closed form, rather than updated through iterative gradient descent. A common formulation is ridge regression, which yields a stable closed-form solution with built-in regularization. This analytic view is attractive in continual learning because it avoids repeated re-optimization, reduces computational overhead, and provides stability across tasks. In our case, the analytical decorrelated prototype-baat classifier can be derived as a ridge regression solution over the projected features h(r) e RRd,, Specifically, we first compute the feature autocorrelation (Gram) matrix:
G= =>
hig E Rdixd.
(12)
We then compute another matrix Cp that consists of the accumulated sum of projected features corresponding to the same class labels, defined by
Cp
hanVen E Rdix/Vl
(13)
==1
The predicted logits are then computed as:
hW adc h(G M(G+31)-14 31) ER
(14)
with 3 the ridge regularization parameter selected by poass-validation-aaaee optimization, which helps stabilize matrix inversion and prevent overfitting over continual learning, and / the identity matrix. As such, W Wade can be interpreted as decorrelated class prototypes.
In this formulation, Wade can be directly interpreted as set of class prototypes. The "decorrelation" property (see Appendix emerges naturally from the ridge regression solution, since the inversion of (G BI) whitens correlated features. Thus, without introducing extra mechanisms, the classifier yields decorrelated prototypes that preserve discriminability across tasks and mitigate representation collapse in continual learning.
16

---

Published as a conference paper at ICLR 2026
D CLOSED-FORM SOLUTION DERIVATION OF ANALYTICAL DECORRELATED CLASSIFIER
To derive the analytical classifier weights. we start from the ridge regression objective in Eq.
arg min || H ade 1p + 3 [aadell
(15)
Wade
where H E Nxd is the stacked matrix of all projected sample features; Y R-Vx contains the corresponding one-hot labels; N = - Ny is the total number of samples accumulated up the current task; || || is the Frobenius norm; and 3 is the ridge regularization parameter selected by validatiodiiobbbae based optimization. Taking the gradient of this objective with respect Wad gives:
(IY HW ade 1 + BllWadel =- - HW adc) 23W adc-
(16)
oWade adc
Setting this gradient to zero yields:
2H Y + 2H HWade + 233 adc 0,
(17)
which can be rearranged into:
HH H BT) H.
(18)
Solving for W de gives the closed-form solution:
W Wade (H H +1) HTY.
(19)
Finally, expressing this solution in terms of the feature autocorrelation (Gram) matrix = H H and the allsmuuccumllteed feature matrix C, H Y yields:
Wade = (G + -1) Cp E R4XM
(20)
which is exactly Eq. 2
E PRIVACY-PRESERVING NATURE OF ANALYTICAL DECORRELATED CLASSIFIER
potential concern is whether maintaining the accumulated statistics G and C, may compromise data privacy, since they are derived from individual features h(x). We emphasize that both matrices statistical summaries rather than raw features: (i) The feature autocorrelation (Gram) matrix G only encodes second-orde relationships (feature covariance structure) without storing individual feature vectors. (ii) The class-accumulated feature matrix Cp aggregates features at the class level, equivalent storing class prototypes rather than per-sample embeddings. As such, the stored information reflects only global distributional properties of the feature space and does not allow direct reconstruction of individual patient images or embeddings. In high-sample regimes (N > dj), the contribution of any single sample is negligible, making pverss-engineering practically infeasible. This property consistent with prior work sharing first- or second-order statistics has been considered substantially safer than transmitting raw data. Compared with conventional rehearsal- methods that store raw features or exemplars, our approach is therefore inherently more sewacy-preserving, since it requires only compact statistical summaries rather than patient-level data.
machine learning and federated learning, where
F PROOF OF THE DECORRELATION PROPERTY
Analytical Formulation. The analytical decorrelated classifier (ADC) is defined as:
W adte (G +1-111
(21)
where =- T is the feature autocorrelation (Gram) matrix, C, = E,, h,y, is the class- accumulated feature matrix, and B is the ridge regularization parameter.
17

---

Published as a conference paper at ICLR 2026
Spectral Reweighting. diag(A )d,)) Then:
Let G
UAUT be the eigendecompositton of G, with A
(G + B1) U diag Ar+
(22)
This expression shows that each cigendirection of the feature space is rescaled by an inverse factor of its variance.
Directions with large variance (often corresponding to domain-shared nuisance factors) are suppressed.
Directions with small variance (often containing class-specific information) relatively amplified.
This behavior is closely related to whitening, which applies G- 1/2 to normalize feature covariance to the identity. While (G+ 31) is not exact whiteming operator, achieves whitening like effect: balancing the contributions of eigendirections and reducing correlations across prototypes. The ridge term further prevents instability when ill-conditioned, which common under searce and heterogencous medical imaging data.
Prototype Decorrelation. Let and wy denote two columns of Wade (the prototypes for classes i and j). Their covariance is:
Cov(w, wy) = UA++
Cp.5
(23)
Since (A + +1) down-weights directions with large eigenvalues, the dominant shared components that typically drive high inter-class correlation are strongly suppressed. As result, the learned prototypes are decorrelated and more discriminative across tasks.
In summary, the ADC achieves decorrelation by:
reweighting feature eigendirections in whitening- like manner,
suppressing shared nuisance components while emphasizing discriminative cues, and
stabilizing inversion with ridge regularization under limited data.
This explains theoretically why ADC prototypes are less correlated, more robust to domain shifts, and consistently outperform the standard prototype-based classifier (Table 5).
Table 5: Preliminary study on analytical decorrelated classifier across representative datasets. PTM: ViT-B/16-IN2IK.
Dataset
fe,, Ja., (D)
IN-A
48.39 - 54.18
CIFAR
81.28 - 86.09
G ADDITIONAL EXPERIMENTS
G.1 RESULTS ON DIFFERENT RANDOM MATRIX
18

---

Published as a conference paper at ICLR 2026
Table 6: Experimental results on different initialization methods for random matrix under CIL settings across general-domain and medical datasets (d1=5do). "K." stands for Kaiming initialization, "X" stands for Xaiver initialization, "U." stands for uniform distribution, "N." stands for normal distribution, and the number is the parameter for initialization. Note that all experiments are conducted without stre adaptation (Original ViT w/ B&LRP), except for the full RA-LDL.
Mothods
Anttug (%)
CIPARTO
Artua (%)
Actuu (%)
IN-4
Antar (%)
Anting (%)
CTB
Actha (%)
Acting (%6
INc
Avtua (%)
Stundard Guussian Full RA-LDE))
92.93
91.55
70.84
68.51
93.83
$9.14
X3.64
78.17
Standard Guussian
91.56
87.61
64.75
53.52
92.63
59.07
75.94
69.94
y/5
87.73
52.52
87.61
92.50
92.44
Methods
Anting (%)
MedNINNSS T-Sab
Antuae (%)
Accau (%)
Skin8
Antuan 1%
Antiu (%)
COVID
Acciae (%)
Actiu (%)
Blood
Jetuu (%)
Standand Caussian (Full RA-ITE))
$4.79
70LA0
5..00
63.83
05.08
58.04
05.07
97.76
Standard Gaussian
$6.30
70.29
75.52
61.03
95.47
90.30
95. 06
97.14
yB
85.77
19:31
67.09
92.31
83.71
94.12
91.33
G.
91.77
82.18
G.2 RESULTS ON DIFFERENT PROJECTION DIMENSIONS
Table 7: Preliminary analysis results the random projection (RP) dimension d1 The average and last performance on eight CIL datasets from both general and medical fields using ViT-B/16-IN2IK as the backbone. All experiments conducted using the same seed for a fair comparison. The best performance highlighted in bold.
First-stage Adaptation
4pmif
<g=5dd
d1=106
dg=16d,
CIPAR
90.15
91.55
91.97
92.17
35,61
87.63
88.60
$9.11
IN-A
55.89
19.11
53.52
60.51
61.09
57.08
$7.27
60.83
CUB
78.05
78.76
71.78
75.91
77.63
78,50
Skin8
56.74
$7.45
62.49
60.85
68.96
62.98
66.24
69
MedMNNST-Suh
67.70
70.60
72.14
72.55
66.22
rays)
Blood
93.99
97.74
91.62
91.58
G.3 SENSITIVITY EVALUATION ON RANK r
Table 8: Sensitivity evaluation rank in Original VIT w/ B&LRP.
MedM
Actug (%)
Acnua (%)
Arrag (%)
Actu (%)
Acrag (%)
Arru (%)
Acrang (%)
Aoqs (%)
92.63
89.07
64.78
53.52
75.52
60.85
84.30
70.29
55.37
70.30
256
92.69
89.12
64.90
55.43
75.42
60.99
84.32
70.31
G.4 EXPERIMENTS COMPARED RECENT METHODS IN GENERAL- DENERAL-DOMAIN DOMAIN CIL
19

---

Published as a conference paper at ICLR 2026
Table 9: Experimental results (10 tasks) compared recent PTM-based CIL methods on general- domain benchmarks. "f" denotes reliance on replayed data. The best and second-best results are marked in red and blue, respectively.
General Tumain
Array
Aectan
Accing (S)
Atcta
Arcun
Artaat (%)
Alrtty
Actiun (8)
trining
14.25
73.47
CodaPrommt
18822
72.09
73.00
LAE
53.08
79.36
42.48
30.00
73.87
61.03
68.82
61.25
SL.CA
93.32
91.30
69.10
62.23
90.22
$5.54
81.52
77.95
XSTAT
94.31
92.30
70.85
62.34
01.33
$9.75
$4.30
70882
EASE
01.31
83.80
65.3
5504
85.08
$1.38
82.50
77.88
MDS
93384
30.35
08.25
59.2
93.1
59.22
82.51
77.82
Simple
91.51
39.67
52.00
91.85
36.77
01.18
$4.25
ADAM-AJaph
75.
Original
91.91
Adiated
77:80
Ailattal RAA.P
G.5 EXPERIMENTS ON CROSS-DATASET LONG-TERM CONTINUAL LEARNING BENCHMARK
Table 10: Experimental results on cross-dataset long-term continual learning benchmark (order:
Skin8 + COVID MedMNIST + BLOOD), following the original intra-dataset task splits in Table
Framework
AccAve
Acct
Jount Training (upper bound)
71.63
SimpleCIL
$7.50
55.11
ADAM-Acspler
63.18
55.52
RA-LDL (OuPS)
74,56
71,52
G.6 SENSITIVITY EVALUATION TASK NUMBER
Table 11: Results with varying task numbers on MedMNIST-Sub dataset.
Method
lask Nuniher
Jeluug (96)
ArMaa (5)
SimpleC
Zooret 2004g
68.07
50.63
SimpleCIL
MOS
Sun Cal 2025
74.59
51.30
MOS
Stiner ai 2012
8
$0.96
61.83
$4.79
70.60
H PROOF OF RANDOM ANCHORS FOR FEATURE CHARACTERISTICS PRESERVATION AND ENHANCEMENT
We formally prove that random projections guided by the Johnson-LLindenstreen lemma effectively preserve original feature characteristies and, when used to increase dimensionality, can improve the structural and statistical properties of features extracted from pre-trained models.
H.1 JOHNOON-LIDEESTRRAUS LEMMA
Lemma 1 (Johnson LoddssnnLinddesstruuss Lemma). Given distortion tolerance < <.<1 < set of points Z1, $1, I'm S Rd, there exists random linear projection B E Bdo Rdoxds, where the target dimension d, satisfies
and a finite
log m
di =
(24)
such that for all i,j E (1....,,), m.) the following inequality holds with high probability:
(1 l): 13 1IBT (xt: 23)11 < + e) x.
(25)
20

---

Published as a conference paper at ICLR 2026
The lemma demonstrates that random linear projection approximately preserves pairwise distances with high probability. As proved the following subsections (H.3). pairwise distances between all data points are uniformly preserved with high probability, maintaining the original geometric relationships and avoiding feature confusion. Similarly, the expected covariance and inner products scale linearly, and cosine similarities preserved in expectation.
Next, we discuss the benefits of increasing feature dimensionality through random projections.
H.2 BENEFITS OF RANDOM PROJECTION INTO HIGHER DIMENSIONS
Theorem (Benefits of Random Projection into Higher Dimensions). Consider a random linear projection defined by Gaussian matrix B E Rdoxd,, with d1 > dg and entries independently sampled as:
1
By N
0,
(26)
di
Given our extracted feature 6(x) R the projection (xA( = BT oTx(xx into higher dimension d1 improves isotropy of feature distributions and reduces intra-class variance relative to inter-class variance, enhancing linear separability.
Proof. We analyze the projected features ( B xxxxx E R
First, we consider the inner product two distinct vectors hRA (x1), hRA (x)) after projection:
E (x)) h (x;) "E [BB (RA(X}) O,
(27)
due independence and zero-mean properties of the random Gaussian projection matrix B₁y- Thus, the random projection helps decorrelate features and increase isotropy. Next, we examine the covariance structure within each class. Suppose the intra-class covariance matrix of the original features <xx)) is denoted as E E Roooxdo After the projection, the covariance matrix becomes:
E BBEB Tr(2) la.
(28)
This indicates that projecting a higher dimension d, makes intra-class distributions more isotropic and reduces per-dimension variance. Simultaneously, the expected squared distance between class means 11, #2 preserved:
|| (#1 - p>)l" (11- = 42) TE [BB (P) - p2) = || - -2211
E
(29)
Thus, increasing the dimension from do to dj effectively reduces intra-class variance without affecting the inter-class separation, thereby improving class separability.
H.3 PROOF OF RANDOM ANCHOR PRESERVING FEATURE CHARACTERISTICS
i) Pairwise Distances.
Proof. Consider a random Gaussian projection matrix B e dd with each element independently drawn from a Gaussian distribution::
By ~ N (o. (
(1.
B[Bay 0, Var(B,)) di
(30)
Consider two feature vectors (br;), )(x,) E Rd and define their difference as: = 6(x;)) - 6(xx). Then, the squared norm of the projected difference given by:
da
B
Bng Uig
(31)
Define the projected component for each as:
da Bp Ug.
y,
(32)
y=1
21

---

Published as a conference paper at ICLR 2026
Since the entries of B are zero-mean and independent, we have:
do
|0
E EY 0, Var(Y,) - E
Σ
Bp.g
(33)
di
Applying concentration inequalities (such as Gaussian concentration), we have, simultaneously for all pairs j), with probability least 6:
(1 8) |||| < || T ull < (1 e) ||0|
(34)
provided the projection dimension satisfies:
m²
log m
dj C² log
0
(35)
8
c²
where C is a suitable constant and m is the number of data points. This shows that pairwise distances approximately preserved.
ii) Variance and Covariance Preservation (Isserlis formula).
Proof. Given the random projection matrix B defined above, the expected covariance and inner products scale linearly, and cosine similarities are exactly preserved in expectation. Let Σ E IR d be the covariance matrix of the input features. The trace of the projected covariance matrix is:
B[tr(BTEB))
Σ Egg B.
(36)
By independence and zero-mean, only the diagonal terms q contribute, giving:
[1/d1, if 0, otherwise,
B Bripl
(37)
so the expectation simplifies to:
di
B[ir(B [EB]
ir(E). do
(38)
iii) Inner Products and Norms (Isserlis formula).
Proof. Consider two vectors E dR and their projected inner product:
di
do
dg
(Bu, Bu) = BB)) (Br)
B Bj
(39)
Taking expectation:
dj dg
do
E (Bu, Bu)]
E. Bap By.p.
(40)
p=1 1=1
1=1
Using the same independence argument as above:
B[(Bu, Bui] do
(u, v).
(41)
Similarly, we also verify norm preservation by setting u 6:
di do
Bu|| = B| (Bu, Bu)
||/|
(42)
iv) Cosine Similarity, Since both the inner product and squared norms scale by 4. the cosine
similarity remains unchanged:
Eicos 0'] = B[(Bu, Boy]
(u. cos 0.
(43)
E Bu
| de ||o|a|2
Thus, the random linear projection preserves covariance, inner product structures, and angular relationships in expectation.
22

---

Published as a conference paper at ICLR 2026
1 PROOF OF LRP BENEFITS
1.1 OPTIMALITY OF RESIDUAL LRP COMPLEMENT TO RA
Theorem 3 (Optimality of Residual LRP Complement to RA), Let 6(x) E Rdo be the feature from the PTM, and let (x) B Φ(x), with B Rdoxdd frozen. Suppose the downstream domain exhibits additional residual variation Ls such that the intra-class covariance becomes Starget Σ Es. Then a trainable low-rank correction hLRP(x), trained in the first session, reduces intra-domain variance in the RA RA- projecced space while preserving inter-class distances.
Proof. After RA projection, the intra-class covariance under domain shift is:
Tr(E + ΣA)
E Starget B
ldj-
(44)
d1
Training RT optimizes:
min E ([(M)))((x)A(x(1(?))) W.W.
(45)
where (xx denotes the domain shift correction. Due to the low-rank constraint, the learned correction emphasizes dominant residual variations. Thus, the residual term complements RA by explicitly correcting domain-induced inducced distortions without altering the preserved global geometry, reducing intra-class spread while maintaining inter-class structure.
I.2 RESIDUAL IMPROVES CLASSIFICATION MARGIN
Theorem 4 (Residual Improves Classification Margin). Let Pl, u2 E Rdo be class means in the feature space. Then the expected squared margin after adding sosd-session-traieel LRP residual satisfies:
E [11b(14) h(u2))1 > E [lIba((1r) h12(12))
(46)
assuming the residual approximates domain-inducee mean shift Apdomain o O.
Proof. Let:
h(p) = B hLRP(H))
(47)
Then:
B[llh(pm) ((pe))1]]
=E[[ (m1 u2) hLRP(HI) hLRP(P2)111
(48)
|||| M2ll / + [1)RP((1)) hLRP(H2))1
+ 2( (M1 - (w)-))), 12), hLRP(
The cross-term expectation is zero due to the independence of B and LLRP. If HLRP approximates the domain-specian mean shift, the added term nonzero, and the margin increases.
J LIMITATIONS AND SOCIETAL IMPACTS
(i) Scope: Our evaluation focuses on 2D classification; extending to 3D, multi-modaa and segmen- tation tasks remains future work. (ii) Novelty: While certain components have been explored in general-domain studies, the key contribution of this work lies in demonstrating that a simple yet carefully integrated strategy can effectively address medical-specific challenges-and even outper- form more complex designs in medical class-incremental learning. (ii) Robustness: While generally stable, RA-LDL shows order sensitivity on heterogeneous datasets. Further calibration could address this. (iv) Backbone Dependence: Performance is bounded by PTM quality: better PTMs should further enhance results. (v) Impact: The lightweight design lowers computational cost and facilitates mriacy--rienddy deployment, but careful clinical validation is essential.
In summary, RA-LDL foundational step: not a final solution, but evidence that minimalist recalibration be surprisingly effective in medical continual learning.
23

---

Published as a conference paper at ICLR 2026
K
LLM USAGE
The authors utilized LLMs as a writing assistant during the preparation of this manuscript. The role of it was primarily focused tasks related language refinement and improving the clarity of the narrative. Its application included: (1) enhancing the conciseness and academic tone of sentences and paragraphs; (2) restructuring complex sentences for better readability; and (3) brainstorming alternative phrasings for key scientific arguments.
is important to emphasize that all core scientific ideas, experimental design, results, and conclusions were exclusively developed by the human authors. The LLM served sophisticated editing and brainstorming tool. All text generated or modified by the LLM was critically reviewed, edited, and ultimately approved by the authors to ensure it accurately reflected our original intent and scientific findings.
24


Reviews:
[{"review_id": "ligDGQDnuk", "reviewer": "Reviewer_wD7K", "summary": "The paper introduces a minimalist approach for class-incremental learning (CIL) in medical image classification called Random Anchors with Low-rank Decorrelated Learning (RA-LDL). The motivation is to address the unique challenges of medical CIL, which involves low intra-class variability and high inter-domain shifts, where complex pre-trained model (PTM)-based continual learning methods from the general domain often fail. RA-LDL combines a feature extraction method, lightweight feature calibration via a frozen random anchor projection and a single-session-trained Low-Rank Projection, and a closed-form analytical de-correlated classifier based on ridge regression.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The empirical evaluation is thorough, benchmarked across four realistic medical datasets as well as standard general-domain CIL benchmarks, and ablation studies clearly show the contribution of each component.\n\nThe work points out an overlooked strategy to CIL that offers better scaling properties/simplicity. It advances methodological understanding by showing that domain-specific challenges in medical continual learning are best addressed with statistical feature recalibration, not greater architectural/training complexity, which is an applicable insight to other fields with similar data properties.\n\nTheoretical justifications for each step (random anchor, LRP, analytical de-correlated classifier) are provided, including proofs of distance and covariance preservation, variance reduction, and prototype de-correlation.", "weaknesses": "While the synergy and systematic evaluation of random anchors, low-rank projections, and de-correlated classifiers are novel for medical CIL, many components are incremental adaptations or well-established in general-domain CIL (Zhuang et al., 2022, McDonnell et al., 2024). More in-depth, quantitative ablation and comparison with these specific prior works could clarify the genuine novelty and highlight improvements beyond re-combination. Some discussion in appendix, but would be nice to see more quantitative evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjuKzQrnYu", "reviewer": "Reviewer_XHJg", "summary": "The paper addresses class-incremental learning (CIL) in medical image classification, where models must learn new disease categories over time without forgetting previously learned ones. The authors argue that existing PTM-based CIL methods—often complex and designed for natural images—fail in medical settings due to low intra-class variability and high inter-domain shifts (e.g., from different scanners or protocols). In response, they propose RA-LDL, a minimalist, representation-based pipeline with three components:\n(a) optional one-time ViT-Adapter tuning for domain adaptation,\n(b) feature calibration via a frozen Random Anchor (RA) projection and a single-session-trained Low-Rank Projection (LRP), and\n(c) an analytical, closed-form decorrelated classifier based on ridge regression.\nRA-LDL requires only one training session, avoids replay or complex routing, and achieves strong performance across four medical CIL benchmarks, often approaching joint-training upper bounds. The method is shown to work robustly with both general-domain and medical-specific pre-trained models (PTMs).", "questions": "Questions:\n1.The authors show that general-domain PTMs, like ViT-B/16-IN21K, outperform medical-specific models such as UniMedCLIP or RAD-DINO. However, I have a question whether the improvements are more due to the general-domain PTM itself rather than the RA-LDL recalibration technique. The paper could further clarify how RA-LDL benefits from this general-domain foundation and whether it can achieve similar results when applied to domain-specific PTMs.\n2.While the manuscript evaluates the model’s robustness to task ordering, further analysis on how RA-LDL performs in highly unstructured or unpredictable sequences of class introduction (i.e., non-sequential class introduction) could be informative. For example, testing RA-LDL under random or shuffled class orders to assess how well it maintains generalizability when the sequence of tasks is not controlled or predictable.\n3.Are there scenarios where RA-LDL underperforms? For example, in highly imbalanced tasks (e.g., Skin8), does decorrelation hurt minority classes?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1.The RA-LDL approach is innovative in its simplicity. While other methods for class-incremental learning in medical imaging often rely on complex architectures, this paper proposes a lighter solution that offers competitive performance, especially under domain shift conditions. It effectively merges generalization and adaptability without extensive tuning.\n2.By focusing on feature recalibration (using random anchors and low-rank projections), RA-LDL addresses low intra-class variability and high inter-domain shifts directly and effectively. The method shows solid promise in medical domains like COVID-19 diagnostics, blood cell analysis, and skin lesion classification.\n3.It only requires one training session with minimal task-specific tuning, making it very practical for real-world medical applications where data is continually evolving.The approach works across various imaging modalities, making it applicable in diverse medical settings.", "weaknesses": "Weaknesses:\n1.it could benefit from a more detailed examination of RA-LDL’s long-term stability and its ability to mitigate forgetting. Specifically, investigating the model's ability to retain previously learned classes as more tasks are introduced would strengthen the paper’s argument.\n2.The “one training session” claim is slightly misleading: while only the first session trains the LRP, the classifier is updated incrementally using accumulated statistics. This is still efficient, but the phrasing could confuse readers expecting fully frozen inference.\n3.How much memory is required to store G and Cp? For large class sets or high-dimensional features, the Gram matrix (d₁×d₁) could become costly. Please report memory usage or discuss scalability.\n4.Why not compare to recent replay-free methods? While Table 2 includes some, a more direct comparison to non-PTM-based CIL would strengthen the claim that PTMs + RA-LDL are uniquely effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fKsSekVafU", "reviewer": "Reviewer_zaiN", "summary": "In this paper, the authors propose a Random Anchors with Low-rank Decorrelated Learning (RA-LDL) approach to address Class-incremental Learning (CIL) in medical image diagnosis. The authors validate their idea with two base models (ViT-B/16-IN21K and BiomedCLIP) and four datasets (Covid, Blood, Skin8, MedMNIST-sub). The gains of RA-LDL are clear.", "questions": "1. What is RA-DL in Table 2? It only appears in Table 2 and without explanation.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The experiments are comprehensive: the authors include various types of medical images and popular base models. In the comparisons, the authors also include most milestone methods in class incremental learning. All these experiments show a clear gain for RA-LDL.\n2. Analytical closed-form decorrelated learning sounds good.", "weaknesses": "1. The method is not novel for the current stage. The main contributions of the paper include a random matrix projection and a LORA-based structure.\n2. The author might need to justify more about the combination of two projectors. I understand the proof the author proposed that they aim to random anchor projections  to preserved the original feature characteristics and LRP residual reduces intra-class variance. However, mathemtically, could it possible to simplify the equation 4 into a single LORA-based structure $h(x) =h_{RA}(x) + h_{LRP}(x) \\approx h_{LORA}(x)$?\n\n3. The contribution of each projector ($h_{RA}(x)$, $h_{LRP}(x)$) is unclear. How do the authors combine each projection, and what is the weight for each part? Does the weight impact the results, and what is the robustness of the weight selection? These might impact the accuracy gain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)