MaRS: Memory-Adaptive Routing for Reliable Capacity Expansion and Knowledge Retention
Authors: Gang Yan
Venue: ICLR 2026 Poster (2026)
Keywords: Large Pre-trained Models
Continual Learning
Slot Expansion
Knowledge Distillation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Large pre-trained models (LPMs) serve as universal backbones for vision and language tasks, but continual learning (CL) with frozen LPMs remains challenging, since shallow adaptation modules face the stability–plasticity dilemma and are prone to catastrophic forgetting. To address this problem, we propose MaRS (Memory-adaptive Router with Statistical control), a modular framework that decouples stable representation from adaptive capacity through three components: a frozen encoder, a slot-based memory router, and a lightweight classifier. On this basis, we design two mechanisms: (i) Statistically-Grounded Slot Expansion (SGSE) formulates expansion as a statistical decision problem, ensuring controlled growth with guarantees on false alarms and detection delay; (ii) Dual-Stage Contrastive–Distillation Adaptation (DCDA) integrates new slots through supervised contrastive learning and knowledge distillation, preserving prior knowledge without raw replay. Experiments on diverse benchmarks show that MaRS achieves state-of-the-art performance in continual learning with frozen LPMs, combining adaptability, efficiency, and retention.

Published as a conference paper at ICLR 2026
MARS: MEMORY-ADAPHHE ROUTING FOR RELIABLE CAPACITY EXPANSION AND KNOWLEDGE RETENTION
Gang Yan
School of Computer Science and Technology, Jilin University gyan8e juu .n
ABSTRACT
Large pre-trained models (LPMs) serve as universal backbones for vision and language tasks, but continual learning (CL.) with frozen LPMs remains challenging, since shallow adaptation modules face the stability-plasticii dilemma and are prone to catastrophic forgetting. To address this problem, we propose MARS Alemory--ddaptvy Router with Statistical control), a modular framework that decouples stable representation from adaptive capacity through three components: a frozen encoder, a slot-based memory router, and a lightweight classifier. On this basis, design two mechanisms: (i) Statistically- Grrouddd Slot Expansion (SGSE) formulates expansion as a statistical decision problem, ensuring controlled growth with guarantees on false alarms and detection delay; (ii) Dual-Stage Con- Adaptation (DCDA) integrates new slots through supervised contrastive learning and knowledge distillation, preserving prior knowledge with- out raw replay. Experiments on diverse benchmarks show that MARS achieves state-of the-art performance in continual learning with frozen LPMs, combining adaptability, efficiency, and retention.
INTRODUCTION
Large pre-trained models (LPMs) such CLIP (Radford et al. 2021 and BERT (Devlin et al. 2019 have transformed modern machine learning. Trained on massive and diverse corpora, they learn general-purpose representations that transfer well across domains. These representations support advances in natural language understanding (Brown et al., 2020 Chowdhery et al. 2023), visual recognition (He et al. 2016, Dosovitskiy et 2001), and multimodal reasoning (Radford et al.) 2021 Liu et al. 2023 The success of LPMs has also established them as universal backbones for downstream applications such as information retrieval, question answering, and zero-shot classifica- tion. A common approach for efficient adaptation to freeze the pre-trained backbone and fine- tune-tune only lightweight task-specific modules (Houlsby et al.,, [019] Lester al.) [221] Hu et al. [2022]
Legate et al. 2023). This parameter-efficient paradigm preserves the generalization ability of the backbone while reducing both computation and memory costs.
In practical applications, tasks and data arrive sequentially, and models must adapt continually while retaining prior knowledge. This challenge is studied in continual learning (CL) (Parisi al. 2019 De Lange et al., 2021 Wang et al. 2024), which aims to learn from a stream of tasks without catastrophic forgetting McCloskey & Cohen 1989 Ramasesh
al. 202T). At its core lies the stability-plaaticir dilemma: models must remain plastic enough to acquire new information while
stable enough to preserve what has already been learned. In the context of frozen LPMs, this dilemma is particularly severe. Because adaptation is restricted to shallow modules, plasticity is limited, and the fixed backbone further amplifies forgetting. As a result, naive parameter-efficient adaptation insufficient for long-horizon continual learning.
To mitigate forgetting, continual learning has developed a wide range of strategies. Replay-based methods (Rebuffi et al. 2017 Lopez-Paz & Ranzato 2017 Chaudhry et al. 2019 Buzzega et al.) 2020) revisit stored or generated samples to reduce drift, but they raise privacy concerns and face scalability issues. Regularization based approaches (Hinton al. 2015 Kirkpatrick et al. 2017] Zenke et al., 2017) Li & Hoiem| 2017] Aljundi et al. 2018) constrain updates to remain close to past solutions, but their corrective signal weakens tasks accumulate. Dynamic expansion tech-

---

Published as a conference paper at ICLR 2026
niques (Rusu et al. 2016 Yoon al. 2018 Dong et al. 2024) add new capacity for novel tasks, but they often rely on heuristic triggers that may cause uncontrolled growth. Prototype-be eaaotyee-based meth- ods (De Lange & Tuytelaars 2021 Liu al. 2025 Zhu et al. 2025) compress historical knowledge into compact memory structures, improving efficiency but showing fragility under distribution shifts. Although these strategies offer useful insights, they are designed for conventional architectures rather than frozen LPMs. In parameter-efficient settings, shallow adapters have limited expressive power, and heuristic retention does not provide formal guarantees.
Recent studies have begun to examine continual learning in the context of large pre-trained models. Adapter-based approaches (Ke et al.) 202 Wang et al., (202) improve efficiency but still suffer from forgetting as tasks accumulate. In the uision-languse domain, methods such as VLM-CIL (Liu et al. 2023), DIKI (Tang et al. 2024), and CoLeCIIP (Li et al. 2025) highlight both the promise and the fragility of frozen encoders. Parameter-efficient modules preserve adaptability, but retention often depends on heuristic replay or task- specifi tuning. Recent designs, including dynamic LoRA adapters (Hu et al. 2022), provide partial relief but still rely on ad-hoc expansion rules and lack formal guarantees. Together, these efforts underscore a persistent gap: current methods demonstrate the feasibility of continual learning with frozen LPMs but do not provide principled mechanisms for expansion and retention.
ranks and mixture-of-expee
In this paper, we address these challenges by proposing MARS (Memory-adaptive Router with Statistical control), modular framework for continual learning with frozen LPMs. As shown in Figure [ the framework has three com- ponents: frozen encoder that provides stable pre-trained representations, a slot-based mem- ory router that organizes knowledge into expand- able capacity units, and lightweight classifier that produces task predictions. By decoupling stable representation from adaptive capacity, the design shifts continual learning control to the routing layer and avoids costly full-model updates.
Tasks
MaRS
Results
Encoder
Clasalfier
SO6E + DCDA
Pre-trained
Trainable
Figure 1: The architecture of MARS. Images are from Tiny- Inn-emmeeNet (Le & Yang 2015).
On top of this architecture, we propose two complementary mechanisms. The first, Statistically- Grounded Slot Expansion (SGSE), determines when and where to allocate new slots. Instead of heuristic triggers, SGSE formulates expansion as a statistical decision problem. Router-aligned novelty detection (Hendrycks & Gimpel| 2017 Liu al.) 2020) monitors representation coverage, while confidence bounds (Roberts) 2000 Brown et al. (001)) ensure that slots are added only when capacity is insufficient, with formal guarantees Adaptation (DCDA), controls how new slots are integrated. It separates representation adaptation from classifier tuning: supervised contrastive learning (Khosla et al. 2020) aligns new slots in the embedding space, while knowledge distillation (Hinton et al. Li & Hoiem 2017; Guo al. (017)) and prototype- bbased regularization (Snell et al. 2017) preserve prior knowledge without requiring raw replay. Together, SGSE regulates when to expand and DCDA determines how to adapt, making slot-based routing both principled and retention- -uaattted. This design addresses the plassity- dilemma in continual learning with frozen LPMs.
false alarms and detection delay. The second,
Dual-Stage Contrastive
2015
We introduce MARS, a modular framework for continual learning with large pre-trained models that separates stable representation from adaptive capacity. (ii) We develop SGSE, a statistically grounded slot-expansion mechanism with formal guarantees on growth and retention. (iii) We design DCDA, dual-stage contrastive distillation method that integrates new capacity while preserving prior knowledge without raw replay.
In summary, our contributions are threefold: (i)
contrastive-distillation
RELATED WORK
Continual learning studies how to acquire knowledge from a sequence of tasks without catastrophic forgetting, Core challenges include interference between old and new tasks, distributional shifts in data or labels, classifier bias toward recently observed classes, and constraints on computation and memory. Surveys provide comprehensive overviews of these challenges and benchmarks (Parisi et al., 2019 De Lange et al. 2021 Wang et al., 2024). and consistently emphasize the dilemma as fundamental problem that underlies most continual learning scenarios.
2

---

Published as a conference paper at ICLR 2026
Early work mitigates forgetting through replay or regularization. Replay-based methods such as iCaRL (Rebuffi et al, 2017). GEM (ooeez-Paz& & Ranzato 2017). A-GEM (Chaudhry et al. 2019).,
and DER++ (Buzzega et al. 2020) rehearse stored or generated samples to reduce drift. While
effective, these methods raise privacy concerns and face scalability limits when storage or generation is constrained. Regularization--aase
approaches constrain parameter updates or distill predictions, including EWC (Kirkpatrick et al.) 2017), SI (Znne et al. 2017), LwF (Li & Hoiem 2017), and MAS (Aljundi 2018). These methods are more memory-efficient, but their corrective signal decays over long horizons or under severe distributional shifts, which limits robustness in practice.
Another direction reduces interference by expanding model capacity or compressing past knowledge. Structural expansion techniques such as Progressive Neural Networks (Rusu et al. (016)) DEN (Yoon) al. 2018); and CEAT (Dong et al. 2024) dynamically add parameters for new tasks. However, they lack principled criteria for when and how much to expand, which often results in uncontrolled growth. Prototype-based methods instead summarize distributions with compact representations, including
dual-bias frameworks (Zhu al. 2021). IPC (Liu et al. 2025), and PASS++ (Zhu et al. (025). These methods are more efficient in memory and computation, but they rely on heuristic allocation rules and tend degrade under distribution shifts, especially in long-horizer learning.
More recently, continual learning with large pre-trained models has gained increasing attention. Models such CLIP and BERT provide strong transferable representations, motivating methods that freeze or partially freeze the backbone while adapting lightweight modules. Examples include prompt-based approaches such I.2P (Wang al. (2022), adapter- and prompt-based vision- language methods (Liu et al. 2023), and parameter-efficien continual learning with CLIP, including DIKI (Tang et al. (0224 and CoLeCLIP eLi al. 2025). These works demonstrate the value of frozen backbones and parameter-efficien adaptation, but they still rely on heuristic expansion strategies and lack statistically grounded guarantees for retention. This gap motivates MARS, which integrates SGSE and DCDA as core mechanisms.
3 PROPOSED DESIGN OF MARS
As shown in Figure [ MARS is designed for continual learning with LPMs. The framework consists of three components: (i) frozen encoder f-) that provides fixed pre-trained features, (ii) slot-based memory router that dynamically assigns inputs to expandable memory slots, and (iii) a lightweight classifier g(-) that produces task predictions. Given input the encoder outputs frozen features hy = fxx) E TR The router then computes routing probabilities that decide which slots should process the features. Each slot is parameterized by affine transformations (Yr, (3;) that scale and shift the features, serving as independent adapters without modifying the encoder.
To ensure stable initialization, all slots are initialized identity mappings with 7; and 3
The router aggregates slot outputs into an adapted representation h. which the classifier g maps to logits. The slot count S begins from So and expands during training as needed. A central challenge is determining when allocate new slots: over-expansion increases cost, while under-expansion leads to interference and forgetting. To address this, we propose Statistically Cariise Slot Expansion.
3.1 DESIGN OF STATISTICALLY-GROUNI
SLOT EXPANSION
SGSE formulates slot expansion as statistical test. leverages the router, a lightweight component of the memory module that compares frozen features with slot keys and outputs probabilities indicating input-slot affinity. By placing statistical bounds on these probabilities, SGSE ensures that new slots are created only when existing ones cannot reliably cover incoming inputs.
Router-Aligned Novelty Detection. SGSE uses the router to estimate the affinity between each input and available slots. Given input x,, the query is computed
(xx)) W.hy Rdd,
(1)
where hy = (xx,) are frozen encoder features. Routing then applies cosine-softmar over normalized keys R; = k:/1|kk/|l
px(x)) = exp(()(xi), 5 exp( (epp((((z))) (x(xx);
q
.g. |
(2)
3

---

Published as a conference paper at ICLR 2026
where T, is the softmax temperature. A smaller T, makes slot probabilities sharper, while larger Tr spreads them more evenly. Following previous practice (Chen et al. 2020). we set Tμ = 0.07, which balances confident routing and robustness. We then define the top-slor confidence (x.))
which measures how confidently the router aligns the input to its best-matching slot. Covered inputs typically yield s; 1, while novel inputs produce lower s due to distributed probabilities. This novelty and out-of -i-oo--distribution indicators (Hendrycks & Gimpel 2017).
matches confidence-base
Proposition 1. Let Ct = maxigs, (G(x;), ki) and assume St > Then keeping {a, : j # i*) fixed, 8t is strictly increasing in C2 whenever A == Ejjr eailte > 0, where i" E arg max, aj and aj = (g. kz)..
Proof. Let slot i attain c = max, aj and set = Ejjt easl"r, Then
ec/Tr
s(c) = E A = 1+ Ac-o/T
(4)
and
ds 1 dc (1 s(c)) >0
(5)
whenever A > 0 (i.c., S. 1).
The monotonicity holds locally under fixed competing similarities, which is the setting used when assessing how the router's confidence varies with affinity. This result shows that 8, is locally monotone in the similarity score C₁, making a calibrated local statistic for novelty. Unlike heuristic thresholds, it provides mathematically justified detector: when the affinity of the top slot decreases while other similarities are unchanged, 8t must also decrease. To stabilize slot semantics, MARS applies slog-weighted affine transformations:
7 LN(hT)
( Pc PiB.)
(6)
where LN(.) is Layer Normalization Ba et al. 2016 To ensure stable and smooth slot representa- tions, we maintain slot statistics using router- weighted exponential moving averages (EMA):
P (1 a) ((-1) + (xx)) LN(hr), 8 = (1 a)
p.
where a E (0, 1) is the smoothing factor. smaller a improves stability, while larger a improves responsiveness. In practice, = 0.05 provides a good balance. Anchors are then defined as
a, %i
+ Br.
99)
max(s..)
with 10-5 for numerical stability. Anchors serve compressed surrogates of past knowledge, enabling memory-preserving distillation without raw data. By compactly representing past distribu- tions and leveraging the classifier's Lipschitz continuity, they provide provable retention guarantees: features close to an anchor induce bounded changes in predicted probabilities (via Pinsker-type arguments (Canonne, 2022 Thus, anchors are theoretically grounded, not heuristic summaries.
Statistical Triggers for Expansion. Although provides an instantaneous novelty signal, thresh- olding directly is unreliable due to noise and non non-stationarity. SGSE therefore tracks the (1-e)- quantile of recent confidences with exponential smoothing:
lt =
(10)
Qi = 3Q.- + (1 (1- 9)gt.
(11)
where 3 E [0, 1) is the smoothing coefficient, and is the short window used for the empirical
quantile. We set w = 10 and = 0.1, which offer a practical short-horizon estimate while avoiding the high variance of very small windows and the excessive lag of larger ones. A larger & provides smoother but slower adaptation, while a smaller 3 increases reactivity. We use 3 = 0.9 to balance stability and responsiveness.
4

---

Published as a conference paper at ICLR 2026
Theorem 1. If { are i.i.d. with mean qf and variance og oo, then
B[Q+] = + + q²); Var(Q.) (1- 33
(12)
- f2 Tg.
(13)
so Qi + + in I.? After a mean shift q² + o < q" at time T, the smallest with E[Qr+kl I 8 for
any 8 (") satisfies
In 11Q-1-90)
[(Q+-10 8-9
$ 1 - B
In
(14)
-In
so the expected detection delay is 0((1 - 3)
This theorem shows that Qe is L'3-consittent estimate of the long-run quantile and that its detection delay is predictable, scaling as O((1 }) To decide expansion, we monitor Bernoulli trials {st 2 Qr) and compute the empirical success rate ia over samples. Expansion is triggered if the one-sided Wilson lower bound drops below a threshold:
LB(p n, 2)
Pe
+
2m
2
6: (1(1-A))
+
4n
(15)
1+
We adopt the Wilson score interval for binomial proportions (Brown et al. 200T), which provides better coverage than Wald intervals in small samples. For expansion decisions, score test with a short evaluation window of n = 20, a standard default in sequential binomial testing that remains stable in small-sample settings, together with the one-sided 95% cutoff 2 1.645.
the Wilson
Corollary 1. If the success probability p Pr(st 2 Qt) is stationary with 2 then for i.i. Bernoulii trials and one-sided Wilson bound with score 2 (level
((2)).
Pr(LB(pe <>) S a.
(16)
Thus, under mild assumptions, the probability of false expansion per test is most
The Wilson bound converts observations into confidence guarantees, ensuring that false expansion is provably controlled at level a (Cor. [T. In this way, SGSE provides statistically calibrated test for novelty: expansions are data-driven rather than noise-triggered. To accelerate specialization, new slots are initialized with the mean query of recent low-8; samples and identity affine parameters, yielding about 15% faster convergence and reduced redundancy. This design places new slots in representative region the feature space, avoiding arbitrary starting points far from incoming data.
Takeaways 3.1. SGSE provides a principled solution to balance stability and plasticity in large pre-trained models. By combining router-aligned novelty detection with statistical triggers, MARS achieves careful and efficient slot growth. Unlike heuristic thresholds, SGSE offers (i) locally monotone and calibrated novelty signals (Prop. 7 (ū) provable convergence with predictable detection delay (Thm. 7 and (iii) explicit false establish SGSE as a theoretically grounded expansion framework for scalable continual learning with frozen LPMs.
guarantees (Cor. [.. Together, these results
3.2 DESIGN OF DUAL-STAGE
ADAPTATION
SGSE determines when to add new slots. The next problem is how to integrate them without forgetting. This is especially important for LPMs because their frozen backbones cannot absorb new tasks. Then, we propose Dual-Stage stages: representation alignment and knowledge retention. New slots are aligned through contrastive learning, while old ones are preserved through anchor-based distillation. This design could help to balance plasticity and stability.
Adaptation, which separates adaptation into two
Stage 1: Feature Adaptation (Memory-Only). Given frozen backbone features hy = fxx,, the memory module adapts them as
h Mem(hr).
(17)
5

---

Published as a conference paper at ICLR 2026
We optimize supervised contrastive loss (Khosla et al. 2020):
exp(sim(h; h,)/7)
Suppon
log
(18)
PO1
Exee exp(sim(h, hT)
JEP())
where features are normalized, Pii) denotes the set of indices in the mini-batch that share the same class label as example i, and т [0.05, 0.2) is the temperature. A smaller makes similarities sharper, while a larger allows more intra-class variation. Following common practice, we set т 0.07. To stabilize adaptation, we add smoothness term that penalizes drift from frozen features:
smooth
lh: hr.
(19)
The Stage 1 objective can be defined as:
( =oupeon Asmooth Samooth:
Lxppenn
(20)
with Asmooth E [0.1 By conducting empirical evaluations, we set Asmooth = 0.3 as it gives the best balance between discrimination and stability.
During Stage 1, only memory parameters (Wg, K, 7, 7, are updated, while the classifier remains fixed. Here, Wg is the query projection matrix and = { is the set of slot keys. Each slot key acts semantic center and guides routing. By freezing g. contrastive learning refines the feature space without shifting classifier boundaries. The contrastive objective increases inter-class separation, while the smoothness term controls feature drift.
Classifier Tuning (Head-Only). In Stage 2, the memory is fixed and only g is updated. The main loss is cross-entropy:
Stage 2:
exp(=i(yil)
LCE = 1 Σ log exp(z(c))
= ghh))
(21)
We regularize the classifier with two distillation terms. The first Learning without Forgetting (LwF) on current inputs:
LLAF
T Σ KL KL (sootmax((33 /T) softmax(==/T))
(22)
where 2pld gold (h:) and T E [2, 5] is the temperature. A larger 7'' smooths distributions and high- lights relative class probabilities (Hinton et al. 2013). It also improves probability calibration (Guo et al. 2017). We set T 3, which balances stability and informativeness.
The second term is anchor distillation on slot anchors A:
Lanchor
T
Σ
(ssfstmmaxx22" /T) softmax(=x/T)))
(23)
T
aEA
where zold gold (a) and 2 g(a). Anchors are surrogate prototypes maintained by SGSE, They store old knowledge without raw replay and follow the idea of prototype learning (Snell et al. 2017
Therefore, the full Stage objective
((2) = LCE + \L.WFLT.WFF + Aanchor Lanchor
(24)
with ÁLWF Ro 1.0 and Aanchor E [0.5, 1.0). These weights reflect the balance between plasticity (cross-entropy: and stability (distillation). Anchor distillation connects SGSE anchors with the following theoretical bound:
Theorem 2. Assume: (i) 9, god Rd R L-Lipschite in logits, (ii) for all anchors E A. S n, and (ii) every old-class feature h lies within distance 5 of some anchor a in feature space. Then for any such h,
KL (ooftmmaa (a)/T) || softmax(g(a)
||softmax(xole (h)/T) softmax(g(h) = of ((v + t0).
(25)
and the old-class accuracy drop is O /n I.5TT).

---

Published as a conference paper at ICLR 2026
Proof. By (ii) and Pinsker's inequality (Canonne 2022). the softmax distributions at each anchor differ by at most ((v)) in f1. By (i), logits vary at most Lg within a 6-ball. After temperature scaling, this variation adds at most ((LL in probability space. By the triangle inequality, the total deviation is O(v" + (L) which yields the stated bound.
This theorem shows that anchor- based distillation gives provable retention. If anchors approximate old features within 5, and if distillation keeps anchor predictions consistent within 1). then the deviation on old-class predictions tightly bounded. Thus, DCDA preserves knowledge without raw replay and remains both memory-efficient and theoretically sound.
Takeaways 3.2. MARS avoids raw replay by encoding knowledge into slots and anchors. SGSE enables principled slot growth, and DCDA integrates new capacity through contrastive alignment and anchor-based distillation. With the encoder frozen, adaptation remains efficient. Empirically (Sec. F.. DCDA improves accuracy by up 20% relative to DER++ (Buzzega al. 2020j, depending on the dataset. Together, SGSE and DCDA offer a principled solution to the stability tradeoff in continual learning with large pre-trained models.
3.3 COMPUTE AND MEMORY COMPLEXITY
At last, we analyze the computational and storage costs of MARS and show how SGSE keeps growth both controlled and predictable.
Per-example Overhead. Each forward pass consists of the frozen encoder f()), followed by the memory router and the slot-conditioned affine transform. Routing costs (Srdka per input because it computes query-key similarities, and affine adaptation costs O(S,dr). Thus the per-example overhead is
Time(z,) O(S = O(S,dr) if d2 S dr.
(26)
Training is efficient because Stage updates only (W,, g K, (3) and Stage updates only g. both of which are much smaller than the backbone.
Per-slot Cost. Each slot stores a key K; E Rds, affine parameters (7i, B:) E R2drr and an anchor a, E Rdr This amounts to (dd++ parameters per slot, plus the head lg|- During inference, routing and adaptation scale linearly with St and remain independent of the frozen encoder [fl.
Lemma 1. With St slots and feature dimension di, the per-input compute cost is ((s((d+drr) (reducing to ((S,dz) if dz I dr),
(27)
and the parameter footppritt is
O(S((d++dr) + lg).
(28)
Complexity Control via SGSE. Without regulation, St could grow linearly with stream length T, leading to uncontrolled complexity. SGSE avoids this by allowing slot expansion only when there is statistically significant evidence that existing slots cannot cover new inputs. This mechanism ensures that growth is linked to true novelty rather than noise. Formally, Cor shows that the false-expansion probability per test at most a, which provides bound on the expected growth:
Proposition 2. For SGSE with Wilson test level a, evaluated every samples over a window n 2 m, let be the stream length, M [(T - w)/ml ] the number of tests, and ST the slot count at horizon T. Then
E[ST]] & So + Ny + a.M
(29)
where Nr is the number of true novelty expansions. Moreover, with probability 2 21-5.
ST < So + Nr + aM Theorem 3. Combining Lemma and Prop. 2 the expected per-example cost at time T is
M In
(30)
E[Time(xT)] ((d, (dd (idx+dr)(ss (So E[NT] + a.M)).
(31)
with high probabiiyyy bound of the same form. The foottt satisfies
E [[Mmr]] = o (dd(dddr)) + +[[r] + -aM) + lgl-
(32)
7

---

Published as a conference paper at ICLR 2026
Takeaways 3.3. When the number of true novelties Nr grows sublinearly with T (for example ( T) or O(T") with p < 1). both computation and memory also grow sublinearly, while scaling linearly with dr and St. In this case. MARS scales smoothly with streaming data and avoids uncontrolled overhead. In contrast, heuristic expansion methods often cause unbounded slot growth and lead linear even superlinear complexity. By grounding expansion in SGSE's ' statistical test, MARS provides controlled growth with both efficiency and scalability.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets and Metric. We evaluate MARS on both vision and NLP tasks using standard benchmarks. For vision tasks, we adopt CIFAR- 100 (Krizhevsky & Hinton 2009), which contains 100 classes with 50,000 training images and 10,000 test images of size 32 x 32, and
(Le Yang
2015), which includes 200 classes with 500 training. 50 validation, and 50 test images per class of size
64 x 64, Following standard class-incremental protocols (Han Guo. 2022 Liu et al. 2024 Pietron et al., 2025), CIFAR-100 is divided into 10 tasks with 10 classes each, and Tiny-ImageNet into 10 tasks with 20 classes each. For NLP tasks, we use 19 aspect-based sentiment classification (ASC) datasets adopted in prior work (Ke et al. 2021b), where each dataset corresponds to a product domain such as laptops, restaurants, cameras, or phones, and is annotated with three sentiment polarities: positive, neutral, and negative. Each dataset is treated one task, which enables evaluation of MARS under diverse domains, different class sizes, and distribution shifts. After training on task 4, the model is evaluated on the test sets of all tasks 1,...,,, l, and the average accuracy A = E "t,s computed, where or is the accuracy on task after learning task t. This produces a trajectory of average accuracy as tasks accumulate, which typically decreases due to forgetting, Unless otherwise noted, we report Ar, the average accuracy after completing the entire sequence. All experiments are conducted with random seeds [1, 123, 1234) on NVIDIA RTX 5090 GPUs
Baselines and Settings. We compare MARS with representative continual learning methods. including EWC (Kirkpatrick et al. 2017). iCaRL (Rebuffi et al. 2017), DER++ (Buzzega et al. 2020), LDC (Gomez- Villa et al. 2024), and PASS++ (Zhu et al. 2025p. To ensure fairness, each baseline is evaluated under two settings. In the standard setting, the entire backbone is trainable as in the original method. In the frozen-encoder setting, the backbone is fixed and only lightweight components such as task-specific heads or adapters MARS and avoids bias toward methods that gain mainly from updating large number of backbone parameters. Replay-based methods (iCaRL, DER++, PASS++) restricted to exemplar budget comparable to the anchor storage in MARS. In addition, all methods use the same encoder, training schedule, and evaluation protocol ensure consistent comparisons.
updated. This matches the capacity used by
Implementation Details. For vision benchmarks, we use CLIP (Radford et al. 2021 as the frozen encoder f()), with its vision transformer (ViT-B/16) producing features of dimension dr. For NLP tasks, we use BERT-base (Devlin et al. 2019), also with frozen parameters. On top of the encoder, the memory router implemented as linear projection Wg that maps frozen features into query space of dimension d4 = 64, which is then compared with the slot key set K = {ks S.S. to compute routing probabilities. We initialize with S. 32 slots, set the quantile momentum to 3 = 0.9, and adopt a Wilson score threshold at 95% confidence. Training follows the two-stage DCDA protocol. In Stage (feature adaptation), using supervised contrastive loss with batch size 128 and temperature - = 0.07, together with a smoothness tether weighted by Asmonth 0.3. In Stage (classifier tuning), train the classifier for 20 epochs with cross-entropy loss and two distillation terms. The learning rate is 0.001, and entropy regularization optionally applied with coefficient 0.1.
update only the memory parameters (W₃z K, K, for 20 epochs
fix the memory and
4.2 EXPERIMENTAL RESULTS
Main Results. Table I reports the average accuracy across benchmarks. Replay-based methods such as DER++ and PASS+ outperform regularization-be based methods such as EWC, but their reliance on small exemplar memories causes performance to plateau as the task sequence increases. On CIFAR-100 and Tiny- Imageee,, these methods converge around 22.54, while MARS consistently achieves 56-58% relative gain of about 3-5%. On ASC, DER++ and PASS++ stabilize near 74-.
8

---

Published as a conference paper at ICLR 2026
Table Average accuracy of different methods under standard and frozen-encoder settings.
CIFAR-100
Tiny-ImageNNet
ASC
Algorithm
Standard
Frozen
Standard
Frozen
Standard
Frozen
Fine-tune
30.74:0..43
30.26:10.20
28.32=0.65
28.2720.43
60.9010.29
61. 61.30-0.80
EWC
47.84+0.58
47.60=+0.40
36.47=0.54
36.38+0.39
70.26+00.6
70.66=0..69
DER+
52.24:10.66
51.72:10.47
40.99=0.37
40.8711.16
75.53:10.27
75.91=0.21
LDC
54.14+00.17
53.95++0.48
43.39+0.63
43,41+0.55
75.11+0.60
75.49+0.23
PASS+
53.67++0.50
52.92+0.52
42.31+0.61
42.53+0.70
74.72+0.20
75.22+00773
ours
57.33-10.48
57.50
09.12:1.36
49.46:10.14
79.45::0.25
0.665
Accuracy (%)
65
5o
Default
Accuracy (%)
65
Defsut:
8-05
No-SGSE
Defaul:
50 16
80
8-0.7
Accuracy
No-Anchor
5o 64
56
No-Stagel
50
5o = 120
50
0 99
50
No-Stage2
45
45
45
8 10
9
9.10
Task
Task
Task
(a) Effect of varying So
(b) Effect of varying B
(c) Design validations
Figure 2: Ablation study results on Tiny-ImageNNe
75%, whereas MARS reaches 78-79%, showing that it retains domain knowledge without raw data. LDC also improves over DER, but its gains remain below those of MARS, suggesting that heuristic consolidation is less effective than statistically grounded slot expansion with anchor distillation. Another important observation is that the difference between the standard and frozen- encoder settings is usually within 1-2%, rather than fixed gap. This shows that improvements do not come from updating the backbone, but from how models allocate and preserve capacity for new tasks. By combining statistical slot expansion with dual-stage adaptation, MARS achieves better balance between stability and plasticity. Through controlled expansion and anchor-base retention, it consistently provides higher accuracy under the same memory budget, demonstrating its suitability for continual learning with large pre-trained models.
Effect of Varying Sg. Figure 2 a) shows that the initial slot number So strongly influences perfor- mance. A small So (e.g., 8)=8) causes accuracy to drop quickly after a few tasks due to limited capacity and strong interference. Increasing So 16-66 64 improves performance, with the best results at Sp=32, which maintains higher accuracy across tasks. Enlarging So to 128 gives no benefit and slightly degrades later accuracy, likely from redundant slots and noisy routing. These results confirm that initialization is important: too few slots reduce plasticity, while too many reduce stability.
Effect of Varying B. Figure b analyzes the smoothing coefficient B, which controls how the statistical trigger adapts to shifts in routing confidence. A small 3 (e.g., 0.5) causes unstable quantile estimates, leading to premature expansions and lower accuracy. As /3 increases to 0.7-0.95, performance improves steadily, with 3=0.9 offering the most robust balance. When / is too large (0.99), the estimator reacts too slowly to distributional shifts, delaying necessary expansions and harming late-task accuracy. These findings validate our choice of 3=0.9, which balances stability and responsiveness for continual learning.
Validation of Design. Figure 2(c) highlights the complementary roles of SGSE, anchors, and the two-stage adaptation. Removing SGSE leads to steep accuracy drop (final accuracy ~41%), confirming that statistically grounded slot expansion is essential for maintaining sufficient capacity. Removing anchors causes a similar decline (final accuracy ~42%) underscoring their importance for knowledge retention without replay. Disabling Stage (contrastive feature adaptation) reduces representation alignment (final ~43%), while omitting Stage 2 (classifier distillation) yields the lowest accuracy (final ~40%), showing that both stages are necessary. Together, these results show that SGSE, anchors, and dual-stage adaptation work together: SGSE regulates expansion, anchors preserve knowledge, and dual-stage adaptation balances stability and plasticity.
Anchor Diagnostics. We further examine the behaviour of the anchor space using three empirical diagnostics. Since anchors and routed features in the same feature space Rdr, cosine similarity provides direct way assess how each anchor relates to the features assigned to its slot. Across tasks, these similarity values remain within the range 0.60-0.85 and vary smoothly as new classes are introduced. To assess temporal stability, we compare each anchor to its counterpart after consecutive

---

Published as a conference paper at ICLR 2026
tasks and obtain stability scores between 0.65 and 0.98, indicating that the updates are gradual rather than abrupt. A nearest neighbor inspection further shows that anchors tend to remain associated with coherent groups of feature patterns, such vehicles, animals, or background textures. Together, these diagnostics suggest that the anchor space preserves a stable and interpretable structure throughout the task sequence. Additional analyses are provided in Appendix A.4
Slot Growth, We visualize how the number of slots changes during training on Tiny-ImageNer, CIFAR-100, and ASC in Figure 3] In all cases, SGSE expands the memory only when the confidence statistic exceeds the Wilson bound for several steps. The slot count grows steadily during the early tasks and then approaches a stable value as learning continues. On Tiny- Imagee,,,,,,t, the slot count increases from So = 32 to about ST = 49. On CIFAR-100, 100, it reaches approximately ST = 44. On ASC, it increases to around ST = 58 more domains are introduced. These results are consistent with the theoretical analysis and show that SGSE provides smooth and controlled capacity expansion.
2
4
6
8
10
12
14
16
18
60
CIFAB-100
Number of Slots
65
Thny ImageNet
50
45
40
35
30
0
3
Task
Figure 3: Slot growth across tasks.
Extended Baseline Comparisons. We include ad- ditional PTM and PEFT baselines under the same frozen-encoder protocol. These baselines include
Table 2: Extended baseline comparisons.
Method
CIFAR-100
Tiny-IN
ASC
L2P (Wang et al. 2022). CODA-Prompt (Smith)
L2P
52.30:10.45
43. 13,80-0.322
73.90-
CODA- CODA-Prompt
54.71++061
45.10+0.58
74.70+00.4
al. 2023), and a representative CLIP-oriented
CLAP4CLIP
55.42:10.50
46.5022.64
method (Jha et al. 2024). All methods use the
MARS
57.50-10.54
49.46 +0.14 79.85++0.66
frozen CLIP ViT-B/16 backbone and have train- able components on the order of 10"h parameters, ensuring comparable effective capacity. Across all benchmarks, MARS achieves the highest accuracy. These results show that the gains of MARS come
from statistical slot expansion and anchor- based distillation rather than from prompting strategies.
Scalability Analysis. We also evaluate the method on ImageNet-100. 100. MARS reaches 49.46% on Tiny-ImageNet, which is 2.96 points higher than the -rienteee baseline. On ImageNet- 100, MARS also performs better than the best frozen- backhone baseline. During this evaluation, the slot count grows from S. 32 to about ST 65. This
Table 3: Performance on larger-scale data.
Dataset
Best Baseline
MARS
Final Sy
Tiny ImageNet 46. 66.50410664
49.46.0.14
ImageNet-100 100
39.67=0.60
42.08=0.53
65
growth remains moderate and shows that SGSE maintains stable and predictable capacity expansion as the dataset size and complexity increase.
Parameter and Inference Cost. We com- pare parameter count and inference time on Tiny-ImageNer with PTM/PEFT baselines under the eonzene setting (L2P, CODA-Prompp CLAP4CLIP). These baselines typically use 0.5M- 0.8M trainable parameters, whereas MARS re- quires only 0.2M, making it substantially lighter. Despite dynamic expansion, the inference overhead remains small: MARS reaches 8.5ms per batch, only a minor increase over the baselines' 7.8-8. ms. Within this group of methods, accuracy ranges from 43.8% to 46.5%, while MARS achieves 49.46%.
Table 4: Parameter and inference cost.
Metric
Baselines
MARS
Trainable parameters
0.5M to 0.8M
0.2M
Inference time per batch
7.8ms to 8. 8.1ms
8.5ms
Final accuracy (%)
43.80 to 46.50
49,46
5 CONCLUSIONS AND LIMITATIONS
In conclusion, we present the MARS framework for continual learning with large pre-trained models, which integrates statistical slot expansion, anchor-based retention, and a dual-stage adaptation strategy. This design improves the stability- plasticity balance while remaining scalable under practical constraints. A key advantage is its reliance on frozen encoders and lightweight modules, making it applicable to both vision and language tasks. Despite these strengths, the framework has limitations. It depends on a reliable frozen encoder, which may not capture fine- ane-grained features in new domains. It also requires careful tuning of hyperparameters that control expansion and adaptation. In addition, although the method reduces reliance on raw data, it does not remove memory costs entirely. Addressing these challenges is important direction for future work.
plability-paastiiity
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was fully supported by the National Natural Science Foundation of China for Excellent Young Scientists Fund Program (Overseas), awarded in 2024.
REFERENCES
Rahaf Aljundi, Francesca Babiloni. Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139-154, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1677.066 2016.
Lawrence D. Brown, T. Tony Cai, and Anirban DasGupta. Interval estimation for a binomial proportion. Statistical Science, 16(2):101-133, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, et al. Language models few-shot learners. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 33, 1877-1901, 2020.
Pictro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: A strong, simple baseline. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 15920-15930, 2020.
Clément L. Canonne. A short note on an inequality between kl and tv. arXiv preprint arXiv:2202.07198, 2022.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. Proceedings of International Conference on Learning Represen- tations (ICLR), 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning (ICML), pp. 1597-1607, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 2023.
Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non- the stationary data streams. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8250-8259, 2021.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, et al. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(7):3366-33 3385, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)) pp. 4171 41866, 2019.
Songlin Dong, Xinyuan Gao, Yuhang He, Zhengdong Zhou, Alex C. Kot, and Yihong Gong, Ceat: Continual expansion and absorption transformer for non-exemplar class-incremental learning. IEEE Transactions on Circuits and Systems for Video Technology, 35(4):3146-315 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
11

---

Published as a conference paper at ICLR 2026
Alex Gomez- Villa,, Dipam Goswami, Kai Wang, Andrew D. Bagdanov, Bartlomiej Twardowski, and Joost Van De Weijer. Exemplar-fre continual representation learning via learnable drift compensation. In Proceedings of European Conference on Computer Vision (ECCV), pp. 473-490, 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of International Conference on Machine Learning (ICML), pp- 1321- 1330, 2017.
Xuejun Han and Yuhong Guo. Overcoming catastrophic forgetting for continual learning via feature propagation. In Proceedings of the British Machine Vision Conference (BMVC), 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1503v 2015.
Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Brianna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of International Conference on Machine Learning (ICML), pp. 2790-2799, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Wang. Lora: Low-rank adaptation of large language models. In Proceedings of International Conference on Learning Representations (ICLR), 2022.
Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for vision-language models. In Proceedings of Advances in Neural Information Processing Systems (NeurlPS), pp. 129 129146-1 129186, 2024.
Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. Achieving forgetting prevention and knowledge transfer in continual learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pp. 22443-22456, 2021a.
Zixuan Ke, Hu Xu, and Bing Liu. Adapting bert for continual learning of sequence of aspect sentiment classification tasks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HHLT)) pp. 4755-4755, 2021b.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samn,, Yonglong Tian, Phillip Isola, al. Supervised contrastive learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pp. 18661-18673, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, al. Overcoming catastrophic forgetting in neural networks. The Proceedings of the National Academy of Sciences (PNAS), 114(13):521-3356, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. In Technical Report, University of Toronto, 2009.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N Course Project, 2015.
Gwen Legate, Nicolas Bernier, Lucas Caccia, Edouard Oyallon. and Eugene Belilovsky. Guiding the last layer in federated learning with pre-trained models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 19832-69848, 2023.
12

---

Published as a conference paper at ICLR 2026
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3045-3055 2021.
Yukun Li, Guansong Pang. Wei Suo, Chenchen Jing, Yuling Xi, Lingqiao Liu, Hao Chen, Guoqiang Liang, and Peng Wang. Coleclip: Open-domain continual learning via joint task prompt and vocabulary learning. IEEE Transactions on Neural Networks and Learning Systems, 2025.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-29477 2017.
Ruiqi Liu, Boyu Diao, Libo Huang, Zijia An, Zhulin An, and Yongjun Xu. Continual learning in the frequency domain. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pp. 15389185411,222 2024.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pp. 2 21464 21475, 2020.
Wenzhuo Liu, Xinjian Wu, Fei Zhu, Mingming Yu, Chuang Wang, and Cheng-Lin Liu. Class- incremental learning with sulf-supe pre-training and prototype learning. Pattern Recognition, 157:110943, 2025.
Xialei Liu, Xusheng Cao, Haori Lu, Jia wen Xiao, Andrew D. Bagdanov, and Ming-Minn Cheng. Class-incremental learning with pre-trained vision-language models. arXiv preprint arXiv:2310.203488 2023.
Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017. David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning, In
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology Learning and Motivation, 24:109-165, 1989.
German Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 11::54-71, 2019.
Marcin Pietron, Kamil Faber, Dominik Zurek, and Roberto Corizzo. Tinysubnets: An efficient and low capacity continual learning strategy. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 19913-19920, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 8748-8763, 2021.
Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In Proceedings of International Conference on Learning Representations (ICIR), 2021.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001-2010, 2017.
Stuart Roberts. Control chart tests based on geometric moving averages. Technometrics, 42(1): 97-101, 2000.
Andrei A. Rusu, Neil Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677 2016.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt Continual decomposed attention- based prompting for rehearsal free continual learning. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11909-11919, 2023.
13

---

Published as a conference paper at ICLR 2026
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017.
Longxiang Tang, Zhuotao Tian, Kai Li, Chunming He, Hantao Zhou, Hengshuang Zhao, Xiu Li, and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vasion-language models. In Proceedings of European Conference on Computer Vision (ECCV), pp. 346-365, 2024.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5332-5388, 2024.
Zifeng Wang. Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, et al. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 139-149, 149, 2022.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In Proceedings of International Conference Learning Representations (ICLR), 2018
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of International Conference on Machine Learning (ICML), pp. 3987-3995, 2017.
Fei Zhu, Zhen Cheng, Yuo Zhang, and Cheng-Lin Liu. Class-incremental learning via dual augmentation. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS),
pp. 14306-14318, 2021.
Fei Zhu, Yu-Yao Zhang, Zhen Cheng, and Cheng-Lin Liu. Pass++: A dual bias reduction framework for innsssincremettll learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(8):7123-7139, 2025.
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)
During the preparation of this paper, we made limited use of large language models writing assistants. Their role was restricted to checking grammar, improving clarity, and polishing exposition. All technical ideas, methods, and experiments were fully developed and validated by the authors.
A.2 DETAILS OF THEORETICAL FOUNDATION
(Monotonicity of &t in c.). This result shows that the top-slot confidence 8: max; (x,) behaves calibrated statistic for novelty detection: when the similarity between the query and its best-matching key increases, the corresponding softmax confidence increases strictly, provided the other similarities are fixed.
Proposition
Proof. Let aj ()(x:), ky). and let E arg max, aj with e == a,.. All other similarities {a, }ite are treated as constants during this analysis. Define
A
A
since S, 1.
Then the maximum softmax confidence is
e0/T
s(c) c + A 1+ Ac-e/T,"
This is logistic-type function of c, strictly between 0 and 1. Differentiating with respect to e gives
ds 1 C/TT
1 (c) s(c)).
de T, (1+ Ae-o/T.)0 Tr
Since T, > 0 and s(c) the derivative is positive. Thus, conditional other similarities being fixed, the top-slot confidence s, is strictly increasing in c, i.c. the maximum cosine similarity. This monotonicity means st faithfully reflects changes in slot affinity, making it suitable indicator. IT
14

---

Published as a conference paper at ICLR 2026
Theorem 1 (EMA quantile tracker under weak dependence). This result analyzes the exponen- tially smoothed quantile statistic Qi that underlies SGSE. We show (i) convergence in mean square to the long-rum quantile and (ii) predictable timescale for detection after mean shift.
Proof. As defined by:
Q. Q- 3Q + B)g::
B E [0, 1),
where {9:} h is a stationary sequence with Elqi] = q" For clarity, first assume {ge} are i.dd. with
variance og. By taking expectations, we have
E[Qu B + =)qa
This is standard linear recursion with solution
E [2] 9- + 8 (Qu -²)).
Hence Q converges in expectation to q² as t - x. Then, for the variance,
Var(Q:) = 3² Var( + (1 3)
Unrolling this recursion,
1-1 Var(Qu) -1 (1
(1 (1-
1=0 1 B2 0 - B2
820
As t -> co, this converges to (1-8)2 1-3a og Thus Qi g" in L If qu are not ii.d but weakly dependent
(e.g., a-mixing), the same result holds with og replaced by the long-run variance. Further, suppose at time r the mean shifts from q² to q' < For 1: 2 O, + 8t (E[Q-) g')-
E[Q+++]
Fix threshold with o' < < q² The smallest integer is such that E[Q++*] 0 must satisfy
8 S E(Q,] q
Taking logarithms,
In EIQ-I-9
k 2
0-0"
Using the inequality In 2 1 - for 3 E [0, 1), we obtain
5 1 In 18 10
1
Therefore, the mean-crossing index (i.e., how many steps until the expected trajectory falls below 8) scales as ((11 - -)- This provides a predictable detection timescale: smaller (1 - 3) (i.e., heavier smoothing) leads to slower adaptation.
Corollary T (False expansion control). This establishes that the Wilson lower-bound test provides approximate per-test false expansion control at level a.
Proof. Let X1..... X, ~ i..d. Bernoulli(p) with = Pr(st 2 Q+) 2 r. Define p. = = Σn X,. The one-sided Wilson lower bound LB(p,; (inii,, n, 2) with 2 = (11 a) satisfies, by score-test theory,
Pr(LB n, 2) < p) > a.
Since p 2 the event (LB < T) implies {LB < p). Therefore,
Pr(LB(fn:nn z) <) < Pr(LB n, z) < <p) a
up to normal approximation enror. Thus the per-test false expansion probability approximately controlled level
15

---

Published as a conference paper at ICLR 2026
Theorem 2 (Anchor-based retention). This theorem shows that, under mild assumptions, anchor- based distillation guarantees bounded deviation between the old and new models' predictions on old-class features.
Proof. Let p(u) softmax(u/T)) denote the temperature-scaled softmax. By assumption (ii), for each anchor A,
(a)p(o""()) (4)) -
By Pinsker's inequality,
[lp(g (a)) - p(g(a))ll S 2n.
Now consider any old-class feature h within distance of some anchor a. By Lipschitz continuity of logits (assumption (i)),
llo(h) - g(a) S Lτ,
I - (a)l & Lg.
And the Jacobian of p(u) is
Vp(u) 7 Diag(p(u)) p(
Its operator norm is bounded by 1/(27) in f/ > la norm. Thus, by the mean-value theorem,
VC
(l((((b)) - p(g(a)) - vC. (CC-1VP(()))l-2 Ilo(b) -((a)ll s 27
Lg,
and similarly
VC
[p و ( (1) S
L5.
2T
Here VC comes from ||l|| Vlllull, and can be absorbed into big-O notation. By applying the triangle inequality. we have
lp old (h)) (()(h))ll < (Ip()")(a)) -(((()(())_ 4.5 2n 45 0(1)
Hence, for any old-class feature, the deviation between old and new softened predictions is bounded by O(V (L Under mild posterior-marge conditions, this ensures the drop in classification accuracy is controlled at the same order.
Proposition 2 (Slot growth bound). This proposition shows that SGSE separates true expansions
(driven by genuine novelty) from false expansions (caused by noise), and that the latter are statistically controlled.
Proof. Let = L(T ((T-w)/m w)/mJ denote the number of hypothesis tests up time T. For each test let Y, E (0, 1} be the indicator false expansion. By Corollary [
Pr(Y 1) a.
Thus
M Σ Yg. f=1
E B|y S a,
[[F] < aM, where F
If we ensure test windows are disjoint (i.e., S m), then the Yj's are independent. By Hoeffding's inequality,
2e
Pr(F
[[ S exp
M
Choosing
M
2 lm(1/8) yields
F < aM
In
with prob. 1 -. 8.
Let Ny be the number of true expansions. Then the total slot count is Taking expectations,
ST S S. Ny +
E[ So + E[NT] aM, and the high- probabiibybylity bound follows from the inequality above.
16

---

Published as a conference paper at ICLR 2026
Algorithm 1 MARS: Training with SGSE and DCDA
1: for each task t = ,, T do
2: Initialize buffers: success buffer B (size n) and low-confidence buffer
3: Initialize quantile tracker Q1, with the first batch.
4: Stage 1: Feature Adaptation (memory-only)
5: for each mini-batch D, do
6:
Extract frozen features by < fxx and queries t- Wyhr, q < g/llgll-
7:
Compute routing probabilities (x) exp (q. kz) TTe) and top confidence s(x).
8:
Update slot statistics μi, G, with EMA (a 0.05) and anchors a,
9:
Update quantile ge from last samples and smooth Qt + BQ (1 - B)q(-
10:
Record Bernoulli trial X X(x x) in buffer B; compute empirical success rate p.
11:
if LL n, 2) Tsace (Wilson lower bound test) then
12:
Expand: Add new slot j with key ky from mean query L; set (7/- B3) = (1, 0)
end if
14:
Update L with lowest-confidence samples in batch.
15:
Compute adapted features h = P(TR) @ LN(hr) + P:B:).
16:
Optimize memory by minimizing (()) Suppon (h;7) T) + Ammouth h hy 11.
17:
end for
18:
Stage 2: Classifier Tuning (head-only)
19:
Store old classifier gold <- 9- for each mini-batch D. do
20:
Compute logits 2 + (h)) old + (h)
21:
Compute anchor logits za + g(a), zold + (ald(a) for e A.
23:
Minimize ( and update only g. end for
24:
end for
26:
return (W K, 3,9))
Theorem (Overall complexity). costs.
Finally, we connect slot growth to computational and memory
Proof. From Lemma
1
Time(x;) θ ((d+ + dr)):),
Mem; = @(ds + dr) + +gg-
Taking expectations and substituting Proposition
E[Time(zT]] = 0 + E[NT| + aMM)
E[Meny] = O dr) ( + aM)) + lgl.
For the biobbbbobybilityy bound, replace ST by its probabilistic upper bound in Proposition 2 which yields the same asymptotic order. Thus both compute and memory scale linearly with slot count, and slot count itself is controlled by SGSE.
A.3 OVERALL WORKFLOW OF MARS
The overall design MARS integrates two complementary mechanisms on top of the frozen LPM backbone. As shown in Algorithm [ SGSE monitors router confidences and decides when to create new slots by formulating expansion as a statistical decision problem with guarantees on false alarms and detection delay. When new slot is added, DCDA controls its integration: Stage aligns slot features through supervised contrastive learning with smoothness regularization, and Stage tunes the without- wwthout-Forgettiie Forgetting distillation current inputs and anchor- based distillation on surrogate prototypes. This workflow ensures controlled slot growth, efficient adaptation, and a provable sability-plasticn balance without updating the large pre-trained encoder.
classifier with Learning -earningwwthouu-
17

---

Published as a conference paper at ICLR 2026
Table
Anchor-feature similarity on Tiny-ImageNet.
Anchor
After Task 1
After Task 2
After Task 3
After Task 4
After Task
A1
0.782 + 0.046
0.759 + 0.038
0.746 + 0.041
0.762 + 0.029
0.755 0.040
A2
0.842 = 0.049
0.825 + 0.050
0.807 = 0.042
0.792 0.034
0.781 0.056
A3
0.603 + 0.029
0.618 + 0.033
0.635 + 0.042
0.648 0.047
0.662 + 0.039
A4
0.701 + 0.027
0.718 + 0.038
0.734 + 0.041
0.725 + 0.039
0.712 + 0.026
Anchor
After Task 6
After Task 7
After Task 8
After Task 9
After Task 10
A1
0.770 + 0.042
0.758 + 0.040
0.749 + 0.045
0.761 + 0.043
0.752 + 0.036
A2
0.794 = 0.027
0.786 + 0.055
0.778 + 0.061
0.791 + 0.034
0.783 + 0.048
A3
0.671 + 0.041
0.658 + 0.041
0.645 + 0.022
0.661 + 0.044
0.653 + 0.051
A4
0.728 + 0.026
0.735 + 0.053
0.742 + 0.049
0.726 0.046
0.732 0.037
Table 7: Nearest neighbor classes for selected anchors on Tiny-ImageNet.
Anchor
Nearest classes
Interpretation
A1
truck, ship, bus, related vehicle classes dog, cat, deer, bird classes with fireuuett sky water textures* bird, airplane, ship
rigid objects or vehicles animal categories
A2
A3
textures background
A4
open or airborne scenes
A.4 ANCHOR COVERAGE DIAGNOSTICS
This section provides additional diagnostics that examine the coverage assumption used in Theorem We evaluate the behaviour of the anchors on Tiny-ImageNet under the frozen-encoder setting. Because anchors and routed features share the same feature space Edr,, we can compare them directly using cosine similarity. We report three diagnostics that characterize anchor-featur similarity, temporal stability, and semantic coherence.
Anchor-feature Similarity. We first study how each anchor relates to the routed features assigned to its slot. We randomly sample four anchors and compute the cosine similarity between each anchor and the router-weitg weighte average of its assigned features after Tasks 1 through 10. The results in Table J]ble show that these similarity values remain in the range 0.600-0.88 and change smoothly as new classes are introduced. This indicates that the anchors stay close to the feature distributions.
Anchor Stability. We next examine how each anchor evolves over the task sequence. We compute the cosine similarity between the same anchor after consecutive tasks and average this value across all anchors. This diagnostic measures the temporal consistency of the anchors and is dis- tinct from the anchor-fcature similarity reported above. The values in Table 6 show that the anchors change smoothly. This behaviour agrees with the exponential moving average update rule described in Section 3.1 These results further support the local coverage assumption that appears in Theorem
Table 6: Anchor stability across tasks.
Metric
Value
Mean stability
0.823
Max stability
0.972
Min stability
0.642
Anchors with stability > 0.7
79%
Semantic Coherence. We also study the semantic coherence of the anchors. For each anchor, we retrieve the Tiny-ImageNet classes whose mean features are closest to the anchor. We then describe the shared visual patterns in these classes. The results in Table sshow that the anchors remain aligned with coherent semantic groups throughout the entire training process. These groups include rigid objects, animals, background textures, and scenes with clear open-space patterns. This behaviour suggests that the anchor space organizes features a stable and interpretable way as tasks accumulate. For classes marked with an asterisk, the descriptive terms refer to shared visual textures such as sky or water rather than official Imng-Im labels.
Across all diagnostics, the anchors remain close to routed features, evolve smoothly across tasks, and preserve meaningful semantic structure. These observations support the practical validity of the coverage assumption in Theorem 2 They also show that the anchor space maintains stable and interpretable behaviour throughout the full task sequence.
18


Reviews:
[{"review_id": "wWNHmL5Sob", "reviewer": "Reviewer_h2XS", "summary": "This paper addresses the problem of catastrophic forgetting in the context of continual learning (CL) with large, frozen pre-trained models (LPMs). The authors argue that the stability-plasticity dilemma is particularly severe in this parameter-efficient setting, where adaptation is confined to shallow modules. To tackle this, they propose MARS (Memory-adaptive Router with Statistical control), a modular framework that decouples the frozen encoder from an expandable, adaptive capacity layer. The core contributions are two-fold: (1) Statistically-Grounded Slot Expansion (SGSE), a mechanism that formulates the decision to add new model capacity (slots) as a statistical test based on router confidence, providing formal guarantees against uncontrolled growth; and (2) Dual-Stage Contrastive-Distillation Adaptation (DCDA), a replay-free strategy that integrates new slots using supervised contrastive learning and preserves knowledge from past tasks via anchor-based distillation. Experiments on diverse vision (CIFAR-100, Tiny-ImageNet) and NLP (ASC) benchmarks demonstrate that MARS achieves state-of-the-art performance, outperforming strong baselines in the frozen-encoder setting.", "questions": "1. The SGSE mechanism is a cornerstone of your method, designed to control the rate of \"false alarm\" expansions. Could you please report the final number of slots ($S_{T}$) created in the CIFAR-100, Tiny-ImageNet, and ASC experiments\n2. Theorem 2 provides a retention bound that relies on the assumption that old-class features are well-approximated by the stored anchors (i.e., they lie within a distance $\\delta$). How is this coverage assumption maintained in practice, especially as the number of tasks increases and the fixed number of anchors per slot must represent an increasingly diverse set of feature distributions?\n3. The DCDA mechanism is entirely replay-free. Have you considered a hybrid approach where a small, fixed-size replay buffer is used to supplement the anchor-based distillation?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Principled and Theoretically Grounded Capacity Expansion: The standout contribution of this work is SGSE. It replaces common heuristic-based triggers for network expansion with a rigorous statistical framework. \n2. Effective and Replay-Free Knowledge Retention: The DCDA mechanism is an elegant and effective solution for integrating new knowledge while preserving old. \n3. High Relevance to Modern Machine Learning Paradigms: The paper is exceptionally well-positioned within current research trends.\n4. Comprehensive and Rigorous Empirical Validation: The authors validate MARS on a diverse suite of benchmarks, including multi-task vision classification", "weaknesses": "1. Dependence on Frozen Encoder Quality: The performance of MARS is fundamentally tied to the quality of the representations from the frozen LPM. \n2. System Complexity and Hyperparameter Sensitivity: The complete MARS system, while modular, involves a significant number of components and associated hyperparameters\n3. Potential Scalability Concerns for the Router: The per-example computational overhead scales linearly with the number of active slots as shown in Theorem 3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uLO01n8l61", "reviewer": "Reviewer_pmFB", "summary": "This paper presents MARS, a modular approach designed for continual learning with pre-trained models (PTMs). The architecture is composed of three key components: a frozen pre-trained encoder, a slot-based memory module, and a lightweight classifier. The authors propose two primary mechanisms to address catastrophic forgetting: (i) Statistically-Grounded Slot Expansion (SGSE), which formulates memory expansion as a statistical decision problem to ensure controlled growth, and (ii) Dual-Stage Contrastive-Distillation Adaptation (DCDA), a replay-free method that integrates new knowledge through supervised contrastive learning and retains past information via knowledge distillation. Experimental results on standard continual learning benchmarks for both vision and NLP validate the effectiveness of the proposed framework.", "questions": "In addition to the issues raised above, I have the following questions for the authors:\n\n- What is the architecture of the lightweight classifier head $g(\\cdot)$? Is it an MLP or a simple linear layer?\n\n- How does the number of slots $S_t$ vary: (i) across tasks, (ii) with different task orders, and (iii) under different class distributions within each task?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The introduction of SGSE, which uses statistical tests to determine when to expand model capacity, is an interesting contribution, as prior methods typically increase sub-modules (e.g., prompts) linearly with the number of tasks.\n- The effectiveness of MARS is demonstrated through evaluations on both vision and NLP datasets.\n- The experiments include detailed ablation studies that probe the impact of key components (SGSE, anchors, and each DCDA stage), as well as analyses of important hyperparameters.", "weaknesses": "- Proposition 1 appears to be both incorrect and irrelevant to the context. It neglects the term $\\frac{dA}{dc}$, even though both $A$ and $c$ are functions of $x_t$. Consequently, the proofs and resulting claims seem invalid. A counterexample can be illustrated as follows: when $S_t = 3$ and $a_1 = 3, a_2 = 3, a_3 = 3$, we have $c_t = 3$ and $s_t = 1/3$. However, when $a_1 = 2, a_2 = 1, a_3 = 1$, then $c_t = 2$ and $s_t = \\frac{e^2}{2e + e^2} > 1/3$. Thus, the claim that $s_t$ is strictly increasing with respect to $c_t$ is false. This represents a fundamental flaw. Moreover, the interpretation in Lines 185–187 appears vague and not directly relevant to the CL context.\n\n\n- The paper suffers from a lack of clarity in its presentation. Key variables are used without prior definition, such as $w$ in Equation (10) and $P(i)$ in Equation (18). The writing structure is also confusing; for example, the Router-weighted EMA in Equations (7-8) is introduced prematurely, disrupting the logical flow of the methodology section.\n\n- The most critical concern is that, while SGSE aims to determine when to dynamically expand model capacity, the paper provides no analysis of how the number of slots $S_t$ grows with the number of tasks. The experiments report only accuracy results. This is a very shallow investigation that undermines a central contribution of the paper.\n\n- Another major issue is that the authors completely ignore most CL methods based on pre-trained models (e.g., L2P [1], HiDe-Prompt [2], NoRGa [3], and SD-LoRA [4]), comparing instead only with conventional CL approaches.\n\n- The paper lacks empirical analysis regarding the number of parameters and computational cost compared to other methods.\n\n- There is no investigation into the effect of different pre-trained models (e.g., iBOT, DINO). This omission is concerning, as the proposed method may rely heavily on the frozen representations of the pre-trained backbone.\n\n[1] Learning to Prompt for Continual Learning, CVPR 2022\n\n[2] Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality, NeurIPS 2023\n\n[3] Mixture of Experts Meets Prompt-Based Continual Learning, NeurIPS 2024\n\n[4] SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning, ICLR 2025\n\n[5] RanPAC: Random Projections and Pre-trained Models for Continual Learning, NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "npSk0HKVCw", "reviewer": "Reviewer_zoZH", "summary": "MARS is a continual-learning framework for frozen LPMs that shifts adaptation to a slot-based memory router plus a lightweight classifier. It expands capacity only when SGSE flags novelty, using an EMA high-quantile tracker of routing confidence and a one-sided Wilson bound to provide predictable detection delay and per-test false-expansion control. Knowledge retention is handled by DCDA: Stage-1 contrastive alignment of memory parameters and Stage-2 head-only training with cross-entropy, LwF, and anchor distillation, avoiding raw replay. Experiments report state-of-the-art performance under this frozen-backbone protocol.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Functionalities are well split into a frozen encoder, slot-based router, and classification head\n2. The proposed method is well-motivated and easy to follow\n3. Principled expansion logic. SGSE uses a smoothed high-quantile of top-slot confidence and a one-sided Wilson score", "weaknesses": "1. Justify using CLIP-ViT when the text encoder is unused and a separate classifier is trained, and add comparisons against a standard ImageNet-pretrained ViT under the same frozen-backbone regime.\n2. Compare against CLIP-focused/frozen baselines (e.g., CLAP4CLIP [A]).\n3. P(i) in Eq. 18 is not defined; specify it explicitly (I'm guessing the index set of positive samples for data i as in supcon).\n4. The method introduces many hyperparameters for both stages, and needs more sensitivity/ablation studies to demonstrate robustness and generalization.\n5. Benchmark coverage. Expand the benchmarks as in other CLIP/PEFT-oriented baselines in the vision domain (CODA-Prompt [B], EASE [C], DIKI) and the language domain (O-LoRA [D]), and consider larger or more diverse datasets.\n6. Report wall-clock time, memory, and inference overhead for the two-stage training vs. prior methods\n7. Provide qualitative results (e.g., nearest-token/phrase visualizations) that anchor capture domain-specific semantics.\n8. Router/slot introspection. Add diagnostics/visualizations such as slot-usage over tasks and growth curves to substantiate controlled expansion and specialization.\n\n[A] Jha, S., et al. Clap4clip: Continual learning with probabilistic finetuning for vision-language models. NeurIPS 2024.\n\n[B] Smith, J. S., et al. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. CVPR 2023.\n\n[C] Zhou, D.W., et al. Expandable subspace ensemble for pre-trained model-based class-incremental learning. CVPR 2024.\n\n[D] Wang, Xiao, et al. \"Orthogonal subspace learning for language model continual learning.\" EMNLP 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIAky0oPM5", "reviewer": "Reviewer_7T3S", "summary": "The paper introduces MARS, a framework for continual learning with frozen large pre-trained models. MARS decouples stable representation from adaptive capacity using a frozen encoder, a slot-based memory router, and a lightweight classifier. It has two core components: SGSE formulates capacity growth as a statistical decision problem with guarantees and DCDA integrates new capacity through contrastive learning and anchor-based knowledge distillation without raw replay. Together these components provide a principled way for efficient adaptation and the authors demonstrates strong results on both vision and NLP benchmarks.", "questions": "- what is the computation time for the model on the empirical experiments? \n- line 247 yielding about 15% faster convergence and reduced redundancy. What is the 15% improvement in relation to? \n- what happens when you perform stage 1 and stage 2 together, i.e. tuning memory parameters together with the classifier parameters (instead of the two stage approach)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **statistical guarantee on slot expansion is novel**: previous approaches rely on heuristic ways to determine parameter growth. SGSE formulates capacity growth as a statistical decision problem and shows statistical guarantees which is a novel way of approaching this problem. \n\n- **retention without replay**.  The proposed DCDA approach preserves prior knowledge without relying on raw data replay, enabling memory-efficient and privacy-friendly continual learning while maintaining high retention performance. \n\n- **nice presentation**. The presentation of the paper is clear and easy to follow.", "weaknesses": "- **scalability**: how scalable is the method? Would this work beyond the small datasets benchmarked in the paper? \n\n- **interplay between alpha and beta parameter**: the alpha and beta parameter in SGSE both controls a balance between a balance between stability and responsiveness. In pratice, how to pick the correct combination of both beyond the default parameter. How would you tune this in practice?\n\n- **training hyperparameter selection**: in the experimental setup, it is said that the stage 1 tuning happens first for 20 epochs and then the stage 2 tuning. How would you know the number of epochs required for each stage and how would one select it in practice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)