FlyPrompt: Brain-Inspired Random-Expanded Routing with Temporal-Ensemble Experts for General Continual Learning
Authors: Hongwei Yan
Guanglong Sun
Kanglei Zhou
Qian Li
Liyuan Wang
Yi Zhong
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Learning
Life-long Learning
Brain-inspired AI
Catastrophic Forgetting
Prompt Tuning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a brain-inspired method FlyPrompt that uses random-expanded routing and temporal-ensemble experts to effectively tackle General Continual Learning problem, achieving significant gains on major benchmarks.

Abstract: General continual learning (GCL) challenges intelligent systems to learn from single-pass, non-stationary data streams without clear task boundaries. While recent advances in continual parameter-efficient tuning (PET) of pretrained models show promise, they typically rely on multiple training epochs and explicit task cues, limiting their effectiveness in GCL scenarios. Moreover, existing methods often lack targeted design and fail to address two fundamental challenges in continual PET: how to allocate expert parameters to evolving data distributions, and how to improve their representational capacity under limited supervision. Inspired by the fruit fly's hierarchical memory system characterized by sparse expansion and modular ensembles, we propose FlyPrompt, a brain-inspired framework that decomposes GCL into two subproblems: expert routing and expert competence improvement. FlyPrompt introduces a randomly expanded analytic router for instance-level expert activation and a temporal ensemble of output heads to dynamically adapt decision boundaries over time. Extensive theoretical and empirical evaluations demonstrate FlyPrompt's superior performance, achieving up to 11.23%, 12.43%, and 7.62% gains over state-of-the-art baselines on CIFAR-100, ImageNet-R, and CUB-200, respectively. Our source code is available at https://github.com/AnAppleCore/FlyGCL.

Published as a conference paper at ICLR 2026
FLYPROMPT: BRAIN-INSPIRES RANDOM-EXPANDED ROUTING WITH CMMPORAL-ENSEMEEE EXPERTS FOR
GENERAL CONTINUAL LEARNING
Hongwei Yan' Guanglong Sun' Kanglei Zhou² Qian Li Liyuan Wang Yi Zhong"* "School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University DDepartment of Psychological and Cognitive Sciences, Tsinghua University SSchool of Medicine, Shenzhen Campus of Sun Yat-Sen University {yanhw22, sg123) (0gg23)mmailll semmalsssssnngeee edu. er, @1gian255@@aill { (zhoukanglei, liyuanwang, (enngyithu@@tsinnngni.
edu. cnn
ABSTRACT
General continual learning (GCL.) challenges intelligent systems learn from single-pass, non-stationary data streams without clear task boundaries. While re- cent advances in continual parameter-efficient tuning (PET) of pretrained models show promise, they typically rely on multiple training epochs and explicit task cues, limiting their effectiveness in GCL scenarios. Moreover, existing methods often lack targeted design and fail to address two fundamental challenges in con- tinual PET: how to allocate expert parameters to evolving data distributions, and how to improve their representational capacity under limited supervision. Inspired by the fruit fly's hicrarchical memory system characterized by sparse expansion and modular ensembles, propose FlyPrompt, train-insppred framework that decomposes GCL into two subproblems: expert routing and expert competence improvement. FlyPrompt introduces randomly expanded analytic router for instance-level expert activation and a temporal ensemble of output heads to dy- namically adapt decision boundaries over time. Extensive theoretical and empir- ical evaluations demonstrate FlyPrompt's 's superior performance, achieving up to 11.23%. 12.43%, and 7.62% gains over state-of-the-a baselines on C00. ImageNet-R, and CUB-200, respectively. Our source code is available at FlyGCI
INTRODUCTION
General Continual Learning (GCL) (Buzzega et al. 2020 De Lange et al. 2021), aims to equip in- telligent systems with the ability to learn continuously from non-stationary. single-past pass data streams, where tasks may not have clear boundaries and can evolve over time. Unlike traditional Continual Learning (CL) (Wang et al. 2024b; Parisi et al., 2019), which assumes well-defined task bound- aries and multiple training epochs, GCL presents a much more challenging problem, as it requires rapid adaptation, robust knowledge retention, and efficient resource usage under conditions of lim- ited supervision and task ambiguity (Fig. [ The ability to effectively tackle GCL. has profound implications for real-world applications such as autonomous agents and personal assistants, where systems must learn from dynamic environments without clear task definitions.
Recent advances I in parameter-efficien tuning (PET) of pretrained models (PTMs) have shown promise in CL (Wang et al., 2022d] c Smith al. 2023), but they still face fundamental limita- tions under GCL conditions. Such methods introduce task-specific prompt experts to adapt PTMs incrementally, and typically rely clear task cues and sufficient gradient updates to allocate and train expert modules (Wang et al., 2024a 2022b). However, those assumptions no longer hold in GCL (Koh al., 2021 Moon al. (2023) We therefore identify two fundamental challenges that remain unresolved ved: (1) how to dynamically route inputs to appropriate experts without task labels or iterative training, and (2) how ensure that each expert maintains strong and adaptive represen- lations under sparse and imbalanced supervision. Both remain non-trivial and underexplored.
"Correspondigg Authors: I. Wang and Y. Zhong.
Due the page limit, we present comprehensive summary of related work in Appendix B

---

Published as a conference paper at ICLR 2026
L2P
DualPrompt
CODMYP
MVP
MISA
FlyPrompt
95-
Jone: 4.52%
75
Average Accuraty
65
Aane:
6..0%
75
Aane: 6.04%
75
55
63-
55
655-
45
CIFAR-100
geNeetRR
CUB-200
15
10300
20000
30030
40000
50000
5000
10000
15030
20000
25000
2000
4000
6000
8000
(n)
Training Steps
(b)
Training Stops
(6)
Training Stops
Figure 1: Any-time average accuracy of GCL methods over three datasets using Sup-21K. Dashed lines indicate task transition. Aauc, the improvement of area-under-curve score by FlyPrompt.
The complexity of GCL has also been extensively studied in biological systems, where organ- isms have evolved efficient strategies for lifelong learning in dynamic environments. The fruit fly Drosophila provides a compelling model: despite having fewer than 100,000 neurons, it exhibits robust memory consolidation, context-aware behavior, and stable learning under minimal supervi- sion (Davis) 2023] Li et al.] [020] Modi et al.] 20200 Owald & Waddell| 2013). These capabilities largely attributed to the mushroom body, central brain structure that encodes sensory inputs via sparse random projections and organizes learning into modular, hierarchical compartments (Aso)
al. 2014b; Dasgupta et al., 2017: Wang et al. 2021 2022a 2023b) Waan & Li. 2025) (see
Fig. Left)). Projection neurons (PNs) from the antennal lobe connect randomly to Kenyon cells (KCs) mushroom body, yielding soattdmimnsiioal sparse codes that support imput separation and
routing even under noisy or overlapping conditions (Turner et al. [008 [Honegger et al., 2011). Fur- thermore, different KC subregions exhibit plasticity on distinet timescales (Aso et al., 2014a] Aso
& Rubin 2016), enabling both rapid adaptation and long-term consolidation (Cervantes-Sandova
ct al. 2013 Bouzaiaaee et al.) 2015; Wang Li 2025). These mechanisms closely mirror the goals of GCL, offering principled inspiration for tackling its core challenges.
Building upon these neurobiological principles and our preliminary analysis in Sec. 2.2, we propose to decompose the GCL challenges into two essential subproblems: (1) expert routing, which aims to assign each input to an appropriate subnetwork (expert) under unknown and shifting task boundaries; and (2) expert competence improvement, which seeks enhance the robustness and adaptability of each expert given limited training and imbalanced class exposure. To address these challenges, we introduce FlyPrompt, brain-inspired framework that integrates two key components: (i) Random Expanded Analytic Router (REAR) that mimics the fruit fly's sparse expansion circuit to rapidly assign inputs to experts in forward-only. closed-form manner, and (ii) Task-wise Experts with Temporal Ensemble (TE²) that captures knowledge across multiple time scales using exponential moving averages, mirroring the compartmental consolidation observed the mushroom body.
FlyPrompt is supported by both theoretical analysis and empirical validation. Across diverse GCL benchmarks, including CiFAR-100, ImageNet-R, R, and CUB-200, consistently outperforms state- of-the-art CL and GCL methods, achieving accuracy improvements of up to 11.23%. 12.43%, and 7.62%, respectively. By integrating biologically grounded design with principled algorithmic struc- ture, FlyPrompt offers interdisciplinary perspective addressing the core challenges of GCL and also exemplifies the potential of the emerging field of NeuroAl (Zador et al. 2023).
2 PRELIMINARIES
In this section, we formulate GCL, and then evaluate PET methods in an instantiated GCL scenario.
2.1 PROBLEM FORMULATION
In CL. model learns sequential tasks t E (1..... ,T) each associated with dataset D, (æt, yt) where H; E Xx and y/ E Vr. The model comprises a backbone (-)) and an output head go (-). which together produce predictions v = ff (x)). The objective is to learn unified mapping from input domains X' = U, X₂ to label spaces y = -, Vi. Classical CL settings impose structural assumptions the input or label space. Domain-incremente
learning (DIL) assumes disjoint input domains with a shared label space (X; n X, = 6. y = D/). while ies-eincremenal and class- incremental learning (TIL,, CII.) assume disjoint label spaces ()), n)) = 0). with TIL additionally
2

---

Published as a conference paper at ICLR 2026
Overall
Ba
80
E
Pg
Dapibuton
0145-10
Learnng Trajectory
(a) GCL as Multi-Expert Learning.
(b) Prompt router accuracy.
(c) Accuracy with oracle router.
Figure 2: Empirical analysis of GCL. schematic illustration of GCL viewed multi-expert collaboration. (b) Prompt selection accuracy for methods with explicit expert routing designs. (c)
Final average accuracy (Assst 1) when using test-time oracle provide the correct prompt identity. Results evaluated across three benchmarks with Sup-2IK. FP, FlyPrompt. RP, Random Projection.
CIFAR-100 5)
ImageNel-R (7
CUB-200 20)
1.0
1.0
Expert D
06
06
-04
16
000
8.
(6)
Expert
CKA
CKA
(c)
Expert ID
CKA
Figure 3: CKA similarity of feature representations between experts of MVP on three datasets.
providing task identity at test time (Van de Ven & Tolias 2019). Under these assumptions, the learning objective can be decomposed into two orthogonal subproblems: task identity prediction (TIP), which selects appropriate task-specific module, and within-task prediction (WTP), which performs classification under the selected module (Kim et al. 2022 Wang et al. 2023a).
challenging condi- tions. Tasks arrive as a one- pass data stream, and their label spaces may overlap with non-negligible probability: Vi A j, P(D: ny; f 0) (Koh al. 2021 This entangles inter-task and intra-task interference, undermining the TIP-WTP orthogonality Especially when using pretrained back- bones, the strong priors encoded PTMs already bias prediction before adaptation, causing task identity and class discrimination to co-evolve (Wang et al., 2023a 2024a). Moreover, GCL intro- duces additional difficulties such as severe intra-task class imbalance (e.g., long-tailed distributions) and limited training iterations. These problems are compounded by memory constraints (Buzzega ict al. 2020 De Lange et al., 202I), where storing past data restricted or disallowed.
GCL, however, lifts these assumptions and operates under substantially
A representative instantiation of GCL is Si-Blurry (Moon et al. 2023), which explicitly partitions global label space into disjoint subset YD and blurry subset y with y yiD U y and y:D nB O. The disjoint class ratio rD = |/P1/ controls the proportion of task-specific classes, while the blurry sample ratio 7B determines how frequently classes in yB reappear across tasks. This flexible design captures the stochasticity and heterogeneity of GCL, which has been validated in recent theoretical and empirical work (Mi et al., 20200 ZZhuang et al. [024 Kang et al.) 2025). We therefore adopt Si-Blurry the default GCL benchmark (see Appendix Dor for discussion about the task/session boundary information and our empirical rationality of using Si-Blurry).
2.2 ANALYSIS OF GCL METHODS WITH EXPERTS
Recent CL and GCL methods increasingly adopt PET techniques on top of PTMs. These meth- ods can be seen as lightweight extensions of architecture-basee CL (Zhu al. 2021 Wang et al.) 2023b), where instead expanding full networks, they introduce trainable modules p (e.g., adapters, prompts, and LoRA) that act semantic-aware adaptation experts to give instructed outputs fo(x; p). A common strategy is to maintain a pool of such experts and design a router to assign inputs the appropriate ones. However, most existing methods, such L2P (Wang al. 2022d), DualPrompt (Wang et al. 2022c), MVP (Moon et al. (023). CODA-P (Smith et al. (023), and MISA (Kang al. 2025 train these routing functions synchronously with the stream of incoming
3

---

Published as a conference paper at ICLR 2026
Output
D2
D2
Ds
Dr
Data Stream
ORNs
Online Head
EMA Head
EMA Head
Temporal Smoothing (ey < (r))
Encoders
Prefrained Model
PNs
Sparse Expansion
Random Projection
KCs
0
0 0 6 5 2
XN
2n Order
Nonlinear Activation
Statistics
Semantie Experts
MSA Block
3
Task- -Wise Prompt Experts
Analytic Router
Regression
4
Input
[MPonttorr o// (eoilarri o/ rengoorng
E1
E,
Es
E
E2
Temporal Ensemble
Figure 4: Method overview. Inspired by the fruit fly's olfactory memory system (Left), FlyPrompt incorporates random-expande analytic router (Middle) and temporal ensemble-based based experts (Right). ORNs, olfactory receptor neurons. PNs, projection neurons. KCs, Kenyon cells.
data, making them vulnerable to distributional shifts and limited iterations. These issues are espe- cially pronounced in GCL, where blurry task boundaries and online constraints prohibit iterative tuning, and class imbalance further weakens the quality of per-task representations.
Based on this observation, we propose to decompose the GCL problem into two critical subprob- lems: expert routing, which determines which expert assign each input; and expert compe- tence, which enhances the quality and robustness of each expert's representation. Compared to the classical TIP-WTP formulation for CL with clearly segmented tasks, GCL's blurry structure and overlapping class distributions make semantic-level experts a more suitable abstraction for adapta- tion (see Fig. 2a Moreover, since the same class may appear in multiple tasks, the correspondence between classes and experts is inherently one-to me-to-many, which further complicates routing.
To better understand these challenges, we conduct a preliminary empirical study from a multi multi-expert collaboration (details in Sec. 4.1.. We first evaluate the routing accuracy of methods that explicilly predict expert identity (e-g., DualPrompt, MVP, MISA, and our FlyPrompt) after all GCL training tasks. A prediction is considered correct if the selected expert belongs to the set of experts previously trained on the true label yt, acknowledging the overlap across tasks. As shown in Fig. 2b existing routers based on similarity contrastive losses still exhibit considerable limitations in expert selec- tion. Next, we evaluate the final average accuracy under an oracle router that always selects a correct expert for each input (Fig. 2c The results reveal that, even with perfect routing, previous methods still exhibit inferior performance, highlighting a second bottleneck: the limited competence of indi- vidual experts. In PTM--based context, such competence depends not only the representation space shaped by each expert module, but also on how well the output head can maintain consistent decision boundaries over time; even when an early expert's encoder frozen, single head that keeps adapting to later data can gradually become misaligned with its fixed representation. To ver- ify this point, our analysis of expert- expert-specific representations using centered kernel alignment (CKA; confirms that experts indeed specialize in distinct feature subspaces, under-
Appendix (.44) in Fig. [3] scoring the need for accurate expert assignment. Together with the observed degradation under an oracle router, these results support decomposing GCL into the interacting subproblems above, and clarify that expert competence must account for both representation quality and decoding robustness.
3 FLYPROMPT: A BRAIN-INSPIRED GCL APPROACH
In this section, we propose FlyPrompt, an innovative brain-inspired approach designed to tackle the key challenges of GCL by explicitly improving expert routing and expert competence. As shown in Fig. 4 FlyPrompt consists of two core components: (i) Random Expanded Analytic Router (REAR) that employs fixed random projections and closed-form updates to assign inputs to experts, inspired by the sparse expansion circuits in fruit flies, and (ii) Task-wise Experts with Temporal Ensemble (TE²) that adaptively refine class boundaries over time to improve expert-level performance, reflecting modularized ensembles architecture in fruit flies' neural systems.
4

---

Published as a conference paper at ICLR 2026
3.1 RANDOM EXPANDED ANALYTIC ROUTER
Recent studies in CL have explored the use of Random Projection (RP) to construct forward-only learners with desirable properties such rapid adaptation and immunity catastrophic forget- ting (Zhuang et al. 2022 2024] MoDonell et al., 2024). When combined with nonlinear activation, RP can significantly improve the linear separability of imput features by capturing high-order interac- tions without backpropagation. Notably, this mechanism aligns closely with the olfactory system of the fruit fly, where projection neurons (PNs) connect sparsely and randomly to a high-dimensional population of Kenyon cells (KCs) in the mushroom body, achieving a nearly 40-fold expansion. Global inhibition then enforces sparsity, allowing robust and efficient pattern separation (Dasgupta et al. 2017 Honegger et al. 2011). This neurobiological design inspires REAR, which mimics the biologically grounded random expansion to enable efficient, non-iterative expert selection in GCL.
Concretely, given pretrained backbone fo(-) that maps input æ an embedding h fo(x) Rd of dimension d, we apply a fixed RP followed by a nonlinear activation:
(xx)) = a (Jo(z)) = (h.R. E RR
where R E RdxM is random matrix with Ra, ~ NN 1), M > d. and (-) is an element-wise activation function (e.g., ReLUU. The resulting feature (p(x) is sparse and high-dimensional.
During online training, we associate each task / with corresponding expert F,. For each incoming batch B, C Dr of size B, we compute the projected features D E R BBBx whose row vectors are (x(()) (xx, y) E J, and update two statisties: the Gram matrix G E RMxM capturing second- order feature correlations; and the prototype matrix Q E R MxT storing expert-wise feature sums:
G G D.T D.
Q-Q+0.0
(2)
where C E R BxT whose vectors are the same one-hot embedding e, E 10,1) for expert E,. We then construct a router matrix U e R by minimizing the following objective:
T
((U) Σ Σ [|+(a:))TT - + A|U
1= (.I.))ED
where À > 0 is the regularization parameter. This objective encourages the router to map sam- ples from task f to the corresponding expert Fg while maintaining numerical stability. Using the accumulated statistics G and Q, the closed-form solution to this optimization problem is given by:
Ô (G + -Q.
(4)
The calculation of Ú is only needed once upon evaluation, therefore this optimization process is efficient and lightweight compared to gradient-based routing mechanisms. At inference time, the routing score s and selected expert E. for input given router Ü is computed
s(x) = T RT.
É(x)) arg max (z)). ITT
(5)
This routing mechanism is efficient, biologically motivated, and requires no gradient updates, mak- ing it well-suited for GCL's online, single-pass constraints. Unlike prior methods based on random expanded features, such as RanPAC (McDonnell et al.) (2024) or ACIL (Zhuang et al., 2022)). which apply closed-form ridge regression directly for final classification on fixed representations. REAR uses random projections solely for instance-level expert routing while keeping each expert's prompts and heads fully trainable. Empirical comparison between the analytic router (REAR) and analytic classifier (RanPAC) under GCL benchmarks is shown in Appendix Tab. 13] And we further demon- strate the superiority of REAR upon alternative routing strategies in Appendix Tab. [7] We then summarize the core theoretical guarantee that explains why REAR yields reliable routing in the expanded sparse feature space. Full assumptions and proofs are included in Appendix
Theorem 1 (REAR, informal). With high probability over the random expansion and the data stream, the population excess risk of the ridge router learned from online statistics admits the fol- lowing decomposition:
R R(U)) S Vlog(N)/M + -VN
for suitable universal constants. Therefore, by increasing the expansion dimension M and the num- ber of samples N, and choosing the regularization parameter X balance estimation error and bias, the population excess risk (and, under a fixed margin assumption on expert scores; see Ap- pendix F. the misrouting probability) be made arbitrarily small.
5

---

Published as a conference paper at ICLR 2026
Interprefation. The first term reflects the approximation error due finite random features; in- creasing M improves the expressive power of the expansion. The second term captures the variance arising from finite data, which diminishes NN grows À increases. The third term represents the bias introduced by ridge regularization, which stabilizes learning but limits expressiveness if it is too large. In practice, this decomposition implies that robust and forward-only routing can be achieved by employing sufficiently rich random expansions and moderate regularization, without requiring task-level or iterative refinement.
3.2 TASK EXPERTS AS TEMPORAL ENSEMBLES
To improve the competence of each expert under dynamic distributions, we draw inspiration from the fruit fly's KCs in the mushroom body and their connections to output neurons. This brain structure integrates multi-timescale plasticity and hierarchical processing across subregions, where KCs mediate short-term memory, a' a'13'' KCs support intermediate memory, and a/3 KCs are critical for long-term memory consolidation (Krashes et al.) [007 [ervantes--anndee
et al. 20033 [ouzaianee 2013). These KC subtypes are sequentially recruited during learning and exhibit compartment- specific modulation by dopamine neurons (Aso et al. 2014at Owald & Waddelll 2015 Aso et al., 2014b][As Aso & Rubin] 2016)), enabling temporally staged memory formation and retrieval. Inspired by this biological design, equip each expert in FlyPrompt with temporal ensemble of oulput heads, implemented using exponential moving averages (EMA) with varying decay rates.
Concretely, instead of using only one shadow head in naive EMA, each expert E, in FlyPrompt maintains a set of n EMA heads with decay rates {az)}=ı where oj * og for all j + k. Let the online head be parameterized as + (W, b) R x RD and the j-th EMA head of expert E, as (W(ii, (i) b(:).. When a new task t begins, its prompt p. is initialized as the average of previously learned prompts warm start:
P: 1 Σ p for L 1,
with random initialization for f = 1. This average-prompt warm start provides more informed initialization under single-pase GCL streams, where each expert only observes limited data, and empirically accelerates convergence and more compatible with blurry boundaries in which classes can reoccur across sessions. The EMA heads of E, are initialized as clones of the current online head. During training, only online head v/ and prompt Pt are updated using the cross-entropy:
(x y) = CE (fo( p:) W + +b+ m. y)
where m E RR is a non-paramettric logit mask initialized for each data batch (X,y). We set m. and for any class c E y encountered in the current batch and set me' = -00 sup- press predictions on unseen labels c' & y. This masking strategy mitigates interference from class imbalance both across and within tasks (Moon et al. 2023 Kang et al. 2025). evaluated in Tab, 15
After each update step, the EMA heads are updated as:
w(())
(1) + aj w + a;) W, 8(1) + ag bi + + (1-aaj a,)
At inference, the REAR module first selects an expert e É(x). Using the associated prompt Pe, we compute logits from the online and EMA heads:
z (0) = fo(æ; Pa) W + b,
(8)
-00) fe p.) W(J + +601,
(I, ,n].
We then ensemble all 1 heads by computing the SoftMax of each and taking their element- maximum, followed by logit masking:
(xx)= max softmax (2() + +m),
((x) arg max (x)).
(10)
je ()
This temporal ensemble mechanism enables FlyPrompt to integrate stable, long-term information via EMA heads while preserving rapid adaptation through the online head, mirroring biological memory consolidation and facilitating robust inference under non-stationary, imbalanced streams.
Here, we also present theoretical guarantee that supports the use of multiple EMA heads in GCL.

---

Published as a conference paper at ICLR 2026
Table 1: Overall performance of representative methods over three GCL benchmarks across PTMs.
CIFAR-100
ImageNet-R
CUB-200
PTM
Method
Aa,, f)
A
Aauc t)
Alost (%6 1)
Azue(%. (%,,
(9..
Seq FT
19.7
10.42
7.51
2.20 2.20
3,47
1. 1.49
Linear Probe
49.69
23.07 23.07
29. 29.2441.26
16.87 16.87
28. 08.96622.
17. 17.38
Seq FT w/ SL
64.90+77.18
62.06g1.6
47 47.20
39. 49.60
56. 16:133
56.50 66.50
L2P
2..23
79.11
44.40g1.00
42 22.03
64.30
61.42
Sup-21K
DualPrompt
76.04+3.32
76.62
46. 1.44
1.04
03
62. 1.78
CODA-P
a9.13
7.070
22.81
48 12.70
66. 1:2.00
62.991
MVP
67.7444..96
63.2220..0
39.50
32 23.63
54. 24.14
50.07 1:
MISA
80.300
80.75
51.5212.0
45. 45.08
65.40.3300
60.20
PlyPrompt (Ours)
83.24
86.76
56.58
55.27
70.64
73.40
L2P
63.88
68.96
47.10
42 12.22
42. 42.96
45.00
DualPrompt
68.02
67.0446.84
52 12.80
47 47.39
46.80 (0.800+2.0
46.39
CODA-P
69.29
69.47+7.19
51.20
44.30
44 14.06:22.5
45. 45.18
Sup-21K/IK
MVP
64.69
51.29
48.99 28.99
38. 6.
44.1042.81
33.97 2.962
MISA
62.9 #7.06
67.99
50.87
27.75
12.76
1.90
FlyPrompt (Ours)
78.48 11.31
80.39 10.04
62.01
56.55 23.04
54.42
55.50 23.50
L2P
56.8228.42
67.61,8870
35.97
36
26.95
14.76
24.51.44162
DualPrompt
66.006..4.02
67.1418.00 00
42 12.48
35.91 35.91.u..s
19.90
21.84::2.30
CODA-P
62.13+7.17
63.38
45.50) L.66
39.44 1.36
17.7244.55
20.82
iBOT:21K
MVP
62.33
48.32
41.55:1.50
29.
29.22
28.73
23.62 02.92.9
MISA
65, 15,30==2.20
67.4 67.4326.75
40.94
36 1.55
18 62
23. 23..6622.22
FlyPrompt (Ours)
75.58
79.36
$7.75
54,39
28.86
36.79
L2P
33.17
62.284s.19
38. 38.2942.6
39 39.86
19 19.20
31.21++.24
DualPrompt
32 32.39 20122
53.5610.10
45.76
39.19
39
29 13.115
30.53
CODA-P
39.29 19.0..4000
61.30
49.56
42.64 42.6441.75
27.57 12283
33.61 33.61144.52 1:4.2
IBOT-IK
37.52+3.62
44.08
44.70+2293
24.93-2-.48
33.81 +3.50
26.32
MISA
34.31a1:9
55.89
43.9143.0
40.09 21:24
27. 27.76
33.744.2.11
Plyprt (Ours)
70.14
74.84
61.50
57.18 1.38
38.75
45.00 :4.19
L2P
47.98
59.13 3.
35.8141.37
36.58 1..5..1..5
21.18 21.18
32. 12.477.m
DualPrompt
52.12+44.0l
55.71
43.03+1.12
35 15:40:..00
27. 80.4
29. 24.40
CODA-P
3-169 2..49
58.91
45 45.16
38 23 2.00
29. 29.27
31.85
DINO-IK
MVP
53.644a..9
41.02:12.00
41.7842.15
32.00
33 .4, 133.441..47
26.02
MISA
52.03
55.98
41. 11.2623325
37 37.50g1.6
27.13
33.05
FlyPrompt (Ours)
65.92.a2.74
72.66
57.29 #2.40
54.72
37.38
44.66 44.66:22.95
L2P
28.17 08.17
39.07
17. 4..31.71
16.27 25.43
12. 12.42
20.00 20.00g7.36
DualPrompt
53.33
58.20
36.69 36.69
30.2 30.24
19. 19.88
21.93
MoCo v3-1K
CODA-P
43.47
58.55g7.19
39 8042.71
31.72+t4.a6
20. 20.00 +2.62
24.10++6.48
54.33
40.84:14.23
46.45.2
26.3772..04
28.48
23.566499.8
MISA
57. 20.06
62.18
38.85
33.47 20.95
25 05.02444.99
27.68 27.68.
4-135
FlyPrompt (Ours)
64.12
71.51
52.32
49.0611.35
27.92 2..53
33.32
Theorem 2 (TE?, informal), For an EMA head with decay a and window L = 1/(1 - a), the
parameter error time satisfies
E W will S (/L))
where ạ² bounds the online noise and P, measures drift. geometric EMA bank contains, at every time, head that achieves near-optimal bias-variance trade-off up to constant factor.
Interpretation. The bound decomposes the parameter enror into a variance term O({2/D,, con- trolled by the effective window size, and a drift-induced bias term ((LP)) which increases with nonstationarity. Larger T. reduces variance but increases bias, creating a bias-vau mas-vvaiance trade-off. A geometric bank of FMA windows ensures that, at any time, one head is near the optimal trade-off for the current drift level. Intuitively, when the input stream contains segments with varying temporal dynamics, such as sudden shifts session transitions or gradual changes within each task, differ- ent EMA heads can align better with different segments, leading to more adaptive predictions. In practice, two EMA heads with windows of 10 and 100 (a 0.9, 0.99) are sufficient (see Sec. 4.2
EXPERIMENT
In this section, we first introduce the experiment setups and then present the experiment results.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison with prominent offline methods on three GCL. benchmarks under Sup-21 K.
CIFAR-100
ImageNet-R
CUB-200
Method
Aaue(% 1)
Amon(% t)
Ague 1)
Aust (%,1) 1)
Aaue f
Anet 1)
S-Prompt++
80.21
83.48
52. 52.14=1 1.66 1.65
49.13
06.21
64.73++2.22
HiDe-Prompt
77.10+3.11
81.77+2.00
53.7 =1.0
49. =3.01
67. 07.05
67. 17..122..50
HiDe-LoRA
80.07+2.41
82.00
55.09
51 61.2996629
67 07.2664: 1
67 67.28 67.28+1.4 +1.45
HiDe-Adapter
79.52+2.81
81.41
53.92_1.32
50.86_5.0s
66.09.1141
61.533
NoRGa
78.8943.33
83.03++1.20
54.12 =1.37
50.09 =3.66
67. 67.1642.44
67. 67.0640.68
SD-LoRA
29.26
78.9142.48
55.51
51.97 3.09
64. 64..122..02
60.57 +0.17
1.330
FlyPrompt (Ours)
23.24
86. 76.73
56.58+1.45
55. 255.7 +0.91
70.64
73. 73.40
4.1 EXPERIMENT SETUP
Benchmarks. We evaluate FlyPrompt under the Si-Blurry GCL, setting (Moon et al. 2023 Kang
ctt al., 2025) using three representative benchmarks: CIFAR-100 (Krizhessky et al. 2009)
(60K
images, 100 classes), ImageNet-R (Hendrycks et al. 2021 (30K images, 200 classes), and CUB- 200 (Wah et al. 201T) (12K images, 200 fine-grained classes). Unless specified otherwise, we adopt the default Si-Blurry configuration with disjoint class ratio rD = 50% and blurry sample ratio TB 10%, trained over five sessions. We report two widely used metrics: average anytime accuracy Annc (evaluated every 1000 batches) and final accuracy Alast (measured after all sessions) (Koh et al. 2021]. Additional experiments of different (LD, rB) and online CL. are provided in Appendix F.1 Unless specified, all results are averaged over five runs (# standard deviation) with different seeds.
Baselines. We compare FlyPrompi against a diverse set of CL and GCL methods: (1) lower-bound baselines such sequential fine-tuning (Seq FT, including the version with a slow learning rate, SL) (Zhang et al.) 2023) and linear probing: (2) prompt-based CL baselines including L2P (Wang
al. (002d)) Prampt (Wang et al.] [202cc) and CODA-P (Smith et al., 2023); (3) state-of-the-
art GCL methods such as MVP (Moon et al. 2023) and MISA (Kang et al. 2025); ; (4) prominent
offline CL. methods S-Prompt++ (Wang et al. 20226). HiDe (Wang et al. 2023a), NoRGa (Le et al.)
2024) and SD-LLRR (Wu et al. (025) in Tabs. and T2 We also implement the online version of
the analytic bascline RanPAC (McDonnell et al. 2024 in Tab. 3 and other variants in Tab, [3
Implementation. We adopt the ViT-B/16 backbone pretrained ImageNet-21 IK and ImageNet-IK, including strong supervised paradigms Sup-21K, Sup-21K/IK (Sup-21K fine-tuned on ImageNet- 1K) (Ridnik et al. [2021] Dosovitskiy et al. 2020), and self- self-super paradigms iBOT-21K, iBOT- 1K (Zhou et al. (021). DINO-IK (Caron et al. (002T), and MoCo v3-1K 3-1K (Chen et al. 2021). We set the projection dimension M 10 and À based checkpoints: 10at (Sup-21K), 10 (Sup-21K/18 MoCo), and 107 (iBOT, DINO). We use n = 2 EMA heads with decay rates 0.9, 0.99. More implementation details of baseline methods and GCL benchmark setup can be found in Appendix
4.2 EXPERIMENT RESULTS
Overall Performance. Tab. Summarizes GCL performance across all benchmarks. Prompt-based CL methods perform well with supervised backbones (e.g., Sup-21K, Sup-21K/IK), but degrade significantly under seff-supervised ones (e.g., DINO, MoCo v3-1K), particularly on fine-grained benchmarks CUB-200. This highlights the challenge of extracting discriminative features without strong pretraining priors. MVP, which incorporates contrastive learning for improved expert selec- tion, outperforms others under the fine-grained benchmark and self-supervised PTMs, reinforcing the importance of prompt routing. However, the contrastive loss yields limited performance gains in other cases due to the absence of replay buffer. Fig. [ presents the anytime accuracy during GCL. MISA benefits from stronger prompt initialization and achieves relatively higher performance at the early stage, but steadily declines due to parameter overwriting, eventually matching weaker baselines like CODA-P. This suggests that while good initialization helps, it alone is insufficient for sustained GCL performance. In contrast, FlyPrompt consistently outperforms all baselines across datasets and PTMs. It achieves up to 1 11.33% 12.43%. and 7.62% improvements in Aquc: 13.53%, 16.49%, and 12.28% in Ajast on CIFAR- 100, ImageNet-R, R, and CUB-200, respectively. As shown in Fig. D FlyPrompt maintains stable, high accuracy throughout GCL, with minimal drops during session transitions. This results confirm FlyPrompt as new state-of-the-art for GCI.
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study of different components in FlyPrompt. "_" indicates not applicable.
FlyPrompt Components
CIFAR-I00
ImageNet-R
PTM
REAR
Prompt Expert
EMA head
Aaue (%, ")
A 13
Asme 1)
Annel (%6, 11
RP- Based Analytic Classifier (RanPACI)
69.91g3.0
29.92
47. 47.29 20.2
47. 47.33
x
71 71.33=2.17
78. 1..2
41.7320.59
37.33=1.71
71. 22.6
1.5
42. 1..50
38.35
Sup-2IK
80.75 1..7
83.65
54.91 11.9121.52
1..5
82.17=2.07
83. 75.7711.0
55. 99. 1.37
53.65
X
81..0022.0
84, 21
55. 76.7 ara
52. 13.76G1..00
83.24 (1.24_.2:00 22
86.76gu.77
56.58 11..1
55.27
RP-Based Analytic Classifier (RanPACt)
69.76
20.4
52.91 =
24.73
x
57 67.94
62. 62.67 8..5
46. 06.22..0.20 0.72
39. 39.06+22.81
60. 60.76
6..3
49. 19.88_.0.00
11.65:11.58
Sup-21K/IK
X
70.09
67.62.5.80
52.07
14.13:3.1
V
72.5 72.51g3.15
68. 56.666...00
56.55
45.90
x
71 11.28_2..6
03.733
53.12
44.69
18.4
80.39
62.01
26.55
Table 4: Effect of REAR and TE? on Aauc(%) performance for PTM- based CL methods using CIFAR-100 under Sup-21K. The numbers in parentheses indicate the difference from the baseline, and the arrow direction indicates an increase (1) or decrease (4). See Tab. 14 for complete results.
Method
Baseline
w/ REAR
TE²
w/ Both
DualPrompt
76.04
20.63 (f 4.59)
76.83 (1 0.79)
82.33 if 6.29)
MVP
67. 6..74
67.44444.59 (1 0.30)
68.91 (1 1.17)
68.93 if 1.19)
MISA
80.35
82.03 (f 1.68)
81.65 (1 1:30
83.60 05 (1 3:25)
S-Prompt++
80.21 22.55
81.43.2.45 (f 1.21)
81.93 ( 1.72)
83.11 22.30 (f 2.00)
HiDe-Prompt
77. 77.10=3.81
78.41 (f 1.31)
77. 77.16443.56 (t 0.36)
78.60 (1 1.61)
NoRGa
78.89
79.37 (t 0.48)
79.16 (1 0.27)
79.37 #2.74 (1 0.48)
Ablation Study. To assess the contribution of each FlyPrompt component, we conduct a compre- hensive ablation study of REAR for prompt selection, multi-prompt across tasks, and TE? for EMA head ensemble. Results in Tab, 3 show that each module provides consistent gains, with the full FlyPrompt achieving the best performance. We additionally include RanPACI, an analytic learner using random projections but no expert modularity, to simulate REAR without multi-expert routing. While this performs competitively under limited training, falls short without expert specialization, underscoring the importance of both routing and competence. Notably, the gain from EMA heads alone is modest unless combined with REAR and prompt modularization, highlighting the synergy among bio-inspired components rather than simple additive effects. We further integrate our REAR and TE2 components into a range of strong baseline models by replacing their routing and output head modules correspondingly. Results in Tab. 4 further demonstrate the consistent improvements when either component added (more results across datasets and metrics are presented in Tab. 14.
Hy Hyperparraeetr Sensitivity. Fig. F key hyperparameters in REAR. Increasing the projec- tion dimension Mf improves performance, consistent with the theory that higher- dignes-dimmesiioall spaces enable better feature separability and router performance (in Theorem D. mirroring sparse expan- sion in the fruit fly mushroom body. However, since the memory cost grows linearly with M, we set M 10,000 as a practical trade-off. The regularization parameter À has smaller impact, with per- formance stable across several orders of magnitude. Full results across other PTMs are provided in Appendix [:5 We further analyze EMA decay rates with temporal ensemble. Tab, 6 shows that two EMA heads of 0.9, 0.99, combined with the online head, achieve the best trade-off across datasets. This aligns with neurobiological findings that the mushroom body maintains short-, mid- and long- term memory modules in parallel. Among various ensemble strategies (Tab. [5). the "SoftMax element-wise maximum" method most effective and used by default. Detailed evaluations D
across
other PTMs and configurations are provided in Appendices F.8 and F.10
Detailed Analysis. Returning to the core challenges identified in Sec. 2.2 we revisit the roles of expert routing and expert competence improvement. As shown in Fig.
methods (e.g., FlyPrompt) that improve in these two areas correlate strongly with better overall GCL performance. In particular, Fig. 2b demonstrates the impact of random projection in boosting routing accuracy, while Fig. 2c highlights remaining headroom for improving expert competence. Despite introducing RP layer and tracking feature statisties, FlyPrompt adds minimal parameter overhead, i.e., just 0.83% more

---

Published as a conference paper at ICLR 2026
10.41
GIFAR-100
95.7
7107
-90
200
09.39
$2.64
00.00
500
10K
[b)
14]
00.40
55.05
Mi 3011
1-: 10K
M- 160
10.12
50.0
09.72
[6]
70
Figure 5: Analysis of hyperparameters in REAR. (a-c) Different projection dimension M with fixed 10t: report Aaue and extra storage cost (bar) given M. (d-f) Different regularization parameter À with fixed M = 10 Dashed lines indicate the optimal choice of each hyperparameter.
Table Performance comparison of ensemble aggregation choices for TE? under Sup-2IK.
CIFAR-100
ImageNet-R
CUB-200
Ensemble Method
Aauc(% ft
Aust (96,
A46,%% t)
Aane(%, f)
Auss (%.,
Mean Max Prob.
81.34
85.1141.0
52.71
53.24
68.49
73.95
82.29
84.95
55.5641.58
53.53
68.00 68.00
66.5621.60
Min Entropy
81.92,2.19
84.23
55.0541.31
52.88
66. 66.7842.58
64. 64.7311.38
SoftMax+Mcan
82.30
85.9820.s
56. 16.6....
55.53
70.77
74 74.80,1.44
SoftMax Max Prob.
83.24
86.7620.73
$6.58 1.447
55.27
70.64.2.85
73. 13.40
Mon Entropy
83.11 12.34
86.50
15.94
54.24
69.8612.80
71.51
Table 6: Performance comparison of different EMA decay rates for TE² under Sup-21K.
Table 7: Computational cost and overall perfor- mance using CIFAR- 100 under Sup-21K.
EMA Decay Rate
A... (3)..)
CIFAR-100
A...
ImageNNt-R
Auao (56.
Method
Total Param. [M_])
Trainable Puram. (M.))
Time Delay
(s/busch..)
(96.1)
A.s.
Online head only
81.00
64.25g1
54.01g1
69.58
L2P
86.01
0.22
5.17
26.23
+0.9
82.81a22s
50.3020044
58.36a1
55.00 $5.00220
DualPrompe
86.35
0.55
4.78
76.04
40.99
82.84
86.41.o.
06.344114
54.07
CODA.P
86.72
01.9
4.75
79.13
+0,999
21.80
.4.30.am
55. 65.15
53
MVP
86.12
0.30
5.35
67.74
83.24.a.3
86.7610
36.58
20.27
MISA
0.58
82.00 pg
66.24
60.35g1.72
g5.50
PlyP
parameters than MISA on ViT-B/16, and incurs negligible increase in computational cost (see Tab. more comprehensive comparison in Tab. and detailed cost breakdown of components in Tab, Together, these findings validate FlyPrompt's effectiveness in resolving the GCL challenges.
5 CONCLUSION
We presented FlyPrompt, a biologically inspired framework for GCL, which addresses the core challenges of expert routing and expert competence improvement under blurred task boundaries and single-past constraints. Grounded in the neurobiological principles of the fruit fly's mushroom body, known for sparse expansion, random connectivity, and multiscale modularity, FlyPrompt integrates a randomly expanded analytic router for non-iterative expert selection and a temporal ensemble of expert heads for robust adaptation over time. Theoretical analysis and empirical results across multiple GCL benchmarks demonstrate its strong performance and scalability.
While these results are encouraging, several limitations of the current work point to promising future directions. For instance, the temporal ensemble relies and adapting these dynamically to data drift could enhance robustness. Additionally, performance under extreme long-tailed distributions warrants further study. Looking forward, GCL is essential for deploying real-world learning systems, such as embodied agents, user-facing A1, and resource- constrained devices, where data dynamic and supervision limited. As continual adaptation is natural strength of biological systems, the underlying principles they offer will continue to inspire future advances in GCL and beyond.
fixed composition of EMA decay rates,
10

---

Published as a conference paper at ICLR 2026
Acknowledgment. This work supported by the ST12030-Major Projects 20222D0204900, the National Natural Science Foundation of China (NO, 62406160, 32530042, 32021002), the Bei- jing Natural Science Foundation L247011, and the Beijing Major Science and Technology Project No. 7251 (00008425003.
Ethics statement. acknowledge that and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Reproducibility statement. We have included the source code with clear instructions, and will release them upon acceptance.
REFERENCES
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In CVPR, pp. 11254-11263, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. NeurlPS, 32, 2019b.
Yoshinori Aso and Gerald M Rubin. Dopaminergie neurons write and update memories with cell- type-specific rules. Elife, 5:e16135, 2016.
Yoshinori Aso, Daisuke Hattori, Yang Yu, Rebecca M Johnston, Nirmala A Iyer, Teri-TB Ngo, Heather Dionne, LF Abbott, Richard Axel, Hiromu Tanimoto, al. The neuronal architecture of the mushroom body provides a logic for associative learning. elife. 3:e04577, 2014a.
Yoshinori Aso, Divya Sitaraman, Toshiharu Ichinose, Karla R Kaun, Katrin Vogt, Ghislain Belliart- Guérin, Pierre- Yves Plaçais, Alice A Robie, Nobuhiro Yamagata, Christopher Schnaitmann, et al. Mushroom body output neurons encode valence and guide memory-based action selection in drosophila. elife, 3:e04580, 2014b.
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Woo Ha, and Jonghyun Choi. Rainbow memory:
Continual learning with a memory of diverse samples. In CVPR, pp. 8218-8227, 2021.
Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations research, 127 2015.
Emna Bouzaiane, Severine Trannoy, Lisa Scheunemann, Pierre- Yves Placais, and Thomas Preat. Two independent mushroom body output circuits retrieve the six discrete components of drosophila aversive memory. Cell Reports, 1188):128-12292 2015.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. NeurIPS, 33: 33:15920-15930. 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pp. 9650- 9660, 2021.
Isaac
Alfonso Martin-Peña, Jacob A Berry, and Ronald I. Davis. System-like consolidation of olfactory memories in drosophila. Journal of Neuroscience, 334699494469984, 2013.
Xinlci Chen, Saining Xie, and Kaiming He. An empirical study of training self self-supervised vision transformers. In ICCV, pp. 9640-9649, 2021.
Sanjoy Dasgupta, Charles F Stevens, and Saket Navlakha. A neural algorithm for fundamental computing problem. Science, 3586366))793-796, 2017.
Ronald L Davis. Learning and memory using drosophila melanogaster: a focus on advances made in the fifth decade of research. Genetics, 224(4): 2023.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. PAMI, 44(7):3366-333) 3385, 2021.
11

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
Dan Hendrycks, Steven Basart, et al. The many faces of robustness: A critical analysis of out-of- distribution generalization. In ICCV, 2021.
Kyle S Honegger, Robert AA Campbell, and Glenn C TuTrnr. Cellular population imag- ing reveals robust sparse coding the drosophila mushroom body. Journal of neuroscience, 31 (33):11772-11788) 2011.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint (0Xi::2066.09685, 2021.
Zhiqi Kang, Liyunn Wang, Xingxing Zhang, and Karteek Alahari. Advancing prompt- prompt-based based meth- ods for
general continual learning. In ICLR, 2025.
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. theoretical study on solving continual learning. NeurIPS, 35:5065-5079, 2022.
Hyunseo Koh, Dahyun Kim, Jung- Buo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. arXiv preprint arXiv:21 10031, 2021.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pp. 3519- 3529. PMIR, 2019.
Michael J Krashes, Alex C Keene, Benjamin Leung, J Douglas Armstrong, and Scott Waddell. Sequential use of mushroom body neuron subsets during drosophila odor memory processing. Neuron, 53(1):103-113 2007.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Minh Le, An Nguyen, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Van Ngo, and Nhat Ho. Mixture of experts meets prompt-based continual learning. Advances in Neural Information Pro- cessing Systems, 37:1 19025-1 19062, 2024.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
Feng Li, Jack W Lindsey, Elizabeth Marin, Nils Otto, Marisa Dreher, Georgia Dempsey, Ildiko Stark, Alexander S Bates, Markus William Pleijzier, Philipp Schlegel, et al. The connectome of the adult drosophila mushroom body provides insights into function. Elife, 9:e62576, 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.0011 2021.
Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. Beyond not-forgetting: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems, 35:16165 16177, 2022.
Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. NeurIPS, 36, 2024.
Fei Mi, Lingjing Kong, Tao Lin, Kaicheng Yu, and Boi Faltings. Generalized class incremental learning. CVPR Workshops, pp. 240-241,200 2020.
Mehrab Modi, Yichun Shuai, and Glenn C Turne.. The drosophila mushroom body: from archi- tecture algorithm in learning eircuit. Annual review neuroscience, 4344):465-444, 2020.
12

---

Published as a conference paper at ICLR 2026
Jun- Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong- Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. In ICCV, pp. 11731-11741, 2023.
David Owald and Scott Waddell. Olfactory learning skews mushroom body output pathways to steer behavioral choice in drosophila. Current opinion in neurobiology. 35:178-184, 2015.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter, Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:54-71, 2019.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In ECCV, pp. 524-540. Springer, 2020.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. NeurIPS, 30, 2017.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21K pretraining for the masses. arXiv preprint arXiv:2104. 10972, 2021.
James Seale Smith, Leonid Karlinsky. Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed attention-bant based prompting for rehearsal-free continual learning. In CVPR, pp. 11909- 11919, 2023.
Joel A Tropp. frerndriendly tail bounds for sums of random matrices, Foundations of computational mathematics, 12(4):389 434, 2012.
Glenn Turner, Maxim Bazhenov, and Gilles Laurent. Olfactory representations by drosophila mushroom body neurons. Journal of neurophysiology. 99(2)734-7766, 2008
Gido M Van de Ven and Andreas Tolias. Three scenarios for continual learning. arXiv preprint 190400990400773. 2019.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
Liyuan Wang and Qian Li. Convergent multi-modular architecturefor adaptive learning in drosophila and artificial intelligence. iScience, 28(11), 2025.
Liyuan Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Chenglong Bao, Kaisheng Ma, Jun Zhu, and Yi Zhong. Afec: Active forgetting of negative transfer in continual learning. NeurIPS, 2021.
Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. In ECCV, 2022a.
Liyun Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical de- composition of prompt-based continual learning: Rethinking obscured sub-optimality. NeurIPS, 2023a.
Liyuan Wang, Xingxing Zhang, Qian Li, Mingtian Zhang, Hang Su, Jun Zhu, and Yi Zhong. In- corporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence, 2023b.
Liyuan Wang, Jingyi Xie, Xingxing Zhang, Hang Su, and Jun Zhu. Hide-pet: Continual learning via hierarchical decomposition of parameter-efficient tuning. arXiv preprint arXlv:2407.05229, 2024a.
Liyua Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. PAMI, 2024b.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre- pre-trained transformers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695. 2022b.
13

---

Published as a conference paper at ICLR 2026
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy. et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pp. 631-648. Springer, 2022c.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pp. 139-149, 149, 2022d.
Yichen Wu, Hongming Piao, Long-Kal Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incre- mental learning. In ICLR, 2025.
Hongwei Yan, Liyuan Wang, Kaisheng Ma, and Yi Zhong. Orchestrate latent expertise: Advancing online continual learning with multi-level supervision and reverse self-distillation. In CVPR, pp- 23670-23680, 2024.
Hongwei Yan, Guanglong Sun, Zhiqi Kang, Yi Zhong, and Liyuan Wang. Domain generalizable continual learning. arXiv preprint arXiv:2510. 16914, 2025.
Anthony Zador, Sean Escola, Blake Richards, Bence Ölveezky, Yoshua Bengio, Kwabena Boa- hen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, et al. Catalyz- ing next- -eneerattnt artificial intelligence through neuroai. Nature Communications, 14(1):1597, 2023.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning on pre-trained model. arXiv preprint arXiv:2303.055 081100 2023.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint urXiv:21/1.07885 2021.
Kanglei Zhou, Ruizhi Cai, Liyuan Wang. Hubert PH Shum, and Xiaohui Liang. A comprehensive survey of action quality assessment: Method and benchmark. arXiv preprint arX1v:2412.11149, 2024a.
Kanglei Zhou, Liyuan Wang. Xingxing Zhang, Hubert PH Shum, Frederick WB Li, Jianguo Li, and Xiaohui Liang. Magr. Manifold-aligner graph regularization for continual action quality assessment. In European Conference on Computer Vision, pp. 375-392. Springer, 2024b.
Kanglei Zhou, Zikai Hao, Liyuan Wang, and Xiaohui Liang. Adaptive score alignment learning for continual perceptual quality assessment of 360-degree videos in virtual reality. IEEE Transactions 2025a.
on Visualization and Computer Graphics, 31(5):2880-22
Kanglei Zhou, Qingyi Pan, Xingxing Zhang. Hubert PH Shum, Frederick WB Li, Xiaohui Liang, and Liyuan Wang. Continual action quality assessment via adaptive manifold-aligned graph reg- ularization. arXiv preprint ar5 10.06842, 2025b.
Fei Zhu, Xu- Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In CVPR, pp. 5867-5876, 2021. doi: 10.1 100/ 200.64443...21.0008888
Huiping Zhuang. Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, and Zhiping Lin. Acil: Analytic -illassiineemettall learning with absolute memorization and privacy protection. NeurIPS, 35:11602-1166 11614, 2022.
Huiping Zhuang, Yizhu Chen, Di Fang, Run He, Kai Tong, Hongxin Wei, Ziqian Zeng, and Cen Chen. Gacl: Exemplar-free generalized analytic continual learning. NeurIPS, 37:83024-83047, 2024.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference machine learning (ieml-03), pp. 928-936, 2003.
14

---

Published as a conference paper at ICLR 2026
A LARGE LANGUAGE MODELS ASSISTANCE
Large language models were used to polish the manuscript. The authors have thoroughly reviewed and edited all content and take full responsibility for the published work.
B RELATED WORK
aims train models on task streams with evolving distributions (Wang iet al. 2024b; 2021] 2022a: Parisi et al. 2019). Canonical CL settings are categorized into task-
Continual Learning (CL)
incremental learning (TIL), class-incremental learning (CIL), and dommin--iccrmmnnall learning
2024 2025), depending on the structure of input and label spaces. TIL and CIL assume disjoint label spaces, with task identity provided only TIL, while DIL shares label space but varies imput domains. Theoretically. CL has been formalized by de- coupling task identity prediction (TIP) and within-task prediction (WTP), which remain orthogonal under clearly segmented tasks and from-scratch training (Kim et al.) 2022 Wang et al., 2023a).
(DIL) (Van de Ven & Tolias
2019 Yan et al.
The rise of pretrained models (PTMs) has shifted CL towards adapting frozen backbones via lightweight modules, known as parameter-efficient tuning (PET) (Lester et al. [021] Li & Liang. 2021 Rebuffi et al.||211 2017] Hu et al. (022). PET-based CL methods often employ either task-shared modules (Zhang et al. 2023 McDonnell et al., 2024] /Zhou et | [2024a] 2025b a: (024b) that re- quire gradual updates, or task-specific experts (Wang et al. 2022d 2023a) that demand effective expert selection (implicitly via external queries (Wang et al.
2022d explicitly via routing func-
tions (Wang et al. 2023a)). Importantly, the strong priors embedded in PTMs blur the TIP-WTP
decomposition, making classical CL theory less applicable (Wang al. 2023a 2024a).
General Continual Learning (GCL) extends CL to more practical scenarios by removing assump- tions of clear task segmentation and offline data access (Buzzega al.) 2020 De Lange al. [021]
Mi et al. 2020). Specifically, GCL emphasizes online learning, where each data point is seen only once; and blurry or unknown task boundaries, where task identities are absent or ill-defined [Aljundi iet al., 22019;; [Prabhu et al.] 2020; Bang et al.) 2021]; Moon et al.) (203)). These properties intro- duce unique challenges in expert selection, knowledge retention, and f'stt adaptation, without task identities or replay buffers. Additional constraints, such as constant memory budgets and anytime inference, further distinguish GCL from traditional CL (De Lange et al. 2021).
To implement the GCL challenges, benchmarks such as Trsk-Free CL (Aljundi et al. 2019a Prabhu et al., 2020) and Si-Blurry (Moon et al.) 2023) have been proposed, progressively relaxing task- awareness and enforcing stream- learning. Correspondingly. GCL methods adapt replay-based sampling (Aljundi et al., 2019b Bang et al. 202I). memory management (Koh et al. [02I). or PET-based designs (Moon et al. 2023 Kang et al. 2025). However, replay methods raise pri-
vacy and scalability concerns, while recent PET-based methods (e.g.. MVP (Moon et al. (023) and
MISA (Kang et al. (025)) ) still suffer from limited representation capacity and lack principled mech-
anisms for prompt expert selection under non-stationary inputs. Consequently, their improvements over naive PTM-based baselines remain modest.
C IMPLEMENTATION DETAILS
C.I TRAINING SETUP
We follow the previous GCL studies (Moon et al., 2023 Kang et al. 2025) for fair comparison. The standard VYT-B/16 transformer backbone has an embedding dimension of d 768. For prompt- based methods. we unify the prompt length to 5 and the position to insert the prompt as the first five layers of VIT. All methods are trained with an Adam optimizer with a learning rate 0.005 and zero weight decay. We set the batch size to 64, the epoch number to 1 (online learning), and the online iteration each batch to 3. All images are cropped and resized to 224 x 224 to fit the ViT format using standard data transformation operations (resize, random crop, random horizontal flip and normalization). Moreover, the logit mask m trick in Sec. Sec. is generally applied to all methods to enhance training stability. All experiment jobs are performed the same Linux server with Intel Xeon Silver 4316 2.3GHz CPUs (20 cores), NVIDIA RTX 4090 GPU. For random seeds, we use the fixed values 1, 2, 3, 4, and for all parallel runs.
15

---

Published as a conference paper at ICLR 2026
C.2 BASELINES
Unless otherwise specified, all baselines share the common ViT-B/16 backbone, single-pass Si- Blurry streams, optimizer, and data preprocessing described at the beginning of this section; below, we highlight method-specific architectures and the GCL-specific adaptations.
Sequential fine-tuning (Seq SL) and Linear probing. Seq FT fine-tunes all parameters of the VīT-B/16 backbone and classifier on the Si-Blurry streams without replay buffers or task-specific heads; SL is an otherwise identical variant with smaller learning rate (i.e., 5 x 10-5, 10 times smaller than 0.005 used by other baselines) to provide a more optimistic lower bound (Zhang et al.) 2023). Linear probing instead freezes the backbone and trains only a linear classifier, with no prompt modules or expert structures; together, these methods serve as simple -TM-based lower bounds.
Prompt-based CL baselines (I.2P, DualPrompt, CODA-Prompt), For L.2P (Wang et al. 2022d), DualPrompt (Wang et al.) [022c). and CODA-Prompt (Smith et al.] 2023). we keep their original prompt-controller designs (key-based prompt pool in L2P, global+task prompts in DualPrompt, and attention-base bastionn-badddd-ee prompts in CODA-
but adapt them to GCL by freezing the ViT-B/16 back- bone and unifying prompt length and insertion position as in Sec. C.T All three methods are trained in single online pass over the Si- Si-Blurry streams with the same update schedule as FlyPrompt, with- out extra replay or offline fine- tuning, so that differences in performance come from their prompt mechanisms rather than from additional data passes.
GCL. baselines (MVP, MISA). MVP (Moon et al., 2023) and MISA (Kang et al. 2025) are im- plemented top of the same VTT-B/16 backbone and Si-Blurry streams. We follow their official configurations for expert/prompt structures and initialization, while enforcing the unified prompt configuration and online training protocol of Sec. C.I As in prior work, they maintain session-wise experts or prompts, but do not use any privileged task oracle beyond the evolving data stream; their routing structures can be interpreted in the same way as FlyPrompt's experts discussed Sec. D For fairnsss, MVP and MISA also use the batch-seen class logit mask in Sec. 3.2 whose effect is ablated alongside other mask types Tab. 15
NoRGa, SD- LoRA). S-Prompt++ (Wang et al.) (0226) introduces prompt experts with a mfxture-of-eexprrts (MoE) structure with linear gating, while DeteeePromppLLLRR/Adaaaeee (Wang et al. 2023a) and NoRGa (Le et al,, 2024) build hierarchical decompositions and stronger MoE-based routing on top of S-Prompt++, and SD-LoRA Wuu et al. (2025) leverages structured low-rank adapters by de- composing expert LoRA into learnable amplitude and fixed direction; all of these are originally designed for offline or task/class-incrementi CL. Specifically, HiDe and NoRGa consist of a two- stage TTP+ (task-ID prediction then within-task prediction) pipeline. To make them compatible with GCL and Si-Blurry, we adapt their TIP step as follows: when the method predicts a class, is allowed to activate all prompts corresponding to the candidate task IDs associated with that class, and we count the prediction as correct if any activated prompt outputs the true label. This is an in- tentionally favorable modification for these baselines. In addition, any feature statistics required by their alignment modules (e.g.. for HiDe or NoRGa) are accumulated online from the stream, rather than being computed from stored per-task datasets. Quantitative comparisons of these adapted of- fline -'M--based methods with FlyPrompt are reported in Tab. 12
Offiin PTM-based CL methods (S- (S-Prompt++,
Analytic random-projection baselines (RanPAC variants). RanPAC (MeDonnell et al. 2024) was originally proposed for offline class-incremental learning, where the PTM is fine-tuned on the first task, frozen, and all task-1 features are recomputed and stored to form stable Gram matrix for closed-form ridge regression; this protocol is incompatible with single-pass GCL and blurry task boundaries. To ensure a fair analytic baseline, we adapt RanPAC into three GCL-compliant variants: RanPACi fine-tunes the PTM on the first Si-Blurry session without storing features and then solves closed-form ridge classifier on the resulting random-feature representations, serving as the main analytic random-projectio the first session and stores all features to approximate the original offline setting while still respect- ing the single-pass constraint on labels; RanPAC simultancously fine-tunes the PTM and collects features during the first session, which yields an ill-conditioned Gram matrix due to representation drift but offers an optimistic upper bound on malytic-classifie performance under our setting. As summarized in Tab. 13 FlyPrompt consistently outperforms all RanPAC variants across datasets.
baseline in our ablations; RanPACt freezes the PTM during
16

---

Published as a conference paper at ICLR 2026
D DISCUSSION OF TASK BOUNDARY IN SI-BLURRY
Table 8: Performance comparison of different numbers of tasks and experts for GCL methods on CIFAR-100 dataset over Sup-21 K. "task-correlated" indicates that the initialization and training of expert parameters are aligned with task/sessions. w denotes the sample budget (window size) of each expert when methods adopt self-triggered expert allocation mechanism. All results are reported as an average of five parallel runs (# standard deviation) with different random seeds.
MVP
MISA
Fly FlyPrompt
of Tasks # Experts
Ause (%,1) f)
(%,,))
(9. t)
Amma%% 1)
A t)
A (%, 1)
5 correlatd))
67.7 67.7444.96
63. =0.22
80.35+2.a9
80.75 10.75=1.2
83. 83.24.2..2
86. 76
S (w 10000)
67. :4.99.9
63. 22 20.76
80.60
81. 131177
83.67 =2.22
85, 85.78+0.e0
5
10 (w 5000)
27.23
(3.477.m 20.75
79.95
8..322 11.15
83,40 S3.40
85. 85.43
(v0 2500)
67.19
64.00 1.84
79.94
81.00
82.26g1154
28.66
(task correlated)
58.23
61.1320.00
75.71
80.2220.47
77.65
+0.50
5 10000)
58.49g...5
61.98 288
76.25
80.62.0.02
77.28
84.63 20.50
10 (w 5000)
58.19
60.80
75.69
80.18
76.9613.2
84.03
20 (tr 2500)
58.46
62. 15
75.7841
80.66.0.61
76.47
83. 83.9920.5
20 (task- correlated)
56.52 2..20
56.87
73.96 73.96
77.98
75.87
81.98 11.12
20
$ (w 10000)
56.59=s.55
56.58 56.58
74.27 =0.07
77.89
76.12
$1.90z0.se
(w 5000)
50.30
66.50
73.8020.82
77.59
g1.12
82. 20.00.00
2500)
56.48ga.15
g6.80
73.6020.85
77.70a1.24
75.36
81.13go
GCL (Buzzega et al. 2020) is defined by a single-pass, non-stationary data stream without task boundaries during training and without a task oracle at test time. The Si-Blurry benchmark (Moon) et al., 2023) that adopt has been carefully analyzed in subsequent work (Kang et al. 2025): by controlling the disjoint-class ratio TD and blurry-sample ratio TB, it generates streams where (i) the number of active classes can vary across sessions, (ii) classes may reoccur across sessions, and (iii)
the number of samples per class and per session is randomized (Mi et al., 2020) In particular, when Tn approaching 0, the nominal "task" or "session" index becomes decorrelated from distributional changes. These properties ensure that Si-Blurry conforms to the core GCL assumptions, rather than reducing to standard task-incremental CIL.
Within this setting, FlyPrompt does not assume any privileged boundary information beyond what is already used by prior GCL methods such as MVP and MISA. The "task" "session" index provided by Si-Blurry is treated as a conceptual device to describe how the benchmark constructs streams, not as a supervision signal for the model. In our implementation, expert indices are aligned with nominal session identities purely for convenience: the same behavior can be reproduced by starting a new expert after a fixed number of observed samples or when user-defined computationall/storgge budget is reached, without accessing the task index. Moreover, the total number of experts T is not hard-coded prior; matrices such as Q E TR- xT and the router head can be dynamically extended from T to T + 1 via prrddo..,iig,. analogous to adding classes in standard classifier. Implementation details for how both GCL methods and offline PTM-based basclines are instantiated under this regime summarized in Appendix C.2
To empirically validate that FlyPrompt and comparable GCL baselines do not gain an advantage from Si-Blurry's session structure, we further compare task-aligned expert management with self- triggered expert allocation mechanism. In the self-triggered setup, each method maintains a fixed sample budget and freezes the current expert while initializing new one whenever the number of observed samples reaches a predefined threshold, fully decoupling expert updates from external task segmentation. We evaluate multiple combinations of nominal task counts (# of Tasks = 10, 20) and expert budgets (# of Experts 5, 10, 20) on CIFAR-100, with corresponding sample budgets chosen cover the 50K training examples. As summarized in Tab. R, seig-triggered expert initial- ization achieves performance on par with, or slightly better than, session-alignes setups for MVP, MISA, and Flyprompt across all tested budgets. This confirms that (i) task-switching signals of- fer no measurable benefit in this benchmark, and (ii) our expert management mechanism does not exploit any extra boundary information.
17

---

Published as a conference paper at ICLR 2026
E THEORETICAL PROOFS
E. PROOFS FOR REAR (THEOREM
NOTATION AND ASSUMPTIONS
We repeat and fix the notation used throughout the proof Theorem T
1. fo X Id is the given pretrained backbone. For an input we write h - fo(x)) R
2. E IRdx is a random matrix with i..d. (o, 1) entries; the j-th column is ry E R2.
3. The random-expanded feature map is
(a (6 (a)), ... PM (m))
p2(x)) oh rz)-
4. Assume embedding- boundedness: ||||22 H for all æ. (This be enforced in practice by layer-norm or clipping.)
5. Activation satisfies these with La- = 1 and linear growth C' = 1.
+ R is L.- Lipschitz and has linear growth [a(=)] S C(1 + |z|). ReLU
6. For training, we accumulate batches (or singletons) to form
G (())))(()) e RRMXM
Σ p(a))c E RRT
where c; E (0, is the one-hot indicator of the target expert; see Eq. 7. Ridge solution (router):
Ô =
A>0. O.
with regularization parameter À as in Eq.
8. For the theoretical analysis, we treat the training pairs (æ;, ye) as i.i.d. draws from an un- derlying distribution over X' x f1...., T), with fixed and finite number of experts T.
9. For the margin-based routing-accuracyy corollary below, we additionally assume that there exists margin y > 0 such that, for the population minimizer U and almost every input æ, the score of the correct expert t" (x) satisfies syz (x) (m) su- (z-( + for all o t" (x)) where su (x) upx)U
At the population level we consider the regularized squared risk
R(U) Ellssr C12 +
where (X, C.) denotes a random variable pair drawn from the same distribution as the training ex- amples (a,, c.), with E X' and C {0. 13" is the one-hot indicator the target expert. We write sy (z) = ( for the router scores as in Eq. (5). The minimizer of R. in the kernel-induced feature space is precisely the U appearing below.
We then state the complete Theorem I here based on the above assumptions:
Theorem (REAR, full). Under the standing assumptions above, form the online statistics G, Q as Eq. (2) and the ridge solution Ö as in Eq. (4). Lei N be the total number of samples used to form G.Q and let U* denote the population regularized minimizer in the feature space induced by the kernel k(h, ht) = (o (h"(h'Tr) r)). Then for any ó E (0,1), with probability at least (over R and the training samples), the excess (population) squared risk decomposes as
R(Ū)- R(U*) < Etear 8) + Eestim (NN (N, N, X, 8) + (A),
where, universal constants C₁ (depending on H, La, C),
(N/6)/ M
1 Eestim (N,A, & O2 VN X
Ejes S
SM CiV
Exeg(A) I3
This assumption not needed for the excess-risk decomposition in Theorem |itselff but hridge.
18

---

Published as a conference paper at ICLR 2026
E.1.1 LEMMA: RANDOM-FEATURE CONCENTRATION
Lemma Let ,....... NND be the finite training set and write h; = fo(w.). Define the
kernel
((,,h) (r- (o(h" (h" r)].
Then, under the standing assumptions, for any å (0,1) and any E (0,
M Cf 1og(N2/6)
e²
(with C₂ depending only on H, Lo. "above), then with probability at least 8 over R.
M p)(()) (p(x))))))))))))))>))())) k(hh;,))) Se.
max ISUJN
Proof. Fix a pair (i,j). Write
Ze = (m))) ((ee(m)))e(zz)))
{= 1= M.
The Zg are independent (across E) mean-zero random variables because columns ry are independent. We will apply Bernstein's inequality for sums of independent sub-exponential variables; to do so, we need variance proxy and uniform tail bound.
From the growth assumption [o(=)) < C(1 + |=|) and ry
NN00, Ia); the marginal (
(h: r²) is sub-Gaussian sub-exponential: more precisely, since h T (o, h l2) S (O, 2) we have for some constants v, b (depending on H, Lo C) that (()) > < 2 eexp(-et) for large t; thus (e( sub-exponential with parameters bounded by functions of H. L,, C. Concretely, can verify
Zele, S
for finite constant depending only on H, Lo. C. where lly, denotes the standard sub- exponential Orlicz norm. Hence, applying Bernstein's inequality for sub-exponential variables yields, for any 0,
M
(-epp-cmmmm mimnn M
7)
Ze
S
2exp
with constants ū, b, > 0 determined by the sub-exponential parameters.
Choose .M.. Plugging Ms and requiring the RHS be S 6/N2 (to union bound over all S 2 pairs) yields the condition
M C+1 log(N220))
for some C,r (combining cases of Bernstein). This gives, for fixed pair (i,
P
Ze
M
N2
Apply union bound over all S 2 ordered pairs (i.j). This yields the claimed uniform bound with probability least 1 1-60 8.
Remarks on applicability of Bernstein. We used Bernstein for independent sub-exponential sum- mands. The summands are independent across random-feature index f; sub-exponentiality follows from (i Gaussianity of re and (ii) Lipschitz linear-growth of o. For ReLU (which is Lipschitz with linear growth) the same argument applies (moments of Gaussian tails control tails (-)).
19

---

Published as a conference paper at ICLR 2026
E.1.2 LEMMA: RIDGE PERTURBATION
We next show that when the empirical feature covariance concentrates around its population coun- terpart and the empirical covarccov concentrates, then the finite-sample ridge solution is close the population ridge solution.
Lemma 2. Let & E N be the feature matrix with rows xp(x); define empirical covariance
1 + D E = N
b 1 N T
where Y E RNXT is the one-hol label matrix (or soft labels). Let the population quantities be == [[() and ( J. Denote population ridge solution
U (E A
Ux -5=
1/155 Ellop S 2 and 16 - bllr S 6b. then
110s F S xeb + 1211|// - -Ellon-
Proof. We write
- - (b -0-0----A--0001---------------30 - AJ)
For the first term use operator norm bound ([5+AT) 'lop 1 1// to get
| -1) b)]l - - - bbb.
For the second term use the identity A-1-B-1- A (B A)B 1 with A B Hence
1 -1 Σ) 10lopp
||-1 - B llop S 1 |lop11 A B Bl|opl B
1lop S )
Thus
- /b1
X2 E Ellop 1E 1b6l
Combining the two terms and tightening constants when
- Ellop - A/2 gives the stated bound.
Concentration of E. and b. We next control the empirical covariance and cross-covariance. Recall
Σ Σ ((x.))((..)
Σ ==](o(()))(())1]
It is convenient to write
N
5 Σ = Σ X x; 1/ N
1 (p(x)))((x)) - -x).
Each X; is self-adjoint and satisfies E[X,] 0. Under the dounded-embedding and activation assumptions (cf. Lemma [}, there exists a constant Cip 0 (depending only on (H, La, C)) such that [|{p(=)]|22 & Cip almost surely. Hence, for every i,
| S 1/ N (11s((s)))<<x))) llop ||E|(op)
1 (C& [Ellop) S N
Lu N
Similarly, we can bound the "matrix variance" term
N 2 = M Epa lo
= N E op
<
E [[(p(z))(x)T
-Ej-
<
Vo
N2
N
op
20

---

Published as a conference paper at ICLR 2026
for some constant Vo 0 depending only (H. L. C). For completeness, recall standard
matrix Bernstein inequality (Theorem 6.1 in Tropp 2012 N
{X,)=1 are independent. mean- zero,
self-adjoint matrices with [Xillop I L almost surely and
8² : Σ
then for all & > 0:
12/2
P
X op
2t)
S 2D exp
122 + Lt/3
where D denotes the matrix dimension (in our case D = M, the feature dimension). Applying this with and our bounds I Lo/N and x² Vo/N gives
N 2 S 2M exp (- Vo + Los/3
P(IÊ - oopop 26 Ie(1
Σ
For (0, 1) the denominator in the exponent is bounded above by constant multiple of Vo, so there exists C" > 0 (depending only on (H, Lo, C)) such that, for all (0, 1), taking
8 c=c'v
N
ensures
P(I - Ellop 2 E) 8
Equivalently, with probability at least 5,
log( M/
3Σ Ellop I CO¹
N
For the empirical cross-covariance = N E ( and its population counterpart = B[p((x)c]] we apply the same argument column- wise (each column is an average of bounded sub- exponential vectors of length M) and obtain
66 - bblp S c" Tog(T/5)
N
for some constant C" > O depending only on the same problem parameters. Adjusting constants account for the two events and taking a union bound, we may assume that both inequalities hold
simultaneously with probability at least 1 1- Choosing large enough to make | Ellop S X/2 and to make [ bll (denoted in Lemma small then yields the desired estimation error term
in Lemma 2
E. LEMMA: ONLINE STATISTICS IMPLEMENT BATCH RIDGE
Lemma 3. If G and Q formed by accumulating per-example contributions
N G G= (=m))(()
Q D=Io(mne
then the closed-form solution ÔT (G+XI)-1Q equals the ridge regression solution computed in butch on features xp(x;) and labels c, Moreover, if the online implementation maintains (G+XI) via rank-l I updates, numerical equivalence holds up to floating Hoating-point precision.
Proof. This is algebraic: batch ridge with design matrix 4 and labels Y solves ÔT = (4 Φb + - Y. But & Σ, (xxx,){((xx)) G and + y Q. The equality follows. For in- cremental numerical maintenance of the inverse, standard Aherman-Morrisee or Woodbury updates apply; also numerically stable Cholesky-update are recommended when M large.
21

---

Published as a conference paper at ICLR 2026
E.1.4 COMBINING LEMMAS: PROOF THEOREM
Proof. The excess population risk decomposes as
R(Ō) R(U)) = R(Ū) - R(U))+ R(UX) - R(U))
estimation
bios
The regularization bias is the usual ridge bias and yields the Eveg (A) term; standard calculus shows is bounded by 1|? up constant factors.
For the estimation error, apply Lemma 2 to relate the empirical ridge solution (which equals the on- line Ó by Lemma 3 to the population ridge solution U/- The two perturbation terms are controlled by ||E- and bl.b. which in turn are bounded by the matrix Bernstein concentration bounds
for 2 and derived above. This yields the stated O I }) behavior for Eestim (explicit constants follow from the above bounds).
VN
Finally, the approximation eπror due to random features is precisely Lemma I replacing the kernel k by its Monte Carlo approximation using M independent features introduces uniform ((Vlog(N (N/))/M perturbation in inner products, which propagates to the excess risk the term Eteas (M, 6) displayed in Theorem
Margin-based routing accuracy. Under the additional margin assumption in the REAR stand- ing assumptions and a uniform bound [|<{(>)]|2 C.p (for some constant C.p depending only on (H, Lo, C)), the excess risk bound above can be converted into a bound on misrouting probability. Let f(x) == arg max, sq (x), and tt (x) arg max, sy- (x), denote the experts selected by Ü and U* respectively. Then
8C2 2g2
P(((X) # (X))
Proof. On the event (f(x)) # to (x)). the margin condition implies
7 S su (z) (x) su- (a)) max su- (x)); s (ar)r/-
su. Hence, 1(t(x) + ts (z)) s (2/7)² ( su-
Given so T and Cauchy- Schwarz yields:
max (s))((); st (2)) s (1(()) U*| s G.1l U'
+1f((z) (x)) (2C//))3110 U
Since the regularized risk is defined R(U) su (X) CIB + || the quadratic ridge term ||| makes R (at least) X-strongly convex in U. In particular, strong convexity implies:
R(O) R(Ú)-R(U)) z (A/2) U.
-0 s (R - R(U))).
Combining these inequalities and taking expectations over X gives the stated bound.
E.2 PROOFS FOR TE (THEOREM 2
E.2.1 NOTATION AND ASSUMPTIONS
We reuse notation from the main text. For a fixed expert (prompt/head), we denote:
W E DD the (population) time-t optimal linear parameter (in the chosen feature space) for that expert.
22

---

Published as a conference paper at ICLR 2026
We the instantaneous (online) estimator after observing the t-th update; model W. W Et.
EMA head with decay a E (0,1). 1):
W (0) (1 a)
We-ks
L(a)
T-a
k20
Discounted path length (drift measure):
Pr =
W Wi-j+1 W/-l
where we define the discount factor to match the EMA decay, i.e., 7 a or compara- ble. This is discounted analogue of the standard path length / variation measure used in dynamic regret (Zinkevich] 2003 Besbes et al., 2013).
Standing conditions: finite Pt. zero-mean noise with bounded variance:
ElE.l = 0,
E E <²
(Optional, for classification calibration) A margin A > and a Lipschitz map-to-logits with constant Cy imply bound on 0/1 error from parameter MSE.
We then state the complete Theorem 2 here based on the above assumptions:
Theorem (TE2, full). Under the standing conditions above, fix t and an EMA decay a with 1/(1 - a.) There exist constants C1, C2 > O such that
w:ll G C CZ.PP.).
E W. (o)
Moreover, if we keep a geometric grid of windows {Ls} with ratio r > (e.g. L; = pi-1), then for every there exists index with
Will S c(r) min (a S 121 L C2
E W (o
where c(r) depends only on the grid ratio (one can take, for example, c(r) max(r2 (x2 r) by the argument below). Additionally, if margin A > 0 holds and the logits map is Lipschitz with constant Cj, then the above parameter MSE implies a classification error bound ((C)e/A)))
whenever E W (a)
W =².
E.2.2 PROOF OF THEOREM
Proof. The proof proceeds via variance-bias decomposition and geometric grid selection argu- ment, followed by calibration from parameter MSE to classification error.
We start by decomposing the error into the variance term. Define the difference between the EMA and the population optimum
W (0) W
a)
Σ Ei-k;
k00
(1- a)
Σ a W
k20
where W is the EMA of the population optima.
Since we have [[G]] 0 and E||&,|| < <², and the EMA weights are aş = (1 - aja*, we can
compute their squared sum explicitly:
(1
a)
1 - a 1
a (1
2k
2
1 a L'
k20
23

---

Published as a conference paper at ICLR 2026
Therefore,
2 L
W. E
Σ K20
CRCI-K
a
#
which matches the variance term C1 L in Theorem 2 (with C1 absorbing the constant factor).
Next, address the bias term. The difference between the EMA of the population optima and the actual population optimum is given by:
W. Wt
ax (W W().
k20
Reordering sums yiclds:
W,
Wi
S
Σ 121
W 1-j++
W = Σ
a A-jtt
k2j
321
(11)
where
A.
W Wi-ll
Using the discounted path length Pt with the same discount factor 7 = a as the EMA definition, can simply rewrite the above bound as
W.
Wi Σ a' Ae-j+1=0
Σ a'' Laa-1A--t11= aP, S LPt,
121
j21
where we used 1/(1 - a) 2 a. Consequently, the squared bias admits the explicit bound
W w < (LP.)²,
which corresponds to the term C2))))) in Theorem 2
Then, by combining the variance and bias terms, we apply the inequality [o|3 s 2||a||2 211122 which yields the claimed MSE bound:
- - < 2 _ ((.P)))
2 we may take the explicit choice C₁ C2 = 2. If we allow y to differ slightly from the constant C2 becomes (a/7)² (bounded if we restrict E [(1 c)a, (1 + c)a)).
Therefore, in Theorem
For the geometric bank, it convenient write the bound in the generic form
f(L) =
L + B(LP,)
Ax Bx
direct derivative calculation (I( shows that the minimizer over is A
A + 2BP,L L
1/3
f((L) L
2BP3
If we maintain a geometric grid L; = - with ratio r >> 1, then for any L there exists an index it
such that Li, E (r.r", Writing L L'n with 17 E [[/r,r] and using the optimality condition
A/L* 2.) we obtain
f(L) DB(L'P.) 2f
2/n
f(L))
31)L-P.)2
The right-hand side maximized over v) []/r,]] at one of the endpoints; simple bound yields
2/n+1)
2 _² 2r
2
max((r.r))
sup nE[1/r]]]
3 max
3''3
3r2
Therefore f(L.)) < c(r) fLL) with the explicit choice c(r) max(r2 r) used in Theorem
(obviously c(r) = 7, given r our case.)
From the general classification error's view, by the Lipschitz-to-logii condition, the induced logit error most:
W W+|
Under the margin condition (A > 0), the standard margin-to-0/1 calibration gives an error bound O((Cye/A))) when E W W < 2
24

---

Published as a conference paper at ICLR 2026
F ADDITIONAL EXPERIMENT RESULTS
F.1 EXPERIMENT WITH DISIOINT AND BLURRY SETUP VARIANTS
Table 9: Performance comparison of rn variants in FlyPrompt. We fixed en 10% and Sup-2IK backbone PTM. All results are reported as an average of five parallel runs (# standard deviation) with different random seeds.
CIFAR-100
ImageNet-R
CUB-200
(%)
Method
Auue (%6,
A (56. t)
Anne 1)
Abaa (%,
A (%,
Aual
1..64
11.7
42.97 s1
42.53 22.53
64.62
62.72 62.7211:22
DualPrompt
72.16-2.10
77.52+1.45
45.08
42.34=1.55
65.42+3.58
62.84+2.21
CODA-P
72.33-2584
78.5624.46
50.31+4.4
48. 18.8823.m
66.6413342
63.7042448
(pure blurry)
MVP
67.10
74.94
39.04
32.28 32.2815.12
56.861a71
55.5016.71
MISA
28.38
83.04
51.78
47.64.0.64
67. 67.38
64.49
PlyPrompi (uurs)
80.12.1.a9
87.11 =0.52
$5.44
55.89 20.86
71.60
74.72=:1.9
76.23-2.r3
79.11+1.45
44.40 1.08
12.03
641.30+2.18
01.42++2.15
DualPrompt
76.04
76.6220.74
46.13
40.80.1.04
65. 25.03
62.43+1.75
CODA-P
19.13
80.91 10.70
51.87
48.09 22.75
66.01 12:00
62.90
50 (mixed)
MVP
67.74
63. 20.00
39.5011.
32.63 12.63
54.6911.14
50.07 00.07.13.50
MISA
80.35-22.09
80. 1:24
51.52::2.09
45.00 =
65. 63.40
60. 60.20
PlyPrompt (ours)
83.24
86.76=0.78
$6.58
55.27
70.64
73.40 :1.88
82.98-2072
20.9
45.48 48.48
43.1220.75
71.7445.58
61.52+1.88
DualPrompt
81.12-2.10
75.94
46.79
41.4220.03
72.81
62.4620.99
CODA-P
82.68
17.97
53.01
49.31 49.31 12.87
73.95
63.90
(disjoint)
MVP
74.92
56.17 56.1712.95
40. 40.43
28. 28.32
63.41
43. 43.80
MISA
85.67=1.01
81.04=1102
53.88 13.88
47 20.7
74.27
62.97
FlyPrompt (ours)
88.25_00.00
85.51 =0.64
57.41
+0.95 +0.95
55.69 #0.33
78.14=3.02
74.34
Table 10: Performance comparison of тB variants in FlyPrompt. We fixed 50% and use
Sup-2IK PTM backbone. All results are reported as an average five parallel runs (#: standard deviation) with different random seeds.
CIFAR-100
ImageNet-R
CUB-200
(9%)
Method
Anne (%. t)
A (%, f)
Auue 1)
Alnet (%, f1)
Agne (%, 1)
Auass (%, t)
1.2P
76. 28.23
79. 41.43
44.40
42.03-1.72
64.30+2.18
61.4242.13
DualPrompt
76.04
76.62 16.62
46. 1.99
40.80-1.04
05.03
62.43
CODA-P
79.13
80.91
51.87
48.09
66.01
62.90
10
MVP
67.7414..66
63.22110.6
39.501141
32.63_3.05
54.6913.u
50.07
MISA
80.35 12.39
80. 11.24
51.52..200
45.08 1.44
65. 13.01
60.20
FlyPrompt (ours)
83.24
86.76=00.73
56,58
55.27
70.64
73.40
L2P
78.48
80.13
43.32
42.27
63.67
63.60
DualPrompt
77.7611.05
19.5010.49
45.11
41.01
64.36
63.7212.22
CODA-P
81.5011.20
10.75
50.55
47.58
11.49
64.97 12.83
30
MVP
71.01=1.70
65.71
38. 1.55
32.43-4.75
54.16444.56
51.0446.80
MISA
82.54=1.08
82.50+0.68
51.69:20.44
47 47.09-11116
67. 17.1341.72
66. 66.53-12.39
FlyPrompt (ours)
84.61
86.89
55.30
55.43
70.19
74.25
L2P
17.44
80.31 10.69
44.39 1.72
43.66 1.00
65,42 12.71
6-L.62 1
DualPrompt
77.4442.64
77.13 ..8
46.23
41.99
66.40+2.72
65.34433.5
CODA-P
81.3942.18
83.1040.97
53.05
50.20=1.74
67. 67.88-22.6
45.44222.52
50
MVP
67.9714.78
58.1111.26
40.68
31.87 2..50
57. 57.25.13.7
53. 53.8613.41
MISA
81.8142.29
22.514
53. 53.27
48.32
68, 68.6842..4
66. 84
FlyPrompt (ours)
83.69 #1.81
86.3 20.73
56.50=1.s7 1.87
55.59
71.95
74.03
FlyPrompt consistently outperforms baselines from the purely blurry regime (rD = 0) to fully disjoint tasks (rD = 1, namely, online CIL) and across different blurry ratios demonstrating robustness under extreme GCL configurations and superior versatility.
As detailed in Tabs. and 10
25

---

Published as a conference paper at ICLR 2026
F.2 GCL EVALUATION METRICS
Table 11: Average accuracy of sessions and forgetting of different GCL methods over three datasets, All results are reported as an average of five runs (# standard deviation) with different random seeds.
CIFAR-100
ImageNNt-R
CUB-200
PTM
Method
Awe (%,,) f)
Fina (%,
Aayg (%, 1)
Pisa 1%6,
(%,
Fi Fi
L2P
75.4122.3
11.53_1.44
47.8220.15
19.16.1.92
65.4413.30
29. 19.15
DualPrompt
75.9612.5
11.42_0.91
49.90
18.24 14:34
66.5413.05
27.18
CODA-P
78.73_a.09
9.95
53.75_2.88
18.49
12.19
27.4313.42
Sup-21K
MVP
64.70=4.14
33. 33.19=2.a3
40.1 10.14=1.39
43.39=4.23
52.42+5.06
47.61 47.61
MISA
79.67
9.67
55. 15.778-1441 1.41
21, 21.4644.25
66, 66,92+3.12
30,06+3.70
FlyPrompt (ours)
5.03
60.38
13.42=1177
72.22 72.22
10.97
L2P
10.98
14.88 6.277
35.88.4.80
15.19
35.05.1931
DualPrompt
66.13
19.18_0.18
56.53_1.22
30.73.7.44
47.97
35.40
CODA-P
66.81 66.81
19.82=6.88
55.01=1.66
35.58
45.38
35.73+s.90
Sup-21K/IK
MVP
63,22
46.98-s.15
50.91
51.11+2.86
46, 16.1341776
62, 62.93
MISA
60.11
11.38__2.00
54.17 =22922
28, 28.06.4.119
43. 13.33
33 95,
FlyPrompt (ours)
76.86_2.20
9.50
64.41 $1.80
21.59 24.78
55.12 55.19
21.71 +3.56
L2P
52.73
12.58 6.22
38.4 1.338
34.17 34.17
14.93
19.09
DualPrompi
62.86.7745
19.22 4.22
46.07=2.16
37. 37.96.9.50
21.96
10.18
CODA-P
59.59-.7.94
22.20 =5.61
49.47=2279
41. 26.19
18.77
28 18.98
iBOT-2IK
MVP
61.48.22.55
50.23 50.23-11.22
44.01=4.18
62.17
34.92-22.26
62. 62.20+5.51
MISA
62.87 62.87=3.77
17.34_6.20
35.62:11.60
19. 18.48844..6
21.18
FlyPrompt (ours)
73.08 -2.18
8,38
31.40
23.40 23.20
30.78
18.85 14.15
L,P
48.97 48.97
16.47 13.47__..80
41.23
33.01 11.01.1.900
19.75
21.26:11.09
DualPrompi
50.16
30.45=3.88
49.51=11m7
35.14
30.29
32. 22.700
CODA-P
25.84
22.72
53.49
40.23
28.85+3.58
27.71+9.66
iBOT-IK
MVP
56.41 +2.00
53.47
46.96=3.97
56.26-5.19
34.97
62. 62.6146.5
MISA
52.58
19. 19.16
48 22.92
33.52-10.12
27. 27.98
29. 3..3
FlyPrompt (ours)
64,94 22.77
11.52
63.77 11.42
20.89 23.75
38.05 66008
21.62
L2P
45.45
14.82_6.s1
38.98
33.00
a:.66
24. 24.23g9
DualPrompt
49. 00.6555..46
18. 60
47.16=1.05
37. 77.65
29. 29.4845.58
31.35 10.61
CODA-P
50.76.55.5
19.43-5588
48 68 -8.77
41.26=20.00
29.43+24.89
32 10.60
DINO-IK
MVP
52.41 1.48
G4.70
44.45-4.11
56.5420..0
34.71+1.92 +1.92
G4. 64.2147.5
MISA
49.81 13307
17.76_4422
46.6 222.2
34.38 11.00
27. 27.4615.11
25. 25.77
FlyPrompt (ours)
63.59 4.56
9.04
60.8311.57 11.57
20.65 _3.08
37.50 s8.33
19.47 14.84
L2P
26.75
19.68 115.1
19.02_2.77
43.59
13.422a.55
26.38
DualPrompt
27.76
14.79 4.09
41.37
35.07 =..93
21.87++4.0
28. 28.42
CODA-P
48.26-25.99
16.97 16.15
44.71=2.45
43.27
22.33++3.4
10.90
McCo-IK
MVP
52.82
55.42_15.0
38. 68.
55.87
30.74
62.97
MISA
53.46
16.19 16.19_..26
45.41 25.41.440
31.42
26,
35. 15.500.7.0
FlyPrompt (ours)
60.16_6.as
60.16
7.69 1:2.42
55.95 $1.60
21.27
22.89
26.52466.3
20.69 13358
Following Moon et al. (023); Kang et al.
2025 ), we consider the standard evaluation metrics Aaues
Austs Aavg and Flast. for GCL performance. We denote Ri.j as the accuracy recorded right after session i with respect to the data in the session j. We then maintain a matrix R whose jth column is the history of evaluation after each session with respect to the data in session j. Firstly we calculate the final average accuracy Alast as:
T
Ajast
Ri
(12)
average running accuracy Asggi
T
Axvg
Ri
(13)
T
i=1
and the final average forgetting Fiast:
Fiast
(max( - RT,s),
(14)
T
where A and Anvg are the higher the better and Finst is the lower the better. They are all conven- tional metrics used in classic CL. research.
26

---

Published as a conference paper at ICLR 2026
Moreover, Aaue (higher the better) is the anytime inference metric proposed by Koh al. (2021) to better evaluate the GCL performance during the learning process. Specifically, the evaluation of GCL accuracy is performed every batches. Throughout this work, we fix b 1000. Denote the total number of the assessments performed as S, then given the history a E TR (where d, shows the accuracy of the model time stamp the data it has observed so far), calculate the area under curve of any-time accuracy Aaue as:
1 Σ Gg 5
Aauc
(15)
==1
In addition these metrics, we also report the backward transfer (BWT) metric (Lin et al. 2022) to quantify how learning later sessions influences performance carlier ones. Using the same notation Rig and T as above, we define
T-1 RTs - R..)).
BWT T-1
(16)
positive BWT indicates that subsequent learning improves performance on previous sessions (pos- itive backward transfer), while negative value implies net forgetting earlier sessions. BWT results of GCL methods are presented in Tab. 12
F.3 COMPARISON WITH PROMINENT OFFLINE PTM-BASED METHODS
Table 12: Extended Comparison of performance, backward transfer, and computational cost across PTM-based CL methods. All performance results are reported as an average five parallel runs (#: standard deviation) with different random seeds, over the CIFAR-10 100 dataset and Sup-21K back- bone. Parameter counts are measured in millions. Time cost is reported in seconds per batch.
Method
Asue (%,
BWT (%,
Total Param,
Trainable Param.
Training Time
Inference Time
L2P
76.23+2.7a
0.10+2.65
86.01
0.22
5.57
0.95
DualPrompt
76.04
2. 2.93_2.42
86.35
0.55
4,78
0.90
CODA-P
19.13
0.833_2.1
86.72
0.92
4.75
0.94
MVP
67.74
18 2..2
86. 12
0.32
5.35
1.27
MISA
80.35
1.7622..8
4.78
0,90
S-Prompe++
80.21 +2.55
0.81
86.26
0.46
6.03
1.18
HiDe-Prompt
77.10 77.10
3.36 22.71
86.81
0.94
6.13
1.27
HiDe-LoRA
80.07
0.36
87,39
1.51
6.80
1.17
HiDe-Adapter
79.52
-2.05 2.05__1.9
87.41
1.53
6.68
1.04
NoRGa
78.89
2.72
86.81
6.69
1.05
SD-LoRA
79.264u.m
-G. 2..62
87.72
1.92
7.24
0.82
FlyPrompi (ours)
83.24
4.35
87.08
0.46
4.96
0.92
Table 13: Comparison between RanPAC variants and FlyPrompt over three GCL benchmarks. All
results reported average of five parallel runs (+ standard deviation) with different seeds.
CIFAR- 100
ImageNet-R
CUB-200
Method
Aaue (%,"
Alast Al,, t)
Aauc (%,
Auass (%,, 1)
Aaue (%, 1)
Atast (%, 1)
RanPACt
69.91 43.88
79.92.0.07
47.1442.18
50.75
60.1845.52
66.21
RanPACt
$7.35
77.654..21
36.90+4.17
44.39
64.522a8.23
71.65_0.17
RanPAC
77.88 +4.28
86.52
53.18 184122222
54.71 +2.48
69.64
+3.99
72.30 _.09
FlyPrompt (ours)
83.24 42.23
86.76
56.58
55.27 20.01
70.64 22.85
73.40
F.4 CKA ANALYSIS OF EXPERTS' REPRESENTATION SIMILARITY
We use centered kernel alignment (CKA) Korblith et al. 2019) to quantify the similarity between expert-specific representations while factoring out the shared contribution of the frozen PTM back- bone. For a given method and dataset, we first fix the backbone fo and, for each expert E, apply its prompt (or expert-specific encoder parameters) to obtain matrix of CLS features Z, Raxd
27

---

Published as a conference paper at ICLR 2026
1/
ImageNat-R (?-
(7-20) 20)
10
14
18
Expert ID
Expert
16
0.0
00
0.0
CKA
CKA
GKA
(6)
Expert
Expert
Expert
Figure 6: CKA similarity of feature representations between experts of MISA three datasets.
CIFAR-100 5)=
ImageNet-R (T= 10)
CUB-200 (T=20)
0.
0.
Expert ID
04
04
02
02
CKA
2 Export ID
CKA
&
CKA
Expert ID
(c)
Expert ID
Figure 7: CKA similarity feature representations between FlyPrompt experts on three datasets.
over a common set a samples. Using the same backbone together with the expert-shared pa- rameters (e.g., the global prompt in DualPrompt), we also compute common representation matrix 7 E Rxdd We then define the residual features of expert E, Z, Zi Zcom which remove the largely stable PTM-driven component and highlight the expert-specific modulation that emerges during online GCL. Based these residual features, we measure the (linear) CKA between experts E, and Ey. as
Z
Ze
CKA Ze) = 1 Z,
(17)
|r 2 Zr l|r
This similarity measure is invariant to isotropic rescaling and orthogonal transformations of the features, and is well-suited for comparing representations across experts. We report the pairwise CKA scores between all experts as a heatmap: diagonal entries capture self-similarity, whereas off- diagonal values reveal the degree of specialization redundancy among experts after removing the common PTM-induced component.
F.5
EXTRA HYPERPARAMETER SENSITIVITY TEST RESULTS
78.24
CIFARC-10
550
02.10
500
95.55
500
77.37
01.58
400
103
100
1000
200
200
200
75.3
100
00.17
100
$1.89
200
150
1K
20x
200
26
5K
10K
250
500
10k
20K
(6)
(8)
34
(4)
J4
78.54
02:25
04.85
36 10X
78.02
00.20
02.62
2
(6)
(A)
(6)
logun
(8)
Figure 8: Analysis of hyperparameters in REAR [Backbone: Sup-21K/IKJ. (a-c) Different random projection dimension M with fixed À 10s; report Aque and extra storage cost (bar) given M. (d-f) Different regularization parameter À with fixed M = 4
28

---

Published as a conference paper at ICLR 2026
CIFAR-100
Image
CUB-200
1.:6-- 101
06.70
A-
28 30
10
100
100
202
200
100
08:44
20x
01.22
25 86
10K
(6)
(0)
(6)
75.20
00.93
82
13.08
10.20
23.07
(6)
(A)
(+)
(0)
Figure 9: Analysis of hyperparameters in REAR [Backbone: iBOT-21KJ. (a-c) Different random projection dimension M with fixed X 10?: we report Aaue and extra storage cost (bar) given M. (d-f) Different regularization parameter À with fixed M = 10
800
ImageNNrt-R
69:11
111
00.54
30.31
10
200
100
100
20c
50.20
200
36
200
10c
20K
(6)
M
(8)
(4)
37-1006
60.75
00.01
37.47
07:20-
02:01
(60
(A)
(e)
(1)
Figure 10: Analysis of hyperparameters in REAR [Backbone: iBOT-1K]. (a-c) Different random projection dimension M with fixed X 1 : we report Ague and extra storage cost (bar) given M. (d-f) Different regularization parameter X with fixed M 104
503
CIFAR-100
ImageNart-it
CUB-200
00.32
57.32
10
82
10
302
300
200
100
100
20K
02
24
200
10k
30k
(5)
(b)
(9)
07:07
10"-10k
20 U.K
3/-1566
00.58
50.05
36 30
66.02
00.10
31.51
69
(A)
(4)
(0
Figure 11: Analysis of hyperparameters in REAR [Backbone: DINO-1K). (a-c) Different random
projection dimension M with fixed À 1 we report Ague and extra storage cost (bar) given M. (d-f) Different regularization parameter X with fixed M = 10
66.7
500
60.06
Image
QUB-200
00.00
1
51
25 50
A
300
60.22
203
200
100
100
61:07
254
68.00
24
200
10k 35k
(6)
M
(8)
(9)
02.14
13K
63.21
51.20
24.00
25.58
20.50
06
00.00
00.53
22 15
(5)
(A)
(+)
(0)
Figure 12: Analysis of hyperparameters in REAR [Backbone: MoCo v3-1KJ. (a-c) Different ran- dom projection dimension M with fixed À 109; we report Agne and extra storage cost (bar) given M. (d-f) Different regularization parameter À with fixed M = 1
29

---

Published as a conference paper at ICLR 2026
F.6 ADDITIONAL ABLATION STUDY
Table 14: Effect of REAR and TE? on various PTM-based CL. methods over three GCL benchmarks. All results reported as an average of five runs (# standard deviation) over Sup-211 21K backbone.
CIFAR-100
ImageNet-R
CUB-200
Setup
Method
Auue 1)
Auaa 1%6, 1)
Anue (%,, 1)
Ane (%6.1 11
Ammn 11
Auaa (%6,
DualPrompt
36.04
76.6220.74
46.13
40.80+1.04
65.03+2.21
62. 02.43:1.1
MVP
67.7414.96
63. 2..00.2:000
39.50.1441
33.6313.95
54..693.114
50.07
MISA
80.35 80.3512.30
80.75 1.27
51. 12:09
45.08
65, 65.4013.01
60,20
Baseline
S-Prompt++
80.21+2.66
83, 48.4881..0
62. 1.45
49.13+1.60
66.61+22.1
63..232..25
HiDe-Prompt
77. 77.10
21.77=2.00
63.77
49 49.87=33.0
67.05++2.57
67.12+0.50
NoRGa
23.89
83.03
14.12
23.09
67.16g2.41
67.06g0.58
DualPrompt
80.6312.25
83.65
53.1611.28
51.11 10.91
65.96
63.81, 63.8
MVP
67.44
62.33
38.87 11.27
31.59 31.5914.11 14.19
53..6513.17
18.25
MISA
83.82 1..
07.30 11.12
54.02 00.00
68.011:2.44
65.55
w/ REAR
S-Prompt++
81.4322.45
83.9320.8 84
54.7441.53
52.30 1.005
66.80
64.72+ 1.97
HiDe-Prompt
78.4112.64
83.4611.44
53.61 53.6111.16 16.16
49.26 49.26.12.36 12.56
67.05
66.99
NoRGa
19.37
83.7911.28
54.7811.00 11.03
50.2419.55
67.26
67.20
DualPrompt
76.83
78.00
47.1 47.11 12.19
12.15
66.47.2.72
65.42
MVP
68.91
34.28=1.00
42.06
35.94
26.98
54. 24.184
MISA
81.6512.24
82.80
54.05
18.46
69.3012244
67.29 12244
w/ TH?
S-Prompt++
81.9312.21
83.98
55.37 11.64
52.91 11.53
67.97 12.51
07.6811.44
HiDe-Prompt
77.4613.56
$2.0911.22
54.83
50.26 12.78
57.777.2.60
69.64
NoRGa
79.16
13.01
a1.58
51.81 23.51
67.97
69.58
DualPrompt
82.33
86. 16.14
54.72+1.29
54.00+0.76
69.65
72. 72.75+2.08
MVP
68.93+.4.0
64.52+1.45
41.59+1.34
35.45+1.44
55.65
51.8442.41
MISA
83.60
86.661...5
59.12.1.02
56.62.0.82
72.38
74.68
w/ both
S- S-Prompi++
83.11 +2.10
86.67
56.57
55.24
70.65
73. 13.42
JiDe-Prompt
78.60 78.60
83. 1.112
54.79+1.12
51.30+2.81
68.27
69.61+0.84
NoRGa
79.37
83.79
55.75+1.31
52.50+a.a3
68.32+2.64
70.03+0.07
FlyPrompt (ours)
83.24
86.7640.78
56.58+1.47
55.27 +0.01
70.64+2.8
73.40+1.55
F.7 DIFTERENT LOGIT MASK STRATEGIES FOR GCL MASKS
Table 15: Performance of different logit mask strategies for GCL methods. All results are reported as an average of five parallel runs (+ standard deviation) with different random seeds, over Sup-211 21K.
CIFAR-100
ImageNet-R
CUB-200
Mask Type
Method
Anne%, 1)
Ama(% 1)
Amnt9% 1)
Aua (%, 1)
Anuc(%, 1)
A (%,. 1)
L2P
62.74 62.74
56.0812.10
31.56g1.05
26. 36.18... 18:00
54..83
47. 17.944.
DualPrompt
06.08
61. 61.98:4.00
11.02g1.45
36 36.08 11.00
36.08
50.71 21
CODA-P
66.15
58.4221.30
40.71
30.56
56.44_2.84
49.50
No Mask
MVP
68.1814485
63. 26
38.79
82.01
54.7422.01
52.88
MISA
09.85.a.1
65. 13 13:0
35. 35.91....4
37. 37.87, 22.49
52. 62.15
FlyPrompt (ours)
78.73
83.6220.50
21.39
48.7221..0
69.222a.04
73.07
L2P
62.16 6..16
54.67
1..10
21.7314.32
52.92.2.51
45.05
DualPrompt
65. 65.48
50 1.70
36.94
29.22:21.6
55.07 07.07
47. 25.83
CODA.P P
65.58
57.0 17.05
39.23-22.0
28. 28.91 2.
55.64=2..7
48. 18.00=5.20 50000
Random Mask
MVP
67.7524.90
63. 23.77
39.45
12.70.+3.0
54.72=a.14
50.01+3.5
MISA
68. 23,
61.55
40.67,
29. 18..88.. 2.
56.06
48. 12
FlyPrompt (ours)
13.62
81.88 10.88
51.67 $1:50
47.26 21.25
68.63 1:2.82
69.52..4.0
L,P
62.222-4.39
23.36
33.80
25. 25.20
53.10=22.0
45.55
DualPrempt
65.291.1.2
57. 2..3
37.31
29. 72.722.:
55.25 55.25
17.67
CODA-P
65.63 65.63=6.40
56. 1.41..1
40 40.13:2.40
29. 10.35.2..00
55. 55.80=2.68
47.71
Seen Class Mask
MVP
67.7224.87
62.99 99.99
39.57=1.44
32.72.4.0
54.72ga.14
50.14 3..00
MISA
08.31 13.00
61. 61.44
41.17, 11.85
29. 14.14
56.44 12.42
48.58
FlyPrompi (ours)
78.75
82.87 20.89
52.39
48.02
69.28
70.91
L2P
76.23=2..5
11.14
44.40 44.40
22.03
64.30a2.1s
61.42:22.1
DualPrompt
76.04ga.32
76. 72.6222.74
46.13
40.80 04
65.03=2224
62 62.43
Batch Seen-Class Mask
CODA-P
79.1313308
80.91 10:m
51.87,281
18.09 2.77
06.01 12:30
62,90
MVP
67.7424..6
63. 22:0..0
39 1.
12.63323.0
54 64.669
50.07 =3.88..0
MISA
80.35 39
80.75
51.52-2.00
45.08
65.40 65.40
60.20 60.20
FlyPrompi (ours)
83.24
86.76
56.58
55.27 10.91
70.64
73.40 1..88
(1) No Mask, standard softmax over all output classes. (2) Random Mask, for each sample (xx, y),
set my O and assign m. 0 or randomly with 0.5 probability for each previously seen class
C # y. (3) Seen-Class Mask, setting m, = for all previously seen classes, -oo otherwise. (4)
Batch Seen-Class Mask (used), setting m. 0 only for the classes present in the current batch y.
30

---

Published as a conference paper at ICLR 2026
F.8 COMPLETE RESULTS OF DIFFERENT EMA DECAY RATES FOR TEMPORAL ENSEMBLE
Table 16:
Performance comparison of different EMA decay rates for TE? across all PTMs. All results are reported as an average of five parallel runs (# standard deviation) with different seeds.
CIEAR-100
ImageNet-R
PTM
EMA Decay Rate
Aaue (%, f)
Aust (%, f)
Aauc (%, f
A (%,
online
81.90 20
84.23 1.32
54.91 11:32
52.58 11:30
0.9
82.81
86.36_o.
56. 11.52
55.09
0.99
82.84
86.41 -0.30
55.9441.68
54.6710.99
Sup-21K
0.999
81. 31.80.22.3
84.39
55.15 15
53.5210.90
0.9.0.99
83.24
86,76
56.58+1447
55.27
0.9,0.99.0.999
82.
86.24
50.3511.72
55.50
online
71.28
69.73
53.12
44.694:3.66
0.9
75.59
77.30
61. 61.56 11.56111.4
57 57.35
0.99
77.96
79.7 79.71
60.96
56.32+1.57
Sup-2IK/IK
0.999
74.13 11.6
74.68
53.90 53.9611130
46.55
0.9,0.99
78.48
80.30
62 01 02.30
56.55 56.55.13
0.9.0.99.0.999
78.44
80.55 :4.12
62.59 #2.22
58.00
online
67.01 sa.st
65.38_7.24
45.32:1144
35.45-14.60
0.9
T2.43
75.46
56.37.2.00
52.37
0.99
75.03
78.08
55.93
51.64
iBOT-2IK
0.999
13.22
69.30 2,94
43. 43.7811221
34..6412.90
0.9.0.99
75.58 2.70
79.36.33447
57.75
54.39 11:20
0.9.0.99.0.999
#4.44
76.61=2.22
55.94443.12
52. 12
online
61.38 12.32
60.58 7.91
50.79
41.92
0.9
68.01=:1 18
71.99_4.62
11.15221151
56.98 1..11
71.50 =1.00
75.20
06 .1.6
56 57 +0070
iBOT-IK
0.999
65.88 19.31
67.07 1.98
50.27 11.40
42.71 12.00
0.9,0.99
10.14
74.84
61.50
=1.00
57.18
0.9.0.99.0.999
67. 67.93
70.69
59.60 59.60 21.8.
55.35
online
58.61
60.76 6.777
47.35.12.24
41.3312.00
0.9
62.95
699.55_..3
56.88 56.83.11.47
53. 68 68.
0.99
66.42=2.51
73.03
56.67+1.77
53.23
DINO-IK
0.999
60.37
64..66
46.38
39.6912.00
0.9.0.99
65.92
72,66
57.29
54.72=1.80
0.9.0.99.0.999
65.27
70.83
55.6611.87
51.91 11.73
online
57.90
62.20
10.81
35.46.13.92
0.9
61. $1.96-4
69.83
51.47+1.64
47. 8841.48
0.99
65.95 1.44..4.40
73.28 16.28.6.90
50.7541.99
47.42 47.42.1.29
MoCo-1K
0.999
61,26
66.42
42.33
36.8222.11
0.9,0.99
64.12
71.51
$2.32
49.06
0.9.0.99.0.999
64.03 14.44
69.92 63
51.6412.55
48.09 10.
F.9 COMPARISON OF DIFFERENT ROUTING ALGORITHMS ON RANDOM EXPANDED FEATURES
Table 17: Comparison routing algorithms based on random expanded features in FlyPrompt. M: expansion dimension (default 10,000); T: number of experts (default 5); H: hidden dimension of MLP (H = 512 is used); K: number of nearest neighbors (K = 10 is used). Time cost is reported in seconds per batch CIFAR-100. All performance results reported an average of five parallel runs (+ standard deviation) with different random seeds over Sup-21K.
Routing
Train Time
Inference Time
Inference Complexity
CIFAR-100
ImageNet-R
CUB-200
Algorithm
Amnc(, f)
(FF.)
Ana (%i
Prototype Similarity
5.58
O(MT))
40.67
63.80
56.2911.72
52.36
67.00
66.66
Naive Bayes
5.30
0.93
12.775.017
85.51
55.8641 as
63.84 -3.0
BLDS L9
60:60
MLP
7.03
1.00
OLMH RT)
81.75
82.70 62.76.1.4
56.31
53.70
08.53
07 07.92.1.1
X-Means
O(KMT)
02.22
35.27
56.00 94
53.08_1.4
68.33
68 24:20
Ridge Regression (ours)
4.96
0.92
OLMIT
$3,24-2
86.76
26.58
55.27 20.01
70.64:245
73.40=..8
(1) Prototype Similarity, cosine similarity to each expert's mean feature. (2) Naïve Bayes, assum-
ing Gaussian-distributed features per expert. (3) MLP, a two-layer MLP router, as in HiDe (Wang et al.) 2023a). (4) K-Means, clusters each expert's features and routes based the nearest center. (5) Ridge Regression (Ours), the REAR analytic router trained once over accumulated statistics.
31

---

Published as a conference paper at ICLR 2026
F.10 COMPLETE RESULTS OF DIFFERENT AGGREGATION METHODS FOR TEMPORAL ENSEMBLE
Table 18: Performance comparison of different aggregation choices for TE across all PTMs. All results are reported as an average of five parallel runs (# standard deviation) with different random seeds.
CIFAR-100
ImageNet-R
CUB-200
PTM
Ensemble Method
Ague (%, f)
Aane %6,7
Aba (%.1)
Aaue (%, f1
Aboo (%6,
Mean
81.34
85.11
52.71 41.36
53.24 12
68.49 68.4942.57
73.95 04.95,:1 19.11.90
Max Prob
82. 22.29
84.95=1.20
55.50
53.53
68.00
66.56.1.m
Min Entropy
81.92+2.19
84.23
55.05
52.88
66. 7822.5
64.73
Sup-21K
SoftMax+Mean
82.30
85.98 20.9
56.16 16.16
55.53
70.77 13.00
74.86
SoflMax+Man
83.24
86,76.0.73
16.58
55. 55.27
70.64
73 13.40
SoftMax Min
83.11+2.34
86.50 -0.64
55.94
54.24+1.44
69. 36622.00
71.51
Mean
29.44
82.82 1
60.84
56.97.22.2
D6.64
60.39
Max Prob
72.58 12.9
71.31
54. 54.28 22.2.
46.44433
47.0513.41
44.42..:.50
Min Entropy
71.27
69.73
53.05
44.80
45.36 45.3613.11
42.82
Sup-21K/IK
SoftMax+Mcan
77.72+148
81.77
60.35+1.58
56.63
57.68 a5.02
62.6014.005
SoftMax+Max
78.48 #1.31
80.39 3.54
62.01
56.55
54.42
55.50
SofiMax+Min
78.30
80.07 =3.08
60.89
43.59
54.12 14.122
54.75
Mean Max Prob
76.59 #1.33
$1.40+2.45
54.92
61.82+1.0
29.31 #5.17
37.60
68.51+1.08
(0.00....1
46. 46.74+1.5
37. 07.6444.220
24. 24.14+3.55
28. 53-g5333
Min Entropy
67.03+3.55
65.
45.33
36 80.-6.44
23. 23.44445 23.44443.39
27.40
iBOT-21K
SoftMax+Mcan
74.69
80 .8.
54.41
52.58
29.09 20.33
36.28
SoftMax+Max
75.58 11.70
79.36 23.47
$7.75.22112
54.39 19
28.86 166.5.56
36.79
SoftMax+Min
24.87
77. 77.60=..77
55.98 11.90
12.03
27.47.5.44
34.57 2:0.12
Mean
10.96
76.38 4..0
18.15
54.24
37.64 #6.08
3.001
Max Prob
62.92+2.4
63.09
52.02.1.32
44.15+1.00
31. 31.53
34.27
Min Entropy
61.36+2.30
2.:73
50. 1.30
42.27
30.53
32, 32.82.44.00
iBOT-IK
SoftMax Meun
67. 17.24
72.
2..0
16.1441.38
53.35
38. 38.08,.. +6.57
44.13+44.27
SoftMax+Max
20.144
74.84-.. 1.4.....0
11.50....66
57.18
38.54.1.77 25.72
45.00
SoftMax+Min
09.72.1.64
73.76
59.87
55.39
37.31 20.33
41.86g.4.0
11.7
73.94.3.00
54.00
51.60
37,91 1...s
44.022...2
Max Prob
59.76+8.26
62.61 27.14
48. 48.12
42.17
31.3640.40
34.50 4..9.0
Min Entropy
58.51
60.79 60.79-6.07
47.14
40.9552..1
30. 30.15 15..44
32.83
DINO-IK
SoftMax+Mean
64.79
72.87
52.9811.49
51.38 1..38
37. 22.22....2
43.74..5.34
SoftMax
65.92
4..52
$7.29 =2.40
54.72
37.38 25.86
44.66
SoftMax+Min
65.48
71. 71.88
55.69
52 25,50++ 1.00
36. 36.4846.75
41 41.43gg2214
Mean
64.29 20.000
72.17 1.s
49.69
46 20.862o
27.92.1519
33.32
Max Prob
58.701s.om
63.89
44.07 10.90
37.06
21.35
24.21 14.551
Min Entropy
67.84.6.22
62. 22.21
42. 42.79.0.s2
35.51
20.59 12.90
22.7 22.75.44
MoCo-1K
Moan
02.90.6.00
71.71
50.56
47.95
26.95 16.22
31.50
MoftMax+Max
16..18
71.51 12.11s844
12.32
49.06 1.33
25.49 14.55
40.4444..50
SoftMax+Min
63.9215.03
71.29=s.50
51.46,41.4
47.93
25. 25.26
29.33
F.11 MORE RESULTS ON SCALABILITY AND EFFICIENCY OF FLYPROMPT
Table 19: Parameter counts, storage and computational complexity breakdown of FlyPrompt com- ponents, M: expansion dimension (default 10,000); T: number of experts (default 5); prompt length (default 20); d: embedding dimension (default 768). Results are reported in millions.
Components
Total Param.
Trainable Param.
Storage
Storage Cost
Computation Cost
G matrix
0.00
0.00
100
O
(
Q matrix
0.00
0.00
0.05
O(MT)
O(MT)
Router Head
0.05
0.00
0.05
O(MT)
O(MT)
Prompts
0.38
0.38
O(ld)
O(Pd)
TE heads
0.77
0.08
0.77
O(dT)
O(dT)
32

---

Published as a conference paper at ICLR 2026
G PSEUDO-CODE OF FLYPROMPT
G.1 PSEUDO-CODE: ONLINE REAR UPDATES AND TE AGGREGATION
Algorithm FlyPrompt: online REAR maintenance and TE? inference
1: Inputs: sessions (D:Y backbone So. random matrix RERdXM, ridge X, EMA decays
online iterations k.
2: Initialize: Gt- G Q-OMXT, prompt set Pt-,, online head (W, logit mask m 0
3:
4: (1) Online Training Phase
5: for (-1 to 7 do
6: Set expert E,: P.+ - D ift>1 else random
7: P - PUPUR {pr}
8: Initialize EMA heads: (W:)) b( ) b), fj E .n.
9:
for each batch (X, CD, do
10:
Set logit mask m. for any class E m,+0, and for e' $ y. me 00
11:
for to A do
12:
Update (W, b) and P. by minimizing CE(fo ( T +b+m. y) EMA for Eg w aa, aa, ( W. (1) ba,,, (,) +(1-a;) ab)),, Vj
13:
14:
end for
15:
H=:o(X:p.); o
16:
Update REAR stats: G Φ; Q-Q+++++
(one-hot C2 for expert E+)
end for
end for
Update closed-form router offline: -T QQ
(only once after the training phase)
(2) Inference Phase
22:
for from the test dataset do
23: Get routing
((a)))((((
24: Select the expert: et arg maxig7 (t()))
25:
Get online head outputs: z (0)) f p.) W +b
Get EMA heads output: 2 -- fe p.) W. ( +b!0)
26:
27:
Get aggregated ensemble output: =(x)) max ( (2()maax((()+m))))
28:
Final output: ū(x)= arg max, (x)
29:
end for
G.1.1
COMPLEXITY
Memory for REAR: storing G E M and Q E MXT is 0(M²).
Time complexity of solving the analytic router: O(M3) (matrix inverse and multiply).
Per-sample cost: forming sp(x) (h R) costs (ddM) (matrix multiply).
33


Reviews:
[{"review_id": "YQ3QIidJGc", "reviewer": "Reviewer_Vwq2", "summary": "the paper introduces a neuro-inspired framework for General Continual Learning that learns online without task labels or replay. It breaks the problem into two parts: routing and expert competence. The Random-Expanded Analytic Router uses random feature projections and closed form ridge updates to select experts efficiently -- no  backpropagation. The Temporal-Ensemble Experts module maintains several EMA classifier heads with different decay rates and it is combining them to balance plasticity and stability. Theoretically REAR is shown to approximate batch ridge regression, and TEE achieves an almost optimal bias–variance trade-off. Experiments on CIFAR-100, ImageNet-R, and CUB-200 under the Si-Blurry protocol demonstrate consistent state-of-the-art results with minimal trainable parameters and runtime overhead.", "questions": "1. plz clarify how REAR maintains the inverse of (G+ lambda I) online. Is inversion recomputed per batch or updated incrementally?\n\n2. Is it that each sample contributes once to G and  Q or multiple times across the three online iterations per batch? If repeated, the estimator corresponds to weighted ridge regression - not the exact form proven in Lemma 3.\n\n3. In Lemma 1 the jump from pairwise concentration to operator-norm bounds on \\hat \\Sigma - \\Sigma is not rigorous. I think this is fixable though.  \n\n4. The paper states a random-feature error rate O(\\log N/M) but this is  is inconsistent with the lemma’s \\tilde O(\\sqrt (\\log N/M)) bound, right? Plz correct or provide an argument for the stronger rate.\n\n5. Thm-1 gives a routing-accuracy guarantee  but it does not make any margin assumption between expert scores. Introduce this assumption and carry the margin constant into the final bound.\n\n6. Thm-2 makes the bias step explicit. But can you show formally that \\sum_j \\alpha^j \\Delta_{t-j+1} \\le L P_t --  and quantify the constant C_2?\n\n7. the claim that a geometric EMA bank achieves near-optimal performance should be supported with a short covering-ratio argument showing the error factor in terms of grid spacing r\n\n9. If there is time in the rebuttal phase, plz evaluate the effect of removing or randomizing the logit mask to confirm that improvements are not due to boundary information.\n\n10. Again, if there is time, it would be good to compare REAR with RanPAC under identical settings to clarify the unique contribution of routing versus analytic classification\n\n11. I would appreciate a list of Assumptions (data i.i.d., λ > 0, bounded feature norms, fixed number of experts) at the start of the theory section", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper identifies two core challenges in GCL a) expert routing and b)expert stability. it addresses each with a distinct, principled component: REAR and TEE.\n2. REAR does not do backpropagation. Instead it maintains online sufficient statistics and is solving a closed-form ridge regression, which is much faster/simpler.\n3. the authors show consistent state-of-the-art results on CIFAR-100, ImageNet-R, and CUB-200 under the Si-Blurry protocol.\n4. I like the math analysis linking random-feature expansion to generalization (thm 1) and characterizing the bias–variance trade-off in temporal ensembling (thm 2)\n5. Less than a million trainable parameters \n6. I very much like the analogy to multi-timescale synaptic adaptation and biologically plausible interpretation of the design", "weaknesses": "1) Thm 1 relies on a pairwise concentration lemma but omits a full matrix-concentration argument and a margin assumption needed to link regression risk to routing accuracy. This may be fixable in the rebuttal period. \n\n2)  Thm-2:  the derivation connecting EMA bias to temporal drift is approximate. The claim of “near-optimal adaptation” is not formally proven. Needs to be clarified\n\n3) Despite the task-free framing that the paper emphasizes, in my opinion REAR initialization and label accumulation still assume known session starts and one-hot session indicators. True?\n\n4) Maintaining and inverting the (M \\times M) Gram matrix can be memory-intensive for large random expansions\n\n5) The method’s logit mask may leak boundary information but this is not analyzed against Si-Blurry baselines such as MVP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MMssaQAm1l", "reviewer": "Reviewer_jMmN", "summary": "Existing parameter-efficient tuning methods struggle in General Continual Learning (GCL) because they cannot effectively allocate expert parameters or improve representations in single-pass, boundary-free data streams. Inspired by the fruit fly's brain, this paper proposes FlyPrompt, a framework that decomposes GCL into expert routing and expert competence improvement. FlyPrompt uses a random analytic router to activate experts and a temporal ensemble of output heads to adapt, significantly outperforming state-of-the-art baselines on key benchmarks like CIFAR-100 and ImageNet-R.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Biologically Inspired Foundation**: The framework is grounded in the neurobiological principles of the fruit fly's brain, offering a novel approach to solving complex GCL challenges.\n\n**Addresses Core GCL Problems**: It effectively tackles two fundamental challenges in GCL: \"expert routing\" (selecting the right parameters) and \"expert competence improvement\" (adapting to new data) under difficult, realistic constraints (single-pass data, no task boundaries).\n\n**Novel and Efficient Components**: It introduces two key innovations:\n\n - A randomly expanded analytic router for non-iterative (fast and efficient) expert selection.\n\n - A temporal ensemble of expert heads to ensure the model robustly adapts to data changes over time.\n\n**Proven Performance**: The method is backed by both strong theoretical analysis and excellent empirical results, demonstrating superior performance and scalability across multiple GCL benchmarks.", "weaknesses": "**Notational Clarity**: There appears to be a notational inconsistency in Equations 2 and 3, where the symbols $\\Phi$ and $\\varphi$ seem to be transposed or confused.\n\n**Comparative Analysis**: The paper would be significantly strengthened by a direct comparison of FlyPrompt against other prominent methods (such as LoRA, Adapters, and MoE). This comparison should explicitly analyze key metrics:\n\n- Parameter efficiency (total and new parameters)\n\n- Computational overhead (training and inference time)\n\n- Backward Transfer (BWT)\n\n**Novelty of Application**: Given that Random Projection is a well-established technique, what is the specific novelty of its application within the FlyPrompt framework? How does its integration into the \"randomly expanded analytic router\" differ from standard implementations and what unique advantages does this specific application provide for the GCL setting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QOcr6UHPLp", "reviewer": "Reviewer_Yd6v", "summary": "The paper proposes FlyPrompt, a framework for GCL inspired by the neural circuitry of the fruit fly mushroom body. It decomposes GCL into two subproblems: expert routing and expert competence improvement. For expert routing, the authors introduce the REAR, which uses fixed random projections and closed-form updates for feedforward expert selection. For competence improvement, they propose TE2, which integrates knowledge across multiple timescales via EMA heads with different decay rates. The method reports SOTA performance on benchmarks.", "questions": "How about the practical scalability and efficiency? Although Table 5 reports only marginal increases in total parameters and per-batch latency, it omits the auxiliary memory and compute burden of $G$ and $Q$", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The separation of GCL into routing and competence subproblems offers a structured approach to tackling its challenges.\n2. The use of principles from fruit fly olfactory memory introduces a novel interdisciplinary perspective to CL.\n3. The paper provides both informal and formal theoretical bounds on routing error and EMA parameter error, enhancing methodological credibility.", "weaknesses": "1. It resemble an ad hoc combination of existing techniques. The proposed FlyPrompt framework appears to be largely a composition of well-established components rather than a fundamentally novel algorithm. Specifically, the REAR combines fixed random projection with ridge regression (a paradigm already explored in prior CL works and analytic class-incremental learning). Similarly, the TE2 employs EMA with multiple decay rates, a standard technique in online learning and model stabilization (e.g., SWA, temporal ensembling). \n2. While the paper provides a biologically inspired narrative grounded in the fruit fly’s mushroom body, the mapping between neurobiological mechanisms and algorithmic design remains largely metaphorical. The performance gains reported (e.g., +11.23% auc on CIFAR-100) are primarily attributable to the strong supervised pretraining (Sup-21K) and the inherent benefits of random feature expansion, rather than the proposed routing or ensemble mechanisms. The ablation study (Table 2) further reveals that a RanPAC-like baseline already achieves 82.17% auc. This suggests that FlyPrompt’s contribution is incremental engineering rather than a necessary or uniquely effective solution.\n3.  While GCL as a research direction is valid, the specific formulation and assumptions in this work appear tailored to a controlled benchmark rather than a pressing real-world problem. The number of tasks $T$ (and thus the number of prompt experts) is assumed known and fixed a priori, which contradicts truly open-world or task-agnostic streaming environments. The evaluation protocol assumes access to task-level metadata during training (e.g., expert identity per session), which may not hold in fully unsupervised or user-generated data streams.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qNn3aLmMEf", "reviewer": "Reviewer_yU5J", "summary": "This paper targets General Continual Learning (GCL), characterized by single-pass, non-stationary data streams without clear task boundaries. Identifying limitations in existing Parameter-Efficient Tuning (PET) methods, it decomposes GCL into expert routing and expert competence improvement. Inspired by the fruit fly's memory system, the paper proposes the FlyPrompt framework with two core components: (1) A Random Expanded Analytic Router (REAR) using fixed random projections and a closed-form solution for gradient-free, rapid input-to-expert (prompt) assignment. (2) Task-wise Experts with Temporal Ensemble (TE$^2$) employing multiple EMA heads with different decay rates within each expert to dynamically refine decision boundaries over time. FlyPrompt achieves strong performance across various GCL benchmarks.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "[S1] The proposed Random Expanded Analytic Router (REAR) uniquely employs a closed-form solution rather than iterative gradient updates to assign inputs to experts. This is advantageous for the strict online, single-pass constraints of GCL, offering a theoretically grounded and computationally efficient alternative to traditional routing methods.\n\n[S2] The framework effectively tackles the complexity of GCL by decomposing it into two manageable subproblems: routing and competence improvement. The Task-wise Experts with Temporal Ensemble (TE²) addresses the latter by leveraging multi-timescale EMA heads, which significantly enhances expert robustness against non-stationary data streams.", "weaknesses": "[W1] Initializing the prompt for a new task by averaging previous prompts may not be beneficial and is likely to show degraded performance when subsequent tasks come from significantly different domains.\n\n[W2] While REAR outperforms gradient-based routers, comparisons against simpler non-learning baselines in the expanded feature space (e.g., k-NN routing) are absent, making it hard to gauge the benefit derived from the analytic ridge regression complexity.\n\n[W3] Experiments primarily use the default Si-Blurry configuration. Performance under more extreme imbalance, higher task overlap, or different types of distribution drift needs further investigation.\n\n[W4] While neuro-inspired, drawing direct equivalences between specific algorithmic components (e.g., EMA heads and KC subtypes) and biological counterparts might be an oversimplification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UWhmvODgS3", "reviewer": "Reviewer_EcKY", "summary": "This paper introduces FlyPrompt, an expert-based framework for General Continual Learning (GCL), where each task is associated with a prompt expert. FlyPrompt proposes two key contributions: a novel strategy named REAR (Random Expanded Analytic Router), which leverages random projection to identify suitable experts at inference, and Task-wise Experts with Temporal Ensemble, designed to track distributional drift. The paper is supported by detailed experiments across standard benchmarks and thoughtful ablation analyses.", "questions": "Please see the Weaknesses section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The decomposition into expert routing and expert competence is empirically well-supported.\n- The results are consistent across a broad range of pre-trained models (Sup-21K, iBOT, DINO, MoCo), rather than being limited to a single backbone.\n- The paper is further strengthened by extensive ablation studies and a comprehensive hyperparameter analysis.\n- The writing is clear, and the overall structure is well-organized.", "weaknesses": "- The paper does not discuss the number of parameters (different from memory in Appendix F) of the proposed method compared to the baselines, which could be a significant concern.\n- In the GCL setting, task boundaries are unknown during both **training** and inference [1]. The methodology described in L216–217, which states that \"we associate each task $t$ with a corresponding expert $E_t$,\" appears to assume that task identities are known during training, thereby allowing the assignment of a new expert. This assumption fundamentally contradicts the definition of GCL. Consequently, the problem being addressed may be closer to Class-Incremental Learning with overlapping classes rather than true task-agnostic GCL.\n- If task boundaries are indeed assumed to be known during training, then several recent SOTA methods like HiDe-Prompt [2], NoRGa [3], and SD-LoRA [4] could be straightforwardly implemented. The paper lacks a comparison to these highly relevant methods, which weakens its claims of superiority.\n- The REAR component, used for identifying task identity, is a powerful module in its own right and is functionally similar to the full RanPAC method [5]. Comparing FlyPrompt (which includes REAR) to methods that employ much simpler routing strategies may therefore be unfair. A more rigorous comparison would involve incorporating REAR into other methods (e.g., evaluating “HiDe-Prompt + REAR”) to properly ablate its contribution. A similar argument applies to the Task-wise Experts with Temporal Ensemble; this technique could also be combined with other baselines for a fairer assessment.\n- In L184-186, the authors motivate their second component by claiming that \"even with perfect routing, previous methods still exhibit inferior performance... highlighting... the limited competence of individual experts.\". However, the paper does not sufficiently diagnose the source of this inferior performance. It remains unclear whether the issue lies in representation drift within the expert prompts $f_\\theta(\\cdot, p_t)$ or in catastrophic forgetting within the final classification head $g_\\psi$. An experiment measuring the representation drift of each expert (e.g., by analyzing the similarity between representations from correct and incorrect experts) would be necessary to clarify and validate this motivation.\n- The “Task Experts as Temporal Ensembles” component is presented as a mechanism to enhance expert competence. However, the core prompt expert is trained using a standard cross-entropy loss (Equation 6). Thus, the observed novelty and performance gains appear to stem from ensembling multiple classification heads rather than from improving the representational quality of the prompt expert itself.\n\n\n[1] Dark Experience for General Continual Learning: a Strong, Simple Baseline, NeurIPS 2020\n\n[2] Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality, NeurIPS 2023\n\n[3] Mixture of Experts Meets Prompt-Based Continual Learning, NeurIPS 2024\n\n[4] SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning, ICLR 2025\n\n[5] RanPAC: Random Projections and Pre-trained Models for Continual Learning, NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)