Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning
Authors: Heming Zou
Yunliang Zang
Wutong Xu
Xiangyang Ji
Venue: ICLR 2026 Poster (2026)
Keywords: continual learning
fly olfactory circuit
class incremental learning
decorrelation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Using a nearly-frozen pretrained model, the continual representation learning paradigm reframes parameter updates as a similarity-matching problem to mitigate catastrophic forgetting. However, directly leveraging pretrained features for downstream tasks often suffers from multicollinearity in the similarity-matching stage, and more advanced methods can be computationally prohibitive for real-time, low-latency applications. Inspired by the fly olfactory circuit, we propose Fly-CL, a bio-inspired framework compatible with a wide range of pretrained backbones. Fly-CL substantially reduces training time while achieving performance comparable to or exceeding that of current state-of-the-art methods. We theoretically show how Fly-CL progressively resolves multicollinearity, enabling more effective similarity matching with low time complexity. Extensive simulation experiments across diverse network architectures and data regimes validate Fly-CL’s effectiveness in addressing this challenge through a biologically inspired design. Code is available at https://github.com/gfyddha/Fly-CL.

Published as a conference paper at ICLR 2026
FLY-CL: A FLY-INSPIRED FRAMEWORK FOR EN- HANCING EFFICIENT DECORRELATION AND REDUCED TRAINING TIME IN PRE-TRAINED MODEL-BASED CON- TINUAL REPRESENTATION LEARNING
Heming Zou'' Yunliang Zang** Wutong Xu' Xiangyang Ji
Depaartment of Automation, Tsinghua University 2 AAcademy of Medical Engineering and Translational Medicine, Tianjin University (zouhm24, xwt22) @ma ils...... tss tsiingua edu.cn cn yunliangzanget ju. ea.edduccm cn, xyji@ta sinlltsinghua...m www.ghuaaauuu.c..o....o
ABSTRACT
Using a nearly-frozen pretrained model, the continual representation learning paradigm reframes parameter updates as similarity-matching problem to miti- gate catastrophic forgetting. However, directly leveraging pretrained features for downstream tasks often suffers from multicollinearrty in the stage, and more advanced methods can be computationally prohibitive for real-time, low-latency applications. Inspired by the fly olfactory circuit, we propose Fly-CL, a bio-inspired framework compatible with wide range of pretrained backbones. Fly- CL substantially reduces training time while achieving performance comparable to exceeding that of current state-of- satte-of-the-ald the-artrrttt art methods. We theoretically show how Fly-CI. progressively resolves multicollinearity enabling more effective similarity matching with low time complexity. Extensive simulation experiments across diverse network architectures and data regimes validate Fly-CL's effectiveness in addressing this challenge through biologically inspired design. Code available at pottps://gee com/ gfyddhyyahaa/ SaayyddhaFFy-Cttt
INTRODUCTION
Artificial neural networks have exhibited remarkable capabilities across various domains in recent years. Nevertheless, real-world applications often require continuous model adaptation handle progressively emerging unseen scenarios, making updates based on sequential incoming data essential. This need has led to the development of Continual Learning (CL). Earlier research primarily focused on training models from scratch (Aljundi et al., 2018; Kirkpatrick et al., 2017: Li & Hoiem, 2017; Zenke et al., 2017). Pretrained models have recently become prominent in CL, owing to their robust generalization in downstream tasks for downstream tasks (Wang et al., 2022a;b; Zou et al., 2025b).
Popular CL methods utilizing traitrained models can generally be classified into three categories: (1) prompt/adapter-base approaches (Jung al., 2023; Smith et al., 2023; Tang et al., 2023; Wang et al., 2022a;b; 2025a; Liang & Li, 2024; Yu et al., 2024; Xiao et al., 2025; Wang et al., 2024b; 2025c), (2) mixture-basee based approaches (Chen et al., 2023; Gao et al., 2023; Wang et al., 2023a;b; approaches (McDonnell et al., 2023; Sun al., 2025; Zhou al., 2023a; 2024: Zhuang et al., 2024). All three paradigms operate without exemplars and significantly outperform traditional training-fron www.-orooratttee methods. Despite their strengths, each approach has limitations. Prompt/adapter-baatt based methods are constrained to trans- former architectures, and updating prompts/adapters inherently risks propagating forgetting within the prompt/adapter space. Mixture-basee approaches require storing previous models, resulting in significant storage overhead and increased computational complexity during model fusion. Compared with parameter-lcariigg approaches, representation based methods perform better by reframing
Zhou et al., 2023b; Che et al., 2025), and (3) representation-base
Equal contribution Corresponding author

---

Published as a conference paper at ICLR 2026
learning similarity matching and avoiding dependence specifie backbone. They form class prototypes (CPs) by averaging features extracted by a frozen pretrained network, with each prototype acting the centroid of its class. However, insufficient separation between features can lead to ambiguous decision houndaries when distinguishing between prototypes, complicating the similarity matching process. This challenge is formally recognized as the multicollinearity problem. Previous studies (McDonnell et al., 2023) have attempted to address this issue, but they still incur substantial computational costs, hindering real-world deployment.
APL
Preprocess
Odors
ORNS
PNs KCs
MBONs
Figure 1: Schematic of the Fly Olfactory Circuit. Odors are first detected and pre-processed by olfactory receptor neurons (ORNs) the antennal lobe, where feature extraction and normalization take place, before being transmitted to projection neurons (PNs). Expansion coding occurs signals move from PNs to Kenyon cells (KCs), with an expansion ratio of approximately 40. Each KC connects to fixed number of PNs (about 6). Lateral inhibition, mediated by anterior paired lateral (APL) neuron, suppresses most weakly activated KCs, exemplifying a winner-take-al strategy. Finally, signals from KCs to mushroom body output neurons (MBONs) involve dense down- projection that reduces dimensionality to select specific actions.
In biological systems, pattern separation a well-recognized circuit motif across cerebellum likee and related networks, including the cerebellum, hippocampus, and fly olfactory system (Lin et al., 2014; Papadopoulou et al., 2011; Stevens, 2015; Zang & De Schutter, 2023). These neural circuits efficiently decorrelate overlapping sensory inputs. Figure 1 illustrates the information processing mechanism in the fly olfactory circuit. The process unfolds in several steps: in response to an odor stimulus, olfactory neurons extract and pre-process odor information represented 50-dimensional vector (PNs). These pre-extracted features are then randomly projected into expanded dimensions (KCs), selectively activating small subset of KCs that receive the strongest excitation while zeroing out others. The high-dimensional features in KCs subsequently converge to low-dimensional MBONs for classification. Numerous studies have highlighted the role of the PN-KC transformation in producing decorrelated representations, and many strategies have been proposed to model this process. By contrast, the downstream KC-MBBN transformation has received far less attention, and a clear theory of its contribution to decorrelation has yet to be established. To address this gap, we theoretically show that the KC-MBON pathway also supports decorrelation under the Hebbian learning rule by demonstrating its consistency with ridge classification (See Section 6).
Inspired by the fly olfactory circuit, we propose Fly-CL, an efficient framework for progressive decorrelation in representation--asse learning with pre-trained models. The pipeline starts with feature extraction and normalization, followed by high-dimensional random sparse projection and top-k operation for collective decorrelation, mimicking the PN-KK process. We then implement prototype similarity matching mechanism mimicking the KC KC-MBON structure during inference, and an efficient streaming ridge classification method to decorrelate parameter weights during training. Empirical results show substantially reduced time consumption versus baseline methods.
Our main contributions are follows:
1. We propose an efficient and biologically plausible decorrelation framework that significantly reduces computational costs compared current SOTA methods while achieving comparable improved performance in CL..
2. Our method's effectiveness and robustness in decorrelation are validated by extensive experiments under various data setups and model architectures, supported by theoretical and empirical analyses.
2

---

Published as a conference paper at ICLR 2026
3. The alignment of our framework with the fly olfactory circuit suggests that biological structures can inspire effective and efficient solutions to Al problems.
RELATED WORK
Representatton-basee methods in CL using Pre-trained Models: Representation-basee methods (McDonnell al., 2023; Sun et al., 2025; Zhou al., 2023a; 2024) demonstrate superior performance over parameter-learmin learning approaches by leveraging features extracted from frozen pre- pre-trained model to compute similarities with class prototypes. They are also more practical for resource-cosstrained deployments (e.g., edge computing), since they avoid updating the entire model. For instance, in smart camera system, such methods enable the efficient addition of new object recognition capabilities without retraining the entire model. While showing promising progress, their computational overhead remains substantial, which may limit their applicability in scenarios requiring real-time responses.
Fly Olfactory Circuit: Information processing cerebellum-iiee circuits, including the fly olfac- tory circuit, involves several stages (Lin et al., 2014; Papadopoulou et al., 2011; Stevens, 2015; Zang & De Schutter, 2023). Theoretical neuroscience studies suggest that most stages exhibit progressive feature separation effects (Hige al., 2015). Algorithms inspired by the fly olfactory circuit have been applied across various AI domains, including parameter-efficient fine-tuning (Zou et al., 2025c), continual learning (Zou al., 2025b), locality-sensitive hashing (Dasgupta al., 2017; Sharma & Navlakha, 2018), word embedding (Liang et al., 2021), and federated learning (Ram & Sinha, 2022).
BACKGROUND
3.1 PROBLEM STATEMENT
In this paper, focus on CL within the context of image classification tasks. We denote sequentially arriving tasks as D {Di,..., Dr}. where each task D; = {(x), )i consists n, samples. Each sample of within task is drawn from the input space X, and its corresponding label y belongs to the label space 3/2 with e classes. The training process involves sequential learning from Dj to Dr. followed by class prediction on an unseen test set spanning the full label space.
To demonstrate the effectiveness and efficiency of our framework, we adopt Class Incremental Learning (CIL), a widely used experimental setup. Unlike traditional Task Incremental Learning, CIL does not provide access to the task ID during the testing process, making it more challenging. In CIL. the model learns mutually exclusive classes within each task, ensuring that the intersection of label spaces satisfies V/ n)) 0. We denote that there are e classes for the first tasks.
3.2 REPRESENTATION BASED PARADIGM IN CL
We study on the basis of the recently popular representation--asee paradigm using pre-trained models (McDonnell al., 2023; Sun al., 2025; Zhou et al., 2023a; 2024), which demonstrates superior performance for CL. Given an input image z/,, it is first compressed into a d-dimensional feature t/ = (=t)) Rdd using pre-trained encoder fo. For each class in task 1, we compute its prototype by averaging features over all training samples belonging to this class:
Ddl
Σ 1 = =)f((a()) E RR,
1
NI
where No - 5 [[y -B i) denotes the cardinality of class i's training set, and IT(-) is the indicator function. During inference, for test sample with feature vector v, the predicted class ÿ is determined by finding the maximum cosine similarity between v and all learned class prototypes:
V T
arg max
|0||
|
(2)
However, significant prototype correlations (E|p, S p. Ct.) > see Figure 3(a)), can severely compromise the discriminative power of the similarity measurement (Belsley et al., 2005). This phenomenon arises because high correlations reduce the effective angular separation between classes
3

---

Published as a conference paper at ICLR 2026
in the embedding space, leading ambiguous decision boundaries. Specifically, when prototypes cluster near a dominant direction R", the cosine similarity metric becomes less sensitive to subtle but critical inter-class distinctions, thereby degrading classification performance.
FLY-CL
In this section, we detail the design motivation and functionality of each component of our Fly-CL framework. A schematic of the overall framework provided in Figure along with the pseudocode for the training and inference pipeline in Appendix A. The decorrelation effect of each component is visualized in Figure using Pearson correlation coefficients of different class prototypes.
Concat
U
H2
Σ
Ridge Selection
Sparse Expansion Codiigg
SVD
Pretrained
C
model
Prototype Calculation
Ridge Classification
Sequential Tasks
Sparse Similarity Matching
Feature Extraction
Projection and Top-k
(Inference-time)
(training-time)
Our framework extracts image embeddings using a frozen pre-trained model, projects them into a higher-dimensiona space via fixed sparse random projection, and filters them through top-k operation (PNs - KCs). Then we utilize learned down-projection for similarity matching during inference time (KCs - MBONs). Right: During the training phase, the parameter C is learned via streaming ridge classification scheme.
Figure 2: Schematic of the Fly-CL Framework. Left:
4.1 SPARSE RANDOM PROJECTION AND TOP-K OPERATION
Building upon the eppresentation-hasee paradigm, it necessary to decouple different class prototypes. Inspired by the decorrelation mechanism of the fly olfactory circuit, emulate the sparse expansion projection from PNs to KCs, followed by winner-take-all inhibition mediated by APL neurons. Given feature embedding u E Rd extracted from the pre-trained encoder, formulate the transformation Z (v) IRd R as:
h' Z(v) = top- top-k (We).
where the fixed projection matrix W E Ix (with >> d) implements weight sparsity: each row contains exactly (p < d) non-zero entries independently sampled from N 1). The top-k operator implements activation sparsity by preserving only the & largest components (k < m) while zeroing out others, formally defined as:
[h]] = {{
[b]; if the magnitude of [h], is among the top-k values of h, otherwise.
(4)
This two-stage process achieves effective decorrelation through the following properties, and its empirical effect is visualized in the transformation from Figure 3(a) to (b).
High--Diensiooal Embedding Enhances Linear Separability: Random projection of low- dimensional features into extremely high-dimensional space can improve the linear separability of the feature representations (Litwin- Kumar et al., 2017).
2. Powerful Inhibition Suppresses Noisy Components: The top-k operation imposes sparsity by suppressing noisy dimensions that may interfere with discrimination through dimensional competition, while enhancing separation by keeping the most discriminative dimensions (Metwally et al., 2006).
Considering computational efficiency, W's sparse pattern reduces the time complexity for random projection from O(mn,d) to O(mnp)) while preserving the representational capacity compared to dense projection. Similarly, the top-k operation reduces similarity matching complexity from (mm)e)) to Oknno))) while simultaneously improving performance.
4

---

Published as a conference paper at ICLR 2026
58
04
20
10
(a) After feature extraction
(b) After random projection
(c) After similarity matching
Figure 3: Pearson Correlation Coefficients of Prototypes at Different Decorrelation Stages in FLy-CL. Heatmaps display Pearson correlation coefficients for 10 randomly selected class prototypes each stage of our pipeline (consistent across visualizations).
We further propose two theorems demonstrate that strong sparsity does not significantly degrade performance. According to Theorem 1.1, as long as p and are not extremely small, the matrix W retains full column rank with probability 1 o(1 which is common approach to demonstrate that a sparse random projection does not result in severe information loss. Furthermore, Theorem 4.2, by proving that the performance degradation between this sparsification operation and the original vector is bounded, we show that & is not extremely small, it can preserve most of performance. For complete proof, please refer to Appendix B.
Theorem 4.1. Given the matrix W E Rmxd where >> d, with each row having exactly non-zero entries, which are randomly sampled from N(0,, 1). Let W E Rddd be any square submatrix of W. Then, for any O, it holds that
P (1dct(WW) 2 (2) 4/2
= =1-o(1).
Thus, for sufficiently large p and d, any submatrix W invertible with probability at least 1 o(1). Theorem 4.2. For top-k sparsification in the expanded dimension m, the performance degradation is bounded by:
E [l L(h',
w)ll ((',,)1l M
where L(-) is a performance loss function for downstream tasks and C, M are constants. To ensure negligible performance degradation, we require:
B| <
Vm
i.e., when &s ((m²) (0 < 1), the error bound decays polynomially with increasing dimension.
4.2 STREAMING RIDGE CLASSIFICATION
Previous studies decorrelation in the fly olfactory circuit have primarily focused PNs-KKCS transformation, where sparse and decorrelated representations have been experimentally observed. In contrast, the downstream KCs+MMBONs transformation has received little attention, and the physio- logical evidence remains inconclusive. This motivates us to investigate whether the KC-MMON pathway also facilitates decorrelation. To simulate the training dynamics of the KC-MBON projec- tions, we employ a biologically plausible Hebbian learning rule. As shown in Section 6, this rule is mathematically equivalent to ridge classification under the CL. setting. Consequently. we implement an online formulation with adaptive regularization, which naturally achieves decorrelation while ensuring computational efficiency and compatibility with sequential data. Ridge classification (Hoerl & Kennard, 1970) mitigates feature collinearity through & l--reuularization. trading increased bias for reduced variance by shrinking correlated feature weights, thereby stabilizing prototype estimation in non-i.i.d. sequential learning scenarios. Let H, E Ra xm denote the concatenation of high- dimensional features for n, samples in task t, and Y, E (0, 10 X represent the corresponding one-hot label matrix for e classes. We maintain two streaming statisties: Gram matrix G E mm
5

---

Published as a conference paper at ICLR 2026
capturing self-correlation, and matrix S Rmxo, accumulating disenssnnnni weights for each class prototype. During each task iteration t, these are updated as follows:
G: x G + H H., St + S - H Y
(5)
The classifier matrix C E R is updated via regularized least squares accordingly:
C = (Ge + Amm)
(6)
Prediction for preprocessed new samples h' E Rm follows:
arg max T'C..... tE(T...,ett
(7)
where C. denotes the i-th column of modulated prototypes.
Adaptive Regularization: Due to the inherent heterogeneity of different tasks, a fixed penalty coefficients À will cause suboptimal performance. For adaptive regularization, vanilla À selection via grid search and cross-vale validation incurs prohibitive computational costs of ((m²) for candidates where the expanded feature dimension m extremely large (McDonnell al., 2023). To achieve our efficiency desideratum, we draw inspiration from an adaptive Generalized Cross- Validation (GCV) (Golub al., 1979) framework that analytically approximates cross volsddaaio error without explicit validation steps that require calculating large matrix inverses.
Given new task data H; E IR xm, we first obtain singular value decomposition (SVD) as H. = U.E.V' where U, E TR and V. E mm are semi-orthogonal column matrices that satisfy U, U, = I,,, V. V, = I,.. and E, = diag(s)..... s,) g contains non-zero singular values with rank(H; time complexity is Om,rm). For candidate regularization coefficients X = {Amins.... Amax) on log scale,
min(ne, m) (typically full rank due to numerical precision). The SVD
the following steps to compute the GCV criterion for each one:
First, in (Ir) time, we get the shrinkage matrix and calculate the effective
by:
Σ XI,
D:
df(A) tr(D.)
(8)
+
Then, reconstruct the prediction value of ridge regression in (In,re)) time by
Y = (vcedidi) D:) @ 1 @ U,YY, where @ denotes the outer product, o denotes the Hadamard product, and vecdiag extracts the diagonal elements into a column vector. Finally, we obtain the GCV value by
(9)
GCV(A) YY, ((Al d((A) 2
(10)
Te The
with time complexity being (ln.).. The optimal regularization parameter is then selected by: X arg min GCV(A). xελ
(11)
Considering the projected dimension m is extremely large, we make the mild assumption that m > and Ice & m. thus r = min(n,, m)
The original / loop complexity is ((nereg) = OlIfa,)) < O( = O (n,rm). Hence, the time complexity determined by SVD, at On, Compared to vanilla cross-validation taking O(Im²), the time consumption is greatly reduced.
Accelerated Prototype Calculation: Upon determining the optimal regularization parameter through GCV, we solve Eq. 6 to obtain class prototypes C, While vanilla matrix inversion via LU decomposition provides a baseline implementation, for the sake of computational efficiency, we exploit the inherent positive-definiteness of G; + A,I to achieve computational acceleration through Cholesky factorization by:
L. G
L-
(12)
where L, denotes the lower-triangular Cholesky factor. This approach reduces theoretical complexity from (33m²) to O O for factorization, with triangular solves requiring half the FLOPs of general linear system solutions. The numerical stability of this method is ensured by the condition number bound x(L,) I <<G + Im), making it particularly suitable for ill-conditioned streaming scenarios where G, may accumulate numerical noise over tasks. The decorrelation effect of the streaming ridge classification is visualized via the transformation from Figure 3(b) to (c).

---

Published as a conference paper at ICLR 2026
Table 1: Performance Comparison on Pre-trained VTT-B/16 Models. We report the average training time per task (Tiain), average post-extraction training time (Tpea), and overall accuracy (4) across three benchmark datasets: CIFAR-100, CUB-200-2011 1, and VTAB.
Method
CIFAR-100
CUB-200-2011
VTAB
Timia(
[pail()
A(1)
Tuiis()
(pua(U)
A(t)
Tpa((C)
A(t)
L2P
263.54+0.10
183.56
87.74+046
52.02+a04
37.03
77.48-1.43
42.10=00
36.45=o02
81 81.24#067
DualPrompt
231 4944
153.66
87.47
46.28:00
31.642006
79.89=1.44
38.144011
31.641006
80,85
InfLoRA
220.82+0.4
140,
91.10=0..6
45.31
30,97=0.2
80.65-073
35.80
29.26 88.73+a.7
SEMA
241.60+0.2
160.87=am
92.04=0.25
+8.21
48.21
32,9620.20
=0.31
45.50
39,822026
91.18+a4
MoE-Adapter
187.91
106.37
90,431:046
37.89
22.61
79.65
32.28
26,0610:14
86.39
EASE
621.26+1.05 583.4:0.76
92.96=0.25
138.78=0.36
122.48+027
89.56=0043
108.35
94.73s1.11
94.02=a15
RanPAC
98.62 84.42:0044
9..21
37.95
33,83
92.67=0.0
63 63.68+0.1
61,442017
14,16
F-OAL
71.05=023 57.13=0.19
91,962.0.9
6.24
2.04:nms
91.13=0015
3.19
+.03+aam
94.68=0.32
Fly-CI
19.07
5.38
93.89=0.12
4.43::.11
0.35
93.84-4.18
2.48
0.34:ama
96.54=0.35
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Datasets and Backbones: We conduct experiments using various architectures, including transformer-based and CNN-based backbones. Specifically, we utilize the Vision Transformer (ViT-B/16) (Dosovitskiy al., 2020) and ResNet-50 (He et al., 2016) as representative architectures. We test our method on five widely used datasets: CIFAR- 100 (Krizhevsky et al., 2009), CUB-200- 2011 (Wah et al., 2011). I), VTAB (Zhai et al., 2019), ImageNet-R (Hendrycks et al., 2021a), and ImageNet-A (Hendrycks et al., 2021b). Further details of the data setup are provided in Appendix E.
Baselines: We compare Fly-CL against eight baselines, including two prompt-based approaches:
L2P (Wang et al., 2022b) and DualPrompt (Wang et al., 2022a), three lora/adapter-based approaches: InfLoRA (Liang Li, 2024), SEMA (Wang et al., 2025a), and MoE-Adapter (Yu et al., 2024), as well as three eepresentationn-useed methods: EASE (Zhou al., 2024), RanPAC (McDonnell et al., 2023), and F-OAL (Zhuang et al., 2024). In Fly-CL, we find that applying data normalization according to the specific combination of backbone and dataset is beneficial; a detailed analysis be found in Appendix C.3. For fair comparison, all baselines use the same data normalization strategy as Fly-CL. Comparisons among the different baselines and implementation details are provided in Appendices D and E, respectively.
Evaluation Metrics:
To assess CL performance, employ four metries; average accuracy (Aa), last stage accuracy (Ar), backward transfer (BWT), and overall accuracy (A). The average accuracy at stage is defined as: A,, = 1 E/= or,i. where a,, denotes the test accuracy on the i-th task after refer the average accuracy at the last stage T as the last
training on the 1-th task. Specially, stage accuracy (Ar). The backward transfer denoted as BWT :- Ei= (at,i at,)). The
overall accuracy is computed as the mean of A, across all T tasks: A 4 A.. To evaluate computational efficiency, we introduce two time-related metrics: average training time per task (Train) and average post-extraction training time (Tpox). Here, Tuain represents the total training time amortized across all tasks, while Tpost is derived by subtracting the average feature extraction time for each task (using the pre-trained model) from Tuain- Tpost is a more precise metric for evaluating Aggorithm-specffe time consumption, as it excludes the shared preprocessing overhead.
5.2 Low LATENCY AND HIGH ACCURACY
The main CL results across various datasets, architectures, and task settings are summarized in Tables 2, and 5. Our framework's 's key strength is achieving CL accuracy comparable to or exceeding SOTA performance with significantly lower computational costs, as measured by both Turinn and Tpost- In Table 1, using ViT-B/16, Fly-CI. reduces Tpust by 91% on CIFAR-100 with only marginal accuracy drop of 0.32% compared to SOTA methods. On CUB-200-201 1 and VTAB, Fly-CL achieves 83% and 67% reductions in Tpost versus the most efficient baseline while improving overall accuracy by
Tuaan and Tpos: measured in seconds (wall clock time); A, and À measured in %.
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance Comparison on Pre-trained ResNet-50 Models. We report the average training time per task (Timin), average post-extraction training time (Tpea), and overall accuracy (4) across three benchmark datasets: CIFAR-100, CUB-200-2011, and VTAB. The best results are highlighted in bold.
CIFAR-100
CUB-200-2011
VTAB
Method
Tirain
[pau(()
A(t)
(pos(()
A(
Trmin
(por(
A(t)
RanPAC 55 68.
46.6510s5
82.72.0.22
58.74_084
54.35.0.99
78.72.1..0
50.15_0.36
47. 17.94
92. 10..00
F-OAL
80.74
71.781035
66.631071
5,19
1.69
60.84.1.67
2.7610.03
0.55
26.1512.0
F1y-CI.
a4.28:a.4 5.25
s4.61
3.90
0.44
80.25
2.53::0.10
0.34
94.00
Table 3: Performance Comparison Pre-trained ViT-B/16 Models using Online Learning Setting. ddenotes methods in online mode. We report the average training time per task (Train), average post-extraction training time (Tpus), and overall accuracy (A) across three benchmark datasets: CIFAR-100, LUB-200-2011, and VTAB. The best results are highlighted in bold.
CIFAR-100
CUB-200-2011
VTAB
Method
w
Tpaa(()
A(t)
Á(t)
Tran
Tpoal
A(t))
RanPAC
1236.74 1223.56:1:0
92. 12.4
242.54=156
238.36a141
91.89
122,89
120.69 93.41
F-OAL.
164.582071
151.27:064
91.48
31.34203:
27.20=a2s
91.60x0m 11.49aa1
9.47*016
28.28+021
FlyCCL
25.4610.
12.57
92.96.014
6.44
2.33
12.5
3.17.005
1.09
96.38
1.17% and 2.38% over the best-performing methods, respectively. In Table 2, with ResNet-50, Fly-CL improves overall accuracy by 1.89%, 1. 1.53%, and 1.20% on CIFAR- 100, CUB (UB-200-2011, 1, and VTAB, respectively, while reducing Tpost by 93%. 74%, and 38% versus the most efficient baselines. These improvements align with transformer- bbssed backbone trends. Notably, F-OAL exhibits significant performance degradation on CNN backbones, presumably due to error accumulation in iterative update mechanism, but Fly-CL does not suffer from this issue. These results highlight Fly-CL's ability to balance computational efficiency and accuracy across diverse CL scenarios, demonstrating its robustness. Results datasets with severe domain shifts are presented in Table 6.
Additionally, the time difference between Tirain and Tpoa in Tables 1 and indicates that feature extraction becomes the dominant time consumer in Fly-CL. For a fair comparison with the baselines, we do not apply additional acceleration techniques here. However, in practical applications, tech- niques like model quantization (e.g., INT8) can further reduce feature extraction time by around 4x without significant accuracy degradation, thereby enhancing the speedup ratio. For hardware-specific deployment, frameworks like TVM (Chen al., 2018) can be utilized maximize efficiency.
Furthermore, Fly-CL can be easily adapted to Online CL setups by updating the G and S matrices and solving Eq. 6 for each batch, without concatenating all batch embeddings within task. The results in Table 3 indicate that batch-mode Fly-CL remains superior to other baselines in training time and also competitive accuracy.
5.3 FACTORS CONTRIBUTING TO COMPUTATIONAL SPEEDUP
Our analysis in Sections 4.1 and 4.2 demonstrates that the proposed framework achieves significant speedup over the vanilla implementation through component-level optimizations. To quantify these improvements precisely, we split the post-extraction training time into three key components (as illustrated in Figure 2) and evaluate Fly-CI. against its vanilla implementation under the CUB-200- 2011 setting in Table 4. The components include: (1) Random Projection: Acceleration via weight sparsity induced by sparse projection versus the dense version. (2) Ridge Selection: Time reduction achieved by GCV, which eliminates the need for explicit cross cross-validates validation. (3) Prototype Calculation: Optimization from LU decomposition to Cholesky factorization. Additionally, the inference stage also benefits from the activation sparsity induced by the top-k operation in similarity comparisons.
5.4 ABLATION STUDY AND HYPERPARAMETER SENSITIVITY ANALYSIS
Using VIT-B/16 as the backbone, we conduct ablation studies by individually removing the projection layer (w/o proj). the streaming ridge classification (w/o ridge), and the data normalization components
8

---

Published as a conference paper at ICLR 2026
Table 4: Time Savings for Post-Extracting Components on CUB-200-2011. We compare the theo- retical time complexity per task (Tineory) ) and the actual runtime per task (T for each component on an NVIDIA GeForce RTX 3090 GPU. The optimized implementations demonstrate significant speedups across all components.
Random Projection
Ridge Selection
Prototype Calculation Similarity Comparison
Method
Tiwor
T actuaal
Tiwcrry
T Tactual
T Tiworry
Tactual
Timcory
T Tactall
vanilla
O (mn,d)
4.203
(lm)) 1.34
O m) O mm²)
0.20_o.m
(mm)ce)
0.21
optimized
O (mmpp)
0.08
(mm)))
0.14:0.01
0.10=0.01
(knie))
0.08
(w/o norm). The results in Figure demonstrate that each component contributes significantly to overall performance. Removing any of these components, or all of them (w/o all), leads to noticeable performance degradation.
We also analyze the sensitivity of the key hyperparameters in Fly-CL: (projection dimension), p (weight sparsity), and (activation sparsity) in Figure 5. Increasing ma improves accuracy, with performance saturating beyond m 10, 000. Notably. Fly-CL does not suffer from the curse of dimensionality, which can be attributed to the fact that random projection into a higher-dimensions space preserves pairwise distances between data points, guaranteed by the Lemma (Johnson et al., 1984). CL performance increases monotonically with p. and no significant performance drop occurs as long as p does take an excessively small value. A sufficiently large k value avoids information loss, while a smaller value suppresses noisy dimensions. Thus, finding an appropriate trade-off can lead to optimal accuracy. Encouragingly, Figure 5(c) shows broad plateau for optimal k selection. Based on our empirical results, we set m 10, 000, p = 300, and 3, 000 default values.
100
average accuracy (%6)
accuracy (%6
average accuracy
95
90
Rl-CL W60 pmi ridgege
Fy
Ry-CL
WEO (Fo]
pro
85
W/S ntgg
dge
nom
W/O
norn
20
50
100
30
number of classes
number of classes
number of classes
(a) CIFAR-100
(b) CUB-200-2011
VTAB
Figure 4: Accuracy Curves from Ablation Studies on Three Datasets. We report average accuracy (A,) for each stage. w/o refers the removal of the specific component.
96
Aa
92
accuracy (96)
02
80
84
90
5000
10000
15000
20000
200
400
600
2000
4000
6000
8000
10000
expand diimension
synaptic degree
(a) Expanded dim
(b) Sparsity level
(c) Coding level &
Figure 5: Sensitivity Analysis for Expanded dim m, Weight Sparsity p. and Activation Sparsity k on CUB-200-2011. We report average accuracy in last task (Az-) and overall accuracy (A). The dots denote the default values we use across experiments.

---

Published as a conference paper at ICLR 2026
6 DISCUSSION: CONSISTENCY BETWEEN RIDGE CLASSIFICATION AND HEBBIAN LEARNING RULES UNDER CONTINUAL LEARNING
In this section, we demonstrate that a biologically plausible local learning rule, constructed from Hebbian and anti- Hebbian plasticity (Hebb, 2005), converges to the same stationary point as ridge classification. This equivalence provides theoretical bridge between the synaptic dynamics in the KC->MBON pathway and the streaming ridge classification algorithm employed in Fly-CL.
We begin by defining synaptic update rule that depends only variables locally available each synapse. Let h' E IR'h denote the pre-synaptic activity vector and ÿ = C, h the post-synaptic response predicted by the current weights C, IRxx,, The synaptic update is defined as:
AC C.+1 C. (h'y h'ğ >C:),
(13)
where 7) > 0 is a small learning rate, y E R is the one-hot ground-truth label. This rule consists of three biologically meaningful components:
1. Hebbian term h'y", which strengthens synapses when both pre-synaptic activity and the target post-synaptic signal co-activate.
2. Anti-Hebbian term -h'ÿ which suppresses correlations between the input and the model's own prediction, implementing an error-correcting mechanism.
3. Weight decay -A corresponding metabolic cost homeostatic constraints.
This formulation biologically plausible: it requires only pre-synaptic activity, post-synaptic activity, and modulatory teaching signal, constituting standard three-factor learning rule commonly observed in neuromodulated plasticity.
Taking the expectation over data samples (h', y), we obtain:
E[AC] 7B((h'
y
E h'y XC,) = =n(S, ((S.--GC,-
(14)
where we define
G: = E h'h"
E IR mxm
S. = B==[hhyy R.xa.
Thus, the expected synaptic dynamics follow the linear recursion:
C.+1 C, + (S, (G, + AI C.))
(15)
Taking the contimuous-time limit r + 0 yields the ordinary differential equation:
dC, (S (G, + CI.)):),,
(16)
dt
where K > 0 rescales time. The stationary solution satisfies:
(G, + ) S
and therefore:
C (G, + A-m))-ss which is exactly the closed-form solution of ridge regression.
(17)
In our representation-hant hased CII. setting, since all samples in the same task are available simultaneously. we do not need integrate the continuous-time dynamics in Eq. 16. Instead, we can directly compute the ridge solution to obtain the optimal stationary classifier.
7 CONCLUSION
In this work, inspired by the decorrelation mechanism the fly olfactory circuit, we propose an efficient CL. framework, Fly-CL. Fly-CI. significantly reduces computational overhead during training while achieving competitive performance compared to SOTA methods. This framework integrates several key components: data normalization, feature extraction, sparse random projection with top-k operation, and streaming ridge classification, each contributing to the overall efficiency and effectiveness of the system. Ultimately, this work demonstrates that neurobiological principles, particularly sparse coding and progressive decorrelation, can effectively address the efficiency- accuracy trade-off in artificial continual learning systems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China ((YB2025XDXM503)
ETHICS STATEMENT
This research strictly complies with the ICLR Code Ethics. Our empirical evaluations are conducted exclusively on standard, publicly accessible benchmarks, entirely free of personal, sensitive, or human-derived information. All data usage respects the original licenses, and we foresee no negative societal impacts regarding privacy, security, algorithmic bias. Furthermore, by advancing efficient continual learning, Fly-CL contributes to sustainable and green Al development by significantly reducing the computational footprint and energy consumption typically required for retraining models from scratch.
REPRODUCIBILITY STATEMENT
The theoretical proofs for theorems are provided in Appendix B. Full details regarding the experimental setup. covering datasets, benchmarks, model configurations, evaluation metrics, and hyperparameters, are outlined in Section 5 and Appendix E. To ensure reproducibility, we rely exclusively on publicly accessible datasets and have made the complete source code available.
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European conference on computer vision (ECCV), pp.
2018.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXwv.2502.13933, 2025.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 482, 2002.
David A Belsley, Edwin Kuh, and Roy E Welsch. Regression diagnostics: Identifying influential data and sources of collinearity. John Wiley & Sons, 2005.
Chang Che, Ziqi Wang. Pengwan Yang, Qi Wang, Hui Ma, and Zenglin Shi. Lora in lora: Towards parameter-efficient architecture expansion for continual visual instruction tuning. arXiv preprint N.Niv::5008.66222. 2025.
Haoran Chen, Zuxuan Wu, Xintong Han, Menglin Jia, and Yu-Gang Jiang. Promptfusion: Decoupling stability and plasticity for continual learning. arXiv preprint arXiv:2303.0722 072222 2023.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 15616664416678, 2022.
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddic Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. Tvm: An automated end-to-end optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 578-55789 594,,994777 2018.
Stuart Coles, Joanna Bawa, Lesley Trenner, and Pat Dorazio. An introduction statistical modeling of extreme values, volume 208. Springer, 2001.
Sanjoy Dasgupta, Charles Stevens, and Saket Navlakha. A neural algorithm for a fundamental computing problem. Science, 358(6364):793-79 2017. doi: 10.112 100i:///eceeemmmm.eeaammm 126/science..amm9888.

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / /929, 2020.
Ronald Aylmer Fisher and Leonard Henry Caleb Tippett. Limiting forms of the frequency distribution of the largest or smallest member of sample. In Mathematical proceedings of the Cambridge philosophical society, volume 24, pp. 1800190. Cambridge University Press, 1928.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang A unified continual learning framework with general parameter-efficient tuning. In ICCV, pp. 11483-11493, October 2023.
Gene Golub, Michael Heath, and Grace Wahba. Generalized cross-validation as a method for choosing good ridge parameter. Technometrics, 21(2):215-22 1979.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Donald Olding Hebb. The organization of behavior: A neuropsychological theory. Psychology press, 2005.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -utto--disttibution generalization. In Proceedings of the IEEE/CVF international 2021a. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021b.
conference on computer vision, pp.
Toshihide Hige, Yoshinori Aso, Gerald M Rubin, and Glenn C Turner. Plasticity-driven individ- ualization of olfactory coding in mushroom body output neurons. Nature, 526(7572)258-262, 2015.
Arthur E Hoerl and Robert W Kennard, Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 1970.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
William B Johnson, Joram Lindenstrauss, et al. Extensions of lipschitz mappings into hilbert space. Contemporary mathematics, 26(18,-206):1, 1984.
Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 11847-11857, 2023.
Pentti Kanerva. Sparse distributed memory. MIT press, 1988.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Barnskkkaaai et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3524 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(1 (0(1)):2935-994) 2017.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35: 109-123, 2022.
12

---

Published as a conference paper at ICLR 2026
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2024.
Yuchen Liang, Chaitanya K Ryali, Benjamin Hoover, Leopold Grinberg, Saket Navlakha, Mo- hammed J Zaki, and Dmitry Krotov. Can fruit fly learn word embeddings? arXiv preprint arXiv:2101.6688 2021.
Andrew Lin, Alexei M Bygrave, Alix De Calignon, Tzumin Lee, and Gero Miesenböck. Sparse, decorrelated odor coding in the mushroom body enhances learned odor discrimination. Nature neuroscience, 17(4):559-505 -588, 2014.
Ashok Kuman, Kameron Decker Harris, Richard Axel, Haim Sompolinsky, and LF Abbott. Optimal degrees of synaptic connectivity. Neuron, (155:1153-1164, 2017.
Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning, pp. 23829-23851. PMLR, 2023a.
Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 40587-40609, 2023b.
Yixiu Mao, Qi Wang, Chen Chen, Yun Qu, and Xiangyang Ji. Offline reinforcement learning with ood state correction and ood action suppression. Advances in Neural Information Processing Systems, 37:93568-9368 93601, 2024a.
Yixiu Mao, Qi Wang. Yun Qu, Yuhang Jiang, and Xiangyang Ji. Doubly mild generalization for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:51436-51473, 2024b.
Yixiu Mao, Yun Qu, Qi Wang, and Xiangyang Ji. Adaptive neighborhhod-ccostrrinne q learning for offline reinforcement learning. arXiv preprint arXiv:25/1.02567, 2025.
Yixiu Mao, Yun Qu, Cheems Wang, Heming Zou, and Xiangyang Ji. Denamics-predicttee sampling for active RL finetuning of large reasoning models. In The Fourteenth International Confer- ence on Learning Representations, 2026. URL https: /wwp//pppneeee.eeeeeee net/ forum? id= voehe2jd8p.
sttts:://pperreviet
Mark McDonnell, Dong Gong, Amin Parvench, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. arXiv preprint arXiv:2307.02251, 2023.
Ahmed Metwally, Divyakant Agrawal, and Amr El Abbadi. An integrated efficient solution for computing frequent and top-k elements in data streams. ACM Transactions on Database Systems (TODS), - 2006.
Maria Papadopoulou, Stijn Cassenaer, Thomas Nowotny, and Gilles Laurent. Normalization for sparse encoding of odors by wide-field interneuron. Science, 332(6033):721-725 2011.
Ameya Prabhu, Shiven Sinha, Ponnurangam Kumaraguru, Philip HS Torr, Ozan Sener, and Puneet Dokania. Random representations outperform online continually learned representations. arXiv preprint arXiv:2402.08822, 2024.
Yun Qu, Boyuan Wang, Jianzhun Shao, Yuhang Jiang, Chen Chen, Zhenbin Ye, Liu Linc, Yang Feng, Lin Lai, Hongyang Qin, et al. Hokoff: Real game dataset from honor kings and its offline reinforcement learning benchmarks. Advances in Neural Information Processing Systems, 36: 22166-22190, 2023.
Yun Qu Boyuan Wang, Yuhang Jiang, Jianzhun Shao, Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Choices are more important than efforts: Llm enables efficient multi-agent exploration. arXiv preprint arXiv:2410.025// 2024.
13

---

Published as a conference paper at ICLR 2026
Yun Qu, Yuhang Jiang, Boyuan Wang, Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Latent reward: Lim-empowered credit assignment in episodic reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 20095-20103, 2025a.
Yun Qu, Cheems Wang, Yixiu Mao, Yiqin Lv. and Xiangyang Ji. Fast and robust: Task sampling with posterior and diversity synergies for adaptive decision-makers in randomized environments. In Forty-second International Conference Machine Learning, 2025b.
Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Björn Ommer, and Xiangyang Ji. Can prompt difficulty be online predicted for accelerating finetuning of reasoning models? arXiv preprint arXiv:2507.04632, 2025c.
Parikshit Ram and Kaushik Sinha. Federated nearest neighbor classification with colony of fruit- flies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8036-8044, 2022.
Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conserva- tive q learning for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36:77290- -7 2023.
Jaiyam Sharma and Saket Navlakha. Improving similarity search with high-dimensional locality- sensitive hashing. arXiv preprint 1812:v88/2.0/844 2018
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Bonianle-Bonilla.. Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-baan based prompting for rehearsal-free -ree continual learning. CVPR, pp. 11909- 11919, 2023.
Charles F Stevens. What the fly's nose tells the fly's brain. Proceedings of the National Academy of Sciences, 112(30):9400 9465)996996996,,,,-- 2015.
Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint (XXi:2309.00117. 2023.
Hai-Long Sun, Da- Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Ye. Mos: Model iessemnncemettll learning. Proceedings of the AAAI
surgery for pre-trained model- Conference on Artificial Intelligence, volume 39, pp. 20699-20707, 2025.
Yu-Ming Tang, Yi-Xing Peng, and Wei-Shi Zheng. When prompt- based incremental learning does not meet strong pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1706-1716, 2023.
Terence Tao and Van Vu. On random +1 matrices: singularity and determinant. Random Struct. Algor, 28(1):1-23, 2006.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Boyuan Wang. Yun Qu, Yuhang Jiang, Jianzhun Shao, Chang Liu, Wenming Yang, and Xiangyang Ji. Lim-empowered state representation for reinforcement learning. arXiv preprint arXiv:2407. 13237, 2024a.
Huiyi Wang, Haodong Lu, Lina Yao, and Dong Gong. Self-expansion of pre-trained models with mixture of adapters for continual learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10087-10098, 2025a.
Liyuan Wang. Jingyi Xie. Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt- based continual learning: Rethinking obscured sub-optimality. arXiv preprint arXiv:2310.07222 0.07234, 2023a.
Qi Wang, Zehao Xiao, Yixiu Mao, Yun Qu, Jiayi Shen, Yiqin Lv, and Xiangyang Ji. Model predictive task sampling for efficient and robust adaptation. arXiv preprint arXiv:2501.1 1039, 2025b.
14

---

Published as a conference paper at ICLR 2026
Yabin Wang, Zhiheng Ma, Zhiwu Huang, Yaowei Wang, Zhou Su, and Xiaopeng Hong. Isolation and impartial aggregation: A paradigm of incremental learning without interference. In AAAI, volume 37. pp. 10209-102177 2023b.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. arXiv preprint S2ti::220.047999 2022a.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang. Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pp. 139-149, 2022b.
Ziqi Wang, Chang Che, Qi Wang, Yangyang Li, Zenglin Shi, and Meng Wang. Separable mixture of low-rank adaptation for continual visual instruction tuning. arXiv preprint arXiv:2411.13949, 1, 2024b.
Ziqi Wang, Chang Che, Qi Wang, Hui Ma, Zenglin Shi, Cees GM Snock, and Meng Wang. Harmo- nious parameter adaptation in continual visual instruction tuning for safety-aligned milms. arXiv preprint arXiv:25/1.200 2.0118, 2025c.
Zehao Xiao, Shilin Yan, Jack Hong. Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiayi Shen, Qi Wang, and Cees GM Snoek. Dynaprompt: Dynamic test-time prompt tuning. arXiv preprint arXiv:2501 16404, 2025.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-expert adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219 23219-23230, 2024.
Yunliang Zang and Erik De Schutter. Recent data on the cerebellum require new models and theories. Current Opinion in Neurobiology, 2023.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995. PMLR, 2017.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1..00.446... 2019.
Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learn- ing with pre-trained models: Generalizability and adaptivity are all you need, arXiv preprint 3303::2003.0733, 2023a.
Da- Wei Zhou, Yuanhan Zhang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for lisiun-language models. arXiv preprint arXiv:2305. 2023b.
Da- Rei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based plass-incremental learning. In CVPR, 2024.
Huiping Zhuang, Yuchen Liu, Run He, Kai Tong, Ziqian Zeng, Cen Chen, Yi Wang, and Lap-Pui Chau. F-oal: Forward-only online analytic learning with fast training and low memory footprint in class incremental learning. Advances in Neural Information Processing Systems, 37:41517-411588 2024.
Heming Zou, Yixiu Mao, Yun Qu, Qi Wang, and Xiangyang Ji. Utility-diversity aware online batch selection for Ilm supervised fine-tuning, arXiv preprint arXiv:2510. 16888.2200...8., 2025a.
Heming Zou, Yunliang Zang, and Xiangyang Ji. Structural features of the fly olfactory eircuit mitigate the stability-plastiicn dilemma in continual learning. arXiv preprint arXiv:2502.014277 2025b.
Heming Zou, Yunliang Zang, Wutong Xu, Yao Zhu, and Xiangyang Ji. FlyloRA: Boosting task decoupling and parameter efficiency via implicit rank-wise mixture-of-experts. In The Thirty- ninth Annual Conference Neural Information Processing Systems, 2025c. URL https: //openreview. www.ew.weeeeeeee.eeee net//oruumiddnngnnna
15

---

Published as a conference paper at ICLR 2026
A ALGORITHM PSEUDOCODE
Algorithm 1 Fly-CI. Training Pipeline
Input: Sequentially arriving data D; { = where t I...., Pre-trained encoder Se. Projection operator Z()) IRd + R Penalty coefficient candidates - Ar). Zero-initialized matrices Go E Rmxx and So E R
Output: Modulated Prototypes C' pmxe
1: for t I....,T' do
2: r- y= min(n,, m)
3: Get compressed embedding for each datum of fo!)) IRd
4: Transform to high-dim sparse embedding ( = top-k (Wo,)
O(mnp)
5:
Concatenate ((ve) get H, xm
6:
G. < G H. H.
O(nmm²)
7:
S. t S + H, Y.
8:
Ur Σx, V, =//ddd(F
O(prm)
9:
for A E A do
10:
D: = E
E
O(rr)
df(A) = tr( tr(D:) = Ei &
11:
> O(Ir)
12:
Y, Uv(vecdiag D2) & 1 O
(lnere))
13:
GCV(A) ne(1 dr(a)
111--1(11))
> O(1mca)
14:
end for
15:
Select X; arg minxes GCV(A)
16:
L, = G. + X
b O
17:
C. Li (L;
> O(m²c))
18: end for
Algorithm 2 Fly-CL. Inference Pipeline
Input: Sequentially arriving data D, {(*), y k where t 1,..., T. Pre-trained encoder fa- Projection operator Z. Modulated class prototypes C₁ E Rmxe.
Output: Predicted labels ý.
1: Get compressed embedding for each datum e = fo( E Rd
2: Transform high-dim sparse embedding Z(v) = top-k (Wo)
O(m)) b O(ke))
3: Compute prediction y = Z(v) C₁
B COMPLETE THEORETICAL ANALYSIS
In this section, we present comprehensive theoretical analysis the sparsification effects on both the weights and activations in the random projection operation.
B. INFORMATION PRESERVING FOR SPARSE CONNECTIONS IN RANDOM PROJECTION MATRIX
A common approach to demonstrate that sparse random matrix multiplication preserves information cquivalently to its dense counterpart lies in proving the matrix's near-preservation of full column rank. For our sparse random matrix W Rm where m d, we prove that W almost surely maintains rank d.
Theorem B.1. Given the matrix W Rmxd where m d, with each row having exactly p non-zero entries, which are randomly sampled from from N(0, 1). Ler W E dddd be any square submatrix of W. Then, for any e >> 0, it holds that
(1det(W)) (2) 4/2 Vilexp(-d//2 = o())
16

---

Published as a conference paper at ICLR 2026
Thus, for sufficiently large p and d, any submatrix W is invertible with probability at least 1 o(1)
Proof. According to aformentioned definition, we have () = and Var(W.,; 2. Consider- ing R 1 2006, Theorem 8.9), that
a W. which satisfies (R5)))) = 0, Var Rij) = 1, we can conclude, based on (Tao & Vu,
PP ( (I)) 2 /dllexp(dd'
o1).
(18)
By using V and det(R) o o- det(WW) for substitution, we complete the proof.
To better demonstrate the information preserv- ing property of our construction, another line of validation is to utilize the random matrix's s distance preservation property. which is theoret- ically guaranteed by the Johnson-Lindenstrauss (JL) lemma (Johnson et al., 1984). Specifi- cally, we conduct an empirical simulation to verify that the normalization of our sparse pro-
Distribution of Pairwise Distance Distortion Ratios (d 768, 11000,000 p 300)
Ideal Ratio (1.0)
Peirs analyzed: 124/56
Bound (1
Ratio within (0.97. 1.0:] 96.87%
20
Dersity
jection matrix +
my W preserves pair-
wise Euclidean distances between feature vec- tors with high probability. For every pair of vee- tors (æ1, 222,, we calculate the Distortion Ratio
10-
[ We randomly defined as Ratio = || 122 15
0.94
0.38 Projected Squared Distance
1.02
1.04
select 500 data points from the extracted fea- tures of CIFAR-I00 dataset and compute the pairwise distances between all point pairs, with the results illustrated in Figure 6. By analyzing the distribution of these ratios across all vector pairs, the resulting histogram shows a strong concentration around the ideal value of 1. If we set to 0.03, then the vast majority of the ratios are empirically confined within the bounds of [1 1 + e]. This concentration strongly validates that our sparse projection structure-comprising only p non-zero (N 1) entries per row-effectively maintains the geometric structure of the high-dimensional data.
Figure 6: Empirical Verification of the Johnson- Lindenstrauss (JL) Property using Sparse Random Projection.
B.2 ROBUSTNESS OF T SPARSIFICATION ON HIGH- DIMENSIONAL EMBEDDINGS
Let the high- -igh-dimensooaal embedding vector be h E R After applying the top-k operation, we obtain a sparsified vector h' E IRm where only the largest absolute values h are retained, and the remaining elements are set to zero. We aim to prove that when k = Q(m²) (with 0 < < 1, i.e. not overly sparse), the performance degradation is negligible.
According to statistic learning theory (Bartlett & Mendelson, 2002) and extreme value theory (Coles et al., 2001; Fisher Tippett, 1928), we start with the following two widely- assumption: Assumption B.2. Assume that the "ennegyy" (squared & (o-norm) of the embedding vector x is concen- trated in a few dimensions, i.e., there exists constant C' > 0 such that:
Σ
h
C K
E
1=
> 1
h
where where h; denotes the i-th largest value in h.
Assumption B.3. Assume that the performance loss function L(h, y) of the downstream task (e.g., classifier) is Lipschitz continuous with respect to input perturbations, i.e., there exists a constant M > 0 such that:
L(h, v) L(h', .2)v < M | h'll.
Regarding the approximation error of top-k operation, we show it is bounded.
17

---

Published as a conference paper at ICLR 2026
Theorem B.4. Under the Assumption B.2, the sparsification envor satisfies:
E 1
E [
Proof. By Assumption B.2:
d
C
h
5
|1
Li=k+1
Thus,
E
|
h113] = EI
h
E
[111]]]
Then, we can quantify the upper bound of possible performance degradation.
Theorem B.5. Under Assumption B.3, the performance degradation due sparsification satisfies:
C V k
E [l v) L(h', y)ll S M
h
Proof. By the Seaphy--Shwwazzz and Theorem B.4, we can derive that
E [[L(h, L(h', v)1l M E
h h
< M
h
h 1
C
M
E [[AI]]
K
From Theorem B.5, we establish in Theorem B.6 that moderate sparsity does not result in significant performance degradation, the error decreases exponentially with the increasing expanded dimension m..
Theorem B.6. For top-k sparsification in the expanded dimension the performance degradation is bounded by:
V C 1(
() v) M
where L(.)) is a performance loss function for downstream lasks and C, M constants. To ensure negligible performance degradation, we require:
E
Vm
i.e., when & = ((m)) (0 < 1), the error bound decays exponentially with increasing dimension.
For example:
If k = ()....), the performance degradation Odd-0.25).
If l 8). the performance degradation is Odd-0.1).
18

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RESULTS
C.I EXPERIMENTS IN LONGER TASK SEQUENCES
To evaluate the long-term stability of FIy-CI., we conduct experiments with task sequences twice as long as those in Table 1. Results are presented in Table and Figure 7. Overall, both Turin and Tpost are shorter than those in Table due to fewer samples per task. Fly-CL improves overall accuracy by 0.54%, 1.21%, and 1. 1.58% compared to SOTA methods, while significantly reducing average post-extraction training time by 89%, 74%, and 59% compared to the most efficient baselines. These results are consistent with the trends observed in Table I and Figure 8, demonstrating the robustness of Fly-CL across different task lengths.
Table 5: Performance Comparison on Pre-trained ViT-B/16 Models with Longer Task Sequence. We report the average training time per task (Fuain), average post-extraction training time (Tpost), and overall accuracy (.A) across three benchmark datasets: CIFAR-100, CUB-200-2011, and VTAB. The best results are highlighted in bold.
CIFAR-100
CUB-200-2011
VTAB
Method
(()
TPoa(L)
A(t)
[Imin(
Tpus (L)
A(t)
Timina(4
Tpea(L)
A(t)
L2P
147.47=0036
107.59=o2s
82.69=031
30.93-a22
23.39
72.83+145
32.17=0.3
29.31
71.84:142
Dualprompt
130.12.1.09
91,051007
83.42..0.6
27.66.0018
20.2810.1
77.93.191
29.4410024
26.621..23
78.4611.
InfLoRA
204..9
84,3610.12
88. 08.14
27.12
19.85
75.67.00.6
29.71:0.1 26.92:0.17
81.09.0.7
SEMA
128.95:0.46
88.57=a.19
88. 0....
27.98
10,52=018
80.82+a.11
30,36=0.27
27.55
84.56#0.4
McE-Adapter
132.67...62
92.31
88 22.222...28
30.25 1019
22.82.1015
78.62.1005
29.2510.18
26.81.0.13
83.2410.5
EASE
350.9220:9 se
1316662000
90. 14
83.38
75.05
91.47 61.47:006
64.682007
57.24
90.26:2041
RanPAC
44
37.03
93. 2..22
20.681032
18. 18.011
92.65.022
41.49.0.34
40.02:0..9
93.6210.3
F-OAL
16.32_002
8.91
92.6310.46
1.65
1.101000
91.481023
2.17
0.71
94.9610.27
Fly-CI.
8.31:044
1.00
94.22:000
2.76::0.6
0.29
93.861027
1.70=a.00
0.29::002
96.54:0.w
100
average accuracy (%)
average accuracy 1%6)
average accuracy
90-
80
80
70
Duaffrrmgt
73
80
BASE
5
15 25
35
45
55
65
75 85 95
10
30 50 70 90 110 1700018
10
15
20
30
25 40 45 50
number of clesses
number of classes
number of classes
(a) C00
(b) CUB-200-2011
(c) VTAB
Figure 7: Accuracy Curves of Different Methods on Pre-trained ViT-B/16 with Longer Task Sequence. The average accuracy (A,) is reported for each dataset. These results align with and extend the quantitative analysis presented in Table 5.
C.2 EXPERIMENTS ON DATASETS WITH SEVERE DOMAIN SHIFT
Table 6 summarizes the results on ImageNet-R R and ImageNet-A under severe domain shift. Compared with existing continual learning baselines, Fly-CL achieves the best overall accuracy on both datasets 983.19% on ImageNet-R and 67.98% on ImageNet-A), comparable with the previous SOTA RanPAC. More importantly, Fly-CL attains these improvements with substantially lower computation cost. Its average training time per task is reduced by order of magnitude compared to prompt-based methods (e.g.. L2P, DualPrompt) and much faster than EASE, while its post-extraction training time is almost negligible (0.21s vs. 67.71s for RanPAC on ImageNet-R) R). These results demonstrate that Fly-CL is not only robust to severe distribution shifts but also highly efficient, making it especially suitable for practical continual learning scenarios where both accuracy and efficiency are critical.
19

---

Published as a conference paper at ICLR 2026
Table Performance Comparison on Pre-trained ViT-B/16 Models with Severe Domain Shift. We report the average training time task (Fimin), average post-extraction training time (Tpos), and overall accuracy (A) across two benchmark datasets: ImageNet-R and ImageNet-A. The best results highlighted in bold.
ImageNet-R R
ImageNet-A
Method
(4)
Tpoo(()
A(1)
Timain
Tpos(())
A(t)
L2P
131.97.0.r
110.5610A4
76. 10.21
56.2810.3
48. 92.1272
48.86.10.6
Dualprompt
=17.80
96.58+0.3
73.92+0046
49.60=a.x
42.31
57.05
onfL.RR
32.322003
41.05+0.23
82.15
25.71 =0:00
18.50.0044
62.32220.28
SEMA
68.85=021 47.34+0.19
81.89+u.17
28.65=022
21.23=0022
61.79
MoB-Adapter
66.37_0029
45.02.1.24
81.76.10.33
26.27:1041
18.89.0.17
61.72.1021
EASE
31 =12200=0.99 272
81.69
80.80=a.19
73.47=002
65.03=+0.28
RanPAC
76.25
40.21
83.02.a..12
32.43=0.13
28.8611.11
67. 67.28: +0009
F-OAL
16.51=0.11
8.80=a04
80.62+t0.25
3.99
1.05
+3.99
Fly-CI
7.55
0.21 =
83.19+..14
3.10+nas
0.15+0.01
67.98=0.17
C.3 DATA NORMALIZATION STRATEGY
While data normalization a well-established technique for improving classification performance in i.i.d. scenarios, its effectiveness in facilitating CL with frozen pre-trained encoders remains unclear. Our results indicate that applying proper architectue--specffic normalization to input images significantly improves the learning performance compared to baseline CL methods (Table 7). The optimal normalization strategies for the included backbones differ. Across all tested datasets, ViT- B/16 (Dosovitskiy al., 2020) benefits more from standard normalization that projects inputs into the [-1, 1] range, while ResNet-50 (He et al., 2016) achieves optimal performance when normalized using ImageNet statistics.
We hypothesize that the imporved performance arises from a reduced feature distribution shift across tasks. Proper normalization preserves the geometry of the pre-trained feature manifold, which is crucial for prototype-based classification, where cosine similarity measures depend on the angular relationships between features. Our empirical results suggest that input normalization may serve as fundamental defense against forgetting by anchoring the feature space topology to the original pre-training distribution.
Table 7: Comparison of CL Performance across Pre-trained Models and Normalization Strate- gies. We report overall accuracy (A). Normalization methods includes: "None" (no data normaliza- tion), "ImageNet" (ImageNet statistics), and caandard""scallle" to the [)).
CIFAR
CUB
VTAB
Backhone
None
ImageNet
Standard
None
ImageNet
Standard
None
ImageNet
Standard
VIT-B/16 |9.66420o2 87.87 93.89=aa2 93.0420.7
90.68+n42 23.84 25.26
95.47+0.22 96.54=a.w
ResNet-50
80.6640.45 84.61 83.09
80.25 75.08
1.07 92.45 94.00 92.76=054
C.4 MEMORY CONSUMPTION
We also compare the memory consumption of Fly-CL against other methods in Table using ViT- B/16 with the same task sequence as Table 1. For fairness, we use batch size of 128 across all methods and datasets. The results show Fly-CL also has the minimal memory cost, strengthening the efficiency of our method.
C.5 MEMORY-TIME TRADEOFF IN IIGI-DIMM PROJECTIONS
We present the trade-off between memory, training time, and overall accuracy in Table 9. The overall accuracy gradually saturates as the dimension increases, while memory and training time grow quadratically. Therefore, chose 10,000 as the dimension in simulations. As long the
20

---

Published as a conference paper at ICLR 2026
Table Memory Usage (GB) Different Methods on Pre-trained ViT-B/16. We report highest peak memory usage of each methods. The best results are highlighted in bold.
Method
CIFAR-100
CUB-200-2011
VTAB
L2P
16.4GB
16,4GB
16.4GB
DualPrompt
13.6GB
13.6GB
13.6GB
InfLoRA
14.1GB
14.1GB
14.1GB
SEMA
12.9GB
12.9GB
12.9GB
MoE-Adapter
20.2GB
20.2GB
20.2GB
EASE
12.2GB
12.2GB
12.2GB
RanPAC
12.2GB
12.2GB
22.8GB
F-OAL
12.2GB
4.9GB
4.5GB
FLy-CL
6.7GB
4.6GB
4.3GB
dimension does not exceed 10,000, both memory and training time consumption remain lower than those previous methods, as summarized in Table
Table 9: Seenory-ime-Accuraee comparison with increasing projection dimension. It's con- ducted the CUB dataset using ViT B/16. We report highest peak memory usage of each methods. The best results are highlighted in bold.
Dimension
1000
2000
5000
10000
20000
Memory
2.8G
2.8G
3.0G
4.6G
9.3G
Turin
4.1940.02
4.20=
4.25
4.43
5.13
90.87:10.9
91.97
92.931041
93.84
93.90
C.6 ADDITIONAL VISUALIZED FIGURES AND EVALUATION METRIC DURING THE TRAINING PROCESS
Here, we present more detailed breakdown of the training processes for ViT-B/16 and ResNet-50. Results from Tables and are visualized in Figures 8 and 9. We list the average accuracy of different methods at different stages across three datasets. We additionally report the last-stage accuracy (Ar)
average accuracy (96)
accuracy (160
95
172
L2P
average
Due Promit
Dunffromp:
EASE
EASE
RarRAC
RanIKC
RanfAC FOOLL
F-OAL
FOL
Dualfromat FASE
FyC
Fly
60
60
number of classes
number of classes
number of classes
(a) CIFAR-100
(b) CUB-200-2011
VTAB
Figure 8: Accuracy Curves of Different Methods on Pre-trained ViT-B/16. The average accuracy (A,) is reported for each dataset. These results align with and extend the quantitative analysis presented in Table 1.
and the backward transfer score (a representative forgetting metric) of Table in Table 10 to provide more comprehensive evaluation.
C.7 ADDTIIONLL SENSITIVITY ANALYSIS ACROSS TASK COMPLEXITY, NUMBER OF TASSS/CLASSES, AND DIFFERENT PRETRAINED BACKBONES
We further conduct additional sensitivity analyses under three complementary settings: (i) ImageNet-
with 10 tasks and 20 classes per task using V examine the effect of task complexity
21

---

Published as a conference paper at ICLR 2026
100
RanHAC F-04L
RanAAC F-OML
averege accuracy (%6
accuracy (%)
90
Cly
80
average accuracy
F.04L
50
number classes
number of classes
number classes
(a) CIFAR-100
(b) CUB-200-2011
(c) VTAB
Figure 9: Accuracy Curves of Different Methods on Pre-trained ResNet-55. The average accuracy (A₂) is reported for each dataset. These results align with and extend the quantitative analysis presented in Table 2.
Table 10: Performance Comparison on Pre-trained ViT-B/16 Models. We report the last stage accuracy (Ar), and backward transfer (BWT) across three benchmark datasets: CIFAR-100, CUB- 200-2011, and VTAB.
CIFAR-100
CUB-200-2011
VTAB
Method
Az(t)
BWT(1)
Ay(t)
BWT(1)
(x(1)
BWT(()
L2P
83.55
-6.23
65.41=184
-13.14
79.12-2.18
-8.04
DualPrompl
81.45
-7.06.1020
67
17.90
-12.12.0.5
79.83
-7.63-1075
InfLoRA
86.5610.46 -5.07.0.16
15,56
-11,531025
87.88_o.7
-4.591024
SEMA
87.47=0.4z
-4.70=0.13
73.66
-9.94.0.99
89.28
-4.06
MoE-Adapper
86.88
1.997.015
68.
1:04:
-12.111028
88.06=044
-4.511019
EASE
87.63=2.20 -4.66=006
10=019
-5.68-0.0
93.30_omr
-2.48+004
RanPAC
90.83 -3.47
88.95
-4.1620.2
89.97-071
-3.791006
F-OAL
87,04 -4.9010.14
85.61 10.50
-5.43.0.44
92.91
-2.661004
Fly-CT.
19.85
-3.822209
89.97=0.18
-3.80=0..
94.61
0.35
-2.01
(Figure 10); (ii) CUB 200-200-2011 with 20 tasks and 10 classes per task using ViT-B/16 to evaluate the impact of larger number of tasks and classes (Figure 11); and (iii) CUB-200-2011 with 10 tasks and 20 classes per task using ResNer-50 to assess the influence of different backbone architectures (Figure 12). Across all settings, the observed trends remain consistent with those reported Figure 5,
65
accuracy (%)
accuracy 1%6)
96
80
60
Ay
42
50
10000
20000
200
expand dimension
synaptic degree
(a) Expanded dim m
(b) Sparsity level p
(c) Coding level
Figure 10: Sensitivity Analysis for Expanded dim Weight Sparsity and Activation Sparsity on ImageNet-A. We report average accuracy in last task (Ar) and overall accuracy (A). The dots denote the default values we use across experiments.
C.8 ADDITIONAL EXPERIMENTS ON LARGER PRE-TRAINED MODELS
In our previous experiments, we primarily adopted Re and ViT-B/16 as backbones for feature extraction. To further evaluate the scalability of Fly-CL on modern foundation models, we employ the vision encoder of Qwen2.5- VL-7B (Bai et al., 2025) to extract visual features. As shown in Table 11, increasing the scale of the pre-trained backbone consistently leads improved performance.
22

---

Published as a conference paper at ICLR 2026
Ar
94
941
accurocy (%)
88
02
accuracy
80
88
86
5000
10000
15000
20000
200
2000
4000
6000
8000
10000
diimension
synaptic degren
(a) Expanded dim m
(b) Sparsity level p
(c) Coding level
Figure 11: Sensitivity Analysis for Expanded dim mi, Weight Sparsity and Activation Sparsity on CUB-200 with Longer Task Sequence. We report average accuracy in last task (A1) ) and overall accuracy (A). The dots denote the default values across experiments.
accurecy (%)
accuracy
accuracy
08
72
60
5000
10000
15000
20000
400
800
1200
3600
2000
2000
4000
6000
8000
10000
expand
degree
(a) Expanded dim m
(b) Sparsity level p
(c) Coding level
Figure 12: Sensitivity Analysis for Expanded dim m, Weight Sparsity p, and Activation Sparsity on CUB-200 with ResNet-50 as Backbone, We report average accuracy in last task (Ar) and overall accuracy (A). The dots denote the default values we use across experiments.
Furthermore, in Figure 13, we analyze the effect of expanding the projection dimension when using Qwen2.5-VEL The trend remains consistent with our earlier findings: performance improves steadily as the expanded dimension grows, but begins to saturate around 10, 000.
accuracy (%))
Table 11: Performance Comparison Different Scale Pre-
trained Models. We report overall accuracy (A) across three
benchmark datasets: CIFAR-100, CUB-200-2011, 1, and VTABB
Model
CIFAR-100
CUB-200-2011
VTAB
ResNet-50
84.6120.16
80.25=0.10
94.00
5000 10000
15000
20000
ViT-B/16
93.89=0.1
93.84=0.18
20.54
expand dimension
Qwen2.5-VL VL-7B
95.06=023
94.68=0211
97.45
Figure 13: Sensitivity Analysis for Expanded dim m on CUB- 200 with Qwen2.5-VL-77 as Backbone.
D DETAILED DISCUSSION OF RELATED WORK
D.1 COMPARISON WITH SEVERAL REPRESENTATION BASED METHODS
We highlight the main advantages of our proposed Fly-CL over several related representation-baae based methods, including RanPAC (McDonnell et al., 2023), F-OAL (Zhuang et al., 2024), and RanDumb (Prabhu al., 2024).
23

---

Published as a conference paper at ICLR 2026
Comparison with RanPAC. RanPAC employs several Parameter-Efficee Efficceett Transfer Learning (PETL) approach (Chen et al., 2022; Jia et al., 2022; Lian et al., 2022) to adapt the pre-trained model to the downstream domain in the first task, alongside ridge classification with explicit cross- validation for all ridge candidates. Although effective, these two components make the entire pipeline computationally expensive (see Table 2, and 5). In contrast, Fly-CI. eliminates the need for PETL and significantly optimizes the ridge classification process. Additionally, we introduce a sparse projection layer with a top-k operation, replacing the dense projection with ReLU, and analyze the impact of data normalization techniques. The speedup for each components can be refered to Table 4.
Comparison with F-OAL. F-OAL is originally designed for online CL and shares similarities with Fly-CL in feature extraction, random projection, and decorrelation. Although it can also be adapted to the CIL. setting with batched data, has several flaws under this circumstance. For instance, F-OAL lacks the top-k operation to filter noisy components after random projection, and its iterative analytic classifier may accumulate errors, leading to significant performance degradation on ResNet-50 (see Table 2). Moreover, while F-OAL is efficient on CUB-200-2011 and VTAB, its computational cost scales more rapidly with sample size compared to Fly-CL, making it less efficient on CIFAR-100 (see Table 2, and 5).
Comparison with RanDumb. RanDumb shares a similar pipeline with F-OAL and is also designed for online CL. Like F-OAL, it does not utilize top-k-like operation, and its fixed penalty coefficient may result in suboptimal performance. Crucially, RanDumb relies on StreamingI.DA, which processes samples sequentially and cannot be parallelized for batch processing. This makes RanDumb significantly slower than all baselines evaluated in Table I, 2, and 5.
D.2 SUMMARY OF OTHER COMPARED BASELINES
L2P (Wang et al., 2022b) utilizes a prompt pool P = P2..P2.... P2...... }Mr where M is the size of the pool, to store task-specific knowledge. Each prompt P; is associated with a learnable key K; for key-value selection. By optimizing the cosine distance (p(xx), kk), where p(æ) is the feature selected by the query function during the training process, I.2P can select the most appropriate prompt to provide information that is specific to the task.
(Wang et al., 2022a) extends the key-value selection and optimization methods of L2P by further encoding different types of information into a task-invariant prompt y and task-specific prompt e. This is shown to be more effective in encoding the learned knowledge. It also decouples the higher-level prompt space by attaching prompts to different layers, which is crucial for the model to reduce forgetting and achieve effective knowledge sharing.
DualPrompt
EASE (Zhou et al., 2024) first initializes and trains an adapter for each incoming task to encode task-specific information. It then extracts features of the current task and synthesizes prototypes of former classes mitigate the subspace gaps between adapters. Finally, EASE constructs the full classifier and reweights the logits for prediction.
InfLRRA (Liang & Li, 2024) is Interference- Low-Rank Adaptation method for continual learning. It injects a small set of parameters to constrain weight updates to specific subspace. Critically, this subspace is designed to be orthogonal the gradients of all past tasks while containing the gradient subspace of the new task, thereby eliminating interference and achieving an effectivee balance between model stability and plasticity.
SEMA (Wang et al., 2025a) introduces a self- eef-exxansion mechanism by dynamically adding modular adapters only when significant distribution shifts are detected. Each adapter consists of a functional module and representation descriptor, which acts as a novelty detector to determine whether existing adapters can handle the new task. SEMA further maintains an expandable mixture router to compose adapters through weighted combination, enabling flexible reuse of old modules while expanding only on demand. This design achieves a sub-linear parameter growth rate while improving the stability-plastice balance across tasks.
MoE-Adapter (Yu et al., 2024) enhances continual learning by injecting a mixture-of -ffeeeer-- adapter structure into transformer blocks. Each MoE-Adapte contains multiple expert adapters, and learned routing network selects or mixes experts conditioned on the input. This architecture allows the model to capture diverse task-specific patterns while mitigating forgetting through expert specialization. By
24

---

Published as a conference paper at ICLR 2026
leveraging the MoE structure, the method improves representational flexibility and achieves stronger adaptation capacity compared to using single shared adapter.
D.3 RELATIONSHIIP WITH KANERVA'S SPARSE DISTRIBUTED MEMORY
Kanerva's Sparse Distributed Memory (SDM) (Kanerva, 1988) is classical high-dimensional computing framework in which memory addresses are distributed in a large binary space, and read/write operations are performed by activating only those locations within neighborhood defined by Hamming similarity. To highlight its conceptual connection to Fly-CL, we rewrite the SDM forward computation using the same notation as Fly-CL:
C (fWw)),
(19)
where Wi is an address transformation matrix, C is a content transformation matrix, and f(-) denotes a binary activation function.
Under this formulation, SDM and Fly-CL share two high-level principles: (i) expansion into high- dimensional representational space, which promotes separability and reduces interference; and (ii) sparse activation, which suppresses noises and enhances discrimination through selective addressing.
Despite these conceptual parallels, Fly-CL differs from SDM in several important ways. First, W in SDM is updated through iterative read/write operations, whereas Fly-CL employs a fixed, randomly initialized projection matrix motivated by the fly olfactory circuit. Second, SDM does not incorporate any decorrelation mechanism analogous to the Fly-CL ridge- based KC-to-MBON transformation. Third, SDM operates in binary space: the activation function f(-) produces binary addresses and similarity is evaluated via Hamming distance, while Fly-CL performs real-valued projections and cosine similarity for downstream matching and classification.
These distinctions illustrate that although Fly-CL and SDM share the broad philosophy of high- dimensional sparse representations, their architectural assumptions, objectives, and operating regimes differ substantially.
E TRAINING DETAILS
E. PRE-TRAINED MODELS
We pre-traine trained ViT-B/16 and ResNet-50 models in experiments. The ViT-B/16 checkpoint we used was first pretrained on ImageNet-21I: and then fine-tuned on the ImageNet- IK dataset. All of which loaded using the timm library. We list the dimensions of the extracted features and the download links for the checkpoints of each model in Table 12.
Table 12: Information Related to the Pre-trained Models We Used in This Work. We list the dimensions of the extracted features and provide corresponding download links for these pre-trained models.
Model
feature dimension
Link
ViT-B/16
768
Link
ResNet-50
2048
Link
E.2 DATASETS
We evaluate our method on three benchmark datasets for CL tasks. Detailed information about these datasets, including download links, is provided in Table 13. For the experiments summarized in Tables for CIFAR- 100 and CUB-200-200
1, 2, and 3, we configure the number of training tasks as T with 10 and 20 classes per task, respectively. For VTAB, we set T with 10 classes per task. In the longer task sequence experiments (Table 5), we double the task sequence length: for CIFAR-100 and CUB-200-200 I, we set T 20 with 5 and 10 classes per task, respectively, while for VTAB, set T 10 with 5 classes per task. For experiments in Table 6, we set T 10 with 20 classes per task.
25

---

Published as a conference paper at ICLR 2026
Table 13:
Details of CIFAR-100, CUB-200-2011, VTAB Datasets. We list the number of training, validation samples and classes for the following datasets, along with the download links.
Dataset
Training Samples
Validation Samples
Classes
Download Link
CIFAR-100 (Krizhevsky et al., 2009)
50000
10000
100
Link
CUB-200-2011 (Wah et al., 2011)
94430
2358
200
Link
VTAB (Zhai et al., 2019)
1796
8619
50
Link
Imagenet-R Hendrycks et al. (2021a)
24000
6000
200
Link
Imagenet-A Hendrycks et al, (2021h)
5981
1519
200
Tink
E.3 EXPERIMENT SETUP
We reproduce the baseline results for I.2P, DualPrompt, EASE, and RanPAC using the code provided by PILOT (Sun et al., 2023), ensuring that the learning parameters for each baseline align with the description in their original papers. For F-OAL, we adopt their official implementation for reproduction.
In our proposed Fly-CL, we set the expanded dimension m 10, 000, p to 300, and k to 3, 000 across all experiments. For ViT-B/16, we apply standard data normalization, scaling each pixel value to the range [-1, 1]. For ResNet-50, we normalize the input images using ImageNet statistics. Given the prior knowledge of bigh multicollinearity in this task, we explore the penalty coefficient range starting from larger values, specifically from 10 to 10 on a log scale for ViT-B/16 and 10 to 10 for ResNet- 50. Since prompt-based methods and PETL techniques are limited to transformer based architectures, we compare Fly-CL only with RanPAC and F-OAL in the ResNet-50 setting. For RanPAC, we remove PETL and incorporate data normalization following their original implementation (McDonnell et al., 2023) for ResNet-50. All experiments are conducted using five different random seeds, and we report the mean + standard deviation.
E.4 ENVIRONMENTS
All experiments were conducted on a Linux server running Ubuntu 20.04.4 LTS, equipped with an Intel(R) Xeon(R) Platinum 8358P CPU 2.60GHz and 8 NVIDIA GeForce RTX 3090 GPUs, using CUDA version 11.7. For model loading, we employed the timm library (version 0.9.16).
F LIMITATIONS AND FUTURE WORK
Our proposed Fly- Fly-CL is theoretically applicable various scenarios requiring feature separation. Its lightweight design further suggests potential utility in a wide range of Continual Learning and Metric Learning tasks. Looking forward, the framework can be broadened in several promising directions. First, incorporating large language model (L.LM) priors (Qu et al., 2025a; Wang et al., 2024a; Qu et al., 2024) could enrich the initial state representations, providing more structured semantic knowledge to guide the decorrelation process, Second, our streaming feature separation mechanism naturally extends to deep reinforcement learning (RL) and RL post-training. By stabilizing representations, can help mitigate out-of- disttribbutinn (OOD) shifts and improve generalization in offline RL (Mao al., 2024a;b; 2023a;b; 2025; Shao al., 2023; Qu et al., 2023), as well accelerate the online RL fine-tuning of reasoning models (Qu et al., 2025e; Mao et al., 2026). Finally, combining Fly-CL with intelligent data sampling strategies (Wang al., 2025b; Qu al., 2025b; Zou et al., 2025a) could synergistically enhance training efficiency and adaptation robustness in complex, non-stationary environments.
Recent neuroscience research (Dasgupta et al., 2017) indicates that the random projection layer in the fly olfactory circuit may not be entirely random. Biological experiments also suggest the presence of certain constraints within this projection layer. Inspired by these findings, a promising direction for future research is to explore structuring the projection layer as an entity with learnable parameters, potentially enhancing its adaptability and performance.
26

---

Published as a conference paper at ICLR 2026
G BROADER IMPACT
Our work provides new perspective for enhancing the efficiency of CL using pre-trained models, which is crucial for real-world deployment, especially with increasingly large modern models. Fly-CL can help AI researchers and developers create more efficient CL algorithms.
On the other hand, the efficiency improvements in CL could potentially accelerate the development of Al systems that rapidly adapt to new domains without proper safeguards. This might lead to: (1) amplified propagation of biases present in sequential datasets, (2) reduced transparency as models continuously evolve beyond their initial training, and (3) potential misuse for generating tailored content scale. We recommend implementing rigorous monitoring frameworks to track model behavior across learning phases.
H LLM USAGE DECLARATION
During the preparation of this manuscript, a large language model was employed exclusively for language refinement. role was limited to rephrasing certain passages and enhancing the overall clarity and readability of the text. All conceptual contributions, theoretical derivations, experimental design, and analysis were independently developed and verified by the authors. The LLM was not involved in generating research ideas, shaping methodologies, producing novel scientific content. The authors bear full responsibility for the entirety of the paper.
27


Reviews:
[{"review_id": "rj6wAOmpin", "reviewer": "Reviewer_4tXN", "summary": "Taking inspiration from the fly's olfactory system, the authors introduce FLY-CL, a way to extract and use features of a pretrained model for class incremental learning.\n\nThe features of a pretrained model are expanded with a fixed random projection, in a similar way to the a layer from \"projection neurons\" to \"Keynyon cells\" in the olfactory circuit of a fly. A top-k activation simulates lateral inhibition. A learned similarity matching down projection emulates the projection to \"mushroom body output neurons\". \n\nThey show how this method reduces catastrophic forgetting when using pretrained models with a set of image benchmarks.", "questions": "you show the performance saturates beyond 10k. Do you expect that saturation point to remain stable as the number of tasks/classes grows much larger?\n\nHave you run experiments that study how the required projection dimensionality scales with increased task complexity or number of tasks/classes? Or if it changes if you use a different pretrained backbone?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The high-dimensional sparse layer does seem to effectively prevent catastrophic forgetting. \n\nNo task identity is required at inference. \n\nworks with unmodified pre-trained models \n\nablation studies do show the need for each part of the model (random projection / ridge regression / normalisation)\n\nFigure 5 shows how important the high dimension layer is, with performance saturating at m>10k. This is crucial since I believe this is the main novel contribution.", "weaknesses": "There can be high memory costs associated with the large sparse layer (the authors do discuss this).\n\nThey do need the model to receive task boundaries and to store \"class prototypes\" for use during inference. \n\nNo adaptation to the backbone so it is dependent on a good pretrained model (a weakness or a strength depending on the circumstance).  \n\nThe paper would benefit from a more detailed analysis of how the key hyperparameters (m, p, and k) scale with task complexity, the number of tasks/classes, and different pretrained backbones.\n\nminor typo:\n037 \"generalization in downstream tasks for downstream tasks\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WcUVZH8q0F", "reviewer": "Reviewer_gFjn", "summary": "This paper proposes Fly-CL, a continuous learning framework inspired by the Drosophila olfactory circuit, designed to address multicollinearity issues in representation learning based on pre-trained models while reducing training time. Fly-CL achieves feature decoupling and efficient classification through mechanisms including sparse random projection, Top-k activation filtering, and streaming ridge classification. Experiments demonstrate that this method achieves or surpasses state-of-the-art performance across multiple datasets and backbone networks while significantly reducing training time, exhibiting particularly strong advantages during post-feature extraction processing stages.", "questions": "1. Could the author establish a simple baseline by using only Fly-CL's projection and Top-k layers, followed by a straightforward linear classifier or k-nearest neighbors classifier, to demonstrate the necessity of streaming ridge regression in your problem setting?\n\n2. Although Fly-CL claims to mitigate forgetting, how does it affect old tasks at the feature space level? When learning new tasks, do the class prototypes of old tasks drift or distort in the high-dimensional space after Fly-CL processing?\n\n3. Theorem B.1 aims to prove that sparse projection matrices W are almost certainly full-rank, which is considered an argument for their ability to preserve information. However, in machine learning, the fact that a random matrix is full-rank does not directly equate to it being a “good” feature mapper. More importantly, the matrix's isometric property or distance-preserving characteristic is guaranteed by the Johnson-Lindenstrauss (JL) lemma. Can the authors provide evidence that your sparse matrix W indeed preserves pairwise distances between feature vectors with high probability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the issue of “persistent feature decoupling in pre-trained models.” While the constituent components (random projection, Top-k, ridge regression) are established techniques, their creative combination and application constitute the contribution.\n    \n2. The paper provides a solid theoretical foundation, demonstrating the information retention capability of sparse projections. The experimental design encompasses multiple architectures (ViT, ResNet), datasets, and evaluation metrics.\n    \n3. This work provides a solution for efficient continuous learning in resource-constrained scenarios.", "weaknesses": "1. I believe the core mechanism of the paper—“random projection + Top-k sparse activation”—shares striking similarities with the fundamental concept of Kanerva's Sparse Distributed Memory (SDM). SDM is similarly inspired by neuroscience and employs high-dimensional sparse representations and similarity matching to address memory and learning challenges. The paper omits discussion with this classic approach.\n\n2. Despite significant efficiency gains, projecting dimensions of m=10,000 may still impose memory constraints on extreme edge devices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mQH5zkGvaT", "reviewer": "Reviewer_uEtz", "summary": "This paper presents Fly-CL, an efficient framework for continual learning with frozen pre-trained encoders. It introduces two key components: (1) a sparse random projection with top-k activation sparsification to decorrelate features and improve prototype separability, and (2) a streaming ridge regression classifier with adaptive regularization via generalized cross-validation for stability and low computational cost. Experiments across ViT-B/16 and ResNet-50 backbones show that Fly-CL achieves comparable or higher accuracy than prior representation-based methods (e.g., RanPAC, F-OAL) while reducing post-extraction training time.", "questions": "see weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is clear motivated and formulated, easy-to-follow\n2. High efficiency with strong accuracy trade-off\n3. Simple and generalizable design, broadly applicable to backbones and datasets.", "weaknesses": "1. The proposed framework shares conceptual similarities with earlier representation-based approaches such as RanPAC and F-OAL, both of which employ random projections and analytic updates. While Fly-CL introduces additional sparsification and adaptive regularization, the methodological advancement over these predecessors appears incremental rather than fundamentally novel.\n\n2. This paper does not empirically demonstrate the effect of reduced prototype correlation.\n3. The experiments primarily report average accuracy across tasks, but omit standard CL metrics such as the final-task accuracy ($\\mathbf{A}_T$) and forgetting measure. \n4. The study focuses on representation-based and prompt-based baselines but do not compared with recent lora/adapter-efficient continual tuning methods such as InfLoRA [A], SEMA [B], and MoE-Adapters [C].\n5. Discuss why Fly-CL slightly underperforms RanPAC on CIFAR-100 despite achieving substantial gains on other datasets. \n6. It is not specified whether the ViT-B/16 model is initialized from ImageNet-21K or ImageNet-1K pre-trained weights. \n\n[A] Liang, Y. S., & Li, W. J. Inflora: Interference-free low-rank adaptation for continual learning. CVPR2024.\n\n[B] Wang, H., et al. Self-expansion of pre-trained models with mixture of adapters for continual learning. CVPR2025\n\n[C] Yu, J., et al. Boosting continual learning of vision-language models via mixture-of-experts adapters. CVPR2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zhZfnqE7vL", "reviewer": "Reviewer_awjr", "summary": "The paper proposes FlyCL, which addresses multicollinearity in continual learning. They use a mixture of sparse random projection, top-k sparsity and streaming ridge classification. They demonstrate impressive computational speedups (90%), while maintaining comparable performance. \n\nThese are real and practical speedups, and this is definitely good engineering. The bio-inspiration however does not add much substance beyond motivation.", "questions": "1. How sensitive is the performance to m, p and k? The defaults chosen seem somewhat arbitrary.\n2. When 10k dimensions are not sufficient, how does this scale to larger models?\n3. Is there a benefit to keeping the projection matrix random as opposed to learning it?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Very strong practical results, the paper shows very significant speedups with barely any loss in accuracy.\n2. The method is clearly general enough to adapt to different architectures and datasets, in a plug-and-play manner.\n3. The work has solid experimental evidence, good ablations, and statistical reporting.", "weaknesses": "1. The fly brain parallel is mostly surface-level (it looks like), and only inspires the sparse projection component.\n2. Some hyperparameters seem to require architecture-specific tuning, although that does not necessarily negate the proposed generality.\n3. Would be useful to examine if this scales to tasks at the scale of modern foundational models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)