RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks
Authors: Nazia Tasnim
Bryan A. Plummer
Venue: ICLR 2025 Poster (2025)
Keywords: reparameterization
weight decomposition
weight reconstruction
task-incremental learning
transfer learning
lifelong learning
incremental learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A novel weight-decomposition and reconstruction framework to facilitate incremental learning in extremely low parameter settings.

Abstract: Incremental learning aims to adapt to new sets of categories over time with minimal computational overhead. Prior work often addresses this task by training efficient task-specific adaptors that modify frozen layer weights or features to capture relevant information without affecting predictions on any previously learned categories. While these adaptors are generally more efficient than finetuning the entire network, they still can require tens to hundreds of thousands task-specific trainable parameters even for relatively small networks. This makes it challenging to operate in resource-constrained environments with high communication costs, such as edge devices or mobile phones. Thus, we propose Reparameterized, Compact weight Adaptation for Sequential Tasks (RECAST), a novel method that dramatically reduces the number of task-specific trainable parameters to fewer than 50 – several orders of magnitude less than competing methods like LoRA. RECAST accomplishes this efficiency by learning to decompose layer weights into a soft parameter-sharing framework consisting of a set of shared weight templates and very few module-specific scaling factors or coefficients. This soft parameter-sharing framework allows for effective task-wise reparameterization by tuning only these coefficients while keeping templates frozen. A key innovation of RECAST is the novel weight reconstruction pipeline called Neural Mimicry, which eliminates the need for pretraining from scratch. This allows for high-fidelity emulation of existing pretrained weights within our framework and provides quick adaptability to any model scale and architecture. Extensive experiments across six diverse datasets demonstrate RECAST outperforms the state-of-the-art by up to ∼ 1.5% and improves baselines by > 3% across various scales, architectures, and parameter spaces. Moreover, we show that RECAST’s architecture-agnostic nature allows for seamless integration with existing methods, further boosting performance.

Published as a conference paper at ICLR 2025
RECAST: REPARAMETERIZED, COMPACT WEIGHT ADAPTATION FOR SEQUENTIAL TASKS
Nazia Tasnim Boston University nimz Lmzziabuu... edu edu
Bryan A. Plummer Boston University bp um@uu.uuueee. edu
ABSTRACT
Incremental learning aims to adapt to new sets of categories time with minimal computational overhead. Prior work often trains efficient task-specific adapters that modify frozen layer weights or features to capture relevant infor- mation without affecting predictions on any previously learned categories. While these adapters are generally more efficient than finetuming the entire network, they still can require tens or hundreds of thousands of task-specific trainable parame- ters, even for relatively small networks. This can be problematic for resource- constrained environments with high communication costs, such as edge devices or mobile phones. Thus, we propose Reparameterized, Compact weight Adaptation for Sequential Tasks (RECAST), novel method that dramatically reduces the number of task-specific trainable parameters to fewer than 50 several orders magnitude less than competing methods like LoRA. RECAST accomplishes this efficiency by learning to decompose layer weights into a soft parameter-share framework consisting of set of shared weight templates and very few module- specific scaling factor coefficients. This soft parameter-sharing framework allows for effective task-wise reparameterization by tuning only these coefficients while keeping the templates frozen. A key innovation of RECAST is the novel weight reconstruction pipeline called Neural Mimicry, which eliminates the need for training in our framework from scratch. Extensive experiments across six diverse datasets demonstrate RECAST outperforms the ofte-oo-ttheart by up to 1.5% and improves baselines > 3% across various scales, architectures, and parameter spaces. Moreover, we show that RECAST's architecture-agnostic nature allows for seamless integration with existing methods, further boosting performancd"
INTRODUCTION
Incremental Learning (IL) aims to learn from a continuous stream of data tasks over time. Gener- ally speaking, the goal of IL methods is to avoid catastrophic forgetting (McCloskey & Cohen 1989 French 1999) while minimizing computational overhead. As summarized in Figure [aa), prior work
IL can be separated into three themes. First, Rehearsal methods that learn what samples to retain on subsequent iterations ensure their task-specific knowledge is retained (Caccia et al. 2020 Rebutti et al.) 2017] Chaudhry et al., 2019) [02T]; Shin et al., 2017), but they can require large amounts of storage as the number of tasks grows. Second, Regularization methods that re- quire no extra storage (Kirkpatrick et al. 2017 Kim et al. 2021 Aljundi et al.) 2018 Nguyen et al.)
2018 Zhang et al., 2020), but can result in network instability due, in part, to the cessticity-stabille dilemma (Grossbergg] 1982). Finally, third, Reconfiguation/AAdpaaaait
methods YYoon et al.) 2018 Rusu et al. 2006 Hung et al. 2019 Mallya et al., 2018) [Singh et al. 2020 Verma et al. 202 Jin & Kim 2022 Ge et al. 2023 Douillard et al. 2022; Wang et al. (002aa) that learn task-specific pa- rameters, but can result in large numbers of new parameters per-task that may not scale well. Even methods designed to be efficient, like those based on LoRA (Hu et al. 2022). still require tens or hundreds of thousands of task- specific trainable parameters for each task (e.g.. (Wang et al., 2024
Zhang et al. 2024 Eirko et al. (023)) making them ill-suited for some applications in resource-
constrained environments like mobile phones or edge devices.
Code: Repository

---

Published as a conference paper at ICLR 2025
(a) Prior Methods
(b) RECAST (Ours)
Rehearsal
Ragularization
Reconfiguration
Frazen Bacibone Memory Butfer
Constrained Backbone
Frozen Backbone Adaplers
Conttcllam-turing
Reseramctorios Model
3S01 9,0000
subbd
Growing Memory Requirement
Network Instability
Large took-wise parameters & uncontrolled growth
Extremnely low parapareeeerr dynamic weight generation
Task1:
Task2: Flowers
Task3: Scenes Aclasses: 67
Task n:
#classes: 200
#classes: 102
Ac Rolasses: 55
Target Tasks
Figure 1: (a) Existing IL methods, i.e. Rehearsal, Regularization, Reconfiguration exhibit various limitations in terms of model complexity, memory requirements, and training overheads. com- parison our proposed method, (b) RECAST can be uses as frozen backbone, allowing efficient reparameterization of any target module with a negligible number of parameter updates (order of 10-9) and can accommodate any number of disjoint tasks.
To address this, propose Reparameterized, Compact weight Adaptation for Sequential Tasks (RECAST), which provides an extremely parameter-efficient approach for IL via reparameteriza- tion of model's pretrained weights. As illustrated in Figure [b), RECAST achieves its compact task-specific representation by finetuning only the coefficients for a set of shared weight templates, enabling us to reconfigure an entire network's weights using fewer than 50 trainable parameters. RECAST framework shares some principles with prior Template Mixing methods (e.g.. (Plummer al. 2022, Savarese & Maire 2019)). However, these methods did not explore incremental learn- ing or reparameterization, focusing instead on hierarchy discovery and parameter allocation. They also required training from scratch, preventing the use of existing pre-trained models.
In contrast, our proposed Neural Mimicry approach allows RECAST to utilize existing large pre- trained weights without costly retraining. This weight reconstruction approach learns to decompose pretrained layers into framework, effectively emulating the original network's weights. This en- ables us to leverage any pretrained model with minimal processing (often just a few minutes). Thus, REACAST addresses gap in reparameterization schemes and IL. research where reconstruction methods remain underexplored where recent work has focused on generating weights for LoRA, typically through diffusion (Wang al. 2024 Zhang et al. 2024 Brkoç al. 2023) or used pre- trained weights to initialize their LoRA components (Liu et al. 2024a Si et al. 2024). Some prior work also learned additional predictor networks from scratch generate weights for subset of modules (Salimans & Kingmal 2016).. In contrast to these methods, our work directly changes the backbone weights rather than learning an adapter like LoRA-based methods, while also being sev- eral orders of magnitude more parameter efficient. provides an overview of our approach.
We evaluate RECAST's effectiveness on diverse datasets a Task-incremental IL setting. We com- pare against 16 baselines comprising ofte-off-the-at IL. methods on both CNN and Transformer architectures, where RECAST reports > 3% gain over prior work. When combined with adapter methods with higher parameter budgets, RECAST further increases performance by 1.5%.
Our key contributions can thus be summarized as follows:
We propose RECAST, A novel framework that dynamically generates module weights using shared basis matrices & module-specific coefficients. This weight-decomposed architecture en- ables task-specific adaptations optimizing only coefficients, leveraging cross-layer knowledge. We introduce Neural Mimicry, a novel weighhtreconsttuutton pipeline, that reproduces network weights without resource-ine resource-intensive pretraining. This scale and architecture-agnostc approach enables easy upgrades to existing architectures. We demonstrate that it builds stronger, more expressive backbones for transfer learning.
2

---

Published as a conference paper at ICLR 2025
Promen Shared Templete
Tasiowise Reparametertzed Module Weight
Decompositonn Through
Dynamic Weight (Section 2:2)
Torect
gitt
Rocarstruction
Taskwise tuned Coefficints
Module
Operation
Figure 2: RECAST decomposes module weights into templates and coefficients, which are learned through reconstruction from pretrained weights (Section 3.2). These components linearly com- bined to dynamically generate weights for target layer (Section 3.1). During TIL training, the templates kept frozen, and only task-specific coefficients are learned to create new layer weights.
RECAST's fine-tuned resource-performanc control, suits efficient continual learning with bounded resource constraints. Its compact task-specific parameters enable easier distribution across large-scale systems and edge devices.
2 RELATED WORK
Our work builds upon and extends several key areas in neural network adaptation, in particular the area of weight decomposition. These methods have previously shown that such modular networks are well suited for task adaptation (Mendez & Faton 2021 Ostapenko et al., 202T). Older works like Weight Normalization (Salimans & Kingma| 2016 decouples the length of target weight vectors from their direction to reparamcterize the network weights. Decomposition has also been used to make DNNs more interpretable (Zhou et al., 2018) and to discover hierarchical structures (Patil et al. 2023). [Savarese & Maire (2019) and Plummer al. (2022) demonstrate the effectiveness of weight sharing through decomposition, similar to our approach. However, these methods either require training models from scratch in their decomposed form or learning additional task-driven priors to accommodate new tasks (Veniat et al. 2020). In terms of parameter-efficient tuning, LoRA now popularly used to adapt to new tasks through low-rank updates (Chen et al. 2022 Liang & Li, 2024 Zhu et al. 2024 Nikdan et al.) 2024). but adds thousands of tunable parameters. In contrast, our approach learns to decompose the weights of pretrained network enable quick adaptation to new tasks with < 0.0002% task parameters.
Our work is also similar to works that generate neural network weights. Earlier works like, Hy- perNetworks (Ha et al. 2017) uses small subnetwork to predict the weights of a larger network. Unlike our reconstruction scheme, this requires end-to-end training on input vectors. NeRN (Ashke- nazi et al.] 2023) builds upon this idea of subnetworks by using positional embeddings of weight coordinates; however, it does not leverage pre-trained knowledge. A recent work, DoRA (Liu et al.) 2024b) shares some similarities with our approach by decomposing pretrained weights into magni- tude vectors and LoRA matrices. However, the weight utilized only during initialization and re- quires significant parameter tuning. Another contemporary work, WWang al. (2024) attempts novel weight generation through autoencoders and latent diffusion, which can be resource-heavy to train. In comparison, RECAST offers a direct, efficient approach weight generation and reparameteri- zation. We also show that RECAST can be combined with existing methods to boost performance.
3 REPARAMETERIZED, COMPACT WEIGHT ADAPTATION FOR SEQUENTIAL TASKS (RECAST)
In Task Incremental Learning (TIL) model sequentially learns tasks {di, d2, da. dn), where each task d; = (X,, Y:) introduces new categories, domains, or output spaces. The objective to maximize classification performance across tasks while minimizing resource usage, i.e. max Dh, P(d.) s.t. R. < Rmaas where P(d.) is the performance on task d;, It the total resource usage, and Rmoo is the resource constraint. This resource-awarenes, is crucial in practice as it enables deployment on edge devices with limited memory. impacts cloud infrastructure costs and real-time processing, and ensures accessibility in resource-limited environments. Thus, TIL poses a complex optimization problem: how can we design system that adapts to new tasks without significantly increasing its computational footprint compromising performance existing tasks?
3

---

Published as a conference paper at ICLR 2025
Algorithm 1 RECAST Framework
Require:
Total number of layers in network
G: Number of template bank groups
1: Number templates per bank
K: Number of coefficient sets
d: Dimension of weight matrices
My: Number of target modules in layer I
1: procedure INITIALIZE(L, G, G,n, K d, Mi)
2: for 9 to G do
3: Tg Tg E IR dddx ,,...,
> Template bank for group
4: end for
5: for / = 1 to L do
6:
for m
M4 do Ct.m C [, E l
7:
K₃ -1,- .....,
Module coefficients
8:
Orth Orthogonallliitaalizziooo (Ct,m)
9: end for
10: end for
11: end procedure
12: procedure FORWARDPASS(2,,)
13:
[T/(G/))
D Get group index Generate weights Apply layer operation
14:
W K EE Ct.m Tag
15:
y= Modull@pperation(1, Wi,m)
16: return y/
17:
end procedure
We address this multifaceted challenge with novel framework Reparameterized, Compact weight Adaptation for Sequential Tasks (RECAST), that strategically balances model plasticity and stability while adhering resource constraints. As illustrated in Figure RECAST introduces a flexible way to adapt the layers of neural networks through the dynamic combination sets of shared weight blueprints (i.e., template banks) and module-wise calibration factors coeffi- cients. In Section 3.1 we present our Weight Decomposition scheme. Section 3.2 introduces Neural Mimicry for direct reconstruction pretrained weights into reusable templates and coefficients.
3.1 WEIGHT DECOMPOSITION
At its core RECAST replaces pretrained layer weights with modules that can dynamically generate weights. A target module has access to bank of templates which is shared with some other layers in the network (discussed in Section 4.2). Each module has its own set of coefficients which are used to generate the weights. Formally, let L be the total number of layers in the network organized into groups, where each group shares a template bank. For each group E I,..., G. we define template bank Tg template gg E Rouu x xd.n x... has the same dimensions the layer being parameterized. For each target module m E M, in layer layer we generate K sets of coefficient vectors. The k-th set of coefficients for module in layer I defined as C = Ctk Ci,m,,: C.k i,m.2s ..... Cr where Cf,m, E is the learned coefficient for template g in group = [t](L/G)] The weight matrix for module m in layer / is computed via:
T9.1, Tg,2,... ... Tg.,. , where is the number of templates per bank. Each
I,..., L. where Mi is the number target modules in
W I.
= Tgi C.
(1)
where represents element-wise multiplication. The final weight matrix for module m in layer / is generated by averaging these K parameter matrices.
Wim
W
(2)
K
1,m.
k=1
4

---

Published as a conference paper at ICLR 2025
Algorithm 2 Neural Mimicry
Require:
M: Original pretrained model
M": Reconstructed Model
L: Number of layers
My: Number of modules in layer
T T Template bank for group containing layer
Cl. Coefficients for module in layer I
1: procedure NEURALLMMICRY
MM*, max.epochs)
2: for epoch to max_epochs do
3: total_loss <- 0
4: for / L do
Layer iteration > Module iteration
5:
for m = 1 to M2 do
6:
W <- Gennrratweiighss (T)) Ct,,)
Weight reconstruction
7:
Lt. f- LLsssLLuntion Li Wi,)
8:
total loss + total loss + Lim
9:
T got)) Ct.m < Update(T ((1,, Cl,m, It,,)
end for
11: end for
12: end for
13: return M
b Return reconstructed model
14:
end procedure
Algorithm describes how template banks for each group are created and as well as the orthogonal initialization of coefficients for each module m in layer I. The forward pass computes the final weight matrix using these components before applying the module operation. The target module then generates the appropriate output (e.g., convolutional feature map for CNN, attention feature vectors for attention module). See Appendix A.4.1 for additional architecture-speciff details.
3.2 NEURAL MIMICRY
We introduce "Neural Mimicry" foundational technique for efficiently emulating complex pre- trained neural networks. This end-to-end pipeline enables the recreation a target model within our framework, bypassing resource- resource-intensvve pretraining. The input to this pipeline is a pretrained target model M, whose module weights we aim to emulate. The output is a set template banks and coefficients that, when combined, approximate the target module weights of M. The Mimicry lies in its iterative refinement process, which progressively shapes the custom model M closely mirror the behavior of the target model M. In experiments the pretrained weights of M are obtained from official releases (see Appendix A.3) To generate weights for each module in each layer, employ Equations
of Neural
[ and 2 as described Section 3.I and then aim reduce the difference between the original pretrained weight (W... and RECAST--gneree
weight (W;, ) The main update step uses gradient descent (Algorithm 2 line 9) on the coefficients and templates &c
(n is the learning rate): Cg,m,5 & Cl.m. Bo mma
OC The process can To(), Toth. V
be formulated the following optimization problem:
Mi ΣΣ m=!
f.
L(Wi, Wi.m))
min (0. C.T
where c (e1...., (k) is set of K-noise vectors sampled from Gaussian distribution (00, o2), where E RRx...... The noise vectors are added with the coefficients only during the recon- struction training process. Here, L denotes total layers, Mi target modules layer 1, n templates bank, and L the weight discrepancy loss. As discussed in Section 4.2 L. should be chosen carefully, and we found Smooth LI loss to be most effective for generating diversified template. It offers smooth gradients for small differences and stability for larger errors, crucial for stable construction (Figures 5 & 4). This outperforms L2/MSE (sensitive to outliers) and KL divergence (computationally expensive) (See Figures & 6 After reconstruction, calculate the layerwise

---

Published as a conference paper at ICLR 2025
feature similarity between M and M using the cosine similarity metrie to evaluate the quality (see Table 6). Overall, this process requires negligible overhead (< minutes, see Table 5).
3.3 PARAMETRIC SCALING STRATEGY
RECAST's design is governed by three parameters: the number of groups (G), templates per group (n), and coefficient sets (K), balancing model expressiveness and efficiency. Increasing improves specialization, while increasing increases expressiveness within groups. The parameter-sharing scheme allows to expand weight configurations without significantly increasing parameters. This design achieves substantial memory savings, S = Lx Mi x dd (Gxx
xG x x X Mi xnxX), Here, Mi refers to the number of target modules per layer, and we can approximate the savings as: n). The savings stem primarily from replacing L x M1 weight matrices with G x n x Mj templates. In our experiments, G and n are calibrated to match the baseline model's 's parameter count but can be adjusted for desired compression. Ad- ditionally, multiple coefficient sets enable a combinatorially large number of weight matrices (n*) per module, covering extensive parameter spaces even with small n. The total learnable coefficients are 1= Mi x n x K. providing flexibility in controlling tunable parameters.
S L x M1 - x d2 x M1 dl (
EXPERIMENTS AND ANALYSIS
Baselines. To evaluate our proposed architecture, we have compared the performance against 16 baselines in total, representing the aforementioned categories of IL strategies. From Regularization approaches, we include EWC (Kirkpatrick et al., 2017). which penalizes changes to critical net- work parameters, and LwF (Li & Hoiem 2016), which uses previous model outputs as soft labels. For Replay Methods, we implement GDumb (Prabhu et al. (2020) with a 4400-sample memory buffer, and ECR-ACE (Caccia et al. 2022) and DER++ (Buzzega al., 2020), both using 120 samples per class. Reconfiguration methods are our primary focus as they are most similar to our approach, particularly adapter-baat based schemes that learn extra task-specific components. Among them, Piggy Back (Mallya et al. 2018) uses binary masking schemes within a fixed architecture, while HAT Serra et al. (2018) learns soft attention mask. CLR Ge et al. (2023 is a purely adapter- based method, that introduces depthwise separable Ghost modules (Han et al. 2020 after each convolutional layer. We also consider several LoRA-based approaches that apply LoRA to various ViT components (Attention (Zhu et al. 2024) or Fully-connected layers (Chen et al.) 2022 In addition, we compare with InfL.oRA (Liang & Li 2024), which has regularization and recon- figuration components, RoSA (Nikdan et al. 2024), which combines LoRA with sparse adaptation, DoRA (Liu et al.) 2024b) that decomposes pretrained weight into magnitude and direction matrices, and a poompt-based adapter technique: I.PP (Wang et al. 2022b). Finally, we include finetuned classifier layers as naive baselines.
Datasets. Following standard experiment setups in similar works by Aljjnndi al. 2019); Ge jet al. (2023) we employ six diverse benchmarking datasets covering fine-grained to coarse im- age classification tasks across various domain including flowers (Nilsback & Zisserman
2008
scenes (Quattoni& & Torralba 2009), birds (Wah et al., 2011). animals (Krizhevsky & Hinton 2009
vehicles (Maji et al. 2013). and other made objects (Krizhevsky Hinton 2009). These
datasets capture the diversity and inter-task variance essential for evaluating IL tasks. Images are standardized to 224 x 224 pixels, applying basic transformations and -ataset-specific normalization. Additional details are in Appendix Impendix/A.11111 Table
Metrics. We report Top-1 accuracy averaged over our six datasets. For RECAST, we report the mean and standard deviation averaged over 3 runs. See Appendix Table for per-task performance.
Implementation Details. We use both ResNet (He et al. 2016) and Vision Transformer (ViT) (Dosoviskkiy et al., 2020) backbones to demonstrate the versatility of our framework as they represent two different branches of image classification architectures (CNN and Transformer). Specifically, we use ResNet-34 and VIT-Small variants for all our methods, both of which have approximately 21 mil. parameters. However, average best performance across the six datasets for larger models is also presented in Figure For fair evaluation, pendent baselines, against our RECAST-Resme models and compare ViT-based baselines against RECAST-ViT model. We trained GDUMB (Prabhu et al. 2020), EWC (Lee et al.) 2019), LWF (Li
compare the ResNet de-

---

Published as a conference paper at ICLR 2025
Table 1: TIL accuracy averaged over six datasets using ResNet-34 comparing non-adapter (top) and adapter methods (bottom). RECAST surpasses non-adapter methods and enhances adapter- based approaches when combined. Parameters: M: model size, buffer, P: total parameters, U: unit attention mask, G: ghost modules Task Parameters and Storage metrics are reported Per Task
Method
Task Params
Storage
Acc@top1 ())
ResNet-34 (He al.
2016
0.0 FIM
M M + FIM
65.2
EWC (Lee et al.
2019)
P +
22.1
LWF (L
Horem 2016
M
19.3
GDUMB (Prabuu et al.
2020)
P
M + R
34.8
ER-ACE (Caccia et al.
2022
P
M + R
36.1
DER++ [Buzzega et al. 2020
P
M + R.
16.2
RECAST (ours)
3 X 10-6P
M
66.3+0.05
HAT (Serra et al.
2018)
P
U
40.0
Piggyback back (Mallya a et al. 2018
M
82.9
CLR (Ge et al. 2023)
4 x 10-3P
M G
79.9
RECAST (ours) Piggyback RECAST (ours) CLR
P
M
83.9-10.
4 x 3p
M+G
80.9 0.3
& Hoiem 2016), and L2P Waag et al. 2022b) using the avalanche library (Carta et al. 2023) Official Py Torch implementations of other methods were modified for TIL settings. We evaluated two RoSA (Nikdan et al., (024) variants: sparse MLP adaptation and full RoSA (LoRA A-ased at- tention). sparsity of 0.001 % ensured active parameters remained under 150, matching RECAST's
efficiency. LoRA methods used rank=1 and task-specific configurations: DoRA (Liu et al.) 2024b) (Q.K.V V matrices), McLo (Q.V matrices), and Adaptformer (MLP layers). RECAST, McLo (Zhu et al. 2024), and AdaptFormer (Chen al. 2022), RoSA (Nikdan et al. 2024). DoRA (Liu et al. 2024b) used AdamW (Loshchilov & Hutter 2017) with 26-3-18 3----5e--- 3----3
learning rates, 1e-6 weight decay, and stepwise LR scheduling (decay by 0.T every 33 epochs) for 100 epochs. Default hy-
perparameters were used for avalanche models and methods like HAT (Serra et al. 2018). Piggy- back (Mallya al.) 2018), and CLR (Ge et al.) (202)), trained for 100 epochs. InfLoRA (Liang & Li. 2024) used class increments of 90 and 10 epochs per task. For RECAST-ViT, we used group size of 6. 2.2to templates per bank, and 2 coefficient sets. Integrated models used RECAST the backbone, with adapter layers modifying features while RECAST generated core layer weights and coefficients adapted per task (See Appendix A.4.2 All experiments were run on single RT'8000 GPU.
4.1 RESULTS
Table T showcases the performance of different methods using ResNet-34 34 backbone. Traditional IL methods such EWC (Lee et al., 2019) and LWF (Li & Hoiem 2016) show significant catas- trophic forgetting, with performance drops of approximately 66% and 70% respectively, compared the baseline ResNet-34. GDUMB (Prabhu al. 2020), ER-ACE (Caccia et al. 2022), despite uti- lizing a large memory buffer, only perform slightly better than these traditional methods but still lag significantly behind simple finetuning, HAT (Seraa et 2018), which learns unit-wise masks over the entire network, achieves further improvement over GDUMB but still lags 39% behind the base- line. This underperformance is likely because HAT designed allocate fixed network capacity to each task, struggling with our diverse task suite. In contrast, RECAST improves upon the baseline ResNet-34 by 2%, while fine-tuning only 0.0003% of the original parameters. This represents significant efficiency advantage while still delivering performance gains. Among adapter methods, Piggyback (Mallya et al., 2018) and CLR (Ge et al.) 2023) show strong performance, exceeding the baseline by 27% and 23% respectively. Combined with these methods, RECAST further enhances their effectiveness. RECAST + Piggyback achieves 1.5% improvement over Piggyback alone. Similarly. RECAST CLR also shows ~ 1.5% gain over standalone CLR. These consistent im- provements across different adapter-based approaches demonstrate the versatility and effectiveness of RECAST complementary technique.
Tablee@summarizes the results for methods using the ViT-Small (Dosovitskiy et al. 2020) backbone, further emphasizing RECAST's strong performance. RECAST improves the baseline ViT-Small by

---

Published as a conference paper at ICLR 2025
Table 2: TIL accuracy averaged over six datasets using a ViT-Small comparing non-adapter (top) and adapter methods (bottom). RECAST improves baseline accuracy by 3 points and enhances existing methods 1%. M: model parameters, P: total parameters, r: LoRA rank, Pl: prompt pool size, To: tokens per prompt, D: embedding dimension, SpA: Sparse Adaptation. Task Parameters and Storage metric are reported Per Task manner (except L2P and InfLoRA)
Method
Task Params
Storage
Acc@top1
ViT-Small Dosovitskiy et al. 2020
0.0 2 x 6P
M
82.4
RECAST (ours)
M
85.0=#0.1
L2P (Wang et al.
2022b)
To * D Mpool
M + Pl
35.9
InAA [iang Li
2024)
x 10-30(pp 3(( 11
M
88.5
AdaptFormer (Chen al.
2022
6 10 10 Pr
M
89.0
MeLo Zhu et al.
2024)
8 X 10 1r
M
88.7
DoRA Liu et al.
2024b)
1.9 10-3Pr
M
89.3
RoSA Nikdan al.
2024
(Full Att)
7 10-3Pr
M
88.6
RoSA Nikdan et al. 2024 MLP)
6 10
M 84.5
RECAST (ours) + Adaptformer
6 10 1PP
M
90.1=0.2
RECAST (ours) + MeLo
8 X 10
M
89.6+0.22
RECAST (ours) + RoSA(Full Att.)
7 X 10 3r
M
88.9 +0.08
RECAST (ours) + RoSA(SpA MLP)
6 x 10 6P
M
85.4
Baneline S7 Rocomi-uudes--1
105A otly Coefictert-aaning LoRA Coetidert-aning
Reconst uotic #1 Couffidarts
87.5
85.0
22
77.
750
ViT Smail
VIT Base
Large
DINO VIT Small
464
1.100
1500
13,000
18,000
22,000
37,000
Number trainable parameters fast
Figure Comparing averaged best Top-1 accuracy across six datasets for ViT-Small across various model config- urations. We find RECAST excels ultra-low parame- ter ranges (24-96) where LoRA struggles. RECAST w/ LoRA improves performance across all parameter ranges, offering complementary advantages.
Figure 3: Averaged best Top-1 accu- racy across six datasets for VIT mod- els of varying scales, comparing Base- line, RECAST, and RECAST with Co- efficient fine-tuning (FT) approaches.
3.5%, adding only 0.0002% task parameters. This represents a significant enhancement with minimal computational overhead. Adapter and reparameterizatioo methods all show strong perfor- mance improvements over the baseline: RoSA (Nikdan et al., 2024) (with SpA MLP) improves by 2.5%. while InfLoRA (Liang & Li 2024). MeLo (Zhu et 2024), AdaptFormer (Chen al. 2022]. DoRA (Liu et al. (2024b), and RoSA (Full Att.) all deliver improvements in the 7.5 8.5% range. However, combining RECAST with these methods yields additional gains. Both RECAST RoSA (SpA MLP) and RECAST + AdaptFormer, show ~ 1.5% gain over standalone RoSA and AdaptFormer alone, and RECAST + MeLo achieves a 1.0% improvement over MeLo. In contrast, prompting methods like L2P (Wang et al. 2022b). which rely instance-level prompts, underper- form the baseline by 56%. struggling to handle diverse domains effectively.
Figure 3 reports the performance of using RECAST over range of architecture sizes including ViT- Small, ViT-Base, and ViT-Large, demonstrating that our approach generalizes. In addition, we also show that our approach generalizes across pretraining datasets by demonstrating that approach can boost performance using a DINO Caron et al. (2021) backbone, as well.
Comparison of Resource Efficiency. Tables and compare various incremental learning meth- ods for ResNet-34 and ViT-Small backbones, respectively. We see that parameter- free methods (EWC, LWF) maintain minimal storage (only the backbone model parameters) but suffer from per-
8

---

Published as a conference paper at ICLR 2025
Frobchis
Enjopy
15.4
66.4
10.00
04.
63
022
Smo00A1
MSE
Smoothi11 Nold
VAS MSE
VAS Smoopits
Smoothe1
MBE
Smoshak1 None
VAE MSE
SmoothL1
Figure 5: Plot showing average classification accuracy of models reconstructed in different ways. VAB-reconstructed models performed slightly worse than the rest, with Smooth LI loss providing the best performance
Figure 6: Plot showing Diversity (Frobenius Norm) and adaptability measurements (En- tropy) of models reconstructed in different ways. VAE models have shallow template di- versity and low internal information content
formance degradation as task diversity increases. Rehearsal-basee methods (GDUMB, DER++ & ER-ACE) and HAT offer improved performance at the cost of increased storage requirements, which methods balance perfor-
may limit scalability for long-term learning. Recent parameter-efficien mance and resource usage. CLR learns 0.3% of backbone parameters, requiring backbone storage plus task-specific depthwise modules. InfLoRA trains the full model for the first dataset and only 0.2% for subsequent tasks. Other LoRA variants train less than 1% per rank. While masking re- quires training a large number of parameters, both LoRA updates and binary masks (PiggyBade RoSA Sparse updates) can be mathematically merged prevent extra storage and communication costs during deployment. Our proposed RECAST method demonstrates unprecedented parame- ter efficiency, learning only 0.0002% of the total backbone parameters while maintaining only the backbone model storage. When combined with other efficient methods like MeLo or Adapt- former, RECAST achieves state-of-the-art performance (90. 10.1% Acc@topl for ViT-Small) with the base storage requirement. This reveals that RECAST not only enhances existing adapter methods but does so in ways that are complementary to their strengths. The impact is delineated clearly in Figure 4 in range of taskwise parameter counts. It compares three adaptation approaches using ViT-Small backbone (82.4% baseline). Traditional LoRA adaptation (blue) becomes effective only above 464 parameters/task. RECAST (orange) achieves consistent 85% accuracy with just 24-96 pa- rameters, showing marginal improvement beyond this range. The combined RECAST+LORA (red) peaks at 91% accuracy, outperforming across all parameter ranges, including areas where LoRA alone falters. To scale traditional LoRA implementations to a negligibly small (< 150) parameters, utilized three techniques: (1) lowering LoRA rank, (2) sharing LoRA matrices across layers, and (3) binary masking for pruning. The figure reveals that RECAST can be used effectively in parameter ranges where other reparameterization methods might struggle.
4.2 MODEL ANALYSIS
How does the number of coefficients impact performance? We analyze the effect of hyperpa- rameters G, n, and K (Subsection 3.3) through two experiments. First, we vary n while keeping == 1, adjusting G to maintain constant parameters (G 4), comparing three sharing schemes: Low (two-layer sharing), Balanced (four-layer sharing), and Extreme (all-layer sharing). Results are shown in Figure 7 Second, we and n while varying K (since |C| x K) as depicted in Figure 8 Both experiments use RECAST MLP and Attention layers of small ViT model, with marker sizes indicating relative coefficient sizes. Results reveal that adjusting template numbers and sharing schemes has greater impact than linearly increasing K, emphasizing the importance of template diversity and specificity for task adaptation. Additional linear combinations show dimin- ishing returns once sufficient coefficients available (see the orange line in Figure 4
Where do we apply RECAST? RECAST is applicable to various layers or modules (Ap- pendix A.4.T), with notable insights highlighted in Figures 7 and g These figures illustrate the

---

Published as a conference paper at ICLR 2025
on RECAST on Attention
RECASI RECAST Attention.
86
Average best Accurac)
Average Dest Accuracy
85-
83
82
2
Low
Balanced
Extreme
Confficient Set
Figure 7: Increasing template numbers (lay- in group) enhances coefficient count and template sharing across the architecture. Low sharing uses G=6, Balanced G=3, and Extreme G=1 (one template bank shared by all network layers). The last setting delivers significant performance improvements.
Figure 8: Comparing the strategies of chang- ing coefficients. The size of the markers cor- responds to the relative number of coefficients. The impact of increasing coefficients by vary- ing the number of coefficient sets K, is much less prominent than changing it through group- ing schemes.
differing performance boosts when applying the method to attention layers versus MLP layers. MLP layers, though simpler, have more parameters that may introduce redundancy, making them more amenable to sharing. In contrast, attention layers are complex, involving multiple computational steps to model relationships between input parts rather than directly transforming representations. This complexity leads to challenging optimization space, hindering effective template sharing and reparameterization. Figure44confrmm this pattern, showing performance spikes when smaller LoRA matrices target MLP layers (Chen et al. 2022) rather than attention mechanisms (Zhu et al. 2024).
What is the best way reconstruct weights? Section 3.2 described the weight reconstruction
methodology for RECAST. Earlier we noted that model expressivity depends more on template bank diversity. investigate this, we generated five models with the same configurations (6 groups, layers each), varying the reconstruction objective: 1) SmoothL1 loss, 2) MSE loss, 3) SmoothL. loss with coefficient noise, VAE with MSE loss, and 5) VAE with SmoothL loss. All models achieved at least 98.5% reconstruction similarity (Appendix Table 6). Figure 5 shows that SmoothL loss (with or without perturbation) provides the most benefit during reparameterization & VAE models provide the least. To understand this, calculated template diversity within groups using Frobenius Norm (diversity metric) and Entropy (versatility metric) (See Appendix (A.3) Just like their classifi- cation performance, SmoothI.1 loss shows roughly similar internal patterns with without noise. The high Frobenius norm for Neural Mimicry (Algorithm 2 with SmoothL1 loss indicates greater template diversity than the rest of the approaches. MSE models showed slightly lower Frobenius norms but higher entropy, suggesting templates are versatile and not overly dependent on dominant components. In contrast, VAEs showed shallow template diversity, limiting adaptability, and lower entropy. indicating reduced versatility. Figure 10 (Appendix A.3) illustrates higher intra- group co- efficient similarity in VAE-bar based models, suggesting multiple layers perform similar operations.
CONCLUSION
RECAST represents a significant advancement neural network architecture design, addressing the challenge of optimizing parameter efficiency while maintaining model expressivity. By decom- posing network layers into templates and coefficients, facilitates dynamic, task-specific weight generation with reduced parameter footprint. Our comprehensive evaluations demonstrate RE- CAST's efficacy across diverse image classification tasks and vision architectures. Based on our model analysis, future research directions may include enhancing model diversity and coefficient versatility, exploring applications beyond computer vision, and investigating alternative approaches for reconstructing novel parameters overcome the limitations of existing pretrained weights. Ad- ditionally, further research is necessary to elucidate the relationship between model complexity and performance, particularly in the context of model compression techniques.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments This material is based upon work supported, in part, by DARPA under agree- ment number PR00112020054. Any opinions, findings, and conclusions or recommendations ex- pressed in this material are those of the author(s) and do not necessarily reflect the views of the supporting agencies.
6 ETHICS STATEMENT
While our work on efficient reparameterization aims to improve model robustness and adaptabil- ity, we acknowledge the potential ethical implications of such advancements. First, the incremental learning capabilities enabled by our approach could be beneficial scenarios like robotics, where systems need to constantly adapt to new categories or environments, well as provide potential benefits in terms of environmental sustainability by reducing computational footprint. While we intend democratize access to powerful AI models, we recognize that easier deployment of these models could also lower barriers to potential misuse, such as in surveillance applications that could infringe on privacy rights. Additionally, as with any machine learning model, there are inherent biases and limitations in the training data and model architecture that users should be aware of. As we reconstruct from existing weights, our reconstructed pretrained weights also likely inherits the implicit biases in those existing weights. Thus, we caution against over-reliance tions without human oversight, especially in high-stakes decision-maie making processes. As researchers, emphasize the importance of responsible development and deployment of Al technologies, and we encourage ongoing discussions about the ethical use incremental learning and efficient deep learning models in various applications.
model predic-
REPRODUCIBILITY
To ensure the reproducibility of our work, we have taken several key steps. The implementation de- tails of our custom ResNet and Vision Transformer architectures are fully described in the main text, with complete code provided in the supplementary materials. We have also provided the full codes for our framework integrated with other adapter and reparameterization methods we have discussed in the main text. Our reconstruction scripts for both ResNet and ViT models, including the loss func- tions and training procedures, are also included in the supplementary material. We have specified all hyperparameters, including learning rates, scheduling, and template bank configurations. The base models used for comparison (ResNet34 and ViT-Small are from widely available libraries (torchvi- sion and timm), ensuring consistent baselines. All experiments were conducted using PyTTorch, with specific versions and dependencies listed in the supplementary materials. We have also included our data preprocessing steps and evaluation metrics to facilitate accurate replication of our results. The supplementary material also includes descriptions of how to the codebase to both reproduce our results, as well as to extend or use by users in plug-and and-play manner.
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Computer Vision ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part III. pp. 144-161, Berlin, Heidelberg, 2018. Springer-Verlag. ISBN 978-3-030-01218-2. doi: 10.1/07/ 978-3-030-01219-9) 9. htt/ttttss:/doi.org///// 1ttp:/111.000//991 0-1----121--9999------
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc. E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL wtR.httpss... httptt:pss//rooceedinnee www.eeeeedeiiggg........ neur wwwp:nnwwwr....ppeeeeoeee wwtpsper.pppppeeeeee files/
Maor Ashkenazi, Zohar Rimon, Ron Vainshtein, Shir Levi, Elad Richardson, Pinchas Mintz, and Eran Treister. Nern learning neural representations for neural networks. In -, Jamuary 2023.
Publisher Copyright: c 2023 11th International Conference on Learning Representations, ICLR 2023. All rights reserved.; 11th International Conference on Learning Representations, ICLR 2023: Conference date: 01-05-2023 Through 05-05-2023.
11

---

Published as a conference paper at ICLR 2025
Pietro Buzzega, Matteo Boschini, Angelo Porrello. Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In Proceedings of the 34th International Conference Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. Online learned continual compression with adaptive quantization modules. Proceedings of the 37th International Confer- ence on Machine Learning, 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights reducing abrupt representation change in online continual learning. In ICLR, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Antonio Carta, Lorenzo Pellegrini, Andrea Cossu, Hamed Hemati, and Vincenzo Lomonaco. Avalanche: A pytorch library for deep continual learning. Journal of Machine Learning Research, 24(363):1-6, 2023. URL http::/tjmmp. org/ papprs/ppps////aaa /wteers//242333/ 0130.110000 htmll
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICLR, 2019.
Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip HS Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. In AAAI, 2021.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: adapting vision transformers for scalable visual recognition. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387100
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 109////R 200...6684....
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. CORR, abs/2010.1 19929, 1929, 2020.
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. 2022 IEEF/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 9275-9285, 2022. doi: 10.11099/V8 2022.00907.
Ziya Erkoç. Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generat- ing implicit neural fields with weight-space diffusion. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14254-14264, 2023.
R. M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3: 135, 1999. doi: 10. 10.101661366466611990019999
Yunhao Ge, Yuecheng Li. Shuo Ni, Jiaping Zhao, Ming-Hsuar Hsuan Yang, and Laurent Itti. Clr: Channel- wise lightweight reprogramming for continual learning. In 2023 IEEE/CVF International Con- ference Computer Vision (ICCV), pp. 18762, 2023. doi: 10.1 N0.100//CCVV5000.22222 01723.
S. Grossberg. Studies of mind and brain. Boston Studies in the Philosophy of Science, 1982. doi: 00.1007/978-94-0099 047007-7758-77
David Ha, Andrew M. Dai, and Quoc V. Le. Hypemetworks.. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
12

---

Published as a conference paper at ICLR 2025
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1577-1586, 2020. doi: 10.1099CCVP442000220000000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10.1109/CVPR.201.9990
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: httt::://openreviert nww/tee/ttte formm?
nd=keeeeee
Steven C. Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning, pp. 11. Curran Associates Inc., Red Hook, NY. USA, 2019.
Hyundong Jin and Eunwoo Kim. Helpful or harmful: Inter-task association in continual learning. In European Conference on Computer Vision, pp- 519-535. Springer. 2022.
Chris Dongjoo Kim, Jinseo Jeong, Sangwoo Moon, and Gunhee Kim. Continual learning on noisy data streams via self- celf--urified replay. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 517-527, 2021. doi: 10. 1..11099CCV499222221110 00055..
J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Barwiiiska,, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, Over- coming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sci- ences, 111:3521135222 -3526, 2017. doi: 10. 10.130///pas.1 161 1611835114,
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
K. Lee, K. Lee, Shin, and H. Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. 2019 IEEF/CVF International Conference on Computer Vision (ICCV), 2019. doi: 10.1109/icc.2019.000 20..0000..
Z. Li and D. Hoiem. Learning without forgetting. Lecture Notes in Computer Science, pp. 614-629, 2016. doi: 10. 11000/9988-3331-4449933
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- low-rank adaptation for continual learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23638-2364444 2024.
Shih- Yang Liu, Chien-Ys Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: weight-decomposed low-rank adaptation. In Proceed- ings of the 41st International Conference on Machine Learning, (CML'24. JMLR.org, 2024a.
Shih- Yang Liu, Chien- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 32100-32121. PMLR, 21-27 Jul 2024b. URL httt:s::/ mmwtiiinnnssmm.......mm.mmmmmmmm parress/235/1iu22400 html
Ilya Loshchilov and Frank Hutter. abs/1711.05101, 2017.
Fixing weight decay regularization in adam.
CoRR,
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. -, 2013.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to mul-
tiple tasks by learning to mask weights. In Computer Vision - ECCV 2018: 15th European Con- ference, Munich, Germany, September 14, 2018, Proceedings, Part IV. pp. 72-88, Berlin, Hei- delberg, 2018 Springer- Verlag. ISBN 978- 000---02300-012-33...333 doi: 10.1007/ 0-107883-300-22-5--- 5.
13

---

Published as a conference paper at ICLR 2025
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower (ed.). : volume 24 of Psychology of Learning and Motivation, pp. 109-165. Academic Press, 1989. doi: https://ooi.or/// 10.
(0161900799-72211888868888 att.icleepp1//50777722222
URL https: pestps:/wwee http///wwws/./ encedi cwmmccccccc...oo ince/
Jorge A Mendez and Eric Eaton. Lifelong learning of compositional structures. In International
Conference on Learning Representations, 2021.
id=ADWG4TT01133
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner Variational continual learn- ing. In International Conference on Learning Representations, 2018.
Mahdi Nikdan, Soroush Tabesh, Elvir Crnčević, and Dan Alistarh. RoSA: Accurate parameter- efficient fine-tuning via robust adaptation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceed- ings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 38187-38206. PMLR, 21-27 Jul 2024. URL https:
//proceedings. wttp:ssl...rrreeeeeeeeeeeeeeeeee
n www.ppreessssa. 1e1/5//iiinannan kdan2 4a. html
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing. Dec 2008.
Oleksiy Ostapenko, Pau Rodrígue7, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393.
Shreyas Malakarjun Patil, Loizos Michael, and Constantine Dovrolis. Neural sculpting: uncovering hierarchically modular task structure in neural networks through pruning and network analysis. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc.
Bryan A. Plummer, Nikoli Dryden, Julius Frost, Torsten Hoefler, and Kate Saenko. Neural parameter allocation search. In International Conference on Learning Representations, 2022.
A. Prabhu, P. H. S. Torr, and P. K. Dokania. Gdumb: a simple approach that questions our progress in continual learning. Computer Vision ECCV 2020, pp. 524-540, 2020. doi: 31.
10.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. 2009 IEEE Conference doi: 10. 20.1.09/
Computer Vision and Pattern Recognition, pp. 413-420 2009. 5206537.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533-5542, 2017. doi: 10. 00.100//VPR.2217.5883
E. Rusu, O. Makarynskyy. A. Gavrilov, and D. Makarynska. Filling gaps in wave records with artifi- cial neural networks. Maritime Transportution and Exploitation of Ocean and Coastal Resources, pp. 1085-1091, 2006. doi: 10. 10.12011999
Tim Salimans and Diederik P. Kingma. Weight normalization: a simple reparameterization to ac- celerate training of deep neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 901-909, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 97815 110838888
Pedro Savarese and Michael Maire. Learning implicitly recurrent CNNs through parameter sharing. In International Conference on Learning Representations, 2019.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4548 4557. PMLR, 10-15 Jul 2018. URL https: http:////ccceednee... m mlr.press/v8b/seee
Ba html
14

---

Published as a conference paper at ICLR 2025
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 1994-3003, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Chongjie Si, Xiaokang Yang, and Wei Shen. See further for parameter efficient fine-tuning by standing on the shoulders of decomposition. arXiv preprint arXiv:2407.05417, 2024.
Pravendra Singh, Vinay Kumar Verma, Pratik Mazumder, Lawrence Carin, and Piyush Rai. Calibrat- ing cnns for lifelong learning. In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 15579-15590. Cur- ran Associates, Inc., 2020.
wwtppurrpppccpppppp..eeeeeeeeeee files/ pdf.
Tom Veniat, Ludovic Denoyer, and Marc' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. CoRR, abs/2012.12631, 2020.
V. K. Verma, K. J. Liang, N. Mehta, P. Rai, and I.. Carin. Efficient feature transformations for dis- criminative and generative continual learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. doi: 10. 10.1199ccypr4443722010.11333
C. Wah, S. Branson. P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011 100-T---2110000 1-001, California Institute Technology, 2011.
Kaili Wang, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network parameter diffusion. In 2024.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transform- ers: An occam's 's razor for domain incremental learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022a.
Zifeng Wang. Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022b.
Ross Wightman. Pytorch image models.
pytorch- sinoorch-imag---e -maag--modes 2019.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In ICLR, 2018.
Baoquan Zhang, Chuyao Luo, Demin Yu, Xutao Li, Huiwei Lin, Yumming Ye, and Bowen Zhang- Metadiff: Meta-learning with conditional diffusion for few-shot learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38115)1166871166955 Mar. 2024. doi: 10.1609/ At///AAA///eee v38.15.29608. URL s wwt.:///..................... http:pppppp/a/atttttt////taa//aa
29608
J. Zhang. J. Zhang, S. Ghosh, D. Li. S. Tasci, L. Heck, H. Zhang, and C. J. Kuo. Class-incremental learning via deep model consolidation. 2020 IEEE Winter Conference on Applications of Com- puter Vision (WACV), 2020. doi: 1 10.11///455557222200000 202.99993....20.99933....
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Yitao Zhu, Zhenrong Shen, Zihao Zhao, Sheng Wang. Xin Wang, Xiangyu Zhao, Dinggang Shen, and Qian Wang. Melo: Low-rank adaptation is better than fine-tuning for medical image diagno- sis. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1-5, 2024. doi: 10. 10.1109/SB16657.22221115533
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DATASET
In Table 3 have summarized the class variations and number of samples for the six datasets have used in our TIL experimens All the datasets were split with 75% 15% train-validation- test split. Simple augmentations like resizing, centercropping, horizontal flips, and normalization have been applied for each.
Table 3: Details on the Benchmarking Datasets
Dataset
Classes
#Sample
Oxford Flowers Nilssbcc & Zisserman 2008
102
6553
FGVC Aircrafts Maju et al. 20133
55
10001
MIT Scenes (Quatton: & Torralba 2009
67
15614
CIFARI RI00 (Krızhevsky & Hinton 2009
100
60000
CIFARIO (Kizhevsky & Hinton 2009)
10
60000
CUBs (Wah et al. (20IT)
200
11789
A.2 TASKWISE PERFORMANCE BREAKDOWN
Impact of Task-order The sensitivity to task ordering varies significantly across different incre- mental learning approaches. Methods that rely on knowledge distillation or parameter importance estimation, such as LwF and EWC, exhibit cumulative degradation as knowledge gets diluted across sequential tasks. In particular, LwF's iistillation-baaee approach becomes less effecttve for later tasks as the knowledge transfer chain grows longer. EWC's reliance on Fisher Information Matrix to protect earlier tasks' parameters lead representation bias, where earlier tasks dispropor- tionately influence the learned feature space. Rehearsal- based methods like GDUMB and DER++ face challenges in maintaining balanced representative memory buffers across tasks, though sam- ple selection strategies help mitigate but don't 't eliminate these order effects. Even masked-based approaches like HAT, despite maintaining unit-wise masks over the entire model, show order sen- sitivity due to the constrained mask allocation being influenced by carlier tasks. In contrast, both RECAST and several adapter-based baselines (AdaptFormer, Mel.o, RoSA, DoRA) maintain ab- solute parameter inference through strict parameter isolation. As shown in Table E these methods achieve consistent performance across tasks, with no significant degradation in earlier task perfor- mance new tasks learned. For instance, RECAST AdaptFormer maintains high accuracy across all domains (99.2% on Flowers, 84.7% on Aircraft, 87.1% 1% on Scene), demonstrating that task-specific parameters can effectively adapt while the shared feature space remains stable. This architectural design choice effectively eliminates task interference, as each task's 's performance re- mains stable once learned, regardless of the order in which subsequent tasks are introduced. The empirical results suggest that methods employing strict parameter isolation are inherently robust to task ordering effects compared approaches that modify shared parameters or rely on knowledge transfer between tasks.
A.3 RECONSTRUCTION ANALYSIS
Pre-trained Model Details We obtained the VIT and DiNO pretrained weights from the official timm repository Wightman (201). Resnet weights are obtained from Pytorch Hub. The specific
models and their corresponding weight URLs are:
ResNet-34:
http:://bbo.laadd..yrreerrrrrrrr
resnet 34-5met33-bb2225599
0.
0 2ht-haapnnn22222ttee00---1-1-1-2
16

---

Published as a conference paper at ICLR 2025
ViT- Base:
3
do 0.
0 0-- ---nagpoat222-ttteeea20------012 22w...11-ee__222.22222
16 _10.-21000000011110000000ooaaoa1
DiNO-Small Caron et al.
dino deitsmalll pretrain www.eeeeeettllee..mommaamaammmee pretraii.ttt
Table 4: Model Performance Comparison Across Different Datasets. For InfLoRA the 6 tasks are generated by taking chunk of 90 classes from the combined dataset so, the classification result doesn't I exactly correspond to each dataset.
Model Type
Flowers
Aircraft
Scene
CIFART00
CIFARTO
Birds
Small ViT
97.4
66.6
82.6
72.5
94.9
81.1
I.2P
0.4
5.7
0.8
56.4
75.0
77.6
InfLoRA
99.8
89.0
88.2
84.8
84.6
84.5
AdaptFormer
99.2
79.6
85.7
87.6
98.1
84.3
MeLO
99.0
84.2
84.0
87.3
98.1
80.0
Rosa (Sparse MLP 0.001%)
97.6
69.1
84.1
78.9
96.2
81.6
RoSA Full
99.4
79.1
85.2
87.4
97.9
82.6
DoRA
99.02
86.4
84.7
87.6
98.2
80.0
RECAST (Small ViT)
98.6
72.5
84.3
79.1
95.7
80.5
RECAST AdaptFormer
99.2
84.7
87.1
88.8
98.3
84.4
RECAST MeLO
99.4
86.3
84.3
87.1
97.9
81.0
RECAST (Sparse MLP 0.001 %)
98.4
72.9
83.9
80.0
96.3
81.0
RECAST RoSA Full
99.2
81.9
84.7
87.5
97.9
82.4
Resnet34
87.6
51.9
66.9
59.8
83.0
42.0
EWC
2.6
12.1
9.0
1.7
42.2
65.4
LWF
1.4
23.1
5.6
1.7
41.1
42,9
GDUMB
38.9
25.6
33,2
4.5
69.7
36.9
DER++
1.3
12.5
26.1
53.9
0.7
2.5
HAT
49.1
71.9
46.5
9.9
30.6
39.8
PiggyBack
93.9
85.4
74.4
82.6
96.6
65.0
CLR
93.9
83.3
73.3
75.9
94.3
59.0
RECAST (Resnet34)
89.8
52.7
65.8
61.4
83.8
44.2
RECAST Piggy Back
95.7
87.3
77.0
81.7
96.3
66.3
RECAST CLR
94.7
83.7
72.1
77.4
95.6
62.1
Table 5: Required runtime and hardware utilization to run Neural Mimicry (Section
3.2
Model Scale
Peak GPU
Average CPU Utilization
Per Epoch Processing (ms)
Total Epochs
Wall Clock
Memory (GB)
Time (seconds)
ViT Small (Rs M)
0.3
3.7%
36.2
1000
37.4
ViT Base ( 86M)
1.2
3.7%
112
1000
102.3
ViT Large (+e 304M)
4.5
3.7%
340
1000
245.0
Resource Requirements The results in Table 5 demonstrate that Neural Mimicry has modest
memory footprint (0.33-4.5GB GPU memory) with minimal CPU overhead ( (377)) It also linearly scales with model size. Furthermore, the complete reconstruction is completed in minutes even for large models. Notably, this step achieves 98-1009 reconstruction accuracy (Table ( and requires significantly less resources than model pretraining.
17

---

Published as a conference paper at ICLR 2025
RECAST is architecture and scale agnostic RECAST reconstruct models across various architecture and scales. In Table we present the results of reconstructed models and the official performance of these models the Imagenet-1K, 1K (Deng et al. 2009) dataset. All reconstructed models in the table have the same configuration 6 groups, layers in each group and 2 sets of coefficients for each target module (except ViT-Large has 12 groups). The inference comparison Imagenet tells us that, RECAST can compietely emulate the feature generation of any pretrained weight, We also report the averaged best accuracy @ topp across the datasets in figure 3 for the varying scales of the VIT models. In all cases, RECAST perfermos slightly better than baseline without coefficient-tumi tuning, and significantly better with coefficient-tningg.
Model
Imagenet Ace Acc@topp top1
Reconstruction
Similarity
Resnet-34
73.3
Recon. Resnet-34
73.0
99%
ViT-Small
74.6
Recon. ViT-Small (MLP)
74.6
100%
Recon. ViT-Small (Attention)
74.6
100%
Recon. ViT-Small (MLP& & Attention)
74.6
100%
Recon. ViT-Small (VAE + MSE)
73.8
98.5%
Recon. ViT-Small (VAE + SmoothL!)
73.0
98.6%
ViT-Base
81.1
Recon. ViT-Base
81.0
100%
ViT-Large
84.4
Recon. ViT-Large
84.4
100%
DINO Small
77.0
Recon. DiNO Small
76.9
99%
Table 6: Accuracy comparison of reconstructed RECAST models over the ImageNetlk dataset, against the original pretrained models from timm (Wightman 2019) demonstrates that both back- bones are empirically the same. Despite this similarity, RECAST provides reparameterization facil- ities with 0.0002% coefficients
Template diversity is important for better reparameterization In Section A.3 we empirically showed that models with higher Frobenius norm value and higher entropy provides better task- adaptable model. We further breakdown this analysis by first defining the metries, and then analyz- ing their layerwise trends for various reconstruction methods.
Frobenius Norm:
Calculates the square root of the sum of the absolute squares of its elements. In the context of RECAST, the Frobenius norm is used to measure the di- versity among templates within a template bank for each group. For a template bank (. in group 9, the average Frobenius norm can be calculated as:
To (T T 9.2.-.... TTo avg frobenius 7(=IIJ 2i=1 2}=++1 -ー the metric sensitive large deviations between templates. This property helps iden- tify when templates are significantly different, not just slightly varied.
[[T
TaIlF.. Squaring the differences makes
Entropy Quantifies the average amount of information contained in a set of values. For each template bank Ty, we calculate the entropy of singular values to measure the balance of importance among the components of the templates. The average entropy for the template bank in group is calculated as: avg entropy, entropy for a single template Tg.;
Σn (TT,))))) where HTT is the
The Frobenius norm helps ensure diversity among T allowing for wide range of possible W A higher score means more difference among the template. The entropy ensures that each Ty is itself balanced and flexible, contributing to the adaptability of the reconstructed weights. A higher entropy corresponds to better information quality within each group. Together, these metrics pro- vide comprehensive view of the template characteristics in RECAST, helping to optimize both the diversity of the template bank and the flexibility of individual templates for efficient and ef- fective weight reconstruction. In Figure 9a and 9b show the change of Frobenius norm and
18

---

Published as a conference paper at ICLR 2025
Entropy. The VAE models perform very low in terms of the Frobenius metrie, although their pattern varies depending on the loss function. This is also true for the models generated through Neural Mimicry í.e. models reconstructed with MSE loss and SmoothL loss performance almost sim- ilar on classification task but differs in entropy pattern and Frobenius norm. Another observation is that noise perturbations don't seem impact these two metrics at all.
19.
Average SV Eneragoy
Smronh)] Kele
WiE 1S55
VAE SmoolLLI
MSE
Smosth11 WAE
Naice
5x33
VAE Smoothit
Group Number
Group Number
(a) Change of Frobenius nomm across the groups
(b) Change of Entropy across the groups
Figure 9: Analyzing the strategies of changing reconstruction schemes by quantifying their respec- tive Frobenius Norm and Entropy metrics (as defined in A.3) across 6 groups of layers of VIT. It can be observed that the VAE-reconstruucted models generally demonstrate significantly low Group- wise Frobenius score and comparatively lower Entropy.
We define another metric the intra-group coefficient similarity to measure how similar the coeffi- cient vectors are between different layers within the same group of RECAST model. For layers I1 and /2 in the same group: sim((11, 12) cosine siniiarrlm (Cn, C12) Where Ct is the flattened and concatenated vector of all coefficients for layer If similarities are generally low within a group, means each coefficient is using the shared templates in unique ways. This could indicate that each layer is capturing different features or transformations. The Figure I0 shows that the later groups in the VAE models share very high untra-group coefficient similarity indicating that layers within each group are using templates very similarly or there are redundancies in the network.
A.4 RECAST ADAPTATION
A.4.1 RECAST-ADAPTER COMPONENTS
Our framework can be adapted to various neural network components. Below we provide the details of how they may be implemented in Neural Networks.
Fully-connected Layers We use RECAST here to generate the weights for the full-connected number of output features and d the
layers network. Here, Ti E RRd dd.. dout number of input features. Coefficient shape, C 1x1 represents a scalar value for each template with additional broadcasting dimensions. After generating final weight using Eq. it's used as f(X W final + b), where E bhachh sizexd.in and Y phateh
sizexd.out pbatch.sizexd.om
Attention QKV Matrices In attention mechanisms, RECAST generates single matrix for the combined query, key, and value projections. This uses a template shape 772 E _dddd where is the embedding dimension and coefficient shape, C/ E Rlxixl The weight generation process is similar the FC layer case. The resulting finat is then used to compute Q, K, and V. For input shape X saateh.siaex.eeeenngg. we can obtain Q, K, E ipbatch_sisex seg.gthhet.
Convolution Kernels For convolutional layers, RECAST generates filter kernels that are applied across the input feature maps. Here, T, E R X x x Cout is the number of output channels, Gin is the number of input channels, and K is the kernel size. The weight generation process follows
19

---

Published as a conference paper at ICLR 2025
Doeder Sowell
Figure 10: Intra-group Coefficient Similarity across the groups of similar configuration models, that have been reconstructed in different ways. The later groups in the VAE/Encoder models share very high intra-group coefficient similarity and that layers within each group are using templates very similarly or there redundancies in the network.
the pattern before. The resulting Wyinal is then used the convolution kernel:
Y
co, W fina)), where X pbotthh.izee ciinx and Y
A.4.2 RECAST INTEGRATION WITH EXISTING METHODS
RECAST complements existing incremental learning methods as a task-adaptable weight generation backbone. For CNN architectures, RECAST generates base weights that Piggy Back's (Mallya al. 2018 binary masks can modulate and provides primary convolution weights that CLR's 's Ge et al. (2023) Ghost modules can refine. In transformer architectures, RECAST produces core weight matrices that methods like Mel.O, AdaptFormer (for query/value and MI.P components). DoRA (for magnitude-direetion decomposition), and RoSA (for sparse adaptations) can then modify.
For example, in standard LoRA, weight matrix W E Rduueddin is modified by adding low-rank update:
WLARA Wo BA
where Rdo.. A Rrxdd, and is the rank.
When integrating with RECAST, instead of using a fixed Wo. we use the dynamically generated weight matrix from RECAST, From Equation 1
K Σ
WRECAST = K
gg.
(5)
A=
20

---

Published as a conference paper at ICLR 2025
where g. are the templates and C.m.; the coefficients. The final integrated approach combines both methods:
combined
WRECAST BA
(6)
The total trainable parameters for the combined approach are:
RECAST: (nn)) coefficients per module
LoRA: O(r(d;n. + dout)) parameters per module
where n is the number of templates, K the number of coefficient sets, and r is the LoRA rank.
Similar principles apply when integrating with other methods:
CLR (Channel-wise Lightweight Reprogramming) Ge et al. (2023):
Ghost(Conv(X, WRECAST)
where Ghost(-) represents the lightweight channel-wise transformations through depthwise- separable convolutions.
PiggyBack (Mallya al., 2018):
W combined WRECAST O M
(8)
where M a binary mask matrix.
DoRA (Liu et al. 2024b):
WRECAST + BA
W combined ||RECAST|!
(9)
| + BA
where || denotes the magnitude and BA is the LoRA update.
RoSA Nikdan al. 2024):
W combincd WRECAST + S Ô WRECAST + (BA)
(10)
where S is sparse binary mask applied to the LoRA update.
21


Reviews:
[{"review_id": "kp3LTgjAHT", "reviewer": "Reviewer_KSLE", "summary": "This paper proposes a Neural Mimicry method to study a very small set of parameters for continual learning purposes. It studies both ViT and ResNet applications. It showed performance benefits over several benchmarks.", "questions": "As above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper studies an important task.\n- Motivating towards neural mimic and learnings from proxy domains are helpful in solving the CL tasks.", "weaknesses": "- Experiments are very small scale. Two target models are only ~21M in size (Ln 305), and does not demonstrate scalability to large models.\n- No forgetting values are presented that are critical aspects of CL learning setups.\n- Fig. 1 is misleading and hard to get the 2*10^-6 parameters - instead this is a ratio instead of parameters. Moreover, it's not thoroughly shown this is a parameter that scales across networks. Would it be the same for LLM? Otherwise stating the method yields <<<1% can be fairly misleading, and in fact wrong if the units are parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TTxVaKQL8k", "reviewer": "Reviewer_GjwD", "summary": "This paper addresses the problem of continual learning with the goal of creating a learning pipeline that reduces the number of learnable parameters, making it feasible for resource-limited devices. To achieve this, the authors propose a Neural Mimicry pipeline, which decomposes model weights into a bank of templates. Using these frozen templates, the model only learns coefficients for subsequent downstream tasks. Experimental results highlight the potential of the proposed method. However, I have several questions, and my detailed comments are as follows.", "questions": "How are the pre-trained weights obtained for further Neural Mimicry decomposition? This seems quite important for the algorithm’s performance. If the model is pre-trained on data that differs significantly from the sequential tasks that follow, learning only coefficients may be insufficient, in my view.\n\nIn Table 1, why the total storage for RECAST + Piggyback/CLR increase linearly? For CNN-based models, why can’t this be further reduced, given that the learnable task parameters are already minimized from  $P$  to  $4 \\times 10^{-3} P$ ?\n\nHow does the algorithm perform with larger models and datasets, such as ViT-base/Large with the ImageNet dataset? Would the Neural Mimicry method still be effective? Further discussion on this would be beneficial.\n\nIn Table 2, why does the total storage remain unchanged? As more tasks are added, shouldn’t total storage also increase with  $M + T \\times 6/8 \\times 10^{-4} Pr$ ?\n\nEfficiency in training/learning on edge devices is also important. It would be highly valuable to include a comparison of wall-clock training time and GPU memory in Tables 1 and 2. Given the significantly reduced number of learnable parameters, the proposed method should enhance training efficiency, which could be seen as an advantage of this approach, but empirical evidence is needed to confirm this.\n\nIt would also be helpful to report the training complexity of the Neural Mimicry process.\n\nFinally, in Table 1, why do the accuracies for EWC, LWF, and GDUMB perform so much worse than ResNet-34?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed idea of incorporating Neural Mimicry into continual learning is novel and interesting. By decomposing model weights into frozen template banks and focusing solely on learning coefficients, the approach significantly reduces the number of learnable parameters. This paper introduces a new pipeline and valuable insights for the continual learning community.\n\nExperiments on both CNN and ViT-small models demonstrate the effectiveness of the proposed method and highlight its architecture-agnostic nature.", "weaknesses": "The writing in this paper could be improved. For the pseudo-code, it would enhance readability to introduce the notations alongside the code for easier understanding. Additionally, more details on how to obtain the pre-trained model weights should be clearly provided.\n\nThe effectiveness of the method on larger models and datasets has not yet been adequately addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bas3c12ELd", "reviewer": "Reviewer_YqZe", "summary": "In this work, the authors propose a parameter efficient task incremental learning method. Specifically, the proposed method reparameterizes the original model by using pre-trained template banks and lightweight learnable coefficients for each tasks during incremental learning. In the experimental results, the proposed method shows better accuracy with less trainable parameters when combining with previous incremental learning methods.", "questions": "Overall, this work appears to introduce a new parameter-efficient task incremental learning method, which significantly reduces the number of learnable parameters while improving accuracy when combined with previous incremental learning techniques. However, the current version has presentation quality issues, and the contribution of the technical aspects remains unclear. My specific questions are as follows:\n\n1. What are the technical differences between the proposed RECAST and previous Template Mixing methods? For example:\n   - How does the design of template banks and coefficients differ in RECAST?\n   - How does RECAST support incremental learning with substantially fewer learnable parameters, whereas Template Mixing methods do not achieve this?\n\n   It's essential to clarify the specific differences to highlight the novelty of the proposed method.\n\n2. How does RECAST balance model plasticity and stability in incremental learning? From my understanding, RECAST reparameterizes the model using lightweight coefficients and frozen template banks for each downstream task, making it a parameter-efficient fine-tuning method rather than one specifically designed for incremental learning. In the experiments, RECAST is combined with existing incremental learning methods to achieve better performance. If this understanding is correct, it would be useful to compare RECAST with other parameter-efficient fine-tuning methods, such as prompt-tuning, LoRA, and DoRA.\n\n3. How does RECAST integrate into existing incremental learning methods, as shown in Tables 1 and 2?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The proposed method can reduce learnable parameter for incremental learning.\n\n2. In the experiments, the proposed methods shows better accuracy with less trainable parameters when combining with previous incremental learning methods.", "weaknesses": "1. The paper is not well written and not well organized. For example:\n    \n    (1) The proposed method and related works are mixed together in Section 2.\n    \n    (2) Subsection 2.1 introduces too many concepts from later sections, making it difficult to follow.\n   \n    (3) The opening paragraphs of Subsections 2.1 and 2.2 are largely redundant.\n\n     (4) The overview of the proposed RECAST method (Fig. 2) should be introduced at the beginning of Section 2, but it isn’t presented until Subsection 2.2.\n\n2. The proposed RECAST method itself does not seem directly related to task incremental learning.\n\n3. The paper lacks sufficient background information, especially on relevant works like Template Mixing methods, making it difficult to assess the novelty of the approach.\n\n4. It is unclear how RECAST integrates with previous incremental learning methods, as shown in Tables 1 and 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fGMsD3twYn", "reviewer": "Reviewer_uU62", "summary": "RECAST is a Task Incremental Learning (TIL) method that efficiently reparameterizes model components to emulate a pretrained target network by decomposing weights into shared templates and module-specific coefficients. The shared templates, stored in a template bank, match the shape of each layer and remain fixed across tasks, while the module-specific coefficients are fine-tuned for each new task. \n\nThrough a process called Neural Mimicry, RECAST selects a target model and combines the templates and coefficients to approximate it. The approach is architecture-agnostic and requires minimal parameter updates across tasks.", "questions": "1.  Does the task order adhere to the sequence in which the datasets are presented in the text (lines 294-296), or does averaging across three different runs involve altering the task order?  \nI believe exploring the latter could be an intriguing study on how task order impacts weight adaptation and whether the learned coefficients are affected by such variations.  \n\n**About LoRA and Figure 3**\n\n2. It’s not entirely clear what is being evaluated in Figure 3. \nWhile a comparison with LoRA is certainly necessary in this work, could you clarify what \"CF\" stands for in the figure? \n\n3. Additionally, rather than comparing to LoRA-only, it might be more informative to use the same baseline using LoRA in one case and RECAST in the other, e.g., fine-tuning the same backbone with the two methods.\nIt’s possible that Figure 3 already presents this setup, as it seems ViT-Small is used as the backbone and lower bound. If this is indeed the case, the figure caption and related section would benefit from clarification, with additional emphasis on these details to enhance interpretability.\n\n4.  Furthermore, the comparison does not appear to fully explore the parameter range for both methods; although the x-axis in the plot represents the number of parameters, the methods are not evaluated across the entire range, and a comparison with the same number of parameters is only partially explored. Could you clarify the reasoning behind this?\n\n**CL Setting**\n5. Is there a specific reason why the authors have chosen to focus exclusively on the Task Incremental Setting, rather than exploring the more challenging Class Incremental Setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Clear presentation of results, focusing on both parameter efficiency and storage alongside accuracy. \n- The method enhances existing adapter-based methods, making it compatible and valuable as a supplementary tool. \n- It is versatile across architectures, with applicability to both CNNs and Transformers. \n- The setting is flexible, allowing adjustment of groups *G* and templates *n* to achieve the desired level of model compression.", "weaknesses": "I recommend adding another baseline based on rehearsal, such as ER[1], ER-ACE[2], or DER++[3]. While the proposed method is more aligned with adapter-based approaches, claiming strong performance in settings utilizing a memory buffer without including one of these established baselines for comparison seems somewhat unfair. \n\nI understand that the accuracy reported is averaged across the six datasets used at the end of training. However, I'm curious if accuracy varies between tasks encountered earlier in training and those seen later. Including a table that shows accuracy results for individual tasks would strengthen the paper’s validity. This would also relate to Question 1 regarding the influence of task order. \n\n\n    [1] Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37–57, 1985 \n    [2] Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New Insights on Reducing Abrupt Representation Change in Online Continual Learning. In International Conference on Learning Representations Workshop, 2022. \n    [3] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark Experience for General Continual Learning: a Strong, Simple Baseline. In Advances in Neural Information Processing Systems, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)