Parameter Efficient Continual Learning with Dynamic Low- Rank Adaptation
Authors: Prashant Shivaram Bhat
Shakib Yazdani
Elahe Arani
Bahram Zonooz
Venue: Accepted by TMLR ()

Abstract: Catastrophic forgetting has remained a critical challenge for deep neural networks in Continual Learning (CL) as it undermines consolidated knowledge when learning new tasks. Parameter efficient fine-tuning CL techniques are gaining traction for their effectiveness in addressing catastrophic forgetting with lightweight training schedule while avoiding degradation of consolidated knowledge in pre-trained models. However, low-rank adapters (LoRA) in these approaches are highly sensitive to rank selection as it can lead to sub-optimal resource allocation and performance. To this end, we introduce PEARL, a rehearsal-free CL framework that entails dynamic rank allocation for LoRA components during CL training. Specifically, PEARL leverages reference task weights and adaptively determines the rank of task-specific LoRA components based on the current task’s proximity to reference task weights in parameter space. To demonstrate the versatility of PEARL, we evaluate PEARL across three vision architectures (ResNet, Separable Convolutional Network, and Vision Transformer) and a multitude of CL scenarios, and show that PEARL outperforms all considered baselines by a large margin.

Published in Transactions on Machine Learning Research (01/2026)
Parameter Efficient Continual Learning with Dynamic Low- Rank Adaptation
Prashant Shivaram Bhat
p.s.bhat@tue.nl
Eindhoven University of Technology (TU/e)
Shakib Yazdani
shakib. www.daaannnnm---naaaaaaaaaaa
Searland University
Elahe Arani
e.aroni@tue.nl
Findhoven University of Technology (TU/e)
Wayve
Bahram Zonooz
b. b.zonooz@hue.rl
Eindhoven University of Technology (TU/e)
Abstract
Catastrophic forgetting has remained critical challenge for deep neural networks in Continual Learning (CT.) it undermines consolidated knowledge when learning new tasks. Parameter efficient fine-tuning CL techniques gaining traction for their effectiveness in addressing catastrophic forgetting with lightweight training schedule while avoiding degradation of consolidated knowledge in pre-trained models. However, low-rank adapters (LoRA) in these approaches are highly sensitive to rank selection as it can lead to suboptimal resource allocation and performance.
this end, we introduce PEARL, a rehearsal- CL framework that entails dynamic rank allocation for LoRA components during CL training. Specifically, PEARL leverages reference task weights? and adaptively determines the rank of task-specific LoRA components based the current task's proximity reference task weights in parameter space. To demonstrate the versatility of PEARL, we evaluate PEARL across three vision architectures (ResNet, Separable Convolutional Network, and Vision Transformer) and multitude of CL sccnarios, and show that PEARL outperforms all considered basclines by a large margin.
1 Introduction
Learning forms the foundation for intelligent systems to adapt to dynamic environments in response to external changes. Unlike conventional deep neural networks (DNNs) that are designed for static data distribution. Continual Learning (CL) is characterized by learning from sequential, dynamic data distribution (Wang et al. 2023b). However, CL has remained significant challenge for DNNs due to the issue of catastrophic forgetting: A phenomenon in which adaptation to new tasks results in significant deterioration of consolidated knowledge Masana et al. 2022 De Lange et al.] 2021 Ditzler et al. 2015 Robins 1995). A number of approaches such as schearsal-bascl (c.g., Buzzoga et al. (2020)), woight-reeularizztion based (c.g., Kirkpatrick et al. (2017)) and irrameter-ssolation based (e.g., Rusu el al. (2016)) have been proposed in the literature to address the problem of catastrophic forgetting in CL. Although CL primarily intended to ensure the resource efficiency of model updates (Wang al. 2023b). the reduction in catastrophic forgetting these approaches is reliant on increases computational and memory footprint due to model surrogates and memory buffer. Extrapolating forward this reliance in real world scenarios is both economically and environmentally infeasible.

---

Published in Transactions on Machine Learning Research (01/2026)
The increased availability of pre-trained models that demonstrate strong generalization capabilities has led researchers to discover early exploratory methods (Douillard et al.] 2022 Jeeveswaran et al. 2023
Gao et al. 2024 Yang et al. 2024), facilitating more efficient integration of new knowledge in CL systems.
However, contimuous finc-tuning on multiple downstream tasks compromises pre-trained model's generalization capabilities (Liu et al.)
2024a). Recently, Parameter Efficient, Fine-Tuning (PEFT) CL techniques (Liang
& Li. 2024 Zhou et al. 2024b) are on the rise as they provide lightweight training schedule while avoiding degradation of consolidated knowledge in pre-trained models. However, it is not clear how one can select the size of their rank (in case of LoRA (Hu et al. 2021)) while their performance is very sensitive to rank selection [Valipour et al. 2022). Although choosing a higher rank usually results in optimal performance, it significantly increases the number of learnable parameters in longer task sequences when using large pre-trained models. Task arithmetic @Tharco et al.) )023), which enables model compression using simple arithmetic operalions with task vectors (difference between pre-trained and fine-tuned weights) (Chitale et al. 2023). allows for selection of higher rank during training without any overhcad during inference. Task arithmetic assumes that fine-tuning a pre-trained model pushes the weights away lowards semantically relevant direction in the weight space, thus enabling model compression using simple arithmetic operations with task vectors (Chitale et al. 2023). However, task arithmetic suffers from number of shortcomings including finding optimal scaling coefficients and suboptimal performance in the absence of a memory buffer (Zhou et al. 2024c).
Typically, when using arbitrary weights, a higher rank usually facilitates learning new tasks but tends to increase forgetting with significant increase in number of parameters, while lower-rank mitigates forgetting with limited adaptation (Lu et al. 2024 Biderman et al., 2024). There often exists a balance between plasticity and stability at moderately small (but not too small) rank, which may enhance the benefits of CL (Lu et al.) 2024). To this end, we introduce PEARL Parameter-Efffiicnt Adaptive Ranking for lifelong learning with LoRA), rehearsal-free, parrameterissolation-bess mitigates catastrophic forgetting through dynamic low-rank adaptation. Specifically, PEARL leverages the information contained in the reference task weights ! and dynamically determines the rank of task-specific, layer-specific LoRA components during training based on the current task's proximity to reference task weights in parameter space. With our dynamic threshold criterion, a close proximity would result. in a lower rank allocation and maximum information re-use while a distant proximity results in larger rank selection, thereby finding an optimal balance between learning and forgetting. To demonstrate the versatility of PEARL, we perform extensive analyses across various datasets, multiple CL scenarios and different architectures, an overview of which can be found in Appendix B.1 Our results demonstrate that PEARL variants consistently outperform the evaluated baselines in all scenarios by a huge margin. Our contributions are as follows:
PEFT vision CL framework that
We propose PEARL, a generic PEFT vision CL framework that mitigates catastrophic forgetting in CL through dynamic low-rank adaptation. PEART is rehearsal-free and grows modestly with each task (Table [ by assigning optimal rank for task-specifie, layer-specific LoRA components, resulting in maximum information reuse.
We propose a simple, dynamic threshold criterion (Eqn. the reference task weights in the parameter space.
3 guided by the current task's proximity to
We conduct extensive experiments across different architectures (ResNet, Separable Convolutional Network, and Vision Transformer), 8 distinct settings, 2 CL scenarios, and 2 CL metrics to clearly demonstrate that PEARL significantly outperforms the considered basclines. To further substantiate our claims, we provide 10+ comprehensive analysis in Appendix C
2 Related Works
Exemplar and Non-Exemplar Based CL: Experience Rehearsal (ER), which stores and replays samples
from previous tasks, has remained a leading approach to addressing the issue of catastrophic forgetting in CL.
Several works build upon ER: DER++ (Buzeeaa al. 2020) proposed consistency regularization technique
to enhance stability and generalization. Guided by the GWT, TAMiL (Bhat et al. 2023) and AGILE (Bhat
We use reference task interchangeably with pre-training when available. Otherwise, refers to the first. task CI..
2

---

Published in Transactions on Machine Learning Research (01/2026)
et al. 2024) incorporate information reuse and task-specific, parameter-cfficiet adapters that re-project the backbone features onto global workspace. However, these approaches face additional computational and memory overhead due to model surrogates and experience rehearsal. Since storing samples from previous tasks is not always feasible, non-cxemplar-basee approaches are gaining traction. Non-Exemplar Class-IL (NECIL) is particularly important scenarios where data confidentiality is crucial due to privacy or security concerns, and where data storage has a limited lifespan. Early contributions to this domain include LwF (Li & Hoiem 2017) and EWC (Kirkpatrick et al., 2017), both of which employ regularization techniques to reduce catastrophic forgetting. More recent methods, such as PASS [Zhu et al., 2021b) and IL2A (Zhu et al.) 2021a), have further advanced NECILL by generating prototypes for previous classes without the need to retain the original images. SSRE (Zhu et al. 2022) introduced a reparameterizztion method to balance old and new knowledge, along with additional sell-training that utilizes external data as an alternative to exemplars. FeTrIL (Petit et al. 2023) proposed a framework that combines a fixed feature extractor with a usendo-feature generator using gcometric transformations to achieve a better
balancc. FcCAM (Goswami eet al. 2024a) addressed the limitations of Euclidean distance when feat ure distributions are non-stationary and suggested using anisotropic Mahalanobis distance as substitute for Euclidean distance when employing frozen feature extractor for prototype classification. Overall, the NECIL benchmark presents a significant challenge as it prohibits the storage of any samples from previous tasks while requiring effective performance in a Class-IL setting. PEARL also aligns with the NECIL benchmark as it operates without any experience rehearsal.
Parameter Efficient Fine- Tuning (PEFT) techniques that have emerged in the space of Large Language Models have been quite pivotal in optimizing model performance with little memory and computational overhcad. Seminal works such Adapter modules (Houlsby et al. 2019). Prompt Tuning (Brown 2020 BitFit (Zaken et al. 2021). LoRA (Hu et al., 2021). and DoRA (Liu et al. 2024b) have demonstrated the efficacy of selective fine-tuning all the while preserving model generalization and enhancing model adaptability. Specifically, LoRA introduces two compact, trainable low rank matrices into each layer's weight matrix, allowing model to adapt to new tasks with minimal memory and computational footprint. Within CL, several approaches leverage PEFT techniques to mitigate catastrophic forgetting: Prompt-based methods such as L2P (Wang et al., (022e) and DualPrompt (Wang et al.) (022c) introduce task-specific prompts to facilitate learning new tasks while preserving consolidated knowledge. S-Prompt (Wang et al., 2022a) and CODA-Prompi (Smith et al. 2023b) extend these works by employing structural prompls to map discriminative relationships and through Schmidt orthogonalization respectively. In order to maintain an optimal balance between computational complexity and accuracy, adapter-based methods such as SEMA (Wang et al. 2025) and DIA (Li et al. 2025) propose self-expandingg dynamic integration of tpsk-specific adapters. Similarly, EASE (Zhou et al. (024b) creates task-specific sub-spaces by employing lightweight. adapter modules for each new task, thereby maintaining resource efficiency, In contrast to adapter- based methods, InfLoRA (Liang & Li, 2024) inserts small number of LoRA parameters to re-parameterize the pre-trained weights and shows that fine-tuning these inserted parameters is equivalent to tinetuning the pre-trained weights within a subspace. C-LoRA (Smith et al., 2023a) incorporates a seff-rogularization strategy, which penalizes updates to parameters already modified by previous tasks. This mechanism helps preserve prior knowledge and enhances stability across sequential tasks. While these approaches have shown relative success, their implementation is tailored specifically to pre-trained vision transformers and do not consider task dynamics, limiting the potential for information reuse and dynamic resource allocation.
Task Arithmetic (Mharco et al. 2023), centered around task vectors, offers a new paradigm for steering the behavior of pre-trained / reference network towards multiple downstream tasks using task similarity. Inspired by recent works on weight interpolation (Frankle et al. 2020 IIharco et al. 2022 Ainsworth et al. 2022). task vectors are defined to be the element-wise difference between pre-trained reference network weights before and after fine-tuning on a particular task. A task vector specifies a direction in the weight space of a pre-trained model, such that movement that direction improves performance on a given task ([lharco et al. 2023). , As these task vectors can be modified and combined together in simple arithmetic operations, new tasks can be learnt via addition while interfering tasks can be forgotten via negation. Inspired by these findings, CCitale et al. (2223) proposed approach to continually train transformer-based vision models using low-rank adaptation and task arithmetic. Although task arithmetic is quite lucrative for its simplicity and intuitiveness, it suffers from several shortcomings such as finding optimal scaling coefficients and sub-optimal
3

---

Published in Transactions on Machine Learning Research (01/2026)
performance in the absence of memory buffer to finetune the network after weight aggregation Zhou et al. 2024ch.
To this end, we propose PEARL, a generic PEFT CL framework that leverages task arithmetic to dynamically decide the rank of task-specific, layer-specific LoRA components during training. PEARL is tailored for custom, compact Cl. enabling maximum information reuse and modest growth in number of parameters with new tasks in order.
3 Method
Our CL setup consists of 1 (1,2,.,,K) sequential tasks wherein the CL model Φg, parameterized by 6, is expected to perform equally well on both current and previous tasks. Each task entails task-specific independent and identically distributed data distribution D, with ((m,,)))) pairs. In rehearsal-free approaches, access to previous task data distributions is limited when learning a new task. This limitation makes CL. espocially challenging, as becomes difficult to maintain performance on older tasks without access to their data distributions. Traditional parameter-isolation methods typically employ lask-specifie, parameter- intensive sub-networks that are learned isolation, resulting a substantial memory and computational footprint. Thus, strike balance between model size and performance in rehearsal-free approaches, is essential to re-use information and enhance the learning of new tasks through parameter-effcieen sub-nctworks. To this end, PEARL is designed to leverage the information aggregated during the reference task training through the dynamic allocation of parameter-efficient low-rank adapters. In the subsequent sub-sections, we detail out the different stages of PEARL development.
3.1 Information aggregation during reference task training
Our CL model parameterized as 0 = 0(0 U 0 L {for } consists of a disjoint set of task-specific parameters and set of reference task parameters respectively. The feature extractor foe incorporates low-rank adapters and normalization layers (only in CNNs), while the classifier Jø encompasses all classes pertinent. to the respective task. In contrast, the reference task feature extractor so- typically self-attention layers, pooling layers, and
comprises of a larger network that integrates convolutional normalization layers, with the classifier gø- representing the classes to be learned during the reference task training. To restrict the empirical risk, PEARL minimizes the following objective during reference task training:
C, Σ (2.9))=b, DED,
Loa (o (Jor (x))),
(1)
where b is training batch, Loe is cross-entropy loss, D, is the data distribution for the reference task, and a-) the softmax function. Since pro-trained models demonstrate strong generalization capabilitios, we expect to aggregale generic information that can be re-used across tasks later during CL. In scenarios where pre-training on a large corpus of data is not possibility, the first task in CL setup can also be used as a reference task.
3.2 Information re-use and dynamic resource allocation during CL
We leverage the knowledge captured in the reference task training and model additional task-specific information pertaining to respective tasks through dynamically allocated low-rank adapters. Let W e for be the weight matrix of certain layer (either lincar, convolutional, or point-wise) belonging to the reference task after its training and W E for be the corresponding weight matrix belonging to current task after finetuning W on D.. The task vector W is given by the element-wise difference between W and W ie., W W* WW. In other words, task vector W can be thought of as a weight matrix signaling the additional information on top of the information contained in the reference task weights. Therefore, PEARL leverages parameter-effieient low-rank adapters to model task vectors to efficiently learn the current task. To this
4

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm PEARL (with pre-training)
1: Input: Pre-trained model 0, = : ),
O,, Data distributions {D, Epochs E R2
2: for tasks t E (1, K} do
3: O₂ + (deepooo (,
for f
Fino-tune 0, on Dj for E1 epochs
6:
for target layers 1 E {1, 2, L) do Compute W W/ - WT Decompose W using Equ. Find dynamic rank ky (Eqn. & Equ
7:
8:
9.
10:
Initialize LoRA components B1 and A (Eqn. 5 for = fo U (B
11:
12:
Re-initialize task-specific weights {for, gor)
13:
A - U { go:
14:
Fine-tune { go+) on Dr for Eg epochs
15:
return model 1
end, we decompose W (flattened and reshaped to 2D if necessary) using Singular Value Decomposition (SV-Decomposition) as follows:
W UEV Σ a.vv
(2)
where the columns of U and V consist of the left and right singular vectors, respectively, n represents the full rank after SV- Dy-Decomposition, and E is a diagonal matrix whose diagonal entries are the singular values of the sum truncated after terms.
Wt. For E 21,2,... n), let W -.
Theorem 3.1 For any matriz W of rank at most k, I - Wtell S I -
where ||- be unitarily invariant norm. The Eckart- Young-Mirsky Mirsky Theorem 3..1 implies that the best low k-rank approximation of matrix can be achieved by selecting top-k singular values and such an approximation maintains the least error in terms of unilaterally invariant norms including the Frobenius and spectral norms. Our novelty lies in finding the best-k that eliminates the need for extensive hyperparameter tuning of the desired and provides an optimum trade-off between performance and growth in number of learnable parameters. To select the best-k, we propose a dynamic threshold on the cumulative explained variance of singular values as follows:
E W [( WT) + (WWr W*)
where T represents a normalized squared error between current task and reference task weights, and o represents element-wise multiplication. Through T, we approximate the proximity of the current task with respect the reference task in the parameter space. is not guaranteed to lie in [0, 1] and T can be as large as extreme cases. Therefore, whenever T L, we simply clip to in our implementation. Once T computed, the dynamic rank for the task-specific, layer-specific LoRA can be found as follows:
E E 2T)
h arg max
+1 +1
(4)
Equation 3 and 4 ensure that the l is bounded and select the rank proportion to the cumulative explained variance of singular values. The +1 term in Equation 4 is to ensure the chosen rank is at least singular

---

Published in Transactions on Machine Learning Research (01/2026)
value, preventing a degenerate zero-rank projection and preserving minimal expressive capacity. Subsequently, the low-rank matrices can be initialized from the SV-Decomposition of W (flattened and re-shaped to 2D if necessary) as follows:
B U 1: R A 2/: [T k
(5)
As A and B are initialized from SV-Decomposition matrices, times, the CL model often struggles to find optimum in the loss landscape resulting in sub-optimal performance. Therefore, before any further fine-tuning on the current task, we re-initialize the weights of A, B along with task-specific normalization (if any) and classification layers. An ablation study on the same can be found in Section 4.5
3.3 Putting it all together
The PEARL framework requires reference task weights to facilitate information re-use across CL tasks. When pre-training is available (Algorithm D the pre-trained weights serve as reference task weights. In the absence of pro-training, the first task in CL is considered the reference task (Algorithm 2 For cach subsequent CL task, a copy of the reference task weights fine-tuned for a specified number of epochs (E₁) on the current. lask data using cross-entropy loss. We compute the corresponding layer-specific task vector as the element-wise difference between the fine-tuned weights and the reference task weights. Depending the backbone architecture, the task vectors computed for convolutional, point-wise convolutional, or linear projections for the key in sell-atlention, as applicable. The task vector is then decomposed using SV-Decomposition through Equation 2 Since task vectors entail additional information beyond the reference task weights, we model them using dynamic LoRA components. Equations 3 and 4 together define dynamic rank specific to the current task and layer under consideration. Given the rank k, matrices B and A are Equation 5 At times, this initialization may lead to local minima, resulting
initialized as described in suboptimal performance. Therefore, we re-initialize the LoRA matrices, along with any task-specific normalization and classification layers, before further fine-tuning on the current task using cross-entropy loss for another Eg epochs. This process is repeated for all target layers across the backbone network. After fine-tuning, these task-specific parameters are kept frozen.
Our CL model is trained simultaneously on both Class-IL and Task-IL settings, as their training regimes are identical. During inference in the Task-LL setting, 0, = {fer, for go is selected based on the given task-id t, and its output is inferred for maximum activation. As task-id not available in Class-IL inference, cach test image passes through every sub- network 6,, and their respective classifier (ge=) outputs are concatenated and then inferred for maximum activation. For more details on training and inference in Class-IL and Task-II.
settings, refer Sections D.2 and D.3
4 Results
Experimental settings: We evaluate PEARL across two distinct scenarios of CL: Class-IL and Task-IL, complete overview of which can be found in Appendix B.1 To facilitate this evaluation, consider a comprehensive array of baseline methods: (i) exemplar-basee based approaches that involve information reuse and task-specific adapters (e.g., TAMiL, AGILE), (ii) NECIL approaches (e.g., PASS, FeTriL, FeCAM), (iii) parameter-isolation approaches that operate within a fixed capacity (e.g., NISPA) and those that expand beyond it (e-g., PNN, PAE), and (iv) PEFT approaches for CL. (e.g., L.2P, C-LoRA, InfLoRA, SEMA, ADA). We also provide a justification for the choice of these baseline in Appendix B.2 The first three categories utilize randomly initialized ResNet-18 18 as their backbone, whereas PEFT approaches leverage the ViT-Base-1 16 model pre-trained on ImageNet-21k. It is important to note that pre-training is often luxury for smaller, custom models. Consequently, in accordance with the training protocols of the bascline methods, we employ randomly initialized convolutional models, while the VIT model is pre-trained. To demonstrate the versatilic ility of PEARL across different architectures, we explore three distinct backbones: (i) Convolutional networks such as RosNet-18 (R18) and a ResNet-18-like architocture incorporating Blucprint Separable Convolutions (BSC) (Li et al. 2022b), and (ii) the Vision Transformer (ViT) Vit-Base-16 pre-trained on ImageNet- For each subsequent. task, we adapt convolutional layer, point-wise convolutional layer and linear layer projecting key

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Comparison of Class-IL performance of exemplar-based and NECIL approaches across different datasets and number of tasks. Baselines use randomly initialized ResNet-18 as backbone. Exemplar- -eampla--basdd approaches use buffer size of 200. NECIL results are taken from Goswami al. (0244)) while for the rest, report an average of 3 runs. Best results are marked in bold while second best are underlined.
Seq- CIFARIOO
Soq-Tiaylmaggele
Method
Venue
T
5
T
10
T
5
T
10
A2%
A%
Ar%
A%
Ay%
A%
Ar%
A%
Sequential fine-tming
17.94
37.54
9.43
25.07
14.41
32.83
7.92
31.55
ER
21.43
41.45
14.38
35.13 13
14.81
33.77
8.51
23.62
DER++
Buzzega et al.
2020
NeurIPS 2020
28.8
47.89
25.42
43.94
17.75
33.89
10:15
27.26
TAMIL
Bhat
2023
ICLR 2023
41.43
58.09
32.20
49.87
26.32
43.91
20.46
40.94
ACILE
Bhat ct
2024
CoLLAs 2024
45.73
57.66
33.23
42.88
20.14
43.79
20.19
43.81
LWF
& Hoicm
2017
BCCV 2016
45.35
61.94
26. 14
16.14
38.81
49.70
27.42
38.77
T12A
Zhu el al.
02 m
NeurIPS 2021
31.70
48.40
25.30
42.01
PASS
Zhu eb nl.
202 1b
CVPR 2021
49.75
63.39
37.78
52.18
36.44
48.64
26.58
38.66
SSRE
Zhu et al.
2022
CVPR 2022
42.39
56.57
29.44
44.38
30.13
43:20
22.48
34.93
FeTHIL
Pobat ct al
2023
WACV 2023
45.11
60.42
36. 36.69
52.11
29.91
43.99
23.88
36.35
FeCAM
GGoswaml et al.
2024a
NeurIPS 2024
47.28
61.37
33.82
48.58
25.62
39.85
23.21
35.32
PEARL (R18))
51.97
63.51
41.11
56.21
40.26
51:27
35.22
47.95
PEARL (BSC)
54.48
65.10
45.92
59.22
41.31
52.32
36.45
48.64
in self-attention using dynamic low-rank adaptation in each of these backbones respectively, More information on backbones and their settings can be found in Appendix D
Metrics: In line with the established benchmarking protocol, we assess the model's performance using A,, reprosenting the accuracy following the 7-th training stage. Notably, we utilize Ar, the performance metric at the conclusion of the final stage, along with A= + E Ar, which computes the average accuracy across all incremental stages, Additionally, we also provide forgetting metrics for our models in Table [I
4.1 Comparison with convolutional CL approaches
Table 1 presents comparison of PEARL against exemplar-based and NECIL approaches across four different Class-IL settings. We report results for both of our convolutional variants. Since the backbone is randomly initialized without pre-training, PEARL convolutional variants use the first CL task as the reference task and assign LoRA components for subsequent tasks based on their proximity to the first task in the parameter space (Algorithm 2. As shown, PEARL (R18) outperforms all considered baselines without relying on experience rehearsal while exhibiting modest model growth with an increasing nunber of tasks (see Table Although larger buffer size might enhance the performance of rehearsal-based approaches, any improvements would likely be outweighed by the increased computational and memory costs (Vijayan et al., 2023b). Additionally, NECIL approaches do not always re-use information across tasks, which often leads to subpar performance. In contrast, PEARL capitalizes on the information contained in the first task for every subsequent task, achieving an optimal balance between performance and model size. While the absence of pre-trained model may limit the generalizability of the first task, the framework's 's ability to rense information and dynamically allocate resources allows PEARL to efficiently assimilate information from both the first and current tasks, resulting superior performance across tasks.
Likewise, PEARL (BSC) outperforms all baseline methods, including PEARL (R18) in all Class-IL IL settings. PEARL (BSC), convolutional variant that employs separable convolutions, is significantly smaller in size compared to other basclines due to its paramcter-cfficiont design. In spite of its modest size, the scparation of point-wise and depth-wise operation enables elficient integration of information in BSCs with little room for redundant/ / interfering information. Coupled with dynamic rank allocation for LoRA components for point-wise filters, PEARL (BSC) enjoys higher performance across all Class-IL. scenarios. The trend remains almost similar in Task-IL scenarios (Refer Table 13 as well with PEARL variants outperforming considered baselines by huge margin.
7

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Class-IL performance comparison on ImageNet-R across 5, 10, and 20 task sequences. Baseline results with standard deviation are taken from Liang & Li (2024) and that of PEARL are averaged over 3 runs with errors indicating standard deviation across these runs. The rest of the results are taken from their respective manuscripts. Best results are highlighted in bold. SeqLoRA is sequential learning baseline which fine-tunes LoRA components multiple tasks without any additional support.
Method
Venue
==5
T
%
Ax%
4%
4x%
4%%
SeqLoRA
70.96 + 0.25
79.14 + 0.32
64.32
0.09
74.78
0.29
56.98
0.29
69.29
0.26
122
Waang al. 2022c
CVPR 2022
64.13 + 0.78
68.66 + 0.41
62.54 + 0.24
67.98
0.27
57.92 + 0.28
64.37
0.29
DualPrompt Waing
2022d
ECCV 2022
67.88 + 0.17
71.16 + 0.31
65.41 = 0.52
09:30
+ 0.43
61.00 + 0.72
65.80 + 0.67
CODA.P (Smith oi al
20233
CVPR 2023
73.09
0:21
76.91
1: 0.21
71.47
0.35
75.82
0.29
67.25
0.30
72.34
0.17
C-LoRA Smith Bl:
20233
CoRR 2023
75.85
0.31
78.85
0.34
71.80
0.45
75.33
0.28
65.71
0.60
70.63
0.85
LAE Gao A..
ICCV 2023
73.84
0.14
77.29
0:45
71.70
0.39 75.65
76.71
0.10
66.98
0.35
73.72 0.05
InfLoRA
Liong
2021
CVPR 2021
77.52 + 0.37
82.01 + 0.12
0.11
80.82
0.21
71.01 + 0.45
77.28
0.15
EASE Zhou ol al.
2024b
CVPR 2024
70.80
T3.37
78.80
SEMA Wang es nl.
2025
CVPR 2025
79.78
84.75
78.00
83.56
74.53
81.75
DIA Li al. A7/2
CVPR 2025
79.03
85.61
76.32
83.51
TSVD
Peng
2025
ICLR 2025
73.58
74.35
79.13
PEARL (VIT)
91.97 +: 0.37
93.78 # 0.36
88.76 +: 0.69
91.67 0.69
81.32 0.79
86.75 0.39
4.2
Comparison with PEFT CL approaches
As PEARL entails dynamic allocation of LoRA components, we compare PEARL with state-of-the-art PEFT CL methods across 5, 10, and 20 tasks in ImageNet-R (Hendrycks et al. 2021), in Table 2 The experiment is conducted using the ViT-B/16 backbone (Dosovitskiy et al.] 2021) pre-trained ImageNet-22k.. As can be seen, PEARL (ViT) shows significant improvement in both incremental and final accuracy, with absolute gains ranging from 9% to 14%. The performance improvement is more pronounced in 5 task sequence and slightly decreases thereafter as we move towards 10 and then to 20 task sequences. Among considered baselines, InfoLoRA is the closest work in terms of design and performance, but falls short of PEARL (ViTy's ' performance by a huge margin across all scenarios. InfLoRA injects a small number of parameters to re- re-parameterize the pre-trained weights and shows that finetuning these injected parameters is equivalent finetuning the pre-trained weights within subspace. Since the reduction matrix B infLoRA is handcrafted rather than being learnt from the training data, InfLoRA suffers performance degradation compared to PEARL (ViT). Additional comparisons wetween our method and adapter-based as well as prompi-based approaches can be found in Table 14 and Table 15
As LoRA components can be plugged in place of any learnable layer, it is imperative to understand the impact of such a low-rank adaptation for different kinds of target layers in ViT. To this end, Table 3 provides an overview of Class-IL performance on ImageNet-R (10T) for different target modules in PEARL (VTT). Our ablation study suggests that adapting linear layers projecting key in teltiontention yields the best performance across tasks. Adapting query modules stands at second when considered individually and third overall. Adapting values, either in isolation or in combination with other target modules, yields subpar performance. Although a combination of query and key is on par with the best results, it entails higher memory footprint when training on longer task sequences. Therefore, throughout our PEARL (ViT) experiments, we apply task-specific LoRA components only to key modules.
4.3 Comparison with fixed rank as a baseline
The PEFT CL approaches offer lightweight training schedules while effectively retaining the consolidated knowledge of pre-trained models. However, LoRA components in these methods are particularly sensitive to rank selection, which can lead to suboptimal resource allocation and diminished performance (Valipour et al.) 2022). Table 4 summarizes the average model size and performance metrics across various ranks for LoRA components. As the rank increases, performance improves at the expense of increased model size; however, beyond a certain rank, the performance gains do not really justify the additional increase in model size. Consequently, tuning the rank is essential to strike the right balance in this trade-off where model growth
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Class-IL performance comparison of low rank adaptation different target modules in ViT on ImageNet-R (10T). The best results are highlighted in bold while the second best are underlined.
Target LoRA Modules
Ar (%)
À (%)
Query
87.43 + 0.41
90.85 = 0.17
Key
88.76 + 0.69
91.67 + 0.69
Value
11.32 2.21
20.50 + 3.15
Query & Value
23.96 0.80
38.09 2.74
Key & Value
32.38 + 0.51
47.13 + 4.17
Query & Key
88.74 + 0.50
91.71 + 0.42
remains modest with optimum performance. To this end, PEARL employs a task-specific, layer-specific dynamic rank allocation enabling optimal trade-off between model size and performance across different. datasets and architectures.
Table Class-IL performance comparison PEARL with static rank allocation for LoRA components. Bold and underline highlight the nearest upper and lower values compared to PEARL. # #Params indicate no.of parameters in millions. As can be seen, PEARL achieves best trade-off between model size and performance without extensive tuning of rank.
R18
BSC Sec- Ingggeeee (10T)
VIT
Rank
CIFAROO (ST)
IntageNet-R (I0T)
Parraax (M)
Ap (%)
(%5)
#Parame (M)
(%)
(X)
#PParans (M)
Ay (%)
A (5))
11.44
45.50
0.53
68.61
0.78
1.86
32.54 1: 0.59
45.33
0.94
91.17
84.50
0.39
89.00
0.12
11.63
19.00
1:000:0000 0.43
61.45 1 0.62
1.98
38.50 4: 0.64
46.3 16.3.100 0.72
91.54
86.18
0.64
90.05 1: (1.41
12.01
92.28
87.52
16
12.74
52.18
0.56
63.34 + 0.62
2.78
36.50
0.15
49.05
+ 0.37
93.76
88.27
0.50
91.46
0.09
02
14.21
32.96
63.76
0.91
37.26
0.29
49.74
0.86
96.71
0.61
92.00
0.21
Dymanic (Ours)
12:36
31.97
0.93
63.51
0.75
2.30
36.45 1. 0.54
48.61 1: 0.43
98.27
88.76
0.69
91.67
0.69
4.4 Comparison with convolutional parameter isolation approaches
Figure [ illustrates a Task-IL comparison between convolutional PEARL variants and various parameter isolation techniques (PNNs, CPG, PAE) well as dynamic sparse methods (CLIP, NISPA, PackNet) on the Seq-CIFAR100 (20T) dataset. The final Task-IT. accuracies after completing the training on all tasks are presented. Parameter isolation methods tend exceed the model's capacity, whereas dynamic sparse architectures develop (ask-specific masks while maintaining fixed model size. PEARL effectively balances these approaches by employing dynamic resource allocation; it expands beyond the model's capacity, albeit more conservatively than other parameter isolation techniques. The results indicate that both convolutional variants of PEARL deliver superior performance across tasks. We attribute this success to effective information reuse and the appropriate allocation of new resources through dynamic resource allocation.
4.5
With and without re-initialization
The PEARL framework entails creation of task-specific, layer-specific LoRA components whose weights are initialized from SV-decomposition of task vectors. As task vectors represent the additional information on top of information contained in the reference task, one would expect LoRA components benefit from such initialization. Table compares the effects of weight re-initialization of LoRA components, BN layers (if any), and classification layers following SV-decomposition in PEARL. In the Without re-init scenario, the weights of the LoRA components are initialized directly from the SV-decomposition of task vectors and are then fine-tuned for the current lask. Conversely, in PEARL, the weights of the LoRA components, as well as the batch normalization (BN) and classification layers, are re-initialized after the creation of the LoRA components. The results indicate that weight re-initializatinn leads to marginally better performance for ResNet-18 on Seq- CIFAR110 (5T), statistically significant performance improvements for ViT on ImageNet-RR (10T), and remarkable two-fold increase in performance for BSC on Seq-Tiny ImageNet. (10T). Overall, the

---

Published in Transactions on Machine Learning Research (01/2026)
Seq-CIFAR100 (20 Tasks)
100
Top-1 (%) Accuracy
90
80
70
60
50
PEARL (R18) 87.19%
NISPA 75.87%
PNN 77.33%
PAE 77.6%
PEARL(BSC)- - 40
CLNP 66.68%
PackNet 67
47%4
CPG 80.89%
1
3
5
6 8
10
11
12
13 14
15
16
17 18 19 20
Tasks
Figure 1: Task-LL performance comparison with parameter isolation approaches on Seq-CIFARIO0 (20T). We report the final Task IL accuracies of each task after training on all tasks. Methods compared are NISPA (Gurbuz Dovrolis, 2022), PNN (Rusu et al., 2016), PAE (Hung al. 2019b), CPG (Hung et al. 2019a PackNet (Mallya & Lazebnik 2018), and CLNP (Golkar ct al.] 2019)
evidence suggests that weight re-initialization positively influences PEARL across various CL archilectures and settings. We speculate that fresh start due to weight re-initialization helps in avoiding local minima and thus better generalization across tasks.
Table 5: Class-IL performance comparison of PEARL with and without weight re-initialization after creation of LoRA components.
Dataset
Variant
Metric
Without re-init
With re-init (Ours)
Ar (%)
51.34 0.71
51.97 + 0.93
CIFACOO (5T)
R18
A (%)
62.37 + 1.20
63.51 + 0.75
Ay (%)
18.71 + 0.79
36.45 + 0.54
Seq- Dog-Tinylmageeet (10T)
BSC
A (%)
29.25 1 0.77
48.64 1 0.43
Ay (%)
78.71
0.38
83.51 + 0.47
ImageNet-R (10T)
ViT
Å (%)
84.79 + 0.63
88.30 + 0.48
Dynamic Threshold
Selected Rank
Actual Rank
25
PEARL (BSC)
1500
PEARL (BSC)
Dynamic threshold
0.15
20
PEARL (R18)
PEARL (R18)
1000
Rank
15.
Rank
10
500
0.05
PEARL (BSC)
5
PEARL (R18)
0
0
0
5
10
15
0
10 Conv layer
15
0
10 Conv layer
15
Conv layer
Figure 2: Visualization of final task dynamic resource allocation in Seq-CIFAR100 (5T). (left) Dynamic threshold of different convolutional layers based on the similarity of current task with respect to reference task for deciding the rank. (middle) Selected rank based on dynamic threshold. (right) Actual rank of re-shaped 2-dimensional weight matrix. As can be seen both dynamic threshold and selected rank show more emphasis for later layers, in line with the actual rank.
10

---

Published in Transactions on Machine Learning Research (01/2026)
4.6 Dynamic rank allocation across layers
PEARL leverages reference lask weights and adaplively determines the rank of lask-specifie, layer-specific LoRA components based on the proximity of the current task to the reference task weights in the parameter space. Figure 2 illustrates the dynamic rank allocation across layers for convolutional variants for the final task in Seq- CIF-RI0 (5T) and that of ViT for the final task in Imagenet-R (10T) can be found in Figure 4 From left to right, Figure presents detailed overview of the layer-wise dynamic thresholds established based on task proximity, along with the corresponding selected rank and actual rank for our convolutional variants. Consistent with the actual rank, our framework allocates lower ranks to carlier layers and higher ranks to later layers. The dynamic rank allocation is also consistent with established conventions in CNN literature, which suggest that carlier layers capture more generic features, while later layers focus on task-specific information. As a result, earlier layers require less task-specific adaptation, leading to lower rank allocations. Conversely, as we approach the later layers, more resources are necessary to adapt the reference task information to the current task, thereby justifying the higher rank allocation these layers.
4.7 Effect of pre-training on PEARL
To demonstrate the versatility of the PEARL framework, we assessed its performance on two architectures: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Given that ResNet- 18 is a widely used model convolutional CL literature, we employed a randomly initialized ResNet-18 to ensure a fair comparison with baseline models. However, the performance of PEARL (R18) is often limited by the generalization ability of the features captured in the first task. Table 6 provides comparison of PEARL (R18) with (Algorithm D and without (Algorithm 2 pre-training. Thanks to the versatility of PEARL, the framework can work with or without pre-training. As can be seen, pre-training on Imagenct- 1K results in superior performance across tasks and across datasets. Wherever possible, we recommend using a pre-trained backbone while employing PEARL for better generalization across tasks.
Table 6: Class-IL performance comparison of PEARL (R18) with and without pre-training. In the former case, PEARL (R18) is pre-trained on M.n Each result is an average of 3 runs with standard deviation alongside. The best results are highlighted in bold.
Method
Seq- CIFAR100 (5T)
Seq TinylmageNet (10T)
A7%
A%
Ar %
A%
Without pre-training
51.97 + 0.93
63.51 + 0.62
35.22 = 0.27
47.95 + 0.34
With pre-training
55.34 + 0.25
65.97 + 0.44
46.15 + 0.54
58.50 + 0.71
4.8 Effect of random class ordering
The Table 7 displays the Class-IL performance on ImageNet-R. (10T) under five different class orders, which differ from those presented in Table 2 In this setting, we randomize class assignments across tasks by setting different random seeds, allowing us to assess the impact of class ordering on PEARL. Although the final Class-IL accuracy (Ar%) is slightly lower than the previously reported result of 98.76 = 0.69, the results remain consistent. and robust. across these new class orders, highlighting the method's 's stability. Furthermore, both incremental and final accuracies show stability across varying class orderings, reinforcing the reliability of the approach. Even across all different class orderings, PEARL (ViT) consistently outperforms the considered baselines in Table 2 by huge margin. In conclusion, the findings indicate that PEARL maintains robust performance and generalizability across varying class orders, affirming its reliability effective framework for CL, even in randomized conditions.
4.9 Growth in model size
Table presents a comparison of parameter growth between PEARL and the complete parameter isolation method. As mentioned earlier, complete parameter isolation involves 100% resource allocation, meaning
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: Class-IL performance comparison on ImageNet-R (10T) across different class orders. Each class order indicates a random mix of classes across tasks. As can be seen, random class ordering has minimal impact on the functioning of PEARL (ViT).
Class Order
# Params (Millions)
A7%
5%
1
92.78
86.63
91.24
2
93.46
86.76
92.24
3
93.41
86.50
91.54
4
93.26
86.73
91.69
5
92.83
86.86
91.77
Average
93.14
86.70
91.70
each task utilizes a separate backbone network. We regard the complete parameter isolation method as an upper bound in terms of both performance and model growth. This comparison is especially relevant because PEARL derived from complete parameter isolation principles. However, it distinguishes itself through efficient information reuse and dynamic resource allocation, allowing for effective management. of model growth. The results show that PEARL requires significantly fewer parameters than the complete parameter isolation method, highlighting its efficiency. Morcover, PEARL demonstratos scalability, maintaining strong performance even in the context of longer task sequences. This positions it as more resource-effcient solution for continual learning tasks compared to traditional parameter isolation approaches.
Table 8: Comparison of parameter growth (in Millions) in PEARL with respect complete parameter isolation in different number of tasks.
Dataset
Backbone
Method
5 Tasks
10 Tasks
20 Tasks
Complete Param. Isolation
55.89
111.73
223.42
R18
PEARL
12.56
13.27
13.55
CIFARIF
Complete Param. Isolation
6.32
12.60
25.14
BSC
PEARL
1.63
1.89
2.30
Complete Param. Isolation PEARL
452.86
905.72
1,811.44
ImageNet-R
ViT
92.46
93.27
95.35
5 Conclusion
We proposed PEARL, rehearsal-free, paramnter-isolation-besss PEFT CL framework that mitigates catastrophic forgetting through dynamic resource allocation. Specifically, PEARL leverages reference task weights and adaptively determines the rank of task-specific, layer-specific LoRA components based on the current task's proximity to reference task weights in parameter space. With our dynamic threshold criterion, a close proximity would result in a lower rank allocation and maximum information re-use while a distant proximity results in a larger rank selection. To demonstrate the versatility of PEARL, we performed extensive evaluation with three different architectures on a multitude of CL scenarios and showed that these variants outperform all considered baselines by significant margin. Extending our work to more realistic settings such general CL scenarios where task boundaries not. known al training time, and exploring alternatives for efficient estimation of task vectors without the need to fine-tune the entire pre-trained model are some of the useful rescarch directions for this work.
References
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. Xix preprint XXi::220000889 2022.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Prashant Bhat, Bharath Renjith, Elahe Arani, and Bahram Zonooz. Mitigating interference in the knowledge continuum through attention-guided incremental learning. Third Conference on Lifelong Learning Agents (CoLLAs), 2024.
Prashant Shivaram Bhat, Bahram Zonooz, and Elahe Arani. Task-aware information routing from com- mon representation space in lifelong learning. In The Eleventh International Conference on Learning Representations, 2023.
Dan Biderman, Jose Javier Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Comnor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. CoRR, 2024.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.1416. 2020.
Pictro Buzzoga, Mattco Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33:15920-15986 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky, New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9516 9525, 2021.
Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip Torr. Continual learning in low-rank orthogonal subspaces. Advances in Neural Information Processing Systems, 33:9900-999 1900009911,111 2020.
Rajas Chitale, Ankit Vaidya, Aditya Kanc, and Archana Santosh Ghotkar. Task arithmetic with loRA for continual learning. In Workshop on Advancing Neural Network Training: Compulational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023), 2023.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEFF transactions on pottern analysis and machine intelligence, 47(7):3366 3385, 2021.
Gregory Ditzler, Manuel Roveri, Cesare Alippi, and Robi Polikar, Learning in nonstationary environments: A survey. IEEE Computational Intelligence Magazine, 10(4):12 25, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9285-9295, 2022.
Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Códric Archambeau. Memory efficient. continual learning with transformers. In Proceedings of the 36th International Conference on Newral Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259 3269. PMLR, 2020.
Qjankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-effict tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) pp. 11483-11 11493, 2023a.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameteer-fficient tuning. In Proceedings of the TEEE/ /CVF International Conference on Computer Vision, pp. 11449 11459, 2023b.
Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28463 28473, 2024.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arxiv preprint WwXii:1903.04470 2019.
Dipam Goswami, Yuyang Liu, Bartlomicj Twardowski, and Joost van de Weijer. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. Advances in Newral Information Processing Systems, 36, 2024a.
Dipam Goswarn,, Albin Soutif- Cormmr-o Yuyang Liu, Sandesh Kamath, Bart Twardowski, Joost van de Weijer, et al. Resurrecting old classes with new data for exemplar- frree continual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 28525 28534, 2024b.
Mustafa B Gurbuz and Constantine Dovrolis. Nispa: Neuro-inspired stability-plasticity adaptation for continual learning in sparse networks. In International Conference on Machine Learning, pp. 5157-8174 PMLR, 2022.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiienn transfer learning for nlp. In International conference machine learning, pp. 2790-2790 PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106 0666566...66.6 2021.
Ching-Y: Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting contimual learning. Advances in Neural Information Processing Systems, 32, 2019a.
Steven CY Hung, Jia-Hong Lee, Timmy ST Wan, Chein-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Increasingly packing multiple facial-informaties modules in a unified deep-learning model via lifelong learning. In Proceedings of the 2019 on International Conference on Multimedia Retrievol, pp. 339-343, 2019b.
Gabriel Iharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. Advances in Neural Information Processing Systems, 35:29262- -92777, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with lask arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
Kishaan Jeeveswaran, Prashant Bhat, Bahram Zonooz, and Elahe Arani. Birt: bio-inspired replay in vision transformers for continual learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 14817-14835, 2023.
Muhammad Gulzain Ali Khan, Muhammad Ferjad Naeem, Lue Van Gool, Federico Tombari, Didier Stricker, and Muhammad Zeshan Afzal. Introducing language guidance in prompt-basod continual learning. In International Conference on Computer Vision 2023. International Conference on Computer Vision (ICCV. 2023), October 2-6, Paris, France. Computer Vision Foundation, IEFE, 10 2023.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representalions, ICLR 2015, Son Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barrinsk,,,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):352) -5226, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Muhammad Rifki Kurniawan, Xiang Song, Zhiheng Ma, Yuhang He, Yihong Gong, Yang Qi, and Xing Wei. Evolving parameterized prompt memory for continual learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(12):13311113300 133399 Mar. 2024.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Jiashuo Li, Shaokun Wang, Bo Qian, Yuhang He, Xing Wei, Qiang Wang, and Yihong Gong. Dynamic integration of tpsk-specific adapters for class incremental learning, 2025.
Jin Li, Zhong Ji, Gang Wang, Qiang Wang, and Feng Gao. Learning from students: Online contrastive distillation network for general continual learning. In Proc. 31st Int. Joint Conf. Artiff. Intell,, pp. 3215-3221, 2022a.
Zheyuan Li, Yingqi Liu, Xiangyu Chen, Haoming Cai, Jinjin Gu, Yu Qiao, and Chao Dong. Blueprint separable residual network for efficient image super-resolution In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2022b.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935 2055,2977,4 2017.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. Tn Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 23638 236377 2024.
Jingren Liu, Zhong Ji, YunLong Yu, Jiale Cao, Yanwei Pang, Jungong Han, and Xuelong Li. Parameter-efficient fine-tuning for continual learning: A neural tangent kernel perspective. arXiv preprint arXie:2407.11190
2024a.
Shih- Yang Liu, Chien- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposod low-rank adaptation. arXiv preprint orXiv:2402.099: 09300953, 2024b.
Haodong Lu, Chongyang Zhao, Minhui Xue, Lina Yao, Kristen Moore, and Dong Gong. Adaptive rank, reduced forgetting: Knowledge retention in continual learning vision-language models with dynamic rank-selective lora. CoRR, 2024.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. Tn Proceedings of the IEFE conference on Computer Vision and Pottern Recognition, pp. 7765-7774 2018.
Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu. DoRA: Fnhancing parfineterefffiienn fine-tuning with dynamic rank distribution. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics ( (Volume 1: Long Papers), pp. 16662-16755, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
Mare Masana, Xialci Liu, Bartlomicj Twardowski, Mikel Menta, Andrew D) Bagdanov, and Joost Van De Weijer. Class-inen incremental learning: survey and performance evaluation on image classification. IEEE Transactions Pallern Analysis and Machine Intelligence, 45(5):5513 55335, 2022.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Liangzu Peng, Juan Elenter, Joshua Agterberg, Alejandro Ribeiro, and Rene Vidal. TSVD: Bridging theory and practice in continual learning with pre-trained models. In The Thirleenth International Conference Learning Representations, 2025.
Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. Fetril: Feature translation for exemplar-free intremenrlllttll learning. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3911-3920, 2023.
Jingyang Qiao, zhizhong zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, and Yuan Xie. Prompt gradient projection for continual learning. In The Twelfth International Conference on Learning Represen- tations, 2024.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123 146, 1995.
Amurag Roy, Riddhiman Moulick, Vinay Verma, Saptarshi Ghosh, and Abir Das. Convolutional prompting meets language models for continual learning. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 100::606.000 04671, 2016.
James Seale Smith, Yen- Ben-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. CoRR, 2023a.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed iitention-based prompting for rehearsal-free free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11909-19913 2023b.
Yuwen Tan, Qinhao Zhou, Xiang Xiang, Ke Wang, Yuchuan Wu, and Yongbin Li. Semantically-Shife Incremental Adapter-Tuning is A Continual ViTransformer. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. Computer Sociely.
Los Alamitos, CA, USA, June 2024. IEEE
Mojtaba Valipour, Mchdi Rezagholizadch, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaplation. arXie preprint or arXiv:2210.074 0.05558 2022.
Mojtaba Valipour, Mehdi Resagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter-efficien tuning of pre-trained models using dynamic search-free low-rank adaptation. In Andreas Vlachos and Tsabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3287, Dubrovnik, Croatia, May 2023. Association for Computational Tinguistics.
Prectha Vijayan, Prashant Bhat, Elahe Arani, and Bahram Zonooz. Trire: multi-mechanism learning paradigm for continual knowledge retention and promotion. arXiv preprint 0ant22300082219 2023a.
Preetha Vijayan, Prashant Bhat, Bahram Zonooz, and Elahe Arani. Trire: A multi-mechanism learning paradigm for continual knowledge retention and promotion. In Advances in Neural Information Processing Systems, volume 36, pp. 73775-73792 2023b.
Huiyi Wang, Haodong Lu, Lina Yao, and Dong Gong, Self-expansion of pre-trained models with mixture of adapters for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition (CVPR), 2025.
Liyuan Wang, Jingyi Xic, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hicrarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 2023a.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint or 0rx::22302.000 00487, 2023b.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658-10677 2023c.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. Advances in Newral Information Processing Systems, 35: 5682-5695, 2022a.
Zifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan, Wei Niu, Tong Jian, Bin Ren, Stratis Toannidis, Yanzhi Wang, and Jennifer Dy. Sparcl: Sparse continual learning on the edge. Advances in Neural Information Processing Systems, 35:20366 20380, 2022b.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022c.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jemnifer G. Dy, and Tomas Pfister. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Proceedings of the European Conference on Compuler Vision, pp. 631-648, 2022d.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the TEFE/CV conference on computer vision and pattern recognition, pp. 139 149, 2022e.
Chengyi Yang, Wentao Liu, Shisong Chen, Jiayin Qi, and Aimin Zhou. Generating prompts in latent space for rehearsal-free continual learning. In ACM Mullimedia, 2024.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-effiiint fine-tuning for transformer- baadd masked longuage-models. arXiv preprint arXiv:2106. 10111990..1019, 2021.
Jingjie Zeng, Zhihao Yang, Qi Yang, Liang Yang, and Hongfei Lin. Peeling back the layers: Interpreting the storytelling of vit. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, pp. 7298-7306, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706868.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng Hc, Yu Cheng, Weizhu Chen, and Tuo Zhao Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
Da-Wei Zhou, Zin Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremeental learning with pre-trained models: Generalizability and adaptivity are all you need. Int. J. Comput. Vision, 133(3): 1012-1032, August 2024a. ISSN 0920-5691.
Da-Wei Zhou, Hai-Long Sun, Han-Jia Yc, and Do-Chuan Zhan. Expandable subspace ensemble for pre-trainod model-based class-incremenntl learning. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pattern Recognition, pp. 23554-23566 2024b.
Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. Metagpt: Merging large language models using model exclusive task arithmetic. arXiv preprint arXiv:2406.11335 11385, 2024c.
Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-lin lin Liu. Class-incremental learning via dual augmentation. Advances in Newral Information Processing Systems, 34:14306-14318, 2021a.
Fci Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self- supervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5881, 2021b.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self-sustaining representation expansion for non-exemplar inccemennrlltttll learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9296 9305, 2022.
A Limitations and Future work
We provided extensive analysis to exhibit the effectiveness of PEARL across multitude of datasets, settings and scenarios in CL. However, PEARL has several limitations: Firstly, as a parameter isolation approach, PEARL scales with the increasing number of tasks. Although this growth is limited, it still results in additional paramcters for each new task, lcading to larger model. Morcover, with every new task, cach image has to pass through every sub-network for Class-IL IL inference, resulting slower inference. Secondly, PEARL assumes knowledge of task boundaries, which are not always available in real-world scenarios. This assumption is common among parameter isolation approaches, as new parameters are typically introduced at the onset of each task. Thirdly, PEARL requires fine-tuning a full pre-trained model on the current task for several epochs. As model sizes grow, this may become impractical on resource-constranned devices. Finally, task vector initialization in PEARL assumes a precise measure of task proximity, but in practice, there can be multiple proximity criteria can be explored and validated. In our future work, we aim to explore alternatives for defining task boundaries and efficient estimation of task vectors without the need to fine-tune the entire pre-trained model.
B Additional Information
B.1 Overview of experiments
We provide a comprehensive overview of the experiments conducted to evaluate the performance of PEARL across various CL benchmarks. To demonstrate the versatility of the PEARL framework, we assess its performance on two architectures: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Given that ResNet-18 is widely used model in convolutional CL literature, employ randomly initialized extend our experiments
ResNet-118 to ensure fair comparison with baseline models. Furthermore, (BSCs). The decision to to include ResNet-11-like architecture with blueprint separable convolutions include a BSC-based network is driven by two key considerations: (i) As CL evolves, there is a growing demand for efficiont models that maintain a modest parameter size. BSCs significantly reduce the paramcter footprint compared to traditional convolutions, resulting more compact models; and (ii) By separating the traditional convolution operation into point-wise and depth-wise operations, BSCs allow us to illustrate the generalizability of our approach beyond conventional model architectures. Consistent with bascline methods, these convolutional models are not pre-trained and are initialized randomly. Since pre-training can be a luxury for small custom models, these experiments highlight that pre-training is not necessary for the effective application of PEARL. In these settings, we treat the first task as a reference task and proceed with parameter-efficient tne-tuning starting from the second task. We evaluate PEARL against three types of baselines: exemplar-basee methods, NECIL, approaches, and parameter isolation techniques. We present. results on standard CL benchmarks, including Seq- CIFAR1000 and Seq- Tog-Tiilllmggee,,, in both Class-IL and Task-IL settings.
Similar to PEARL, parameter-efficient fine-tuning (PEFT) approaches in CT leverage techniques to adapt pro-trained ViT models for new tasks. Consequently, we consider PEFT CL. approaches as primary basclines for our PEARL (ViT) version. In alignment with these baselines, we utilize the ViT-base-16 model, which has been pre-trained on the ImageNet- dataset. The larger size of ViT-base-1l 16 enables us to experiment with a more extensive datasct. Accordingly, we evaluate PEARL (VIT) on the rendered ImageNet datasct (ImageNet-R) in Class-IL setting across various task sequences.
B.2 Justification for considered baselines
To facilitate extensive evaluation of PEARL, we consider a comprehensive array of bascline methods: (1) exemplar-based approaches that involve information reuse and task-specific adapters (e.g., TAMIL, AGILE);
18

---

Published in Transactions on Machine Learning Research (01/2026)
Table 9: Summary of experiments.
Architecture
Model
Reference task
Setting
CL scenario
Baseline
Metric
Convolutional
R18
Seq- CIFAR100 (5T, 10T, 20T)
Class-IL
Exemplans based
BSC
Not pre-traiued
Seq. Mog-TTnllaageee (5T, 10T)
Task-II
NECIL
A (%) Ay (%)
Param isolation based
Transformers
VIT
Pre-trained
ImageNet-R (5T, 10T, 20T)
Class-IL
PEFT CL
(ii) NECIL approaches (e.g., PASS, FeTriL, FeCAM); (iii) irrameter-isolation approaches that operate within a fixed capacity (o.g., NISPA) and those that expand beyond it (c.g., PNN, PAE); and (iv) PEFT approaches for continual learning (e.g., L2P, C-LoRA, InfLoRA). Firstly, inspired by the GWT, exemplar- baselines entail shared network for information reuse and task-specific attention modules that re-project the representation from the common representation space to the global workspace. Likewise, the PEARL framework includes shared reference task network for information reuse and task-specific, layer-specific low-rank adapters for orienting towards the current task. Secondly, PEARL is a rehearsal-free framework; it does not store previous task samples either directly or indiroctly through generative networks. Therefore, it aligns well with the NECIL benchmark and associated approaches. Thirdly, we note that low-rank adaptation introduces task-specific parameters that are trained on the correspouding task and kept frozen afterwards, akin to parameter-isslation approachos. As PEARL exhibits partial parameter isolation with shared reference lask network and low-rank adapters, we also provide comparison with parameter isolation approaches. Finally, since PEARL readily utilizes LoRA (Hu et al.) 2021), prominent PFFT technique, we offer an in-depth comparison with PEFT continual learning approaches. In addition, we also provide a discussion on related dynamic low-rank adaptation techniques beyond continual learning in Section B.
To the best of our knowledge, PEARL is the first dynamic low-rank adaptation framework tailored specifically for generic or custom-built small-scale models in continual learning. With a variety of the aforementioned baselines and analysis, we provide comprehensive understanding of PEARL across architectures, datasets, and continual learning scenarios.
B.3 Discussion on PEFT LoRA counterparts
PEFT CL approaches with static rank allocation: Among PEFT continual learning approaches, InfLoRA (Liang & Li 2024 and C-LoRA (Smith et al. 2023a] are prominent techniques that involve low-rank adaptation of pre-trained networks, similar to PEARL. Each of these methods are tailored for CL scenarios and addresses unique challenges. InfLoRA aims to minimize the interference of new tasks on previously learned ones, thereby improving knowledge retention. However, its reliance on a handerafted reduction matrix B, rather than a data-driven learned representation, limits its flexibility. C-LoRA, on the other hand, depends on pre-trained model as its foundation, making it reliant on the availability of such models for its application. It applies LoRA to both the key and query attention mechanisms, ensuring efficient adaptation for new tasks. Additionally, C-LoRA incorporates a regllarguliiinnion strategy, which penalizes updates to parameters already modified by previous tasks. This mechanism helps preserve prior knowledge and enhances stability across sequential tasks. Despite its relative success, caution is warranted when training C-LoRA on large task sequences, as this approach suffers from scalability issues. In line with standard practices in continual learning benchmarks, we provide results for SeqLoRA, a sequential finetuning baseline without. any safeguards against catastrophic forgetting. SeqLoRA experiences significant forgetting, underscoring the difficulty of maintaining performance across tasks in continual learning.
The PEARL framework proposed in this paper entails dynamic rank allocation for LoRA components, making it efficient and scalable even with longer task sequences. Secondly, duc to efficient dynamic resource allocation, all LoRA parameters are learnable through gradient descent. Unlike InfLoRA, this flexibility allows PEARL to capture the nuances of the corresponding task more effectively without any hindrance. With regard to C-LORA, PEARL does not entail self-rogularization to bring forth stability and consoquent reduced task interference. We speculate that task dynamics as played out in the parameter space allow PEARL to counteract any task interference, thereby effectively mitigating forgetting across CL. tasks. Overall, we find
19

---

Published in Transactions on Machine Learning Research (01/2026)
that dynamic resource allocation coupled with the flexibility to learn task muances based on task proximity has endowed PEART with superior performance across multitude of CI scenarios.
Parameter efficient fine-tuning techniques such as LoRA distribute trainable parameters equally across all layers, which may reduce efficiency when layers vary in their contribution towards task adaptation. To address this, adaptive low-rank adaptation strategies have emerged that dynamically adjust parameter budgets to improve fine-tuning efficiency, similar to PEARL. AdaLoRA (Zhagg et al., 2023) adaptively allocates the parameter budget among weight matrices according to their importance score. The importance score is computed by parameterizing the incremental updates in the form of SV-decomposition and the singular values of unimportant updates are pruned. DyLoRA (Valipour et al. 2023) proposed a dynamic low-rank adaptation in which LoRA blocks are trained for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. More recent methods, such as DoRA (Mao et al., 2024) and Dynamic Rank-Selective LoRA (Lu) ct al. 2024), have emerged, with DoRA decomposing high-rank LoRA layers into structured single-rank components, while Dynamic Rank-Selective LoRA adaptively assigns ranks to LoRA modules based on their relevance to the current data in CL in vision language models. Although these approaches address similar challenge as PEARL, these are not tailored for CL in custom, compact models.
PEFT approaches with dynamic rank allocation:
AdaLoRA uses SVD to measure the importance of each matrix component, pruning lower singular values. However, this adaptive adjustment introduces a more complex implementation compared to PEARL, and potential instability if rank adjustments are too frequent. DyLoRA addresses the limitations of static rank selection by training LoRA blocks with a range of ranks, randomly sampling a rank during each training iteration and optimizing the model for that rank. Unlike DyLoRA, PEARL does not require setting a predefined rank range before training, offering greater flexibility during training. DoRA introduces a decomposition strategy that splits high-rank LoRA layers into multiple single-rank components, with dynamic rank allocation and a regularization penalty to ensure stable pruming. However, DoRA does not result in significant reduction in mumber of parameters compared to high-rank LoRA allocation. Dynamic Rank-Selective LoRA introduces dynamic importance weights, represented by a learnable vector, which assigns importance to each rank during training. These weights, inspired by the diagonal elements in the singular value matrix of SV-Decomposition, are learned directly from the data while promoting sparsity by minimizing the & norm of the importance weights. However, Dynamic Rank-Selective LoRA directly merges parameter updates into the pre-trained weights without explicitly considering correlations between tasks. Overall, the three former approaches are not specifically tailored for CL and lack results for direct comparisons with PEARL variants. Although Dynamic Rank-Seloctive LoRA focuses on CL, it is tailored for vision-language models, limiting the possibility of a direct comparison with PEARL. To the best of our knowledge, PEARL one of the carly approaches tailored specifically for efficient custom, compact CL.
B.4 Potential sensitivity of PEARL
PEARL is quite sensitive to the stability of task vector and noise contained in the resultant SVD decomposition thereof. In cases where task vectors are poorly conditioned, the dynamic rank estimation can be imprecise resulting in over or under allocation of dynamic rank. Figure Bb) provides a depiction of resource allocation dynamics for ImageNet-R (10T) using PEARL (ViT) across three distinct runs. We note that PEARL relies solely on the proximity of a task relative to the reference task weights in parameter space. Tt is important to clarify that task proximity, as defined in PEARL, does not correspond to semantic similarity between tasks. Different random seeds can alter the training trajectory, thereby changing the proximity of tasks relative lo the reference task. As a result, the distribution of parameter allocation can vary significantly across tasks in different runs.
B.5 Relation to Orthogonal-Subsppe Methods
Orthogonal-ssusppace approaches (e.g., Wang et al. [023c]; Chaudhry et al. (2000) address catastrophic forgetting in CL by enforcing goometric constraints that keep updates for new tasks orthogonal to subspaces associated with previous tasks, typically using SVD-based decompositions or orthogonalized gradients. These methods effectively reduce interference but require explicit construction, storage, and maintenance
20

---

Published in Transactions on Machine Learning Research (01/2026)
of task-specific subspaces, causing computational and memory costs to grow with the number of tasks. Moreover, they operate in the full parameter gradient space, which can be expensive and less practical for large modern architectures. We compare against this line of work because it shares the core objective of preserving prior knowledge while providing task-specific flexibility, making a natural conceptual baseline for evaluating PEARL. In contrast. to explicit orthogonality constraints, PEARL works entirely within the parameter-officiet LoRA space and introduces dynamic rank allocation based on task similarity, allowing it adapt capacity without building or maintaining orthogonal subspaces. This results in more scalable and lightweight mechanism for mitigating interference, while still offering the ability to adapt to diverse tasks in rehearsal-free continual learning setting.
C Additional Analysis
C.1 Comparison with complete parameter isolation
At its core, PEARL functions parameter-isolation CL framework that dynamically allocates resources based on the current lask's proximity to the reference lask in the parameter space. The theoretical upper bound for PEARL represents a model with 100% resource allocation for each task. To illustrate this, we compare PEARL with the Complete Parameter Isolation method, which entails a separate backbone for each task. For instance, in Scq-CIFAR100 (5T), the Complete Parameter Isolation method would necessitate a distinet ResNet-18 backbone for each task, enabling maximum resource allocation and complete parameter isolation. Table 10 provides comparison of PEARL against complete parameter isolation with maximum resource allocation. Across settings and architectures, PEARL stays close to the performance of complete parameter isolation method while being extremely parameter efficient.
Table 10: Class-II (Ar%) performance evaluation of PEARL with respect to model with complete parameter isolation. We treat such as model as an upper bound for PEARL. Elficiency is computed as a ratio of Class-IL ())%) Vs #Params (M). As can be seen, dynamic rank allocation leaves minimal footprint while staying close to the performance of complete parameter isolation method.
Dataset
Backbone
Model
# PParams (M)
Class- IL (Ay %)
Task-IL (Az %)
Efficiency
Complete param isolation
55.89
53.57 + 0.66
80.69 0.54
0.96
R18
PEARL
12.56
51.97 + 0.93
80.04 10.54
4.14
Seq-CIFARION (5T)
Complete param isolation
59.22 0.29
83.82
0.19
9.37
BSC
PEARL
1.63
54.48 1: 0.35
81.55 1 0.71
33.42
Complete param isolation
111.79
38.5 0.32
73.54
0.53
0.35
R18
PEARL
12.32
35.22 + 0.27
71.49 + 0.22
2.86
BeeyTTnylmageeer (10T)
Complete param isolation
12.65
40.63 + 0.48
73.3 = 0:27
3.21
BSC
PEARL
2.3
36.45 t 0.53
70.95 + 0.44
15.84
Complete param isolation
905.72
87.58 1 0.49
97.28
0.19
0.09
ImageNet-R. (10T)
ViT
PEARL
93.27
88.76 + 0.69
97.19 - 0.18
0.95
C.2 Forgetting analysis
We provide additional metrics our experiments comprchensively evaluate the performance of PEARL. Table II includes metrics such forgetting, stability, and plasticity, which offer valuable insights into the model's behavior. It is noteworthy that most baseline models do not report these metrics, limiting our ability to compare their performance across these dimensions. Forgetting measures the model's ability to retain knowledge from previous tasks by calculating the average decline in accuracy for task at the conclusion of CL training relative to its initial accuracy; lower forget values indicale superior knowledge retention. Stability (S) reflects the average accuracy on previously learned tasks at the end of training, illustrating the model's s performance on carlier tasks. In contrast, plasticity (P) evaluates the model's capability to learn new tasks effectively, determined by the average accuracy during the initial training of those tasks. The trade-off between stability and plasticity is quantified by the formula 2xSxP which measures how well the
P+S
21

---

Published in Transactions on Machine Learning Research (01/2026)
method balances these two aspects. Together, these metrics offer holistic view of PEARL's performance, emphasizing its strengths and trade-offs in various Class-IT, scenarios. Across both Seq-CIFAR100 (5T) and Mog-TinnllmgeNee (10T), PEARL (BSC) demonstrates superior performance and a more favorable trade-off between stability and plasticity compared to PEARL (R18). A notable exception is observed in the forgetting metric for CTFAR (5T), where PEARL (BSC) experiences slightly higher lorgetting. However, the enhanced stability and plasticity PEARL (BSC) effectively compensate for this increase in forgetting. On ImageNet-R (10T), the PEARL (ViT) method demonstrates strong performance across key continual learning metrics. It achieves high average accuracy (Ar) of 88. 76,,, excellent plasticity (91.67%), and minimal forgetting (3.63%), indicating well-balanced trade-off between stability and adaptability.
Table 11: PEARL Class-IL performance metrics across datasets.
Dataset
Method
Ar (%) t
Forgetting (%) 4
Stability (%)**
Plasticity (%) f
Trade-off
PEARL (R18)
51.97 + 0.34
13.34 = 0.39
51.19 + 0.68
62.64 + 0.90
56.34 = 0.77
CIFARI0 (5T)
PEARL (BSC)
54.47 + 0.27
14.47 = 0.34
53.21 + 0.41
66.06 + 0.54
59.01 = 0.32
PEARL (R18)
35.22 0.49
14.26
0.56
35.41 1 0.25
48.06 1 0.32
40.78
0.14
Beg-TinylmageNet (10T)
PEARL (BSC)
36.45 1 0.68
13.44 1 0.65
36.56
1 0.53
48.54
1 0.95
41.70
0.59
ImageNet-R (10T)
PEARL (VIT)
88.76 0.69
3.631 0.17
88.18 + 0.77
91.67 + 0.71
89.89
0.74
C.3 Hyperparameter Tuning
PEARL framework provides robust way to find the best tradc-off between model size and performance in PEFT CL. As such, our framework is devoid of any specific hyper-parameters. In order to provide a better picture of PEARL's performance landscape, Table I2 presents Class-IL (Ar%) performance for various learning rates. The best results for each datasct and model combination are highlighted in bold, providing a clear indication of the optimal settings. For Seq- CIFAR100 (5T), the results show that a learning rate of 0.001 yielded the highest performance for both PEARL (R18) and PEARI. (BSC). Notably, the performance of PEARL (R18) is better than its BSC counterpart in all other learning rates except 0.001 and 0.01. Therefore, when generalizing to datasets beyond those considered in this paper, we urge the readers to extensively tune learning rate of PEARL (BSC) for optimum performance. In the Seq-Tiny ImageNet (10T) scenario, the optimal learning rate for PEARL (R18) was found to be 0.0001, achieving an Ar of 35.22 35.22%. For PEARL (BSC), a slightly higher learning rate of 0.001 led to the best performance of 36.45%. PEARL (BSC)'s 's performance across different learning rates further confirms the susceptibility of PEARL (BSC) changing learning rates. For the ImageNet-R, (10T) dataset, the results demonstrate that the learning rate of 0.03 provided the best performance for PEARL (ViT), resulting in an Ar of 91.38%. As PEARL (VIT) is pre-trained on ImtgeNet-21k, it is more robust to changes in the learning rate.
Table 12:
Comparison of Class-IL performance (Ar (%)) for various learning rates. The best results for each dataset and model combination are highlighted in bold. The error terms indicate the standard deviation across 3 random runs.
Class II. (Ay%) for different learning ratrs
Dataset
Model
0.1
0.01
0.03
0.001
0.005
0.0001
PEARL (R18)
30.95 : 6.12
42.69 + 1.78
41.62 = 0.76
51.97 + 0.93
47.25 + 0,28
48.59 = 0.28
Seq-CIEAR110 (TT)
PEARL (BSC)
10.44 = 1.59
52.18 + 0.46
40.63 = 3.02
54.48 +: 0.36
47.60 + 0.53
45.53 = 0.48
PEARL (R18)
14.77 # 1.41
27.18 + 0.75
15 0.98
34.63 : 0.67
28.57 + 0.99
35.22 + 0.27
Seq TinyImageNet, (10T)
PEARL (BSC)
3.47 + 1.08
33.93 4 0.60
18.69
1.51
36.22 4 0.53
36.45 + 0.53
29.30 0.39
ImtgeNet-R (10T)
PEARL (VTT)
91.23 # 0.18
91.06 + 0.58
91.38
0.50
83.51 # 0.47
88,76 + 0.69
83.51 0.47
C.4
Comparison against CL approaches in standard benchmarks.
Table 13 presents a comparison of PEARL convolutional models in both Class-IL and Task-IL scenarios
for Seq- CIFAR100 (5T) and Seq-TTinyImageNet (10T). Rehearsal-based methods, including those that rely
22

---

Published in Transactions on Machine Learning Research (01/2026)
Table 13: A benchmark comparison with CNN-based prior works on Class-IL (A7% ) and Task-IL (A7% ) The best results are highlighted in bold while the second best are underlined. The error terms indicate a standard deviation across random runs. The rehearsal-basee baselines employ a buffer size of 200.
Seq-CIFAR100 (GT)
Seq- Beq-TinyImageNe (10T)
Method
Venue
Class IL (Ay%
Task-IL (Ay%
Class- IL (Ay% ) Task-IL (Ay%
JOINT
70.56 4:028
86.19 +0.43
59.99 10.19
82.01 10.10
SGD
17.49 +0.28
40.46 +00.9
7.92 +0.26
18.31 +0.68
PNNs (Rusu et al
2016
NeurIPS 2016
74.01 +1.11
67.84 #:0.29
ParkNel Mailya Lazebrik
2018
CVPR 2018
72.39 +-0.37
60.46 +:1.2
NISPA Curbuz & Dovrolis
2022
NeurIPS 2022
65,36 4 19
59.56 10.32
SparCI- LWC
Wang ot al 27226
NeurIPS 2022
59.53 +0.25
59.56 #0032
ER
21.40 +0.22
61.36 +-0.5
8.57 +0004
38.17 +-2.00
DER ++
Buzzega et al
2020
NeurIPS 2020
29.60 11.14
62.49 1100
10.96 1:.17
40.87 16
ER-ACE Cacoia et ml.
2022
ICLR 2022
35.17 11.17
63.09 11.23
11.25 +0.54
44.17=1.02
Co Chaa et. al. 2021
ICCV 2021
31.90 +:038
55.02 00.36
13.88 +:040
42.37 0.074
OCDNet
LL of a.. 2022
IJCAI 2022
44.29 40049
73.53 40.24
17.60 -0.97
56.19 11.31
TAMEL [Bhat of
ICLR 2022
41.43 440.75
71.39 +0.17
20.46 +0.40
55.44 +0.52
THRE Vijaau
20/24
NeurIPS 2023
43.91 10.18
71.66 = 0.44
20.14 10.19
55.95 10.78
AGILE
Baat
2021
CoLLAs 2024
45.73 10.15
74.37 +0.34
20.19 11.65
53:47 1160
PEARL (R18)
51.97 0.93
0.54
35.22 0.27
71.49 + 0:23
PEARL (BSC)
54.48 0.36
81.55 0.72
36.45 0.53
70.91
0.44
exclusively on experience replay (e.g., ER, DER++), exhibit subpar performance in both Class-IL and Task-IL contexts. In contrast, approaches that utilize one or more model surrogates (e.g., Co²L, OCDNet, TAMiL) demonstrate improved performance, which aligns with the increase overall model size. Parameter isolation methods (e.g. PNNs, PackNet) that require task-identity inference show great promise in Task-IL setting. However, these methods struggle in Class-IL scenarios due to challenges related to inter-task class separation. Conversely, both PEARL models consistently outperform all examined baselines across all scenarios. The robust performance across varied Task-IL scttings highlights the versatility and effectiveness of the PEARL framework, underscoring its potential as a leading solution in continual learning tasks.
C.5 Performance in longer task sequences
It is essential for CL models to excel in longer task sequences, as real-world applications often require them to adapt to a variety of tasks in succession. Figure showcases the Class-IL performance (A7%) of PEARL over extended task sequences of 5, 10, and 20 tasks in CIF-CIF Parameter isolation approaches, including PEARL, provide maximum stability by fixing parameters associated with previous tasks. Although there is no capacity saturation, the difficulty of distingui between larger set of classes, leading to decrease in performance. This is reflected in the steady drop in performance as we progress from 5 to 10 and then to 20 tasks, highlighting the challenges of maintaining knowledge retention in incrcasingly complex Class-IL task sequences.
the number of tasks increases, forgetting becomes more pronounced due to
C.6 Dynamic rank allocation across layers in ViT
PEARL introduces a PEFT CL framework that incorporates dynamic rank selection for LoRA components during training. PEARL leverages reference task weights and adaptively determines the rank of task-specific, layer-specific LoRA components based on the proximity of the current task to the reference task weights in the parameter space. Figure illustrates the dynamic rank allocation across layers for convolutional variants for the final task in -IF-CI1011 (5T). From left to right, Figure 2 presents detailed overview of the layer-wise dynamic thresholds established based on task proximity, along with the corresponding selected rank and actual rank for our convolutional variants. Similarly, Figure 4 presents the dynamic rank allocation across layers for the final task in ImageNet-R (5T) for PEARL (VIT). In the VIT model, earlier layers are primarily responsible for extracting low-level image features. These features lay the foundation for subsequent high-level feature extraction and semantic understanding in later layers. As these features relatively less complex and can be re-used across tasks, PEARL. puts less emphasis them through lower
23

---

Published in Transactions on Machine Learning Research (01/2026)
CIFAR-100 5 tasks
CIFAR-100 in 10 tasks
CIFAR-100 20 tasks
90
PEARL (R18)
PEARL (R18)
PEARL (R18)
Top-1 Accuracy (%6
PEARL (BSC)
Top-1 Accuracy (%)
80
PEARL (BSC)
Top-1 Accuracy (%)
80
PEARL (BSC)
70
60
60
50
40
40 60
100
20
40
80
100
60
80
100
Number of classes
Number of classes
Number of classes
Evaluation of Class-TL (A7%) ) performance under longer task sequences in Seq-CTFAR100. In line with other parameter isolation approaches, there is steady drop in performance we move from 5 tasks to 10 tasks and eventually to 20 tasks.
Figure 3:
rank allocation. On the other hand, intermediate layers handle more abstract and complex features, such parts of objects and entire objects, necessitating higher rank allocation. Therefore, PEARL puts major emphasis on intermediate layers with higher rank allocation. The top layers of the ViT model are designed to capture high-level semantic features and global contextual information. By later layers, the model has likely already captured the most significant semantic features relevant to the task, and additional layers contribute marginally smaller improvements to the final output quality. Therefore, PEARL allocates a lower threshold and corresponding lower rank in later layers. In line with the ione findings in ZZng et al. 2024), PEARL displays a bell-curve shaped dynamic rank allocation across layers thereby optimizing resource allocation and maximizing information reusc.
Dynamic Threshold
Selected Rank
Actual Rank
50
PEARL (VIT)
PEARL (VIT)
PEARL (VIT)
Dynamic threshold
0.60
800
40
0.55
780
Rank
30
Rank
0.50
20
760
0.45
10
740
5
10
0
5
10
0
5
10
ViT layer
ViT layer
ViT layer
Figure 4: Depiction of final task dynamic resource allocation ImageNet-R (5T). (left) Dynamic threshold of different VIT layers based on the similarity of current task with respect to reference task for deciding the rank. (middle) Selected rank based on dynamic threshold. (right) Actual rank of Key weight matrix. As can be seen both dynamic threshold and selected rank show more emphasis for later layers, in line with the actual rank.
C.7 Additional comparison with adapter-based methods
Table 2 providees an extensive comparison of different adapter-based and LoRA-based PEFT CL approaches. In addition, Table [4] provides comparison against additional adapter- based methods in the Class-IL setting on ImageNet-R (10T) using a VIT-B/16 backbone pre-trained on ImageNet-21K. As can be seen, PEARL (ViT) produces superior performance across both the metrics and beats the baselines by large margin. Although LoRA-rank allocation is dynamic in our framework, PEART. (ViT) leaves slightly more number of
24

---

Published in Transactions on Machine Learning Research (01/2026)
parameters compared to adapter-based approaches. Although adapter-based methods are parameter efficient, We note that their performance is sub-par and can be even worse in longer task sequences.
Table 14: Following Tan et al. (2024), we compare additional adapter-based methods in the Class-IT. setting on ImageNet-R (10T) using a ViT-B/16 backbone pre-trained on ImageNet-2IK.
Method
Ar (%) 1
(%) f
Params (M)
Adamn-ss
Zhou et al.
2024a
66.61 1 0.09
74.36 1 1.00
0.20
LAE
Gao et al.
2023a
72.29 + 0.14
77.99 = 0.46
0.19
ADA
Famis el al.
2022
73.76 + 0.27
79.57 + 0.84
1.19
SSTAT
Dan et al.
2024
79.38
+ 0.59
83.63 + 0.43
1.19
PEARL
88.76 1: 0.69
91.67 1: 0.69
1.75
C.8 Additional comparison with prompt-based methods
Table 15 provides an extensive comparison prompt-based methods the continual learning (CL) setting on ImageNet-R. Following HiDee-Prompt (Wang et al. 2023a we evaluate our method using the final forgetting measure (FFM) and cumulative forgetting measure (CFM), defined as follows. Let A., denote the accuracy on the i-th task after learning the t-th task.
After learning all 7' tasks, the forgetting metrics are defined as:
T-1
FFM-
Σ
max
(Ai,e - AiTr)
(A58)
T
LE
CFM T-1T-1 i= (e(1..J-11)
1
ΣΣ
(Ais Ass)
(A59)
As shown in the results, PEARL (ViT) consistently achieves the best overall performance across both FFM and CFM metrics. HiDe-Prompt and EvoPrompt rank second and third overall, respectively, across all metrics and task settings. Notably, our method demonstrraes stable and competitive performance across varying task numbers (5, 10, and 20), highlighting its robustness in different continual learning scenarios.
Table 15: Following Wang et al.
2023a we compare additional prompt-based methods in the Class-IL setting on ImageNet-R using a ViT-B/I6 backbone pre-trained on ImageNeet221. Bold indicates the best. result and underline indicates the second-best best result.
5 Task & 40 classes/Las Lask
10 Task @ 20 classus/
20 Task @ 10 classes/ Lask
Method
FFM
CFM
FFM
CFM
FFM
CFM
1,2P /Wang al. 2022e
3.94 + 0.16
3.55 + 0.20
5.01 + 0.40
4.41 0.43
10.76 + 0.45
7.88 + 0.17
DualPromnt Wang 2022d
3.32 0.16
0.25
4.61 0.07
3.70 0.18
7.30 0.18
5.16 0.34
CODA-P Smith al.
23b
0.65
0.98
7.94+0.0
0.79
8.23 0.86
6.95 + 0.70
LGCL KKhan et al. 2023
0.34
2.50 - 0.38
0.38
0.33
3.38 0.558
7.35 1: 0.65
5.05 1 0.32
WWang al.
2023a
3.15 0.46
2.64 0.16
2.29
2.33
2.23 1 0.38
PGP Qiao al. 2024
3.36 0.23
2.85
4.53
0.33
7.17 0.21
5.09 0.25
Evo Proprompt. [Krriaaan
2024
1.39
0.32
4.22
0.52
6.72
5.67 0.26
CPrompt
(Ca al. 2024
0.00
5.49
6.10
1.33
0.28
5.54
ConvPrompt (Rty ct al.
2024
3.42 1 0.05
0.16
0.00
3.11
4.87
3.57 1 0.25
PEARI.
2.28 + 0.13
0.78 0.07
3.63 0.17
1.57 +.17
5.93 + 0.25
2.49 + 0.07
C.9 Inference time comparison in Class-IL
Table 16
presents an overview of per-image inference times on ImageNet-R. (20T) for various baselines and PEARL (VIT). It is evident that adapter-based methods exhibit slower growth compared to LoRA-basse
25

---

Published in Transactions on Machine Learning Research (01/2026)
methods, resulting in lower computational effort during inference. However, shown in Table 2 and 14 adapter-based approaches experience higher degree of forgetting, leading to reduced performance across tasks. In contrast, LoRA-based methods, including PEARL (ViT), involve slightly larger number of parameters and necessitate forward passes through each sub-network during inference. While PEARL (ViT) achieves superior performance across lasks, the additional overhead during inference remains a limitation of LoR R---aaeed approaches, including our own.
Table 16: Per-image inference time on ImageNet-R (20T) for cach method.
Method
Inference Time (ms)
L2P Wang et al. (2022c)
9.53
DualPrompa
Wang et al.
(2022c)
9.51
CODA-P Smith et al. 2023b
9.47
Adam ssf
Zhou et al. 2024a
10.03
SEMA Wang et al. (2025)
7.39
PEARL (ViT)
14.67
C.10 Task-wise resource allocation
As PEARL incorporates dynamic rank allocation based on task proximity in parameter space, we expect to see a non-uniform distribution of resources across different tasks in CL. Figure
((a) presents an overview of distribution of normalized resource allocation across tasks for all PEARL variants. As can be seen, resource allocation quite dynamic with highest variance scen PEARL (ViT) in ImageNet-R (10T). Guided by the task proximity in the parameter space, PEARL dynamically allocates resources thereby ensuring optimum trade-off between performance and model growth. In addition, also provide a comparison of normalized resource allocation across three runs of PEARL. (ViT) on ImageNet-R (10T) in Figure bb). We duly note that PEARL relies solely on the proximity of a particular task with respect to the reference task weights in the parameter space. We also clarify that task proximity as defined in PEARL does not correlate with the semaantic similarity of tasks. With different random seeds, the training trajectory and thus the proximity of tasks with respect to the reference task changes. Therefore, the distribution of parameter allocation exhibits significant. variation across lasks in different runs.
D Experimental setup
D.1 Backbones
To demonstrate the versatility of PEARL across different architectures, we also explore three distinct backbones: (i) Convolutional networks such ResNet-18 and ResNet-18-like architecture incorporating
Blueprint Separable Convolutions (BSC) (Li et al. 2022b), and (ii) the Vision Transformer (VIT) Base 16 pre-trained on ImageNet-21k.
With regard to the PEARL (R18), the backbone is same as the one used in Mammoth CL benchmark Buzzega et al. 2020). In line with previous works, the backbone is randomly initialized and trained for CL. tasks. However, we introduce PyTorch ModuleLists all throughout to make casier to add / remove convolutional and BatchNorm layers at will. The ResNet-18 backbone consists of 4 layers with 2 blocks each. Each block in turn consists of two 3x3 convolutional layers with BatchNorm and ReLU activation in between and after. ResNet-18 18 also entails shorteut connections in the form of 3x3 convolutional layer followed by a BatchNorm layer and ReLU activation. Within PEARL (R18), we reserve separate classification and BatchNorm layers for each task. As the backbone is not pre-trained, we treat the first. task as the reference task and conduct dynamic allocation from the second task onward. LoRA components are allocated for every convolutional layer. Their rank is dynamically decided after SVD dccomposition of respective task vectors. Even after dynamic allocation and further training, LoRA components, BatchNorm layers and a corresponding classification layer kept separate for each task.
26

---

Published in Transactions on Machine Learning Research (01/2026)
Seq-CIFAR100 (10T) R18
(Norma lizedd
ImageNet-R (10T) VIT Run
#Params (Normaiin Farams (Nommall
1.0
1.0
0.8
0.6
SeqTinylmageNet (10T)
#Params (Normalizes llrred))
ImageNet-R (10T) VIT Run
1.0
0.9
0.6
0.8
0.4
0.7
0.2
(Normalized)
ImageNet-R (10T) VIT
ImageNet-R (10T) VIT Run
1.0
1.0
0.8
0.8
0.6
Task 1
Task
Task 3
Task 4
Task 5
Task 6
Task
Task
Task 9
Task
Task 1
Task
Täsk 3
Task 4
Task 5
Task 6
Task 7
Task 8
Task 9
Task
(b)
Figure 5: (lsskwwie resource allocation in parameter space (normalized number of parameters) for PEARL variants. (a) Visualizes resource allocation across Seq-CIFAR100 (10T), Beg-TinyImageNet (10T), and
ImageNet-Re (10T). (b) Shows resource allocation dynamics for ImageNet-R (10T) using ViT across three runs.
Li et al. 2022b) introduced blueprint separable convolutions (BSC) to optimize and reduce redundancy in convolutional operations. Essentially, BSC entails a Ixl operation (point-wise filter) followed by depth-wise convolution layer. This scparability reducos the memory and computational footprint resulting far more efficient model. Such a model well suited for parameter isolation CL approach as it reduces overall model size when dealing with longer task sequences. PEARL (BSC) backbone consists of layers with 2 blocks each. Within cach block, the organization of layers is same that of ResNet-18 except that traditional convolutional layers are replaced by BSC layers. As the BSC backbone is custom, small network, pre-trained model is not readily available. Therefore, we treat the first task as the reference task and conduct dynamic allocation from the second task onward. LoRA components are allocated for every point-wise filters as they have the majority of the footprint terms of number parameters. Since depth-wise filters have relatively lower footprint, we allocate LoRA components and select the rank of the task vector after decomposition. Same as in ResNet-18, even after dynamic allocation and further training, LoRA components, BatchNorm layers and a corresponding classification layer kept separate for each task.
VIT-B/16 backbone pre-trained on the ImageNet-211K dataset. To support approach, PyTorch ModuleList structures are employed to facilitate the integration of task-specific LoRA components. LoRA weights are inserted in all layers of the ViT backbone, and as shown previously in Table 3 adding LoRA weights to the Key matrix yields better results. The rank of these LoRA components is dynamically determined by performing SVD decomposition on the corresponding task vectors. During inference, parameter-efficiet task-specific LoRAA weights that are used to predict image labels in both Task-IL and Class-IL. scenarios.
For PEARL (VIT),
27

---

Published in Transactions on Machine Learning Research (01/2026)
D.2 Datasets and settings
We evaluate PEARL across two distinct scenarios of CL: Class-IL and Task-IL. In both these settings, the model encounters a new set of classes in each task and must learn to distinguish all classcs encountered thus lar after each lask. In practice, split CIFAR-100 (Krizhevsky al. 2009), Tiny InageNeeeNet (Le & Yang) 2015 and ImageNet--R (Hendrycks et al. 2021 into different tasks depending the number of tasks as in Table 17
Table 17: Dataset organization across different tasks and settings.
Dataset
Setting
Image size
Tasks
Classes per task
5T
32x32
Seq- CIEAR100
10T
32x32
10
10
20T
32x32
20
5T
64x64
5
40
Seq-TTiny ImageNtt
10T
64x64
10
20
5T
224x224
40
ImageNet-R
10T
224x224
10
20
20T
224x224
20
10
PEARL is a CL framework that addresses catastrophic forgetting by using parameter isolation through the dynamic allocation of LoRA components. It creates a distinct sub-network for each task. When a new task is introduced, a sub-network instantiated from the reference task and trained using a cross-entropy loss function. After a few epochs of fine-tuning on the current task, task vectors are computed and decomposed using SV-Decomposition. The LoRA components are initialized with rank that determined dynamically according to the criteria set forth in approach. These components are then re-initialized and trained for the current task. Once the training for task is complete, the associated sub-network-ww which includes the LoRA components, BatchNorm layers, and classification layer is frozen.
The model is trained simultaneously on both Class-IL and Task-LL settings, as their training regimes are identical. During inference in the Task-IL setting, the appropriate task-speeific LoRA weights are selected based on the given task identity, and its output is inferred for maximum activation. However, inference in the Class-IT setting is more complex, as task-identity available. In this case, each test. image passes through every sub-network, and their respective classifier outputs are concatenated. Since each task-specific sub-network is trained independently and the activation magnitudes produced can be imbalanced, the performance in the Class-IT, setting often lags significantly behind that in the Task-II. setting. Although PEARL is highly efficient in terms of parameter allocation, it introduces inference overhcad in Class-IL scenarios, similar to C-LoRA. Unlike InfLoRA, where the learned parameters can be folded back into the original model weights for more efficient inference, PEARL stores task-specific LoRA sub-networks for each task. This task-wise separation, while effective for reducing interference between tasks, results in slightly increased storage and retrieval costs during inference, especially as the number of tasks grow beyond a reasonable limit.
D.3 Training details
For experiments concerning PEARL (ViT) on the ImageNet-R dataset, use the Adam optimizer (Kiggma & Ba, 2015) with hyperparamcters 31 = 0.9 and 32 0.999. The batch size is set to 128. The ViT model is fine-tuned on the current task with a learning rate of 5 x 10-3 for 25 epochs (E1). Following this, the LoRA weights are trained separately for an additional 25 epochs (Eg) using a learning rate of 5 x 10-4, We observed that a cosinc-decaying learning rate schedule outperforms a constant learning rate. Regarding the LoRA parameterization, the rank is determined dynamically based on the criteria outlined in our approach, and the LoRA alpha parameter is set to twice the rank.
28

---

Published in Transactions on Machine Learning Research (01/2026)
With regard to PEARL convolutional variants, batch size of 32 for all experiments. Furthermore. we fine-tune the reference task weights for 15 epochs (Fi) with best learning rate found in Section C.3 Following this, the LoRAA components are initialized with dynamic ranks and are fine-tuned further for 50 epochs (E2).
Algorithm PEARL (without pre-training)
Input: Randomly initialized model Φo, Data distributions (D. Epochs Ei, E2 Reference task training using Equ.
3: 0r {for, ,, 0 0,
4: for lasks LE .., K} do
5: 0g + [deepcopy([nn 9o
for +
Fino-tune 0, on D. for E₁ epochs
for target layers 1 E 21, [.] do Compute W = W/ -WT WW
10:
Decompose W using Equ.
11:
Find dynamic rank k (Equ. & Equ
12:
Initialize LoRA components By and A (Eqn. 5 for = for U {Br Ar}
13:
14:
Re-initialize task-speeific weights (oc
15:
0 - U {for, go:
16:
Fine-tune (Jor, on Dr for Eg epochs
17:
return model 0
D.4 PEARL Algorithm in the absence of pre- vre-training
Algorithm provides an account of PEARL's functioning when pro-trained model is readily available. However, pre-training can be a luxury for custom, compact models. Therefore, we design PEARL to seamlessly work even when pre-training is not available. Algorithm 2 illustrates the functioning PEARL in the absence of pre-training. The only difference being, the first task treated as reference task and all the subsoquent tasks leverage first task information for elficient CL. Although is possible to use more number of classes in the first task for better knowledge consolidation, first task in PEARL contains same number of classes the rest of the tasks.
29


Reviews:
[{"review_id": "aa6UW2Ftiu", "reviewer": "Reviewer_7EMy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper targets the setup of continual learning (CL), primarily in a low-rank space where models struggle to retain previously learned knowledge while learning new tasks. The authors propose PEARL (expaded as Parameter-Efficient and Adaptive Resource Allocation for Continual Learning), that uses low-rank adaptations to modify pre-trained models in a resource-efficient way, reducing the disruption to the model's performance on previous tasks while adapting to new ones in a continual fashion. \n\nContributions:\n* The paper introduces a new formulation for a dynamic rank allocation mechanism for LoRA components, which adapts the rank based on the similarity between the current and previous tasks. The primary motivation is to efficiently reuse information from earlier tasks by assigning lower ranks to earlier layers and higher ranks to task-specific later layers, which is claimed to ensure minimal computational load while maintaining task-specific adaptations as the learning continues for tasks in sequential order. \n* Another key contribution is the task-specific LoRA components, which are applied exclusively to the key modules of transformer models (though they have been explored in retrieval, knowledge merging, etc. in the past), the presented formulation focuses on being efficient for long task sequences in continual settings. \n* The proposed method, PEARL, also leverages SV-decomposition of task vectors to initialize the LoRA components, for a strong starting point in task adaptation. This initialization is claimed to accelerate convergence. \n\nStrengths\n\n* One of the proposed methods primary focus is on resource efficiency. The dynamic rank allocation and task-specific LoRA components allow it to adapt to new tasks without a significant increase in model size. The detailed set of experiments are provided that demonstrates robust performance across multiple architectures (e.g., ResNet-18 and Vision Transformers) and tasks, outperforming other parameter-isolation techniques in terms of both accuracy and efficiency (as shown in the tables). \n\n* Another notable strength is that the proposed formulation helps in the ability to benefit from pre-training, which adds to the model’s generalization ability and helps it perform well across multiple tasks, and may be helpful for the CL community. \n\n* The limitations highlighted in the paper capture the open ends both in terms of application and experimentation. \n\nWeaknesses:\n\n* There are some lines of work that also explore LoRA on similar lines (e.g., Chaudhry et al., 2020; Wang et al., 2023; Farajtabar et al., 2020), making use of orthogonal subspaces formed using SVD formulation. The current version of the paper does not discuss the initial works/prior arts/parallel works in detail, and it would be good to add a comparison (empirically, if possible) or a discussion on similar lines to help the reader understand the primary gap that the proposed method fills. \n\n* In the presented method, the task vector initialization assumes a precise measure of task similarity, but in practice, there can be multiple similarity criteria that are left underexplored, moreover, the accurate task vector estimation is a challenging problem, and more evaluation on that front would be good to add in the paper for more reliability in the presented approach.  \n\n\n[1] Wang, Xiao, et al. \"Orthogonal subspace learning for language model continual learning.\" arXiv preprint arXiv:2310.14152 (2023). \n\n[2] Chaudhry, Arslan, et al. \"Continual learning in low-rank orthogonal subspaces.\" Advances in Neural Information Processing Systems 33 (2020): 9900-9911. \n\n[3] Farajtabar, Mehrdad, et al. \"Orthogonal gradient descent for continual learning.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.", "claims_and_evidence_yn": "No", "claims_explanation": "* The results in Tables 1 and 2 show that the presented method gives performance improvements when compared to existing approaches across different scenarios (Class-IL, Task-IL, and with different architectures like ResNet-18, ViT, and BSC). \n* The paper also includes suitable ablation studies (e.g., Table 3 and Figure 2) that show the importance of dynamic rank allocation for LoRA components, illustrating how it optimally balances model size and performance.\n\nOverall, the experimental setup, including detailed comparison across multiple datasets and tasks, with ablation experiments justify the primary motivation/claims made in the paper.", "audience_yn": "Yes", "audience_explanation": "* The paper presents a new framework that is rehearsal-free, efficient, and adaptable across architectures.  \n* The paper show empirical results highlighting gains over the compared state-of-the-art methods.\n* In terms of audience, the paper also has a Broad applicability in current literature, since handling multiple sequential tasks without catastrophic forgetting is essential for scaling ML systems in the real world.\n\nIn general, the paper targets a relevant problem in continual learning, and the increasing focus on efficient adaptation in large pre-trained models being the primary focus aligns with the TMLR scope.", "requested_changes": "Minor Changes/Edits:\n\n* The algorithms, both 1 and 2, could be explained in more detail; in the current version, it becomes difficult to get the entire thing in one go. Several steps (e.g., the exact data flow between reference-task parameters and newly allocated LoRA modules, the re-initialization step, and the treatment of normalization layers) are a little ambiguous. It would be good to add precise definitions, and clarify the sequence of operations would make the methodology easier to reproduce.\n\n* In the finetuning schedules, the difference between E1 and E2 is not completely clear; it would be good to describe whether reference-task weights are frozen or partially updated during different stages. Since in training/finetuning, there is a huge instability, and PEFT methods in general are a little sensitive to finetuning setups, it would be good to add all the details in the next iteration of the paper. \n\n* The current description suggests passing every test sample through all task-specific sub-networks. This has an added inference cost, especially for large ViT models. If possible, it would be good to explicitly quantify inference-time FLOPs and latency, comparing them with baselines for a more clear picture of the improvements, which would also help in the utility of this work for future works.", "broader_impact_concerns": "The paper does not contain any explicit Broader Impact Statement section. However, given the scope of work, there are no major concerns about the ethical implications of the work that would require adding a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AYArYEgsDN", "reviewer": "Reviewer_22Px", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a rehearsal free CL method (PEARL) that uses finetuning with LoRA modules and assigns a different low-rank to each task and layer. The main idea is to treat a pre-trained model (or the first CL task) as a “reference network” and for each subsequent task to compute a task vector (diff between current fine-tuned weights and reference weights). An SVD of this task vector gives singular values whose cumulative energy is compared against a normalized distance between current task and reference. This yields a layer-specific dynamic rank, aiming to increase plasticity for layers where the new task induces large weight changes, and reduce plasticity where the task is closer to the reference. These LoRA modules are then re-initialized and trained only for the current task, while the reference network remains fixed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main empirical claims are supported by the reported experiments. The implementation appears technically sound at a high level. There are some gaps however in the definition of the dynamic threshold. See below:\n\na) the fraction in Eq4 (where the denominator is the total singular-value mass) is in [0,1] but T is not guaranteed to lie in [0,1]. T can be as large as 2 in extreme cases. The paper does not mention clipping T to [0,1] or handling the case T>1 and Eq4 is ill-defined if the inequality “cumulative variance ≥ T” never holds.\n\nb) Also the +1 in Eq4 is not motivated. It forces the chosen rank k to overshoot the minimum rank satisfying the variance criterion, which may or may not be desirable. Plz explain.", "audience_yn": "Yes", "audience_explanation": "CL is an active research area and the sub-area of replay-free parameter-efficient methods is of emerging interest. So yes, I have no doubt that the paper will be found interesting by some researchers.", "requested_changes": "- plz clarify and correct the definition of the threshold T. The paper should state T's range, explain the behavior when T>1, and specify whether T is clipped or normalized.\n\n- Provide a justification for the “+1” term in the rank selection rule. the motivation should be explained or the formula revised.\n\n- *Important* Include a comparison against at least one recent dynamic-rank LoRA CL method (e.g., FM-LoRA, CoDyRA). Both methods are highly relevant and address very related problems. \n\n- Explicitly analyze (or at least discuss) the potential sensitivity of the dynamic rank to noise in the SVD of Wc, especially for small matrices or early convolutional layers where task vectors may be low-energy or poorly conditioned.\n\n- Provide an ablation isolating the contribution of the “reference task” choice (e.g., first task vs pre-trained model). It is VERY important to see whether the dynamic ranks or performance depend strongly on which task is used as reference.\n\n- Many baseline results are taken from prior papers with different training pipelines. At minimum, report the exact data preprocessing and augmentation used for PEARL and explain any known differences from the baselines’ setups.\n\n- Class-IL inference requires passing each test sample through all task-specific branches. Inference time grows linearly with the number of tasks. It would be useful to quantify this cost and compare it to alternative PEFT-CL approaches that avoid multi-branch inference.\n\n- Provide a clearer comparison of parameter counts versus rehearsal methods. Although rehearsal methods use a buffer, PEARL uses per-task adapters. It would be helpful to include a table converting both into a unified memory footprint metric.\n\n- Expand the analysis of failure modes. For example are there cases where PEARL selects very small ranks and significantly underfits or very large ranks that approximate full fine-tuning? A controlled experiment showing such cases would help show the limitations.\n\n- Improve the clarity around re-initialization. Currently LoRA components are initialized from the SVD and then immediately re-initialized. The manuscript should explain when and why the SVD-based initialization is beneficial given that it is then discarded.", "broader_impact_concerns": "I do not see any such concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uUlgYEnHXd", "reviewer": "Reviewer_teHE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Parameter-Efficient Adaptive Ranking for lifelong learning with LoRA (PEARL), a rehearsal‑free continual learning (CL) framework that dynamically allocates the LoRA rank per task and per layer using information from reference task weights. For each new task, the method first fine‑tunes a copy of the reference network to obtain (current) weights. Different from standard LoRA, the method computes a dynamic threshold to reflect proximity between the current and reference weights, and then accordingly sets a layer‑wise rank to fine-tune the parameters through LoRA to avoid catastrophic forgetting while keeping learning new tasks.\n\nStrengths/contributions\n\nThe paper is well written. The proposed data-driven rank selection method for LoRA is simple to apply and works with or without pre-training. It also shows strong empirical results across models and tasks. Besides the performance, the paper further provides analyses on several aspects (e.g., static vs. dynamic ranks, forgetting metrics and re-initialization benefits).  \n\nWeaknesses\n\n(i) If I understand correctly, the proximity score $T$ is used as a threshold in [0,1] for Eq.(4), but the given normalization may exceed 1. This lacks more explanations or clarification.  \n\n(ii) My major concern is about the computation (also mentioned in Section 5): the method requires a full‑model fine‑tune per task before training LoRA, I was wondering if in practice how much additional computation cost is introduced compared to a standard LoRA (with fixed rank)?\n\n(iii) As mentioned in Section 5, the model assumes knowledge of task boundaries, which restricts the practical applicability of it.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper presents comprehensive performance experiments, together with analyses and ablation studies that provide additional insights (see Section 4 and Appendix B).", "audience_yn": "Yes", "audience_explanation": "Dynamic, proximity-aware LoRA rank allocation is timely for the CL and PEFT communities. The method is evaluated on CNNs and ViTs, spans both Class-IL and Task-IL settings, and delivers practical procedures with sizable gains. Comprehensive ablations and analyses further inform future research and applications.", "requested_changes": "(i) Clarify the proximity score $T$ as discussed in the weaknesses.\n(ii) Discuss the computational cost of Algorithm 1. (Experiments are not necessary)", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)