PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction Tuning
Authors: Weicheng Meng
Jingyang Qiao
Zhizhong Zhang
Shaohui Liu
Yuan Xie
Venue: ICLR 2026 Poster (2026)
Keywords: Large Multimodal Models
Continual Instruction Tuning
Catastrophic Forgetting
Knowledge Distillation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A continual instruction tuning method based on progressive knowledge compression and integration.

Abstract: Continual Instruction Tuning (CIT) enables Large Multimodal Models (LMMs) to rapidly adapt to new tasks without retraining, but it suffers from the catastrophic forgetting problem. By adding new branches, model extension provides a great idea to accommodate novel knowledge while causing huge memory consumption. To jointly address forgetting and memory explosion, we propose the Compression–Integration–Learning (CIL) pipeline, which draws on the memory consolidation processes during human sleep. Compression streamlines old parameters to release capacity. Integration merges knowledge from similar tasks to restore the performance loss due to compression. For example, based on LLaVA-7B, the forgetting is reduced from 11.29 to 5.09. Learning reallocates released capacity for new task-relevant parameters. Next, based on the characteristics of LMMs at different learning stages, we establish the progressive learning process, further reducing forgetting from 5.09 to 3.39. Moreover, to adapt this process, we decompose LoRA into a set of rank vectors and introduce an extremely fine-grained architecture, LoRA Rank Pool (LRP), with the goal of flexible knowledge employment and editing. Finally, we combine all components, and yield **P**rogressively **C**ompressed **L**o**R**A (PCLR). Extensive experiments demonstrate that PCLR owns a memory budget close to non-extension methods while outperforming extension methods in performance. The implementation code is available at https://github.com/SII-HITclearlove777/PCLR.

Published as a conference paper at ICLR 2026
PCLR: PROGRESSIVELY COMPRESSED LORA FOR MULTIMODAL CONTINUAL INSTRUCTION TUNING
Weicheng Meng". 12, Jingyang Qiao Zhizhong Zhang Shaohui Liu",, Yuan X
Harbin Institute of Technology, Harbin, China Shhnngaai Innovation Institute, Shanghal, China
*East China Normal University, Shanghal, China
*Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China 24b903022 @ stu.hit.ee www.e.uuenn 5227590101 @ www..ennu.ddu.nn zzzhang @ we..eennu.euuc..
shliu @ wehit.edu edu.cn, yxie celo@ss.comm es.ecnu.edu.c
ABSTRACT
Continual Instruction Tuning (CIT) enables Large Multimodal Models (LMMs) rapidly adapt
tasks without retraining, but it suffers from the catas- trophic forgetting problem. By adding new branches, model extension provides great idea accommodate novel knowledge while causing huge memory con- sumption. To jointly address forgetting and memory explosion, we propose the Compression- Intigoatiinntiiinnnnn antegrationLLeaming (CIL) pipeline, which draws on the memory consolidation processes during human sleep. Compression streamlines old pa- rameters to release capacity. Integration merges knowledge from similar tasks to restore the performance loss due to compression. For example, based on LLaVA- 7B, the forgetting reduced from 11.29 to 5.09. Learning reallocates released capacity for new task-relevant parameters. Next, based on the characteristies of LMMs at different learning stages, we establish the progressive learning process, further reducing forgetting from 5.09 3.39. Moreover, to adapt this process, we decompose LoRA into a set of rank vectors and introduce an extremely fine- grained architecture, LoRA Rank Pool (LRP), with the goal of flexible knowl- edge employment and editing. Finally, we combine all components, and yield Progressively Compressed LoRA (PCLR). Extensive experiments demonstrate that PCLR owns a memory budget close non-extension methods while outper- forming extension methods in performance. The implementation code available at
com wtw.eeeeeealllle7/eeeo00
INTRODUCTION
Large Multimodal Models (LMMs) have gained widespread adoption due to their exceptional cross-
modal comprehension and generation capabilities (Lu al. 2024 /Zhnn al. 2023) The training process follows two-stage paradigm: Pre- Training (PT) and Supervised Fine- Tuning (SFT) (Bai
al. 2025 Chen et al. 2024c). Within SFT, instruction tuning markedly improves the ability of models to follow human intent and become an industry standard practice (Dai et al.) 2023 Achiam al. 2023). However, with continuously evolving data sources and task requirements, the need for
frequent retraining is costly and impractical (Scialom et al., 2022 Luo et al. (023). Thus, Continual Instruction Tuning (CIT) becomes promising way to learn evolving knowledge for LMMs (Chen)
et al.
2024a
Xie et al. 2025).
CIT methods include static-structure (non-cxtension) and model extension (cxtension) Yu ct al. 2024a). Static methods alleviate forgetting by constraining parameter updates, but they suffer from the trade-off between stability and plasticity (Li & Hoiem 2017: Aljundi et al. 2018 Zhu et al,
2024 Extension methods append task-specific modules to isolate interference (Wang et al. 2022a
Yu et al. 2024b), but this incurs unbounded memory with tasks growing.
'"Corresponding Authors
PProjecct Leader

---

Published as a conference paper at ICLR 2026
SclenceOA
Sc
SciencegA
KEVO
OCRVOR
OCIVOR
$1.40
Imagetet
HLLP
ImaroNeet
VgAv2
vgAu2
voNu2
64.77
44.03
12477
1794
30,6
13.07
46.2
$3.9
Groudingg
STAT
Grounding
5300
Granding
Basciin
Dus
VIEVIZ
Ous
(a) Comparison on LLaBA-7B
(b) Comparison on LLaVA-131 13B
(c) Comparison Qwen- VL
Figure 1: Radar chart of comparisons on Final Accuracy between baseline (LoRA) and ours.
To mitigate memory growth, recent methods focus conditional extension, assuming task interfer- ence correlates with feature distribution differences. Before training, feature distribution similarities between previous tasks and the new task measured. If similar previous tasks, independent parameter group is instantiated to avoid interference; otherwise, the task is assigned to the clus- ter whose feature distribution is closest to the new task and the corresponding parameter group trained within a regularization constrained update scheme (He et al. [023] Qiao et al. 2025a). With treating parameter groups as routing experts, these frameworks mitigate high-interference tasks and reduce the need for extension. Nevertheless, they only implicitly postpone structural extension, and memory will still undergo unbounded growth with a long sequence of tasks.
The memory overhead of the model extension is in fact unnecessary. During continual learning, the LoRA parameters from distinct tasks contain linearly depen- dent rank vectors (as shown in Figure 2), and these rank vectors are compressible. However, current tension methods ignore fine-grained rank-level relation- ships, which constitute the key driver of memory plosion. To address the problem, we develop fine- grained Mixture-of-Expertt structure, LoRA Rank Pool (LRP). which is inspired by AdaLoRA (Zhang et al. 2023) and L2P series (Wang et al. 2022cc bj Smith al.
Relative Redundancy Rank Ratio of Different Task LoRA
1.0
VizWiz
0.065
0.159
6.657
GOA
0.202
0.6
TRxXVVA ImageNet
0.635
0.579
0.051
0.4
2023b) (Appendix Mp. TRP decomposes LoRA into rank experts, providing maximal flexibility for knowl- edge employment and editing. This design reduces stor- age overhead by minimizing parameter extension dur- ing training and pruning parameters post-training.
002
0,386
0.067
0.001
TextVQA
ImageNet
VizWizz
Figure 2: From TextVQA to VizWiz, (VizWiz, GQA) represents 38.2% ranks in VizWiz's LoRA are redundant versus GQA. Details are shown in Appendix
To save memory, we introduce Compression after ex- tension: pruning redundant rank experts in LRP to lease capacity. Inspired by hippocampal reactivation during sleep that consolidates memory (Wilson & McNaughton, 1994| Stickgold 2005 Srinivasan
et al. 2025), we propose fff-learning-phase Integration: using knowledge distillation to reactivate learned experts and fuse their internal knowledge, thereby mitigating the performance degradation induced by compression. Combined with Learning on new tasks, this yields the Compression- Integration-Learnin (CIL) pipeline. The memory saved by compression is set equal that added by learning, aiming to fundamentally eliminate the issue of unbounded extension. Finally, we es- tablish the progressive learning process, effectively mitigating catastrophic forgetting in long- term CIT. Different from previous work, our approach allows all experts to contribute to new task learn- ing, which promotes knowledge transfer and integration. In our method, task-specific experts that support single task are progressively transformed into mixture-of-expertss that support multiple tasks, during continual learning. The evolution visualized Appendix F
We evaluate the proposed method on LLaVA-1.5 (Liu et al. 2024) and Qwen-VL VL (Bai et al. 2023b) by using the Continual Instruction Tuning benchmark (Chen et al. 2024a). Furthermore, we extend evaluation long-term multimodal CIT benchmark, Continual-NExt Experimental results demonstrate that PCLR significantly enhances the continual learning capabilities of LMMs. In sum- mary, the contributions of this work follows:
2

---

Published as a conference paper at ICLR 2026
We develop the Compression Cumpression-Integrrtiono--- Intgration-LLarning (CIL) pipeline D balance stability, plasticity, and memory efficiency in CIT. Additionally, can be combined with other reg- ularization strategies to further improve the long-tenn CIT performance.
We propose an extremely fine-grained Mixture-of-Expeee Expeetssss (MoE) structure, LoRA Rank Pool (LRP), enabling maximal freedom to employ and edit knowledge.
To the best of our knowledge, PCLR achieves state-of-the- performance with significant baseline improvements (as shown in Figure [T. In addition, it also owns the superior per- formance on Continual-NEx1 (15 tasks), the longest known multimodal CIT benchmark.
RELATED WORK
Large Multimodal Models: Benefiting from the advanced understanding and generation capabili-
ties of LLMs (Bai et al. [223a] Touvron et al., 2023b Dubey et al.] (2024), LMMs achieve rapid de-
velopment (Zhan et al. 2024 Wang et al. 2024 Team 2024
Ge Ge et al. [2024), particularly in Visual
Large Language Models (VLLMs) (Chen et al.
2024b Bai et al. 2025 Li et al. 2024a). VLLMs
employ linear projection layers (Touvron et al.
2023a Bai et al. 2023a) and Q Q-Former (Liu et al.,
2023 Zhu al.] 2023) cross-modal bridge modules to connect visual encoders to LLM back-
bones (Touvron et al. 2023a Bai et al. 2023a), enhancing multimodal reasoning.
Multimodal Continual Instruction Tuning: CTT methods can be categorized into three paradigms.
Regularization methods constrain gradients or parameters during training (Smith et al. (203a Zhu al. 2024). Structure-based methods enhance performance through architectural modification and dynamic extension (Yan et al. [022 Tha et al.) 2024; Yu et al.) (202b)). Replay methods mitigate catastrophic forgetting by replaying high-quality historical samples (Chaudhry al., 2018: 2019
Yoon et al. 2021; Zhang et al. 2024). To support multimodal CIT research, the CoIN (Chen et al.,
2024a) benchmark introduced. Representative methods: Model Tailor (Zhu al.
2024) updates
critical parameters via sparse masking. Eproj (He et al. (023) and LCCIA (Qiao et al.
2025a) imple-
ment grouped parameter learning based on task similarity. Specifically, Eproj introduces similarity- driven dynamic regularization, while LCIA incorporates the dynamic exponential moving average.
3 PRELIMINARY
Given the LMM M, series of tasks T = T2,, ,Tr and the cor- responding instruction datasets D = {D1, D2. ... DnJ. multimodal continual instruction tuning refers to sequentially training M each new task T, (1 < t n) with access only D, (or limited access to previous datasets), where D. = (V M₁ 1' n4 denotes the size of the training set for the 1-th task, and V?² M3, T denote the visual inputs, the textual messages, and the instruction for sample i, respectively. The objective is to acquire new knowledge while preserving performance on previous tasks during continual learning.
Problem Definition:
1=
LoRA Tuning: LoRA directly interacts with the frozen weights of the original model and has strong performance on complex tasks (Hu et al. 2022). It uses matrix factorisation AW = =BBBT to represent weight updates during fine-tuning, where is scaling factor, AW Rd X A E Xr B E Reome and din, dout, r represent the input dimension, output dimension, and latent dimension, respectively. The forward process is defined as: y zW AxA...
MoE for CIT: The Mixture-of- -ffee-tstss (MoE) architecture inherently excels in multi-task learning due to its sparse activation mechanism (Guo al. (025). Recent methods leverage data features as queries for sequence-level routing: Eproj (He et al. (023) encodes visual-textual features to assign
the task-specific routing expert, while LCIA (Qiao al., 2025a) identifies instruction patterns to allocate experts. The general formulation can be shown as:
xW +
gate(s(z))
(I)
where s(x) E R' denotes the expert routing scores, t is the number of total experts, & is the number of activated experts, and A, B, are the LoRA weights of the i-th expert. The operator gate(u, k) selects the top-k entries of vector v, setting those positions and others to O. Moreover, to
3

---

Published as a conference paper at ICLR 2026
Ouiput
Hüdden Embeeding Ouput
Shared Rank
LoRA-B
R-1
Opened Kank
LLM
Closed Rank
Score
Linear Module
LoRA-A
Inpui
Cate (Score)
1lidden Embbelding Input
What object image?
Cosine Similarity
Key Pool
Enceder
LLM Embredding
An Sequence
Frozun Puram
Training Param
Query
Connect
Figure 3: The pipeline of LRP. First, extract the query from inputs. Next, compute similarity scores with the key pools and gate the scores. Then, apply scores jointly with LRP weights in the forward pass. LRP is semi-frozen during training: new parameters trained while previous ones frozen.
accommodate continuously growing knowledge, the MoE can be extended by adding new experts, and refer this Dynamic MoE in this paper.
4 METHOD
4.1 OVERVIEW
Our method comprises two coupled components: the LoRA Rank Pool (LRP) architecture and the Compressoon-Intrratonn-LLCoeeee Integration-Learinng (CIL) pipeline. LRP factorizes LoRA adapters into rank vec- tors, each paired with a Iearnable key. forming an atomic expert that is inserted into linear lay- LRP is similar to Dynamic MoE, which uses static features activate experts and preserves past knowledge during training by freezing parameters (as shown in Figure B3. For the learning paradigm, we design CIL emulate the memory cycle of human lifelong learning (as shown in Figure 4). Compression, prunes experts at a preset retention rate to release capacity. Integration, applies improved distillation algorithm align the post-compression LRP with its original state, thereby compensating for new experts with the released capacity, and initializes them with similar previous experts to enhance forward transfer. Finally, we build progressive synergy mechanism combining the CIL. pipeline and the CIT process to enhance long-term CIT performance (the algorithm is shown in Appendix
loss by merging similar experts. Learning, trains
4.2 LORA RANK POOL
Although Dynamic MoE achieves outstanding performance on CIT, its experts tend acquire over- lapping (redundant) knowledge. To address this problem, we decompose experts to the atomic by splitting the LoRA weight A into column vector set {a1, a2, ... an and the weight 8 into row vector set {b] 62" 5/ Then, we introduce globally shared components A. B, to accu- mulate global knowledge. Thus, we obtain the LoRA Rank Pool formulation from Eq. [
y xW B.AA,B Bm ggate(s(x)))))
where n the number of total rank experts, is the number of activated rank experts, B, the factor of the shared ranks part, 3m is the factor of the mixture-of-experts part.
Next, we define the score function s(x) Kg, where K E gu q E IR are the L2-regularized key pool and query. For each input text embeddings with the visual output ([CLS) token or mean-pon mean-pooling). Each rank expert in the LRP is assigned learnable key, and all keys form K (key pool). We use cosine similarity scores and gating to select top-r relevant experts. This enables the model to employ distinct experts to acquire different knowledge and selectively activate them during inference.
construct (query) by concatenating the mean-pooling of
4

---

Published as a conference paper at ICLR 2026
Krnnlegge from
Preparing LRP
Compression
Krowlegge from Tock
Key
Expen Rark
Expert
Knonleigo from Task
Knowleige flom
Takk
Esper
Kaswleige from Tack &
Key
Exeen Kark 4
Koy
Pipen Ranc
Shared Ranks
Shared
Stared Knowladge
Rank
Keys
Espen lanic
Ranks
Sterge Knowledge
Key
bepen Kank
Key
Piper Ranic
Kaunledge traafer
Simibriy-Neane
Tadk 3
Key
Expert Rati
Task Masthing
LoRA Fool
LokA Raik
Learning
Integration
Exper Bank
Expen Rank
Ope
Espert
Key
Key
Exper Rank
Key
Isperi Renk
Stared Ranks
Key
Papen Ront
Espert Renic
Key
K
Exprt
Expert Rask
Key
Rank
Key 5
Exper Rank
Key
Ispen Rani
Key
Exper Rank
Rark'
Koy
LoRA Bank Pool
Exper
Ronk
Expert
Task
Key
Exper lunk
Expert
Key
Teacher Model
Studeut Model
(lik by Original Model)
(lit Congressed Model
Figure 4: Compression- Compression-Ineegratinn
(CIL) pipeline. Compression is a training-fiee free process that streamlines the original LRP. Integration is distillation process that aligns the compressed LRP with the original LRP. Learning is supervised fine-tuning process that initializes the L.P from the integration- processed LRP and then adapts the new task.
integration-processeed
In (). the mixture-of-expert. part forms n-term matrix polynomial. Sequential execution of all terms would cause significant computational overhead, making parallelization necessary. We aggregate the discrete ranks into A. and B.n- We broadcast the scoring vector s(x) to match the dimensions of zAm and derive a parallelized formulation:
y xW B.IA. +BnF (xAm, gate (Kg,r)² Bir
where the operator F (U,u) broadcasts v Re match the shape of U Baxb resulting in two matrices with identical dimensions, followed by an element-wise multiplication.
To jointly improve downstream task performance and learn the keys of labeled experts within the LRP, we combine the cross-entropy loss with the query-key cosine similarity loss:
log Po (ye x. N.y.e.. q) + Σ |1, (K.)lll)
(0) = T
(4)
where À is balancing hyperparameter, and denotes the number of key pools, which equals the LMMs layer count. To preserve previous knowledge, we freeze all previous keys and ranks while optimizing only the newly added ones.
4.3 COMPRESSIONIITTERRTTOON
Conventional learning faces the dilemma of whether to learn within the previous parameter space or allocate new one. The former suffers from the trade-off between stability and plastic- ity, whereas the latter leads to unbounded memory growth. To address this problem, we propose the waaressiioo-nntgaattiooooooooooe Learning (CIL.) pipeline, which compresses previous knowledge when learning tasks. We decompose each CIT task into three main phases: compressing, integrating past knowledge, and learning new knowledge (as shown in Figure Notice that CIL is initiated once the total rank reaches predefined value.
Compression: At this phase, we introduce a compression retention rate a E (0, 1]. discarding portion of redundant rank experts and their corresponding keys. As result, the pruned LRP maintains its rank and key count at a times the values of the original one. Although compression may lead to performance loss, it crucial for alleviating the memory pressure of the learning system.
5

---

Published as a conference paper at ICLR 2026
Integration: Removing key-value pairs will degrade knowledge from previously learned tasks. To compensate for this loss, we perform integration on the current dataset to adapt the LRP using the distillation loss (Hinton et al. 2015 Huang & Wang 2017| Gou et al., 2021). Specifically, we for- mulate the Query-based Kullback- Leiblr Divergence Loss (QKLD Loss) to align the compressed LRP with the original one. The process freezes the teacher model (IRP before compression) and up- dates the student model (LRP after compression). Through distillation, rank experts in LRP absorb the knowledge from pruned experts, and the common representations across different tasks flow into the shared knowledge space, ultimately mitigating the knowledge loss caused by compression. The pseudo algorithm of compression and integration can be referred to in Appendix 0.3
LRP activates specific rank experts based on the query of the input. However, the datasets from previous tasks are not accessible during integration. Considering that we optimize the query-key cosine similarity during learning, and its optimum is achieved when each new key equals the mean of all queries for the corresponding task. Therefore, the learned keys can serve as surrogates for the queries from previous tasks, and we define them as fake queries. During integration, we compute Kullback-Leible Divergence (KLD) to quantify the performance loss associated with the fake query:
Σ Poors (2e I<e, log
Ponne (Ira xet. 9) Po..p (Iz
DKL (q)
(5)
where Por is the original LRP, iz is the compressed LRP, xr is the input. To emphasize optimization for tasks with larger performance degradation, we define the sampling probability of fake query:
VKLL (q)
(g) EnEr.n VDKE (4)
(6)
where Kors is the original key pool. Finally, we define the QKLD Loss:
L (0.sp) E (1) =E Pk))DD (g).
qEK
Learning: During the learning phase, shared and previous ranks are frozen preserve past knowl- edge, while new key-value pairs (rank experts and their corresponding keys) are trained using the capacity released through compression to optimize the new task. By decomposing LoRA into rank vectors, we enhance forward transfer: new parameters are initialized using ranks from similar tasks. This is achieved by computing the queries of subset of training samples before learning, taking their mean as an identifier to match to key-value pairs in the LRP, and initializing the new rank components with the values corresponding to the keys that exhibit the top-r cosine similarity scores. The pscudo algorithm of learning can be referred to in Appendix U.2
4.4 PROGRESSIVE LEARNING PROCESS
In this section, we introduce the progressive learning process to further improve the continual learn- ing performance of LMMs. In the early CIT, the rank space has not attained its allotted capacity, rendering compression an unnecessary expense. Thus, we disable compression until the capacity filled. This optimization reduces training cost and avoids unnecessary performance degradation without altering the final number of rank experts and the model memory.
As the number of tasks increases, LMMs accumulate more knowledge and strengthen their capa- bilities. This means that when learning new tasks, the number of additional trainable parameters progressively decreases, because much of the relevant knowledge has already been encountered. Consequently, the learning focus shifts from acquiring entirely new knowledge to organizing and consolidating acquired knowledge. In the CIL. pipeline, overlap between new tasks and earlier knowledge (reusable knowledge) increases. Meanwhile, internal representation becomes increas- ingly compact and resistant to further compression (high knowledge density).
Accordingly, in the later CIT, we allocate fewer new ranks and employ lighter, high-retention compression scheme to minimize performance degradation. This encourages LMMs to rely more on frozen knowledge during the late learning phase and has negligible impact new task performance. The progressive learning process, through deliberate planning of the learning path for LMMs, further enhances stability, plasticity, memory efficiency, and temporal efficiency of its evolution.

---

Published as a conference paper at ICLR 2026
Table 1: Comparisons between ours and baselines on LLaVA-1 CoIN benchmark.
Methel
Viner
2/T
Slasgghett
Accunsyy on Each Task
1-00A
5.Vi2W12
6.Gnossiing
T.V01e2
S0CRV0
Avg RCCCTT
Ovcrall Beselt
Forgeling: New.ACOYT
Zeoo duot
29.91
2.58
0.31
2.58
0.50
0.50
0.68
0.17
7:12
Mtdll Itak
56.77
40:35
05.55
56.65
$3.90
20.00
-9.90
55.65
57.08
Taka
al 2012
11:36
18:34
16:24
96.75
17:45
0.61
47.90
5716
18.14
17:30
61:66
1xF
2E 2017
TRAMI 16
63.14
19.50
6.90
34.33
14.53
2.48
40.67
60.35
33.31
22.32
$2.55
EWC
201
PNAS
07.41
40.41
5.38
25:05
37:88
2.67
41.27
61.00
36.34
20.51
54.65
MoB1
24
58.92
38.59
8.85
37.10
68.25
2.45
41.3
25.35
35.56
2571
55.36
51.29
06.75
53:36
46.28
55.21
IRPUPCE
ICMLL:24
$5.41
64.37
62.21
53.23
56.00
56.35
32.30
65:34
55.77
$7.89
51.83
62.71
65.65
[H
A3
57.53
67.67
63.54
60.79
65.54
58.34
80.8
58.14
57.61
64.17
62.19
65.16
4.5 INTEGRATION WITH REGULARIZATION METHODS
To further enhance the performance of our proposed PCLR on long-term CIT while minimizing unnecessary performance degradation and training overhead, we integrate with regularization methods. Specifically, the long sequence of tasks is partitioned into groups of contiguous tasks. Within each group we perform continual learning in a unified rank space combined with regulariza- tion methods, while across groups we retain the CIL process, thereby avoiding unnecessary exten- sion and compression. This optimization enhances the long-term CIT performance of PCLR while markedly reducing training overhead. We present cases of the integration of PCLR with LwF (Li & Hoiem 2017) in the experiments, and the pseudo algorithm can be referred to in Appendix U.4
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
CoIN Benchmark: To verify the performance of our method in challenging scenarios, adopt LLa (Liu et al. \QwenV (Bai et al.] 2023b) as the base model, and insert the PCLR module into the linear layers of the LLM backbone network and the cross-modal bridge. We use
the CoIN (Chen et al. 2024a) benchmark and keep the order of ScienceQA (Lu et al.) 2022).
TextVQA (Singh et al. 2019). ImageNet (Russakovsky al. 2015). GQA (Hudson & Manning
2019) 4 VizWiz (Gurari et al. (018)), Grounding (Mao et al. 2016). VQAv2 (Gooaa et al. (2017), and
OCRVQA (Mishra et al., 2019). These visual language datasets encompass a diverse range of task types, including selection, classification, grounding, and open-ended question answering.
Continual-NExT Benchmark: In order to further verify our method the long-term CIT setting, we use the Continual-NExT (Xie et al. (025)) benchmark, and keep the order of ArXivQA (Li et al.) 2024b), GeoChat (Kuckreja et al.| [2024). IconQA (Lu et al. (202)) ClevrMath (Lindström & Abra- ham 2022), CodeQA (Liu & Wan Wan||2021 200I), ImageNet (Russakovsky et al. 2015). Flickr3kk (Plum- mer et al. 2013), , Doc VQA (Mathew et al. 2021). TextVQA (Singh et al. 2019). MathQA (Amini
et al. 2019 ChartQA (Masry et al. 2022), Path VQt (He et al. 2020). Grounding (Mao et al., 2016 Science (Lu et al. 2022), and WikiQA (Yang et al.](215) These visual language and
pure language datasets cover multiple fields such as coding, mathematies, remote sensing and med- ical images. The experimental setup is shown in Appendix B
Comparison Methods: Compared baselines include LoRA (uu et al. 2022), zero-shot and multi-
task; regularization methods: (1) LWF (i & Hoiem, 2017). (2) EWC (Kirkpatrick et al. 2017)
(3) GEM (Lopez-Paz & Ranzato 2017). (4) MT (Zhu et 2024). (5) PGP Qiao al. 2025b).
(6) CIA* (w/o Instruction Grouping) (Qiao et al. (025a). (7) SEFE (Chen et al. 2025): static-
structure method: (1) MoELoRA (Chn al. 2024a). (2) AdALoRA (Zhang et al.) 2023); extension
methods: (1) Eproj (He et al. 2023), (2) CIA QQia et al., 2025a), (3) ProgLoRA YYu et al.) 2025);
replay method: (1) Experience Replay (Roinick et al. 2019). The experiments all adopt the setting
of training each task for 1 epoch. The method details are provided in Appendix S
We utilize three popular evaluation metrics: the average accuracy (Avg.ACC), the forgetting (Forgetting) and the new accuracy (New.ACC), which are shown in Appendix E
Evaluation Metrics:
5.2
MAIN RESULTS
As shown in Table we discover that PCLR achieves the highest overall performance on the CoIN benchmark. Compared the former best regularization method SEFE, the Avg.ACC improves by
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparisons between ours and baselines on LLa 1---h, ContimualNNExT benchmark.
Mechod
1:ArsiiQQ
2-GrsClla
Accurscy un Each Tank & Overall Roulls
4-Clevhhaah
S-Cal:OA
6-Inugghee
7-5lick2166
S-DxVVQA
9-TaaVVOA
Zero-bot
67.67
20.27
0.25
18.10
14.55
57.39
70.50
10.51
ICLR'22
47.70
TRANUTS
Keipzzi
PRAS
91.73
49.30
$2.03
51.73
NIW
91.03
48
6.76
76.30
Repbyrinn
Beaiis 2nng
NDW
5485
40.07
9461
0.36
$4.70
MotLoRA AKA (1 RUAMN]
21
56.00
91.36
65.76
48.90
3.82
82.19
17:37
16.33
59.51
CIA Oba Sylibita
JCML/25
56.35
93.41
49.76
48.20
4.23
$3,00
16.54
16.95
$1.32
PCT
10.20
51.30
37.17
7.86
R5.19
IKAM
21010
64.10
PCLK-LWE (PCLR variant
61.16
96.37
02.40
64.70
8.68
97.46
20,65
19.84
63,97
11-Chum04
13-Gmundinn
14-ceMncOQA
15-WRIUA
MOC1F
Forgotiing()
7emsddet
9.80
Malis
30.26
72.00
85.21
23.35
LaKA
ICER
3530
1804
5038
09:33
19:36
22.27
25.00
11:02
55.91
LWF
TRAMI 16
30.05
18.64
52.79
64.11
87.05
24.96
44.24
12.29
55.70
PWC
A d 2017
PNAS
15:41
19.00
50.22
69.0
89.51
26.17
06.95
0.93
56.01
GEM
Lapo For Kilaro 20
NIPS
35.21
17.68
51.36
67.23
39.56
23.85
45.25
10.19
55.74
Replig 100
NIPS119
31:42
14.00
1964
56.98
15.62
23.85
65.36
11.43
56,03
McELoRA Caan F1120
21PS/24
34.17
18.52
49.04
67.65
83.25
22.50
46.99
9.06
$4.51
CIA*
IOMB/24
12.46
17.07
40:47
69:86
89.12
23.54
66.43
8.10
54.95
PCLK
69.76
89.65
PCLK-1.9 (PCLR voriant)
62.83
83,00
$6.89
3.62 and the Forgetting decreases by 8.55, demonstrating strong continual learning ability. Addition- ally. some early approaches (c.g.. LwF and EWC) tend to overemphasize the mitigation of forgetting, while they suffer from severe plasticity reduction. Recent methods (such as MT and EProj) place greater importance on maintaining plasticity, contributing to significant improvements in NeW.ACC. However, these methods still struggle to achieve an optimal balance between New.ACC and Forget- ting. Notably, our proposed method, PCLR, achieves the highest New.ACC and the lowest Forget- ting among all compared approaches, demonstrating its superior performance. Furthermore, PCLR surpasses both the previous best extension method (EProj) and the leading dynamic update method (CIA*). To validate PCLR's 's stability and scalability on larger models and diverse architectures, we conduct evaluations on LLa 13B and Qwen- VL. Detailed results are provided in Appendix[A] Additionally, we present several visualizations in Appendix
As shown in Table our method exceeds the best of other methods (MoELoRA), improving Avg.ACC by 0.95 and reducing Forgetting by 0.35, demonstrating that PCLR sustains strong per- formance on the long-term Continual-NExT benchmark. To further enhance PCLR's 's efficiency and performance, we merge it with LwF to form PCLR-LwF. Specifically, we group adjacent tasks and perform LwF fine-tuning within each group (details shown in Appendix (Bddd Appendix(K). This integration with a simple baseline yields notable gains over the original PCLR, improving Avg.ACC by 4.73 and reducing Forgetting by 3.13. Additionally, the results of Qwen2.5-VL-Instruce (Bai
et al. 2025) are provided in Appendix A The visualization is shown in Appendix G
5.3
ROBUST EXPERIMENTS
To verify that PCLR exhibits strong robustness under different CIT settings, we conduct two sets of experiments on instruction templates and learning orders. Note that, in double-row tables, the upper row denotes the immediate accuracy (evaluate after the current task), and the lower row denotes the final accuracy (evaluate after the final task).
Table Results of ILL 1110A-15--77 1.5-7B on different instruction templates.
Accuracy Each Task
Overall Results
Type I-ScienceQA
2-TexIVQA
3-ImageNet
4-GQA
5-VizWiz
6-Grounding
7-VQAV2
8-OCRVQA
Avg.ACC Forgetting New.ACC
Origin
83.47
61.29
96.50
59.97
$8.32
34.02
65,75
61.92
65.16
78.33
$8.24
86.08
58.14
57.61
33.04
64.17
61.92
"3.447"
61.36
96.57
"0003"
56.53
"34.3"
'63.6 63.85"
B1.66
Diverse 76.92
65.25
56.16
$6.36
57.93
57.54
33.06
65.75
61.96
84.58
6039
96.40
59.39
58.21
334.4
63.67
60.99
10Type
61.98
3.52
65.06
80.24
58.62
84.81
57.85
$6.59
32.29
61.M4
60.99
Different Instruction Templates: As part of the text input, different instructions can influence query encoding and current task learning. We select three instruction templates (details are shown in Appendix|] [) for comparison, as shown in Table 3 We observe that changes in the instructions incur only a negligible effect, and our method can be adapted to different instruction templates.
Different Learning Order: In CIT, interfering tasks suffer from severe catastrophic forgetting when learned in adjacent order without parameter grouping. PCLR smoothly transitions task-specific ex- perts to mixture-of-expertt through the CIL. pipeline, which can resist this interference phenomenon
8

---

Published as a conference paper at ICLR 2026
Table 4: Results of LLaVA-1.5-77 on different task orders.
Order
SSSceescOA
QA
3-ImageNer
Accuracy
4-GQA
Each Tack
Grounding
7-VQAV2
S-OCRVQA
Avg.CC Forgeting New.CC
Overall Resulis
Origin
83.47
78.33
58.24
61.29
86.08
96,50
58.14
59.97
58.32
57.61
3304
$4.22
64.17
65.75
61,92
61.92
62.19
3.29
65.16
L-CCRQQA
2-VQAV2
3-Goounding
4-
S-GUA
6-hmageNe
7-TaaVQA
8-ScienaQA
Avg.ACC
Rargetingg New.ACU
$8.97
65.00
34.40
96
60.40
83.68
Reverso
56.78
62.66
26.08
55.85
57.00
94.02
60.74
83,88
62.18
303
TOQA
Grounding
Amaggeeet
&OOCRYAA
SclencQAA
AggACC
Porgetting New.AC
Alphabet
59.70
60.14
27.50
33.10
83.86
96.69
6149
50.15
62.72
K3.97
59.9
60.52
57.10
$6.89
64.34
64.34
60,02
4.63
64.67
to certain extent. In order to verify that PCLR has strong adaptability to different CIT orders, we compare three settings with different task orders. As shown in Table 4 task order changes induce slight knowledge conflicts, and have minor impact on the overall continual learning performance.
5.4 ABLATION STUDY
To evaluate each PCLR component, we start from the LoRA baseline, add components incre- mentally, and compare continual instruction tuning performance. Results are shown in Table 5 The experimental results demonstrate that each proposed component is effective in enhancing ac- curacy and reducing forgetting of LMMs. To validate the superiority of the LRP architecture, we compare the LoRA baseline with the LRP (which only performs compression and learning).
We can observe that the introduction of the LRP architecture increases Avg.ACC (+25.92) and reduces Forgetting (-26). In addition, hatprssiioo-nnttgaation--eeeee pare the (CIL) and Compression-Leandes Leaminng (CL) pro- cesses to validate the effectiveness of the integration process, and the results show that integration increases AvC.ACC (+6.12) and reduces Forgetting (-6.2). Moreover, observe that the introduc- tion of the progressive process further increases Avg.ACC (+ 1.441) and reduces Forgetting (-1.7). In summary, compared with the baseline, PCLR increases Avg.ACC (+33.45) and NeW.ACC (+3.8), reduces Forgetting (-33.9), and achieves an optimal balance between plasticity and stability.
Table 5: Ablation study results.
Method
ACC())
Forgetting()) New.ACCIt)
LoRA(Baseline)
28.74
37.29
61.36
retain LRP and com-
woo Integration Process
54.66
11.29
64.54
w/o Progressive Process
60.78
65.23
PCLR(Ours)
62.19
3,39
65.16
5.5 THE IMPACT OF PROGRESSIVE LEARNING
To investigate the impact of different compression strategies on CIT, we designed five methods: Ag- gressive (non-progrsssive). Conservative (non-progressive), Reverse Progressive, Centralized Com- pression, and Progressive Compression (Ours). These methods differ only in their compression strategies while maintaining the same memory usage (details are shown in Appendix
Table Results of LLa 1.5-7B on different compression strategies.
Final Accurtcc on Each Task
Overall Results
Strategy
I-SeienceQA
2-TexIVQA
3-maggNeet
4-GQA
5-VieWi, 6-Grounding
7-VQAV2
X-OCRVQA
Koggtting
New ACC
Aggressive
76.02
$7.03
82.61
55.81
$7.21
30.71
63.36
62.92
60.78
5.09
65.23
Conservative
92.28
58.77
25.88
64.16
1.77
Reverse
79.13
$8.79
87.41
58.51
$3.39
25.67
64.44
62.76
61.26
3.21
64.07
Centralized
77.72
53.78
88.93
60.05
55.71
28.71
62.93
60.16
61.00
3.38
64.37
Ours 78.33
58.24
86.08
58.14
37.61
33,04
64.17
61.92
62.19
3.39
65.16
As shown in Table 66 Aggressive Compression improves performance on new tasks but lacks the ability to consolidate old knowledge (stability). with Forgetting increased (+1.7). Conservative Compression reduces forgetting but limits the ability to learn new tasks (plasticity), with New.ACC decreased (-1.73). Reverse Progressive Compression exhibits subpar performance in both stability and plasticity, with Avg.ACC decreased (-0.93). Centralized Compression also demonstrates overall lower performance, with AvC.ACC decreased (-1.19). In contrast, Progressive Compression dynam- ically adjusts the compression retention rate, outperforming other strategies in reducing forgetting (Forgetting = 3.39) and enhancing new task learning ability (New.ACC = 65.16), achieving the best overall performance (Avg.ACC- 62.19). By optimizing the learning trajectory based changes in task knowledge density, Progressive Compression strikes a balance between stability and plasticity, providing an effective design solution for system development.

---

Published as a conference paper at ICLR 2026
Table 7: The results of LLL 1.5--BBB on different integration data usage. For the double-row table, the upper row denotes the final accuracy, and the lower row denotes the integration cost.
Accuracy and Integration Cast on Each Task
Evaluation Metries
IDU
I-SeieneeQA
2-TaUVQA
3-ImageNet
4-GQA
5-VizWiz 6-Grounding
7-VQAV2
8-OCRVQA
Avg-ACC Forgetting Avg
81.63
60.84
36.55
55.83
52.95
34.51
67.64
65:27
$9.40
8.86
83,49
61.33
91,09
58.11
54.41
35.92
67.47
64.17
min
14 män
7 miin
12m
min
83.33
61.35
93.15
55.94
35.76
67.29
64.12
min
26 män
13 min
23 min
18 min
61.47
56.82
34.31
20%
49 min
24 min
min
35 min
65.14
2.27
38 min
Origin
81.82
61.99
93.98
173
59.49
24 min
57.98
121
56,03
141 min
66,19
64.67
65.51
2.08
115 min
5.6
EFFICIENCY ANALYSIS
All LRP computations are fully parallelizable with the training cost close to the MoELoRA with one expert (Appendix D. For PCLR, the primary additional cost arises from the integration. We control the integration strength by varying the amount of integration data usage. Specifically, Integration Data Usage (IDU) is set 5 levels: 0, 5k, 10k, 20k samples, and Origin. Here, 0 denotes no integration, and Origin refers to the full dataset for the current task. Experiments are conducted on LLaVA-1.5--38 Results are reported Table (Avg.ICost denotes the average integration cost in minutes). In this paper, we use the Origin level to obtain the best performance.
In conclusion, PCLR does not incur significant time cost, and appropriately reducing the IDU will not cause serious performance degradation. When computing resources are sufficient, the IDU can be increased pursue the best final performance.
CONCLUSION
In this work, we introduce PCLR, which provides an extremely fine-grained LoRA Rank Pool (LRP) (CIL) pipeline to balance stability, plasticity, and mem- with a during CIT. LRP provides maximal flexibility for expert employment and editing. CIL ory efficiency progressively compresses LRP, trading off minor performance drop to eliminate unlimited model extension. On CoIN and Continual-NE: benchmarks across multiple LMMs, it delivers superior overall performance to former regularization methods while remaining competitive with extension methods. Robustness studies demonstrate the stability of the method. Ablation experiments confirm the necessity of each component. Efficiency analyses guide our selection of the optimal amount of integration data usage, balancing performance and speed. Owing to computing resource limits, we currently focus on image- text CIT. In the future, we aim to extend our method to modality extension scenarios. Additionally, we will design effective spatial structures in block-wise manner to further improve memory efficiency (feasibility can be found in Appendix D)
ACKNOWLEDGMENT
This work supported by the National Natural Science Foundation of China (Grant No. 62476090, 62302167, U23A20343, 62502159), Science and Technology Commission of Shang- hai Municipality (Grant No. 255111027000 Natural Science Foundation of Shanghai (Grant No. 25ZR1402 135), Natural Science Foundation of Chongqing (Grant No. CSTB202NSCQ-JXX0007 Open Project Program of the State Key Laboratory of CAD&CG (Grant No. A2501), Zhejiang University, Open Research Fund of Key Laboratory of Advanced Theory and Application in Statistics and Data Siience-MOE. ECNU.
CSTB205NSCO-GPX0433
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report arXiv preprint MxXiv:2303.08772 2023.
Rahaf Aljundi, Francesca Babiloni. Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European confer- ence on computer vision (ECCV), pp. 139-154, 2018.
10

---

Published as a conference paper at ICLR 2026
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedzionss Yejin Choi, and Hannaneh Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation- bassdd for- malisms. arXiv preprint 1rXiv:/905.13333 2019.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond, 2023b. URL hhttps://arxi...
www.oorg//.....oooooooeeoeoeoebe 2308 12966
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprini wrXiv:/8/2.000 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 1908. 2019.
Cheng Chen, Junchen Zhu, Xu Luo, Heng T Shen, Jingkuan Song, and Lianli Gao. Coin: A bench- mark of continual instruction tuning for multimodel large language models. Advances in Neural Information Processing Systems, 37:57817-57844 2024a.
Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang. Guangneng Hu, Horace Ho Shing Ip. and Sam Kwong. Sefe: Superficial and essential forgetting eliminator for multimodal continual instruction tuning, 2025. URL htttp:://aaxiyy. org/ hbtp//bs//2// 2505//555.... 02486
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Yc, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14185-24199 2024c.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See htips://vicuna Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems, 66:49250-492677 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv 2407, 2024.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi- multi-granularity comprehension and generation. arXiv preprint arXiv:2404. 1 /4396, 2024.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A
survey. International journal of computer vision, 129(6): 1et.118999911 2021.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
11

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. Continual instruction tuning for large multimodal models. arXiv preprint arXiv:2311.16200 2023.
Xuchai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.102880 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXivv 2017.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for vision-language models. Advances in neural information processing systems, 37:129146- 129186, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grrassk-Barwinnka. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-352 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of 2009.
features from tiny images,
Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad Shahbaz Khan. Geochat: Grounded large vision-language model for remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 27831-27840, 2024.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024a.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multi- modal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.0023: 2024b.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Iransactions on pattern analysis and machine intelligence, 40(12):2935-294) 2017.
Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.09 05358, 2022.
Chenxiao Liu and Xiaojun Wan. Codeqa: A question answering dataset for source code compre- hension. arXiv preprint arXiv:2109.08363) 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
12

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296-26306, 2024.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. et al. Deepseek-vl: towards real-world vision- Assionlanguuae understanding. arXiv preprint 4r35v:2403.05525, 2024.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.132 2021.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.00 08747, 2023.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11-20, 2016.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint 1.21:22033.10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirhan Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEEE, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang. Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-phen to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp- 2641-2649, 2015.
Jingyang Qiao, Xin Tan, Yanyun Qu, Shouhong Ding, Yuan Xie, et al. Large continual instruction assistant. In Forty-second International Conference on Machine Learning, 2025a.
Jingyang Qiao, Zhizhong Zhang, Xin Tan, Yanyun Qu, Wensheng Zhang, Zhi Han, and Yuan Xie. Gradient projection for continual parameter-efffcient tuning. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 2025b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-87633 PmLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 1908. 10084, 2019.
13

---

Published as a conference paper at ICLR 2026
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2111252) 2015.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. arXiv preprint arXiv:2205.12399, 2022.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization text-to-image diffusion with c-lora. arXiv preprint ar arXiv:2304. 06027, 2023a.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Riscante-Bonille Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11909-11919, 2023b.
Sakthivel Srinivasan, Iyo Koyanagi, Pablo Vergara, Yuteng Wang, Akinobu Ohba, Toshie Naoi, Kaspar Vogt, Yoan Chérasse, Noriki Kutsumura, Takeshi Sakurai, et al. Transient reactivation of small ensembles of adult-born neurons during rem sleep supports memory consolidation in mice. Nature Communications, 16(1):7210, 2025.
Robert Stickgold. Sleep-dependent memory consolidation. Nature, 43707063)11222-1278,P 2005.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.
arXiv preprint
arXiv:2405.098/ 098118 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprini arXiv:2307.09288 2023b.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint 2409 18869, 2024.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5699 2022a.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference computer vision, pp. 631-648. Springer, 2022b.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022c.
Matthew A Wilson and Bruce L McNaughton. Reactivation of hippocampal ensemble memories during sleep. Science, 265(5172):676-6777 1994.
14

---

Published as a conference paper at ICLR 2026
Yuan Xie, Jingyang Qiao, Weicheng Meng, Qingsong Hu, Jie Jin, Zhizhong Zhang, Xin Tan, and Jingyu Gong. Continual- next: A toolbox of multimodal continual instruction tuning, August 2025. BRLLtt:::///ttttut
com Bttp///Co-SSSICtiiaaaaaaa---
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3014-3023, 2021.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zähan Qiu. Qwen2.5 technical report, 2025. URL fttts:://ariis wttp://rg/bbb....... htttsss//aaxx..rrrr////////11111 15115
Yi Yang, Wen-tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open ppen-domain ques- tion answering. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 2013-2018, 2015.
Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXtv:2106.01085, 2021.
Dianzi Yu, Xinni Zhang, Yankai Chen, Aiwei Liu, Yifei Zhang, Philip S Yu, and Irwin King. Recent advances of multimodal continual learning: A comprehensive survey. arXiv preprint arXiv:2410.05352, 2024a.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang. Huchuan Lu, and You He. Boosting continual learning of vision-language models via ofxture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219-23230, 2024b.
Yahan Yu, Duzhen Zhang, Yong Ren, Xuanle Zhao, Xiuyi Chen, and Chenhui Chu. Progressive LoRA for multimodal continual instruction tuning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Findings of the Association for Computational Linguistics: ACL 2025, pp. 2779-2796, Vienna, Austria, July 2025. Association for Computa- doi: 10. 10.18663/112025.fnning
tional Linguistics. ISBN 979-8-89176-256-53
URL
wttp://wwwoooo..../eeeeeeee// findings-acl.... ott.attc.......o..........aaaaa. 1437
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint arXiv:2402. 12226, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint arXiv:2303. 105/2, 2023.
Xinni Zhang, Yankai Chen, Chenhao Ma, Yixiang Fang, and Irwin King. Influential exemplar replay for incremental learning recommender systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 9368-93777 2024.
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language genera- tion via generative vokens. arXiv preprint arXiv:2310.00223, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint orXiv:2304.1059/ 2023.
Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. arXiv preprint arXiv:2402. 12048, 2024.
15

---

Published as a conference paper at ICLR 2026
A SUPPLEMENTARY RESULTS OF CONTINUAL INSTRUCTION TUNING
Table Comparisons with baselines on LLa CoIN benchmark.
Method
Vooue
--Inggeeet
Accurncy on Each Jank
4.0/Q)
S.VEWW
+OCCVOO
Overal Resulis
Foogetionl.1
LokA
X2
60.13
41.11
16.82
31.13
32.67
2,50
46.33
61.30
35.08
32.00
64.47
MT
ICML:24
101.3
06:73
-66710
04:35
-0.19
33.16
03.74
05.44
57.47
11.30
07.32
PGP
K3
$2.90
04.64
49:15
02:53
08.43
37:37
05.57
66.83
59.15
10:11
67.98
Eho
37.65
58.93
92.31
60.22
38,27
33.77
64.39
68.80
61.42
5.84
66.53
C14
(100
27050
24
83.94
01.40
97.05
02.61
43.99
38.72
06.29
65.75
05.10
2.31
07.12
Dark
3302
61.99
93.98
30.8
37.33
3033
66.00
64.07
45.31
2785
07.32
Table 9: Comparisons with baselines on Qwen-VL, CoIN benchmark.
Mathal
3lIcaacNe:
Accurwy on Fach Task
4.000%
5-CKVVX
Aog-MCCT1
Onerail Ramalls
Porectinul
New.NCCTT
$1383
47.30
2017
64.30
59.67
44.04
57.79
38.18
41.76
8.69
60.05
06.42
41.33
12.16
49.83
3605
43.06
3333
61.2
Table 10: Comparisons with baselines on S-we2.55VLL-Innee Continual NEXT benchmark.
Method
Venu:
I-AminUA
2-Creot
S-Lont2A
Accuracy Each Task & Overall Besulis
4-Ceunbath
S-UndeUA
6-ImageNer
7-Hidkr9K
5-DocV124
9-1exrVAA
15.40
2.75
83:34
Let
TRAME 16
71.60
91.35
58.50
95,00
3.49
75.53
21.42
83.77
71.89
BWC
Sunppnna 24 2 2017
PNAS117
73.28
92.00
67.87
89.20
3.53
75.06
20.53
90.61
53.82
Repla Ronnn i wang
NIPS 19
71.37
87.57
76.57
08.83
2.45
83.20
20.61
86.30
71.25
MolLLok CZzt 20240
2IP5
72.19
91.53
74,36
97.66
2.05
90.75
21.08
85.51
79.55
PCLR
97.37
8:45
74.06
21.11
16.27
PCLR-LwE (PCLR
5,93
20.75
86.19
80.35
I0-MaliQA
11-ChaniQ
12
13-Cnounding
Arg-ACC.11
Furgstingg. 3 New ACC1
Zeno Zeno-bost
0.03
75.00
33.22
72.23
$2.95
2.90
47.59
Hu 2017
XCLR'22
9136
67.04
TRAMI'16
PNAN
Roplay Donnik 2112
19PS119
90.90
AEX Cime Mirka
NIP5'34
87.68
16.90
58.94
11.81
PCER
43.55
76.76
35.44
86.24
$7.27
7.74
64.26
518
69005
PCLR-L.W (PCLR variant)
45.16
76,08
46.43
85.55
$8.06
9.32
45.76
3.74
69.36
As shown in Table g we discover that PCLR achieves the highest overall performance on the CoIN benchmark, LLaVA-1.5-13B. Compared to the former best regularization method PGP (Qiao et al.) 20256), the Avg.ACC improves by 6.38 and the Forgetting decreases by 8.03, demonstrating strong continual learning ability. Furthermore, our method outperforms the extension methods Eproj (He jet al. 2023) and CIA (Qiao et al.) 2025a). Experimental results show that our method is robust across model scales and maintains strong performance on larger and powerful models.
As shown in Table 9 we discover that PCLR achieves the highest overall performance on the CoIN benchmark, Qwen-VI. VL. Compared to the former best regularization method EWC (Kirkpatrick et al., 2017]. the Avg.ACC improves by 13.12 and the Forgetting decreases by 4.47, demonstrating strong continual learning ability. Experimental results show that method is robust across models of different architectures and maintains strong performance even when the base architecture varies.
As shown in Table TO our method exceeds the best of other methods (Replay), improving Avg.ACC by 2.77 and reducing Forgetting by 1.22, which demonstrates that PCLR sustains strong perfor- mance on the long-term Continual-NExT benchmark, Qwenn.5-VL-IIntrruct. Similar the main paper, we introduce PCLR-LwF, which divides the whole task sequence into five consecutive groups, and applies LwF in each group. This integration with simple baseline yields notable gains by im- proving Avg.ACC by 1.52 and reducing Forgetting by 1.44, while preserving the same parameter number as PCLR.
In summary, extensive experiments on Large Multimodal Models (LMMs), including L.. 7B, LL5VA-1.5-13BB and Qwen-VI. on the CoIN benchmark, I.LVA- and Qwen2.5-V6 Instruct on the Continual-NExT benchmark, demonstrate the superiority of PCLR. The method achieves an effective balance between plasticity, stability and memory efficiency. Importantly, PCLR prevents unbounded parameter growth via the Compessionn-Iiteeaainn-LLLeeee Moreover, its modular design enables seamless integration with regularization methods such as LwF, demonstrated by the PCLR-LwF variant, which further improves training efficiency without sac- rificing performance. These results establish PCLR as a scalable, efficient and robust framework for multimodal continual instruction tuning in LMMs.
(CIL) pipeline.
16

---

Published as a conference paper at ICLR 2026
B DETAILS OF EXPERIMENTAL SETTING
Overall Setting: For each dataset, we set the training epoch to 1. The LLM backbone and cross- modal modules are incorporated into the PCLR framework. Throughout all phases, only the inserted PCLR modules are trainable, and the LMM remains frozen. During learning, the learning rate is set 10-4 for LLaVA series, 2 x 10-5 for Qwen-Viz VL, 1 x 10 for Qwen2.5-VL-Insteee VL-Instruct During integration, only data from the current task is used, the learning rate is set to 5 x 10 for LLaVA- 1.5, 1 x 10m5 for Qwen-VIe while 10 for LLaVA-1.5-hr and Qwen2.5-Ve. ---n25--L--nnstruct Weight decay is set to O. The maximum input embedding length is fixed as 2048. We employ gradient checkpointing and mixed-precision training using TF32 and BF16. We use DeepSpeed ZeRO-2 for distributed training. All experiments are conducted on 4 80-GB GPUs. During learning, employ a rank-64 extension and later switch to rank-32 extension. During compression, we adopt retention rate of 100% (no compression) for the first three tasks, then set it to 75%, and finally adjust it to 87.5%
LoRA Rank Configuration: In the main paper, all baselines fine-tuned using LoRA with rank 128. However, due to the MoE. architecture PCLR, it is infeasible to ensure a fair comparison of both total and activated ranks simultaneously. To balance parameter efficiency and memory cost while preserving the MoE structure, we adopt: total rank 256, activated rank 64 (activated during forward passes both training and inference), and optional shared rank 32 (activated during forward passes in both training and inference, trainable during Integration and frozen during Learning). This setting balances total rank and activated rank maximize overall fairness.
Leamiing in Group by LwF
Old Ranks
Old Ranks
S Old Ranks
Compression & Integration
New Ranks
New Ranks
New Ranks
Integration Ranks
Integration Ranks
Compression & Integration
Learning in Group by LwF
Old Ranks
Old Ranks
Old Ranks
Ranks
New
Figure 5: PCLR-LwF Overview. In long-term CIT, the task sequence is divided into several groups, with the tasks in each group being contiguous in the original sequence. Within each group, we apply LwF for learning. Between the groups, the CIL process is adopted.
Combination with LwF: the main paper, we propose an optimization for long-term continual instruction tuning: instead of running CIL after each task, perform constrained updates in the newly extended rank space using regularization methods such as LwF (Li & Hoiem 2017). for a subset of tasks. As described in the main paper, the Continual-NExT benchmark, group consecutive tasks into a cluster and apply LwF for constrained updates within each cluster, with a weight of 1. This approach reduces training cost and improves performance while maintaining the same total rank as the standard PCLR. The PCLR-LwF pipeline shown in figure 5
Equivalent Setting for Fake Query Sampling: The main paper states that fake queries are sam- pled from the key pool. However, during continual compression, the number of keys from earlier tasks decays exponentially. This produces unfair sampling of synthetic task-related queries. Af- ter completing each task, we average its newly generated keys and place the resulting mean vector into a global task key pool. Each task thus retains single representative fake query, which keeps cross-task sampling balanced during the integration.
17

---

Published as a conference paper at ICLR 2026
C THE CASES OF CONTINUAL INSTRUCTION TUNING
In this section, present evaluation cases after completing the CoIN (Chen al. 2024a) bench- mark by using Pretrain, MoËLoRA, PCLR, and Ground-Trune
Model after continual tuning, test ScienceQA
Answer from Pretrain
The correct answer is C. the Arctic Ocean.
A
Answer from MoELoRA
Q
Which ocean is highlighted?
A. Atlantic Ocean
A
Answer from PCLR
B. Indian Ocean
C.the Arctic Ocean
D.the Southern Ocean
Answer with the option's letter from the given choices directly.
A
Answer from Ground- Tround-Truth
Model after continual tuning, test TextVQA
A
Answer from Pretrain
The plane features the FXQ centre.
A
Answer from MoELoRA
Soaring Centre Husband & Bosworth
Q Which centre featured on the planc?
A
Answer from PCLR
Reference OCR token: FXQ, THE, SOARING, CENTRE, HUSBANDS, BOSWORTH, PuCHACZ Answer the question using a single word or phrase.
soaring centre
Answer from Ground-True
the soaring centre
Answer from Pretrain
Model after continual tuning, test ImageNet
The object in the image is dog.
A Answer from MoELoRA
dog.
A
Answer from PCLR
Q What is the object in the image? Give the object the image in the form of single word or a concise phrase.
Pomeranian.
A
Answer from Ground- Truth
Pomeranian
Figure 6: From top to bottom, these are cases on ScienceQA, TexIVQA, and ImageNet after learning the final task of the CoIN benchmark.
18

---

Published as a conference paper at ICLR 2026
Answer from Pretrain
Answer from MoELoRA
Answer from PCLR
Answer from Ground-Trun
Which kind of fast food is the plate?
Hot dog.
Sandwiches
Hot dog
Hot dog
What type of clothing the picture is striped?
The shirt that the person is wearing is striped.
Q
None
Shirt
Shirt
A
What kind of clothing is black?
The shorts that the person is wearing black.
None
Jeans
Pants
A
What type of baked good is to the right of the hot dog that is on the left?
There is no baked good to the right of the hot dog that on the left.
Q
None
Bun
Bun
A
What is the man to the right of the fence holding?
The man to the right of the fence is not holding anything.
Hot dog
Plate
Plate
What the vegetable that is the right of the tomato that is the left?
There no vegetable the right of the tomato that on the left.
Onion
Onion
Onion
Figure 7: The case on GQA after learning the final task the CoIN benchmark. This is case of multiple rounds of dialogue.
19

---

Published as a conference paper at ICLR 2026
QuestionI:
Can you tell what the error message on the Windows screen?
Model after continual tuning, test on VizWiz
When the provided information is insufficient, respond with 'unanswerable'. Answer the question using single word or phrase.
Pretrain: unanswerable
MoELoRA: unanswerable
PCLR: starting windows
Ground-Trun starting windows
Question2: What is the title of this book?
Pretrain: I'm sorry, but I cannot answer your question as there is no book visible in the provided image. Please provide a different image question.
MoELoRA: Windows 7: The Missing Manual
PCLR: unanswerable
Ground-Trun unanswerable
Model after continual tuning, test on Grounding
Identify and provide the coordinates of the bounding box described in the sentence: bananas in the middle.
Question1:
Question2:
Give the bounding box coordinates described in the sentence: top left bunch.
Pretrain: I'm sorry, but cannot provide the bounding box coordinates for the sentence V'lop left bunch!" as it is not clear what region or object being described. Please provide more context specific sentence for me to assist you.
Pretrain: I'm sorry, but cannot provide the bounding box coordinates for the sentence \"bananas in the middle'" as it is not clear what region object is being described. Please provide more context or specilic sentence for me to assist you.
MoELoRA:
MoELoRA: (0.,,,0522.061100666
PCLR: [0.32,0.39,0.66,00776 Ground-Trath: [0.35.0.4.0..6..0.66]]
PCLR: [0.0,0.24,0.55,0.499
Ground-Truth: [0.0.0.22.0.24.0044]
Figure 8: From top to bottom, these are cases on VizWiz and Grounding after learning the final task of the CoIN benchmark.
20

---

Published as a conference paper at ICLR 2026
D LAYER SENSITIVITY ANALYSIS IN PCLR
In this section, we investigate the sensitivity of different L.M layers to compression and integration in PCLR for continual learning. Based on the 40-layer LLaVA-1.5-133 Transformer architecture, we divide it into 5 groups of 8 consecutive layers (Lower: layers 1-8, Middle-Lower: layers 9-16, Mid- dle: layers 17-24, Middle-Upper: layers 25-32, Upper: layers 33-40). In each experiment, we skip the Compression and Integration stages for one group (opened group), while keeping the other four groups unchanged. The results in Table [T show that task-specific sensitivity varies across layers, with the highest performance degradation observed in upper layers compression (-0.53@ Avg-ACC) and the lowest in mid-level layers compression (-0.15@ Avg.ACC). This observation highlights the layer-specific characteristies in PCLR.
Table 11: The results of LLaVA-1.5-138 on different opened groups.
Accuracy Each Task
Evaluation Metries
Layers
ScienceQA
TextVQA
ImageNet
GQA
VizWiz
Grounding
VQAV2
OCRVQA
Avg.ACC
Forgetting
Origin
83.82
61.99
93.98
59.49
57.98
36.03
66.09
64.67
65.51
2.08
Lower 86.02
62.28
94.28
60.40
56.80
35,33
66.51
63.99
65.70
1.56
Middle-Lower
63.18
60.43
56.54
36.21
66.05
62.93
65.74
1.36
Middle
83,99
62.49
94.89
59.87
58.76
34.93
66.14
64.19
65.66
1.73
Upper
83,80
62.04
95.07
59.58
58.44
36.70
66.16
64.31
65.76
1.67
Upper
83.85
61.89
96.46
60.02
59.23
36.13
66.15
64.56
66.04
1.32
We hypothesize that lower layers focus on input comprehension, mid-level layers focus on reason- ing, and upper layers focus inslluition-folloiingg output generation. Since reasoning processes across tasks often share a high degree of similarity (general logic patterns), mid-level compres- sion causes minimal interference. However, upper layers face significant task-specific divergence in output generation (single-choice answers in ScienceQA v.s. classification answers in ImageNet w.s. bounding- answers in Grounding), leading to pronounced conflicts during compression- integration. Notably, the compression of lower layers is especially sensitive to ScienceQA because its inputs mix text-only and image-text samples, unlike tasks that are exclusively image-text. This indicates that heterogeneous modality distributions can amplify compression-induces performance drops in the lower layers.
Our findings motivate layer-aware compression strategy for future CIL optimization: applying maximal compression retention ratio to upper layers (high conflict), moderate compression retention ratio to lower layers (moderate conflict), and minimal compression retention ratio to mid-level layers (low conffict). This optimization is expected to achieve further performance improvements while maintaining similar memory budget.
E EVALUATION METRICS
We emphasize that our evaluation of prediction accuracy is based on a comparison between the outputs of LMMs and the corresponding ground-truth annotations. This evaluation protocol in [Ceen let al. (0224)). provides a rigorous and consistent criterion. For short textual outputs we apply Truth Alignment, and for longer textual outputs compute semantic similarity using Sentence- Transformers (Reimers & Gurevych] 2019) and accept when the similarity score is at least 0.8.
We adopt 3 primary metrics to comprehensively evaluate continual instruction tuning performance:
Average Accuracy (Avg.ACC) measures the average test accuracy across all datasets, reflecting the overall performance of models throughout the continual learning process.
Forgetting (FOR) quantifies the decline in performance on previously learned datasets after training on new datasets. serves an indicator of stability (retaining previous knowledge).
New Accuracy (New.ACC) computes the average test accuracy on newly introduced datasets. serves indicator of plasticity (adapting to new tasks).
21

---

Published as a conference paper at ICLR 2026
Overall, these metries generally defined follows:
Avg.ACC
Σ
ATi
(8)
T
1=1
FOR
max Az,t Ar. jeRt,1
New.ACC 1 Σ A
(10)
where denotes the total number datasets, AT, represents the accuracy of the i-th dataset eval- uated on the model after training the T-th (final) dataset, Aj.i is the accuracy of the i-th dataset on the model after training on the j-th dataset, and Ar the accuracy of the i-th dataset evaluated immediately after its own training.
F VISUALIZATION OF EXPERT TRANSITION DURING CIT
In this section, we visualize the expert transition process during continual instruction tuning (CIT) within the PCLR method. We follow the task sequence defined by the CoIN benchmark: ScienceQA - TexttVA + ImageNet - GQA - VizWiz - Grounding - VQAv2 - OCRVQA. We present the rank experts activation patterns the last five tasks. Specifically, the invocation rate of the rank experts on the current task and previously learned tasks shown as below.
Rank Experts Invocation Rate of Learned Tasks
0.8
Invocation Rete
SclenceAA
imaggeNet
0.4
4GG
Rank Expert Index (0-255)
Figure 9; The invocation rate of rank experts after learning the 4-th task GQA.
Rank Experts Invocation Rate of 5 Learned Tasks
0.8
Invocation Rate
1-SckinoG QA
3-maggeeet
4-G0A
0.4
0.2
Rank Expert Index (0-255)
Figure 10: The invocation rate of rank experts after learning the 5-th task VizWiz.
22

---

Published as a conference paper at ICLR 2026
Rank Experts Invocation Rate of Learned Tasks
1.0
Invocation Rate
1-Scenoo
2-TextVo
3-mageNet
0.4
5-Viatiz
Grounding
0.0
50
150
250
Rank Index (0-255)
Figure 11: The invocation rate of rank experts after learning the 6-th task Grounding.
Rank Experts Invocation Rate of 7 Learned Tasks
0.8
Invocation Rate
1-Scencc0A
2 TacVVA
imageNet
S-NuWi
0.44
6-Grounding
Expert Index (0-255)
Figure 12: The invocation rate of rank experts after learning the 7-th task VQAv2.
Rank Experts Invocation Rate of 8 Learned Tasks
0.8
Invocation Rate
2-TexiVA
3-mageMet
4-0GA
e-Grounding
VOANZ
8-OCRVOA
0.2
50
100
150
200
250
Rank Expert Index (0-255)
Figure 13: The invocation rate of rank experts after learning the 8-th task OCRVQA.
G THE RESULTS OF CONTINUAL-NEXT BENCHMARK
We present the performance of PCLR and its variant PCLR-LwF on the Continual-NExT bench- mark, and visualize the final accuracies of two representative models (L.LVVA-1.5-hf and
23

---

Published as a conference paper at ICLR 2026
An
AnaivQA
Geachat
NikiA
Geachat
Scleycecc
longe
Grounding
CronMath
Grounding
CododA
Imaggetett
MathOA
TEXEVOA
DocVQA
PCLR
Baseline
TEXIVOA
DocVGA
PCLR
Baseline
PCLR
(a) Comparison on LLaVA-1.5-hf
(b) Comparison on Qwen2.5- --L-5------nnttrrct
Figure 14: Radar chart of comparisons on Accuracy between baseline (LoRA) and ours.
using radar charts. As shown in Figure 14 PCLR achieves significant base- line improvements. Notably, PCLR-LwF incorporates LwF (Li & Hoiem 2017) which is regularized reduce the compression demand, achieving improved computational efficiency while further im- proving the performance of PCLR. Thus, PCLR enables compatibility with other continual learning algorithms, offering practical solution for deployment in real-worke applications.
H THREE TYPES OF TUNING ORDER SEQUENCES
To evaluate robustness under different Continual Instruction Tuning orders, we employ three task ordering strategies:
1. Original Order:
Tasks are presented the default sequence: ScienceQA, TextVQA, Im- ageNet, GQA, VizWiz, Grounding, VQAv2, OCRVQA.
2. Reverse Order: The sequence reversed relative to the original: OCRVQA, VQAv2, Grounding, VizWiz, GQA, ImageNet, TextVQA, ScienceQA.
3. Alphabetical Order: Tasks are sorted alphabetically: GQA, Grounding, ImageNet, OCRVQA, ScienceQA, TexIVQA, VizWiz, VQAv2.
THREE TYPES OF INSTRUCTION TEMPLATES
To evaluate robustness under different instruction templates, adopt three types of instruction templates. Detailed examples of these templates are provided in Table 20
Original Instruction Template: Each task associated with a single instruction, and multiple tasks may share identical instruction formats. This setting reflects minimal prompt diversity.
2. Diverse Instruction Template: Each task uses a single, uniquely designed instruction. Instructions are carefully tailored to reflect the semantic and structural characteristics of individual tasks, maximizing prompt diversity.
3. 10Type Instruction Template:
Each task is assigned approximately ten distinct instruction variants. While instructions are diverse within each task, certain templates may shared across related tasks, simulating balanced scenario between diversity and generalization.
24

---

Published as a conference paper at ICLR 2026
J DETAILED SETTINGS FOR DIFFERENT COMPRESSION STRATEGIES
All settings are maintained at total rank = 256, activated rank = 64, and share rank = 32.
Aggressive: The compression retention rate is fixed 75%, and the number of new rank experts is
fixed at 64.
Conservative: The compression retention rate is fixed at 87.5%, and the number of new rank experts fixed at 64.
Reverse: The compression retention rate changes from 87.5% 75%, with the number of new rank experts fixed at 32 - 64.
The first eight tasks are not compressed, and after all tasks are learned, 50% retention rate compression is applied (forgetting is calculated based on 8 tasks). In terms of integration data, compared the Ours (integrating the current dataset after each learning cycle), it is relatively singular (the last integration only shows OCRVQA data).
Centralized:
Ours: The compression retention rate changes from 75% 87.5%, with the number of new rank experts fixed at 64 + 32.
The dynamic adjustment strategy we adopt is based the distillation loss during the integration phase: if, within the 50th-70th training steps of the integration process, at least 25% of the fake queries (i.e., 25% of the learned tasks) exhibit KL divergence losses exceeding predefined thresh- old (set to 0.05 in this paper), it is determined that the current compression intensity is too high, causing significant knowledge conflicts between tasks. This occurs because the optimization ob- jectives of different tasks generate severe discrepancies over the same rank components, leading to interference in task-specific information.
In such cases, the system increases the retention rate and re-executes the compression-integratio process, reducing the degree of information fusion between tasks to mitigate the loss of historical knowledge. Simultaneously, during the learning phase of subsequent tasks, the number of newly added ranks correspondingly reduced (e.g., from to 32) to maintain the total parameter capacity constant.
K
TASK GROUPING OF PCLR-LWF
Task Grouping for LLa VA-1.-5.
Group 1-ArxivQA, 2-GeoChat, 3-IconQA
Group 2: 4-ClevrMath
Group 3: 5-CodeQA, 6-ImageNet
Group 4: 7-Flickr30k, V-DA 9-TextVQA
Group 5: 10-MathQA, 1--haarQA, 12-PathVQA
Group 6: 13-Grounding. 14-ScienceQA, 15-WikiQA
Using a compression retention ratio of 66.67%, the final PCLR consists of 256 expert ranks.
Task Grouping for Qwwen2.--VL-Instructt
Group 1: 1-ArxivQA, 2-GeoChat, 3-IooQQA
Group 2: 4-ClevrMath, 5-CodeQA, 6-aageee
Group 3: 7-Flickr30k, 8-DocVQA, 9-TextVQA
Group 4: 10-MathQA, 1-ChartQA, 12-Path
Group 5: 13-Grounding, 14-Scienc, 15- 15-WiKIQA
Using compression retention ratio of 80%, the final PCLR consists of 256 expert ranks.
This different grouping strategy is motivated by the observation that the LLaVA-1.5-hff after training on ImageNet performs worse ClevrMath. To address this, place the ClevrMath task in
25

---

Published as a conference paper at ICLR 2026
dedicated group. The memory budget can be mitigated by either reducing the compression retention ratio or increasing the number compression times.
L COMPARISON OF TRAINING COST WITH MOELORA
Table 12: The Training Cost of LLaVA about different numbers of experts.
Training Cost Each Task (Unit: Minutes)
Number
I-ScienceQA
2-TexiVQA
3-ImageNet
4-GQA
S-iizWiz G-Grounding 7-VQAV2 8-OCRVQA
Average
10.05
33,55
91.22
118.12
17.78
79.58
93.44
135.68
72.43
2
17.73
"50.97
130.6
173.58
27.94
115.71
136.53
188.86
105.24
21.25
"68.1 14
189.05
245.97
"3..25"
169.12
205.52
286.92
152.65
Ours
34.43
94.82
115.49
19.85
93.68
121.87
177.12
83.27
As discussed in the main paper, all computations in our fine- Ane-grained LRP, are fully parallelizable. On LLaVA-13BB we compare LRP (total rank 256, activated rank 64, shared rank 32) with MoELoRA (total and activated rank of 128) variants having 1, 2, and experts, summarized in Table 12
The forward computation of MoELoRA is defined as:
xW + + B (T))
(11)
where s(x) E I' denotes the expert routing scores, t is the number of total experts, k is the number of activated experts, and A,, B, are the LoRA weights of the i-th expert.
The proposed LRP forward computation in the main paper is formulated as:
(12) where As; B, represent the globally shared components, and Am, Bm represent the mixture expert components. The operator F (U, broadcasts v E R to match the shape of E R followed by an element-wise multiplication. Here, K E _nxd, ,9 Rd are the L2-regularized key pool and query, n is the number of total rank experts, r is the number of activated rank experts.
y = xW + A,,A, + 3m F (x:Am gate (Kg,r) r)*) BT
MoELoRA consistently actívates all ranks and performs serial summation across experts, whereas LRP avoids serial computation by directly integrating expert scores into the LoRA formulation. Specifically, we set the number of experts match the LoRA rank to align the expert score dimen- sion with the LoRA rank. This design enables parallel integration of expert scores into the LoRA forward computation. This yields efficiency comparable to MoELoRA with expert because GPU matrix multiplication is faster than serial multiplication and summation over smaller matrices the same total rank.
M INSPIRATIONS FROM ADALORA AND THE L2P SERIES
AdaLoRA /Zhang et al. 2023) converts each LoRA adapter into a SVD-style representation, en- abling dynamic rank adjustment under fixed parameter budget and thereby improving computa- tional allocation efficiency. The L2P series (Wang et al. 2022c b Smith et al., (2023b) decomposes conventional prompts prefixes into subcomponent representations paired with learnable keys. further introduces dynamic parameter allocation mechanism, markedly improving performance and efficiency in continual learning for the vision domain (Krizhevsky et al. 2009 Peng et al.) 2019).
Inspired by AdaLoRA, we likewise decompose LoRA updates into rank vectors but remove explicit rank constraints, and adopt an L.2P-like key-value matching scheme, we associate each rank vector with a learnable key and allow dynamic extension during CIT. This simultaneously (i) extends the L2P series to the more parameter-efffiient LoRA paradigm, facilitating adaptation to LMMs, and (ii) generalizes adaptive rank allocation principle AdaLoRA to a Mixture-of Mfxture-of-EEpprtt (MoE) structure tailored for continual learning, In summary, we introduce the LoRA Rank Pool (LRP), extremely fine-grained MoE architecture that offers maximum ficxibility for knowledge employment and edit- ing enhance stability and plasticity.
26

---

Published as a conference paper at ICLR 2026
N EXPLANATION OF RELATIVE REDUNDANCY RANK
In this section, we provide a detailed definition of the relative redundancy rank, namely the relative redundancy rank of M N for the LoRA pair (M, N). We select an arbitrary identically named weight pairs {[AMs, BAs), (AN, Bx)}, and perform an orthogonal triangle decomposition on Ay:
QR = AN.
(13)
where Aws AN E R4axx, Rddar and Q is a set of orthogonal vectors of AN- Subsequently,
we apply 1.2 normalization to every column of A AM. producing Ars For each E Rddx in Am, we use Q to reconstruct:
a" QQTa.
(14)
Therefore, can obtain the reconstruction loss for the part (a, AN):
La = ||,1
(15)
La E [0, 1] reflects the degree of linear dependence between and the vectors in AN, a smaller value indicates higher degree of linear dependence. When La 0, and AN are linearly dependent. When C. = 1, the inner product between a and every v E AN is zero.
Similarly, can obtain the reconstruction loss for the part of (6, BN), Ls E [0. Next, we define the reconstruction loss for the rank (a, b) E (AM, BAr) with respect to LoRA Bx):
[ C. Co:
(16)
L E [0, 1] reflects the degree of linear dependence between the rank component (a, b) E (AM, Buy and the LoRA weight pair (Av, Br). According to the theoretical insights from LoRA (Hu et al. 2022] and AdaLoRA (Zhang et al. (023), rank components that are linearly dependent are redun- dant, and they carry knowledge similar to other rank components, which is the knowledge redun- dancy referred to the main text.
We define the threshold the LoRA weight (Ax, Bx), if its score L < a, this component deemed a redundant rank. Next, obtain the redundancy rank ratio of (AM, By) relative to (Ax, BN)) (the corresponding LoRA weight pairs ((AM, BM), (AN, Br))a. Finally, by computing L of all attention modules, obtain the relative redundancy rank ratio for NM (M relative N).
0.001. For each rank component (a, 6) (AM, BM) relative to
O THE EFFECT OF EPOCH ON PCLR
To verify the impact of epoch count on PCLR's performance and forgetting resistance, we conducted experiments the LLaVA-1.5-70 model and the CoIN benchmark, setting training cycles of 1, 3, and epochs to compare the performance of PCLR and PGP:
Table 13: Results of LLa VA- 1.5-7B on PGP and PCLR methods with different epochs.
Methods-Rpoch
Accuracy on Fach
Overall Resnlis
Science(AA
TextVQA
ImageNet
GQA
VizWiz
Grounding
VQAv2
OCRVQA
Avg.CC
Forgetting
New.ACC
PGP-T
85.17
56.85
12.26
61.74
49.43
12.74
63.74
62.20
55.77
12.94
67.09
PGP-3
85:38
57.14
32.59
62.15
66.06
62.43
56.07
12.76
PGP-5
85.50
57.43
32.75
62.31
49.88
33.46
66.22
62.67
56.28
12.63
67.33
PCLR-1
78.33
38.14
37.61
33.04
64.17
61.92
62.19
3.39
PCLR33
84.56
59.73
91.66
57.31
58.12
35.65
65.03
62.06
64.27
2.75
66.68
PCI R-5
86.04
$9.24
93.29
55.64
57.81
36.8
64.41
61.69
64.37
2,36
66.43
Note: To ensure fairness, the integration data usage (IDU) for PCLR-1, PCLR-3, and PCLR-5
remains the same.
PGP shows limited improvement over longer epochs: Although PGP restricts updates through gra- dient constraints, during longer epoch training, the trainable parameters gradually skew toward the optimal solution for the current task, leading to overwriting of historical knowledge and maintaining high forgetting rates (12.94 12.76 + 12.63).
PCLR's CIL mechanism demonstrates stronger performance over longer epochs: During training with 3 and 5 epochs, PCLR's forgetting rate remains significantly lower than PGP (2.75 v.s. 12.94,
27

---

Published as a conference paper at ICLR 2026
2.36 v.s. 12.63), and the Avg.ACC. continues to improve (62.19 + 64.27 - 64.37), the Forgetting continues to reduce (3.39 + 2.75 + 2.36).
We can analyze the reasons for the performance improvement from the perspective of CIL. Learn- ing phase: New tasks only update newly added rank experts, avoiding interference with historical knowledge, so increasing training epochs does not lead to forgetting. Integration phase: A well- learned set of parameters has less noise, and it is easier for compatible representations to emerge across different tasks, which is beneficial for integration. Knowledge distillation merges new and old knowledge, ensuring stability improvement. The time spent in this phase has positive feedback relationship with the effectiveness of forgetting resistance.
In summary, PCLR's 's design not only adapts to tasks with varying training lengths but also excels in multi-epoch training, ensuring efficient knowledge integration and low forgetting rates.
P MORE ABLATION EXPERIMENTS
In this section, we discuss how to balance CIT performance and memory efficiency from two per- spectives: knowledge density and memory scale. For more intuitive comparisons, we use total rank experts + shared ranks to measure the static storage parameters (SSP) and activated rank experts + shared ranks to measure the inference activated parameters (LAP).
Table 14: Results of LLaVA-1.5-7B on different knowledge density.
Method
Allocation
Final Accurayy on Each Tisk
Overall Rosalis
SSF
TXP
ScienceOA
TestVQA
ImageNNt
GQA
VizWaz
Grounding
VQ,22
OCRVQA
Avg.ACC
Forgetiing
NeW.ACU
Dause Space
192+32
64-32
72:01
53.68
88.32
58.19
55.23
30..22
04.38
63.40
00.75
4:83
64.98
Sparse Spros
330+32
64-32
8071
99.36
90.81
56.05
31.32
65
62.84
Dynomic Space
386+32
64-32
81.54
59.37
93.05
$6.44
57.05
30.80
64.46
62,80
63.19
2.06
65.00
Ours 256+32
64-30
78.33
58.24
86.08
58.
57.61
33.04
64.17
61.92
62.19
3.39
65.16
Dense Space:
With total rank experts 192, the initial compression retention rate is adjusted to
66.67%, and the compression strategy is 66.67% - 88.33%, with new ranks changing from 64 -
32. 1/3 of the experts are activated during inference, resulting in dense knowledge compression and significant overlap among rank experts. Consequently, Avg.ACC decreases (60.75), Forgetting increases (4.83), and SSP reduces by 22.22% compared to ours.
Sparse Space: With total rank experts = 320, the initial compression retention rate is adjusted to 80%, and the compression strategy 80% 90%, with new ranks changing from 64 - 32. 1/5 of the experts are activated during inference, with weaker sharing among rank experts. Consequently, Avg.ACC
increases inn.raCess (62.84), Forgetting decreases (2.65), and SSP rises by 22.22% compared to ours.
Dynamic Space: Total rank experts dynamically change from 256 + 384. with 32 ranks compressed and 64 ranks added each step. This setting achieves the highest Avg.ACC (63.19), the lowest Forgetting (2.06), and the largest final SSP.
The data shows that forgetting largely depends on the knowledge density of the PCLR parameter space (activated rank experts / total rank experts). Higher values tend to favor knowledge integration, while lower values favor expert specialization. Knowledge density a flexible parameter designed to balance memory efficiency and continual learning capability. It can be adjusted based on memory constraints (there is no universally optimal setting; it depends on memory conditions).
In summary, the optimal compression strategy involves transition from aggressive to conserva- tive, and the initial compression rate can be adjusted as needed. We do not recommend treating it as fixed parameter.
Table 15: Results of LLaVA-1.5577 on different memory scale.
Mothoe
Rank Allocation
SSP
IAP
SclenceOA
TesTVOA
ImaggeNet
Final Accuracy Each Task
GOA
Grounding
VQAr2
OCRVOA
AvgACC
Overall Resulis Forgetting
NewACC
Scale
12640
32e0
78.87
38,23
52.16
36.77
35.81
20.76
03.30
60.14
00.38
4,13
64.19
Shared
25610
6440
58.09
83.57
56.93
64.41
62.40
62.03
3.63
65.21
Ours
256432
64633
78.33
58.34
8608
58. 14
$7.61
33.04
64.17
61.92
62.19
3.39
65.16
Small-Scale Parameter Space: Compared to No Shared, memory usage is halved, but performance declines across the board: Avg.ACC (62.03 - 60.58), Forgetting (3.63 - 4.13), and New.ACC (65.21 + 64.19). This reflects insufficient plasticity and severe parameter contention, leading to increased forgetting.
28

---

Published as a conference paper at ICLR 2026
No Shared Ranks: Compared to ours, the lack of shared ranks during integration prevents the absorption of global knowledge, resulting in higher forgetting (3.39 + 3.63).
The data in the table indicates that the scale of the parameters is positively correlated with model performance. Small-Scale, despite having the same total parameter space as regularization-be methods, only utilizes 1/4 of the activated/trained parameters, yet it still outperforms PGP (55.77) and SEFE (58.57), which are among the best regularization-base methods. Both No Shared and Ours, with fewer activated parameters and fixed memory budget, surpass the extension-based method Eproj (60.79) in performance.
Q MEMORY EFFICIENCY COMPARISON
Using the LLaVA-1.5-7b model on the CoIN benchmark, we further compared PGP, Eproj. PCLR- small (0 shared, 32 active, 128 total), and PCLR-ours (32 shared, 64 active, 256 total), trainable parameters, and final performance (all statistics exclude the base model and focus solely on the adapter components).
Table 16: PGP, parameters and accuracy across tasks.
Tasks
Metric
ScienceQA
TextVQA
ImageNet
GQA
VizWiz
Grounding
VQAv2
OCRVQA
Total Params
340.80M
340.80M
340.80M
340.80M
340.80M
340.80M
340.80M
340.80M
Activated Params
340.80M
340.80M
340.80M
340.80M
340.80M
340.80M
340.80M
340.80M
Trainable Params
340.80M
340.80M
340.80M
340.80M
340.80M
340.80M
340.80M
340.80M
Final Accuracy
85.17
56.85
32,26
61.74
49.43
32.74
65.74
62.20
Table 17: Eproj, parameters and accuracy across tasks.
Tasks
Metric
ScienceQA
TextVQA
ImageNet
GQA
VizWiz
Grounding
VQA+2
OCRVQA
Total Params
340,80M
660.61M
980.43M
980.43M
980.43M
1300.24M
1300.24M
1300.24M
Activated Parums
340. 80M
340.80M
340.80M 80M
340.80M
340.80M
340.80M
340.80M
340.80M
Trainable Params
340.80M
800.
340.80M
340. 80M
340.80M
340.80M
340.80M
Final Accuracy
78.51
57.53
92.35
55.93
44.67
36.59
63.74
57.00
Table 18: PCLR- parameters and accuracy across tasks.
Tasks
Metric
ScienceQA
TextVQA
ImageNet
GQA
VizWiz
Grounding
VQAv2
OCRVQA
Total Params
85.79M
171,58M
257.37M
343. 14M.
343.16M
343.16M
343.
343. 16M
Activated Params
85.79M
85.79M
85.79M
85.79M
85.79M
85.79M
85.79M
85.79M
Trainable Params
85.79M
85.79M
85.79M
85.79M
85.79M
85.79M
42.90M
42,90M
Final Accuracy
78.87
58.23
82.16
56.77
53.81
30.76
63.86
60.
Table 19: PCLR-oure parameters and accuracy across tasks.
Tasks
Metric
ScienceQA
TextVQA
ImageNet
GQA
VizWiz
Grounding
VQAv2
OCRVQA
Total Params
171.58M
343.16M
514,74M
686.31M
766.70M
766.70X 706.70M
766.70M
766.70M
Activated Params
171.58M
171.58M
171.58M
171.58M
251.97M
251.97M
251.97M
251.97M
Trainable Params
171.58M
171.58M
171.58M
171.58M
171,58M
171.58M
85.79M
85.79M
Final Accuracy
78.33
58.24
86.08
58.14
57.61
33.04
64.17
61.92
PCLR imposes strict upper bound on total ranks, halting parameter growth after early expansion compression-integratin. In contrast, Eproj expands continuously, risking parameter explosion. PCLR outperforms both PGP and Eproj in average accuracy (62.19 55.77 60.79) and forget- ting (3.39 v.s. 12.94 5.42), demonstrating its ability to control memory while maintaining strong continual learning performance.
When the total parameter count constrained to a level similar to PGP, PCLR still achieves com- petitive performance with an Avg.ACC of 60.58, close to Eproj (60.79) and significantly higher than PGP (55.77). These results demonstrate that PCLR can effectively utilize limited resources, providing robust performance under strict parameter budget constraints.
29

---

Published as a conference paper at ICLR 2026
R MODELS
2024): LLaVA-1.5 is foundational Large Multimodal Model (LMM) that integrates Vicuna (Chiang et al. 2023) as its large language model (LLM) backbone and CLIP- VTT (Radford et al. 2021) the visual encoder. The cross-modal bridge module employs linear projection layers to align visual features with linguistic representations. Specifically, CLIP-ViT ViT extracts image embeddings, which are then projected through línear transformation to match the hidden dimension of Vicuna. This design enables efficient fusion of visual and textual information for downstream tasks such visual question answering (VQA) and image captioning. In this paper, -LaVA-1-5--31 and LLaVA-1.5-7BBhh (a fine-tuned version of LLaVA-
LL. (Liu et al.
we use L.aVA-11.-7E 1.5-7B).
Qwen- VL (aii et al. 2023b): Qwen- VI is Visual-Large Language Model (VLLM) developed by Alibaba Cloud, combining QwenLM (Bai al. (023a) as its LLM backbone and Vision Trans- former (ViT) for visual encoding. Its cross-modal bridge module utilizes Q-Former (a transformer- based architecture) designed for multi-modal feature fusion. The Q-Former attends visual and textual tokens, enabling context-aware interaction between modalities.
Qwen2.5-VL (Bai al. 2025): Qwen2.5-VL. is an advanced version of Qwen-VL (Bai et al., 2023b), featuring Qwen2.5-LM (Yang et al. 2025) as its LLM backbone and a cross-modal ar- chitecture. Its modall bridge module works as follows: RMSNorm first normalizes the visual features, followed by three-layer MLP that projects them into the linguistic space. This replaces the Q-Former Qwen-Ve VL with lightweight yet powerful structure, enabling more efficient pa- rameter allocation. Qwen2.5- Siven..5-VL-Instten VL--Instruef on curated, high-quality instruction data, improving its adaptability to downstream tasks.
upon Qwen2.5-VI. through supervised fine-tuning
S METHODS
LoRA (Base) (Hu et al. 2022): LoRA is parameter-efficient fine-tuning method that introduces low-rank decomposition matrices into the weight matrices pretrained models. By freezing the original model weights and using low-rank matrices to capture task-specific information, LoRA significantly reduces the number of trainable parameters while preserving the model's generalization ability. It is widely applied in natural language processing and computer vision due to its low memory consumption, fast training speed, and support for multi-task adaptability.
MoELoRA (Chen al. 2024a): MoELoRA combines the Mixture-of- Ex-peetts (MoE) mechanism with Low-Rank Adaptation (LoRA) to enhance adaptability in dynamic task environments. It dy- namically allocates expert resources and adjusts modality-speciff adaptation weights, achieving improved multi-task performance. In our CIT setting, the number of experts is set to 2 per MoE layer. It is static-strrccure--aseed
Learning Without Forgetting (LwF) (Li Hoiem 2017): LwF addresses catastrophic forgetting through knowledge distillation. A pretrained model guides the current model retain historical task knowledge without requiring access to old datasets. It enforces alignment between the output distribution of the current model and the teacher model by combining next-token prediction loss and distillation loss with a weighted sum of two losses of two losses. It is eegularization-basee
Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017): EWC introduces a regularization term based on the Fisher Information Matrix (FIM) to protect critical parameters from previous tasks. By computing parameter importance (diagonal elements of FIM) and imposing constraints during new task training, EWC balances old and new task performance. It is regularization-based.
Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato 2017): GEM prevents forgetting by maintaining episodic memory of historical task samples and enforcing gradient constraints. projects the current task gradient into a subspace compatible with all historical gradients via
"http::/huggginffcccoo/// http:/ggg...ggacccooooo/////////
30

---

Published as a conference paper at ICLR 2026
quadratic programming, ensuring task updates do not degrade old task performance. It is regularization- begularization-baseddea and replay-based. We set 100 samples per task for replay.
Experience Replay (Rolnick et al. 2019); Experience Replay mitigates catastrophic forgetting by replaying subsets of historical data during new task training. A replay buffer stores representative samples, and the model alternates between learning new data and reinforcing old knowledge through mixed training. It is replay-based We set 100 samples per task for replay.
Prompt Gradient Projection (PGP) (Qiao et al.. 2025b): PGP proposes gradient projection
approach that enforces model parameter updates to be orthogonal to the previous feature subspace, thereby preserving historical knowledge and enabling adaptation based.
tasks. It regularization-
Model Tailor (MT) (Zuu et 2024): MT restricts training the critical parameters while com- pensating for variations in the trainable parameters. It is regularization-basee
Eproj (He al. 2023): Eproj is an advanced dynamic model adaptation method. It groups high- conflict tasks and handles low-conflict tasks via regularization. It is extension-
Dynamic EMA (CIA*) (Qia al. 2025a): CIA* derives optimal balance weights from the trade- off premise and EMA update, satisfying plasticity-stability conditions. The weights are adaptively determined by gradients and learned parameters. It is regularization-bant
Dynamic EMA + Instruction Grouping (CIA) (Qiao et al. 2025a): : CIA extends CIA by intro- ducing instruction grouping to avoid high-conficct tasks. Based on instruction semantic similarity, it determines whether to retrain or extend parameters and allocates the most suitable parameters for testing instances. extension-based.
PCLR (Ours): PCLR introduces two key innovations to address the challenges of continual instruc- tion tuning (CIT) in Large Multimodal Models (LMMs). First, we decompose LoRA weights into LoRA Rank Pool (LRP), enabling fine-grained control over expert ranks and achieving flexible pa- rameter allocation. This design supports arbitrary compression ratios while preserving task-specific adaptability. Then, propose the -nnressioon-nteerationn Learning (CIL) pipeline, which bal- ances plasticity and stability through three stages:
(1) Compression: It prunes rank experts to reserve space for new task learning.
Integration: It fuses knowledge from similar experts via distillation enhance synergy.
Learning: It trains new experts in the released space without memory explosion.
PCLR-LwF (A Simplified Variant of PCLR) To further enhance the performance of PCLR on long-term CIT while minimizing unnecessary performance degradation and training overhead, propose PCLR-LwF, a simplified variant that combines PCLR with LwF (Li & Hoiem 2017). We group consecutive tasks into clusters, apply LwF within each cluster for continual instruction tuning. This approach preserves the core advantages of PCL. and significantly reducing the cost of integration.
T DATASETS
ScienceQA (Lu al. 2022 ScienceQA is a multimodal science question-nnswering dataset de- signed to evaluate the ability of models to perform reasoning by integrating visual and textual in- formation. The training dataset comprises 12,726 samples, with 6,218 18 instances the image-text modality and 6,508 in the text-only modality. The testing dataset contains 4,241 samples, distributed as 2,017 image-text instances and 2,224 text-only instances.
TextVQA (Singh et al. 2019): TextVQA targets text recognition in visual question-answeriigg The dataset includes real-world images with diverse text formats (handwritten, printed). The training dataset comprises 34,602 samples, all of which belong the image-text modality. The testing dataset contains 5,000 samples, all of which are in the image-text modality.
ImageNet (Russakovsky et al., 2015): ImageNet is a large-scale image classification dataset. The training dataset comprises 129,833 samples, all of which belong to the image-text modality. The testing dataset contains 5,050 samples, all of which are in the image-text modality.
31

---

Published as a conference paper at ICLR 2026
GQA (Hudson & Manning 2019)
GQA emphasizes real-world visual reasoning. It evaluates un- derstanding of object relationships and perform multi-step inference. The dataset includes both synthetic and real-world images with scene graphs for structured reasoning. The training dataset comprises 72, 140 samples, all of which belong to the image-text modality. The testing dataset con- tains 12,578 samples, all of which are in the image-text modality.
VizWiz Gurari et al., 2018): VizWiz is a visual question-answerte -nswerinn dataset for visually impaired users. The dataset focuses on practical, everyday visual queries. The training dataset comprises 20,523 samples, all of which belong to the image-text modality. The testing dataset contains 4,319 samples, all of which are in the image-text modality.
Grounding (Mao et al., 2016): The Grounding tests the ability of models to align natural language instructions with objects in images. It includes image-text pairs where the text describes object locations or attributes, and the task is to predict the corresponding bounding boxes. The training dataset comprises 55,885 samples, all of which belong to the image-text modality. The testing dataset contains 30,969 samples, all of which are in the image-text modality.
VQAv2 (Goyal et al. 2017): VQAv2 is a foundational visual guestion-answering dataset. em- phasizes balanced answer distributions and diverse topics. The training dataset comprises 82,783 samples, all of which belong to the image-text modality. The testing dataset contains 214,354 sam- ples, all of which are in the image-text modality.
OCRVQA (Mishra et al. 2019): OCRVQA combines optical character recognition with visual crestion-answering It evaluates models' ability to parse text from images and generate answers based on the extracted content. The training dataset comprises 165,348 samples, all of which belong to the image-text modality. The testing dataset contains 99,926 samples, all of which are in the image-text modality.
ArXivQA (Li et al. 2024b): ArXivQA is multi-modal dataset for scientific paper analysis. It in- cludes images, formulas, and text from arXiv papers, with questions requiring cross-modal reason- ing for tasks such figure interpretation or explaining derivations. The training dataset comprises 90,000 samples, all of which belong to the image-text modality. The testing dataset contains 10,000 samples, all of which are the image-text modality.
GeoChat (Kuckreja et al. 2024): GeoChat focuses on geospatial reasoning using maps. It includes map images and natural language questions about locations, terrain features, or symbols. The train- ing dataset comprises 25,362 samples, all of which belong to the image-text modality. The testing dataset contains 3,000 samples, all of which are in the image-text modality.
IconQA (Lu et al. 202T): IconQA targets abstract icon understanding. It contains icon-text pairs where models must match icons to their descriptions, testing semantic parsing of symbolic visual elements. The training dataset comprises 29,859 samples, all of which belong to the image-text modality. The testing dataset contains 3,000 samples, all of which in the image-text modality.
ClevrMath (Lindström & Abraham 2022): ClevrMath includes synthetic images and math prob- lems (geometry, arithmetic) and requires models to perform compositional logic and computation. The training dataset comprises 40,000 samples, all of which belong to the image-text modality. The testing dataset contains 3,000 samples, all of which are in the image-text modality.
CodeQA (Liu & Wan 2021): CodeQA is programming question- answering dataset. It pairs code snippets with questions, testing the ability of models to understand and explain code logic across multiple programming languages. The training dataset comprises 150,896 samples, all of which belong to the text-only modality. The testing dataset contains 18,997 samples, all of which in the text-only modality.
Flickr30k (Plummer et al.) 2015): Flickr30k is an image captioning dataset. It supports fine- grained visual-linguistir alignment tasks. The training dataset comprises 30,000 samples, all of which be- long to the image-text modality. The testing dataset contains 1,783 samples, all of which are in the image-text modality.
DQA Mathew et al. 2021): Doc focuses on document understanding. It includes scanned documents with questions that require text localization and semantic interpretation in complex lay- outs. The training dataset comprises 39,463 samples, all of which belong to the image-text modality. The testing dataset contains 5,349 samples, all of which are in the image-text modality.
32

---

Published as a conference paper at ICLR 2026
MathQA (Amini et al. 2019): MathQA is mathematical problem solving dataset with text-based questions. It emphasizes interpretability and logical reasoning for algebraic and geometric problems. The training dataset comprises 29,837 samples, all of which belong the text-only modality. The testing dataset contains 2,985 samples, all which are in the text-only modality.
ChartQA (Masry al. 2022 ChartQA evaluates quantitative reasoning charts. Questions
involve trend analysis and numerical comparisons. The training dataset comprises 28,299 samples, all of which belong the image-text modality. The testing dataset contains 2,500 samples, all of which are in the image-text modality.
Path VQA (He et al. 2020): PathVQA is medical pathology dataset. It focuses disease identifi- cation and cellular structure analysis for clinical applications. The training dataset comprises 19, 6,65 samples, all of which belong to the image-text modality. The testing dataset contains 6,719 samples, all of which are in the image-text modality.
question-answeriig dataset with question- answer pairs from Wikipedia. The training dataset comprises 20,360 samples, all of which belong to the text-only modality. The testing dataset contains 633 samples, of which are in the text-only modality.
WikiQA Yang al. 2013): WikiQA is an
33

---

Published as a conference paper at ICLR 2026
Table 20: The list of instructions for each task.
Task
Original
Divens
p0
dredl,
Scleet lac aasus for ile gie clatup resenal with lae kiks of dence oplioe.
Demmmn me comoo opton troe prosated cheise and septy wotn compprarinn leter
Asore with optivn's leler Boot the ginat slateo diccilly.
Anewee with the opuint's kdta frxe Ua gance chicks sinadly
sdouth the choiss from the
balow
lettrr comect
ScheeegA
Frves thegiie clstioo cheese la ocnnt usoner soquaal will de kiia of laa cletee Seare
the unweer froe sytina cogund wrth
asscisaed
trove the given chotces otler the orrnet seever and rely wrth the later of chooes optine. ideratt commect optice mom the chrise procoted reppost the kiter comect optine
insicaing
Asaver ties spa with jus wordd
glaase
Usecne word consiig plase to rogued the gestios
Prondde using oelly one worl
deyeliivve plane
phrae.
Actover the peceaoo using siegle
Captart the eacaace of your reiponn sitgle
Use snnlt werd or sted pleas respond the ceeeios.
respeme
plane
jirmee.
Provile your answer one soed skin dos pelio planse urals
Azaner the qumito
test doverriiee pleasc
Cagauro the esseret of imsorse in siigle soed concli planse. Sumerana the atrect in the imaee in steele word bnse pteae
imnaee roing
Che ebject
single
Clvvo obrect image woed coeciit phross
Exprci the obicct in ventier dovettiiie plasse.
boseo dae etject the truae with one worr arn deet, deseppiie parast Mentty type coment mraa usity
concio plraa.
Dearth the conaent of inage using word ooad plase.
Tigiess the ryjr truge sinle werl hhon deadlpii plans Chasily the
only
pleaa.
Use wast or aobed placs ctadly creest the issaee
Royeed in las caseies ssuu wnr
parast
Ancur the spectee with sigle eved hixt plasss. Rapped
word dost strae.
GQA
unitals soid ylariio
beielly saing only wort are glosst
Respaed the quesies with just wadd bidd pleae Ascert spetion adeg degle worl condie grece yeur repures with
Ropped the Ropped in the quesict oiis uo werd concin plans. Asavic gapiis with use weed ir plane.
word brid
Reiqued in spartin binng
oved be crecisl phoses.
Reepoed spartion with stigle wond hext gaans
Ansvec the gucaticn using unigle
Proside pasiinci vogrease with anigle parmar.
Regetal with use wad comche plaso.
VieWis
Aapvees le quoiioss will joi oue word or brie plasse.
pirase
Pronid perranseriin
Reply the quoiiv mong wont omder ptreet
will single
Manif and pumntc He bounding bis condiatt Mal Jocigliin givin'as Bi SEakARe bonnc and perrade toending has
hand tn regon deacthed unimice. that protis bersding conetturee regice
Ploae proside the lourdin condinaie agiion the dourthi..
Plezee poriide da bounding coniivate of repion
Provice tie cato the boarlng loo fst cscopced lhe regirs dosilbed loe sexticao Give louralig hos coodintes
Grondlig
desonhes.
Desmmies itantyy and pernie the oventrats
heansing his comtraere bardd desorption the
barsing hox do desethed in the sestenes.
Proste the coundrais the hoandigg haus the regice decetted Maniff and give conniitaos of the houading
dowiibed by the sentenee.
Arsea the gesiice wsng o single and plasse
Use seed gestice single
placs.
perau
Ansowr the gandies wusd planac:
Anouartte spection seep plaie..
Azover the queetiee usize sot one west omeoe shraee.
your
queston usng
planse
Piesido woed
The
using only with wond best ptrme.
or heaf giaas
Ascont the cpeetic
Resgoal the queaton using ssigle word or csncis plasse
Respord to the enc with angle wort or ahet plame. Paesido
using
plese
rels
Use une eved ora bazi plazae ao snaver the quosee..
Assover the cuceiinn woid placon
Candane your saanec worder consile glane
Haply
usita une woed or concne plaaa.
Use o deglt wved or please avoser the coroiee
Piesidd arower raiigg only ward
giass
Rerpand
planse.
Asreei the survice asing single vord plaza.
34

---

Published as a conference paper at ICLR 2026
U ALGORITHM
Algorithm 1: PCLR
Input: Pretrained LMM F, number of task N, training set {( o 1 12 12. learning loss function L1, integration loss function L2, progressive compression
fi= learning rate ?h
retention ratio function a, preset total rank number T'm-
Output: LRP module M.
Initialize: LRP module M Ø, fake query pool Tk- Ø.
for f 1.....N do
.M = L(,, (1), yf I M, :,,())
L1). # Learning (L).
2. Collect all trained keys and calculate their mean Qk-
Tk TK }.
if' (Trnk((]) 2 Tm then
M = CI( F, (at, of Tk, M, a(t), 12, L2).
# Compression and Integration (CT).
end
end
Algorithm 2: Learning (L)
Input:
Pretrained LMM F, current task dataset D, LRP module Mald, learning rate 7/, learning loss function Li according to Eq.
Output: LRP module M.
Initialize: Mnew-
if Mold * Ø then
1. Sample subset from D and calculate their average query q.
2. Compute the cosine similarity scores between the keys in Motd and the query select
the top-r indices based on the scores (where represents the rank of Mnew), and Mnew set to the value of the corresponding vector set.
end
Extend rank space M = cat aat(Mold, Mnewl, freeze the old part, and activate the new part. for epoch do
for f 1,.., do
Draw mani-batch B = {at, yi 3
2. Calculate / = (F,, B).
Using / backward.
4. Update M with learning rate f)1
end end
Algorithm 3: Compression and Integration (CI)
Input: Pretrained LMM F, visible datasets D, task key pool K, original LRP M, retention ratio a, learning rate 7/2, integration loss function La according 5q.
Output: post-integration LRP M.
Initialize: Load as fake query pool, initialize probability table of fake query, freeze F and
M, using a to get post-compression M to initialize M.
for epoch do for t = 1, I,.., do
1. Draw mini-batch B = (af, y )
Sample q from K with probability table P.
3. Calculate integration Loss
C2(F, M, M, B,g).
4. Using / update P by Eq. 6
5. Using I to backward.
6. Update M with learning rate 722-
end end
35

---

Published as a conference paper at ICLR 2026
Algorithm 4: PCLR-LwF
Input: Pretrained LMM F, number of task N, training set {(at, y 1 1==1 learning rate n,
12, learning loss function Li, integration loss function L2, progressive compression retention ratio function a, preset total rank number Tms LwF weight 3, Conflict detection function E.
Output: LRP module M. Initialize: LRP module M 2, fake query pool Tk 2, 0.
for f, 1 1..... N do
if (FF, M) then
1. M L(F, ( M, M, C1).
# Learning (L).
0.
end else
M LLWF (F, (a), of M/ M, m,m,i, 2. n = n +
# Learning with LwF (LLwF).
end
Collect trained keys and calculate their mean qi-
2. Tk Tx U {qx}-
M 2 Tm then
. CI(P, (st v Tk, a(t()), 12 f2).
# Compression and Integration (CT).
end
end
Algorithm 5: Learning (LLwF)
Input: Pretrained LMM F, current task dataset D, LRP module M, learning rate 7/,, learning
loss function L1 according to (4. continual LwF number n, LwF weight S.
Output: LRP module M.
Initialize: Copy M as Motd, and freeze it, set KLD Loss LKL, 1.2 regularization function l2.
for epoch
for f = 1..., do
Draw mini-batch B {at, 9/36 yi
2. Calculate = Li (F, M, B) + BLKI M, Mard, B).
Using backward.
4. Update M with learning rate 7)1-
end end
1. Take out all the keys Kold of Mold and all the keys K of M. For all (kotd,,, k;) E (Kala, K), k 12(n kotd, + k:).
3. Assign all k; E K back.
36


Reviews:
[{"review_id": "Pn5ERUDpAN", "reviewer": "Reviewer_v6DS", "summary": "This paper proposes a method named PCLR for CIT of LMMs, introducing the LRP and a CIL pipeline, and demonstrates superior performance over existing approaches on the CoIN and Continual-NExT benchmarks. Despite the appealing experimental results, the work suffers from significant weaknesses in theoretical grounding and systematic efficiency analysis, which undermine its scientific rigor and practical deployment value.", "questions": "1. Could you provide the theoretical basis for decomposing and obtaining the LoRA rank pool? The current method lacks relevant theoretical support. Although LoRA contains \"linearly dependent redundant rank vectors,\" linear dependence does not equate to redundant knowledge: even if two rank vectors are mathematically linearly dependent, they may carry different information at the semantic or task representation level.\n2. Why is the rank level the optimal granularity? Why not coarser or finer? It is recommended to include related ablation experiments.\n3. Since this paper's method is based on AdaLoRA, should AdaLoRA be included as a baseline in the experiments?\n4. In Table 2, the parameter count for LLaVA-1.5-hf is not reported—does this refer to LLaVA-1.5-7b-hf?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is fluent, the figures are clear, and the content is easy to understand.\n2. PCLR introduces a progressive strategy that dynamically adjusts compression and learning strategies according to the learning stage, thereby optimizing long-term learning performance.", "weaknesses": "1. The theoretical foundation heavily relies on intuitive analogies and lacks formal justification. The paper extensively borrows the neuroscience concept of “memory consolidation during human sleep” and maps it onto the three-stage CIL pipeline. However, this analogy remains purely heuristic, with no accompanying verifiable computational model or theoretical guarantees.\n2. Moreover, the paper lacks critical comparisons of efficiency and parameter counts, making it impossible to assess the true validity of its claimed memory advantages. Particularly given its emphasis on “solving memory explosion” the absence of parameter statistics and GPU memory usage curves constitutes a fatal flaw.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3xwN6lR6me", "reviewer": "Reviewer_U4Ez", "summary": "This paper proposes a Compression–Integration–Learning (CIL) pipeline and a LoRA Rank Pool (LRP) architecture for multimodal continual instruction tuning. The method releases model capacity by compressing redundant rank vectors, integrates old knowledge through knowledge distillation, and incrementally learns new tasks, thereby mitigating catastrophic forgetting in multi-task continual learning. Experiments demonstrate that PCLR significantly reduces forgetting on LLaVA and Qwen-VL models, outperforming existing methods on two benchmarks.", "questions": "- What is the retention rate used during the Compression stage? What is the rationale behind this choice?\n- For the first task, are all experts trained, and is compression applied only after the second task begins?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The performance is impressive, achieving state-of-the-art results on both datasets.\n- The CIL process exhibits a high degree of innovation.", "weaknesses": "- The capacity limit of the expert pool may constrain the method’s effectiveness in ultra-long continual learning sequences. When experts from earlier tasks are excessively compressed, complete prevention of forgetting becomes difficult. Therefore, the method may not be well-suited for extremely long task sequences.\n- The Integration stage requires additional training, which could lead to longer overall training time compared to existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ucoxLVtDA", "reviewer": "Reviewer_pBRi", "summary": "This paper addresses catastrophic forgetting and unbounded memory growth in Multimodal Continual Instruction Tuning (CIT) for Large Multimodal Models (LMMs) by proposing Progressively Compressed LoRA (PCLR). Through integrating the fine-grained LoRA Rank Pool (LRP) architecture and Compression–Integration–Learning (CIL) pipeline (inspired by human memory consolidation), the authors demonstrate that PCLR balances stability, plasticity, and memory efficiency: LRP decomposes LoRA into rank vectors with learnable keys for flexible knowledge reuse, while CIL prunes redundant experts, fuses knowledge via distillation, and trains new experts in released memory. Experiments demonstrate that PCLR outperforms state-of-the-art baseline methods in terms of average accuracy and forgetting rate, and its variant PCLR-LwF further enhances the performance of long-term CIT.", "questions": "1. In Section 4.4, Progressive Learning mentions that \"as the number of tasks increases, fewer new ranks are allocated and a lighter, high-retention compression scheme is adopted\", but the quantitative criteria for adjustment are not clearly defined. Is \"a certain degree of task increase\" based on a task quantity threshold or a model knowledge density indicator? What is the basis for reducing the rank from 64 to 32 in the Experimental Setup Section?\n2. When comparing with the LoRA baseline—where LoRA uses a rank of 128, while PCLR adopts a \"total rank of 256 and activated rank of 64\"—the authors claim to \"balance parameter efficiency,\" yet fail to align the \"effective trainable parameter scale\" between the two methods. Since PCLR involves fewer parameters actually participating in computation, a critical question arises: Does the \"shared expert reusing historical knowledge\" implicitly increase the effective parameter count? It is necessary to supplement direct comparisons of parameter scale and computational cost between the two methods.\n3. In the experiments, all tasks were trained with 1 epoch, and the impact of the number of epochs on PCLR was not verified. If a new task is highly complex and requires multi-epoch training, should the compression-integration frequency of CIL be adjusted? Will multi-epoch training exacerbate the forgetting of old tasks, and will PCLR's anti-forgetting ability still be superior to that of baseline methods?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It innovatively proposes the LRP architecture, decomposing LoRA into fine-grained rank vector experts with shared components and dynamic gating, which enables flexible knowledge reuse and reduces redundancy, addressing the memory explosion issue of traditional extension methods.\n2. The CIL pipeline, inspired by human memory consolidation, balances catastrophic forgetting and memory efficiency through compression-pruning redundant experts, integration-compensating performance loss via distillation, and learning-allocating capacity for new tasks, achieving a stable-plastic-memory balance.", "weaknesses": "1. The paper lacks sufficient analysis on why the progressive learning process adjusts the number of new ranks and compression retention rate in specific ways (e.g., 75% to 87.5%), with no clear theoretical or experimental justification for these hyperparameter choices.\n2. It fails to compare PCLR with more multimodal continual instruction tuning methods published in 2024 and 2025, such as SEFE[1], HiDe-LLaVA[2], ProgLoRA[3].\n\n[1] SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning, ICML 2025\n\n[2] HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model, ACL 2025\n\n[3] Progressive LoRA for Multimodal Continual Instruction Tuning, ACL 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)