Efficient Few-Shot Continual Learning in Vision-Language Models
Authors: Aristeidis Panos
Rahaf Aljundi
Daniel Olmeda Reino
Richard E. Turner
Venue: Accepted by TMLR ()

Abstract: Vision-language models (VLMs) excel at tasks like visual question answering and image captioning, but their reliance on frozen, pretrained image encoders like CLIP often leads to persistent vision errors that degrade downstream performance. Moreover, real-world deployment demands that VLMs continually adapt to new, scarce data in a few-shot setting without forgetting prior knowledge. To meet these challenges, we introduce LoRSU (Low-Rank Adaptation with Structured Updates), a lightweight and robust technique for few-shot continual learning of VLMs’ image encoders. Our approach leverages theoretical insights to identify and update only the most critical parameters, achieving significant resource efficiency. Specifically, we demonstrate that LoRSU reduces computational overhead by over 25x compared to full VLM updates, without sacrificing performance. In experiments on VQA benchmarks under a few-shot continual learning protocol, LoRSU demonstrates superior scalability, efficiency, and accuracy, offering a practical solution for dynamic, resource-constrained vision-language applications.

Published in Transactions on Machine Learning Research (12/2025)
Efficient Few-Shot Continual Learning in Vision-Language Models
Aristeidis Panos
ap2313@cam.ac..
University of Combridge
Rahaf Aljundi Toyota Motor Europe
Daniel Olmeda Reino Toyota Motor Europe
daniel. www.:eddw.eeetttttooooeeeeur.e.o
Richard E. Turner University of Cambridge
ret26@com.ac.uk ac.uk
Reviewed on OpenReview: https: // openreview. net/ forum? id- sQIu92WWOV
Abstract
Vision-language models (VLMs) excel at tasks like visual question answering and image caplioning, but their reliance on frozen, pretrained image encoders like CLIP often leads to persistent vision errors that degrade downstream performance. Moreover, real-world deployment demands that VLMs continually adapt to new, scarce data in few-shot sctting without forgetting prior knowledge. To meet these challenges, introduce LoRSU (Low- Rank Adaptation with Structured Updates), a lightweight and robust techmique for few-shot continual learning of VLMs" image encoders. Our approach leverages theoretical insights to identify and update only the most critical parameters, achieving significant resource elficiency. Specifically, we demonstrate that LoRSU reduces computational overhead by over 25x compared full VLM updates, without sacrificing performance. In experiments VQA benchmarks under few-shot continual learning protocol, LoRSU demonstrates superior scalability, efficiency, and accuracy, offering practical solution for dynamic, resource- constrained vision-language applications.
1 Introduction
Large Language Models (LLMs) have revolutionized natural language understanding and generation, enabling significant advancements across diverse applications. As intelligent agents are increasingly expected to operate in real-world multimodal environments, integrating visual understanding becomes essential. Vision-Language Models (VLMs) extend LLMs by incorporating visual information, either through pretrained vision encoders or end-to-end multimodal training. These models have demonstrated state-of-the-art performance vision language tasks such as visual question answering (VQA) and image captioning. highlighting their potential for general-purpose multimodal reasoning (CCen al. 2024 Wang et al. 2024a).
Approaches relying pre-trained image encoders typically variants of the CLIP model (Radford et al. 2021), which is kept frozen in the vision-language binding process (Liu et al. 2024). CLIP is widely deployed vision transformer with strong zero-shot capabilities across various tasks and domains. However, several
existing works have highlighted various weaknesses CLIP on out-of-domain data (Liu et al.) 2024 Zhu
ct al. 2023 Chen et al. 2023 Li et al. 2023 Tong al. 2024). When deploying VLMs as visual assistants new domains, it is expected that VLMs can be updated using a few images gathered from the target environment whenever deficiencies noted.

---

Published in Transactions on Machine Learning Research (12/2025)
Q: What this person doing? GT: Cutting vegetables.
Q: What this person doing?
GT: Cooking a stove
Maintaining VLM's Knowledge
Few-shot Update
Efficient Updates
Significant Performance Gains
"The person standing kitchan sink, washing dishes."
LLaVA:
"The standing kitchen, preparing food. They are using pot to something, possibly stirring the pot."
LLaVA-LoRSU:
No Replay
Figure 1: (Ist column) Incorrect response of the pretrained LLaVA to a sample from the TSI dataset. (2nd column) A sample of few-shot data used for fine-tuning LLaVA. (3rd column) Correct response LLaVA to a test TSI image after fine-tuning with LoRSU. (4th column) set of desiderata for few-shot continual learning with VLMs that our method satisfies.
Continual learning allows model to be continuously updated as data from new tasks domains are encountered (De Lange et al.) 2021). Recent literature on continual learning (CL) of vision-language models focuses on updating either the LLM (Srivastava et al. 2024) and/or language projection layers (Das et al. 2024), while maintaining a frozen image encoder, as noted in a recent review on continual learning of VLMs (Huang et al. 2025).
In vision-language models, the LLM component provides reasoning and factual knowledge, while the image encoder's role is to extract robust and accurate visual features. In this work, argue that adapting VLMs to new visual domains or tasks is more effective and efficient when the image encoder is updated rather than the LLM. Limiting updates to the vision encoder can lead to unprecedented stability in the performed updates.
To enable reliable few-shot continual learning (FSCL) under large vision-language models, we posit a set of desiderata capturing four main aspects: 1) Maintaining the VLM's knowledge, since models capture vast body of generic knowledge, updates on new domains/tann ttakks should not drastically deteriorate this knowledge (as also noted in He et al. (2023a); Zhang et al. (024))) ); 2) Significant performance gains, it has been noted that continual learning solutions, when coupled with pro-trained models, can lead to worse performance than zero-shot or very marginal improvement (Janson et al. 2022); 3) Efficient updates, given the large model sizes, few-shot updates that require large compute budget become impractical; and 4) No replay (Rolnick ct al.) 2019), as replaying a set of examples that cover the model's previons knowledge can significantly increase the update memory and compute footprint.
Under this set of desiderata, we introduce novel parameter-effiiies fine-tuning (PEFT) method called LoRSU (Low-Rank Adaptation with Structured Updates). We show, for the first time, that it is possible to perform continual few-shot updates with zero replay of previous data or storage of previous models, while achieving strong performance gains of up to (~ 20%) with negligible deterioration of the model's previous knowledge (~~ 1%).
Our method localizes the updates to specific layers in the vision encoder, namely the Attention and MLP layers, and selects a small set of parameters with the highest sensitivity to the received few-shot data. This approach limits the updates
small number of relevant parameters, striking balance between adaptability to the new task and preservation of information from previous tasks.
The third column of Figure demonstrates the correct response of LLLVA after fine-tuning the image encoder separately using our method on a few-shot sample from the TSI dataset (Das et al. 2019 (shown in the second column). This is contrasted with the incorrect response of the pre-trained VA, depicted in the first column.
Through extensive experiments, we demonstrate that updating the image encoder is essential for improving the performance of the VLM that relies it. More importantly, this approach is computationally efficient,
2

---

Published in Transactions on Machine Learning Research (12/2025)
the image encoder has significantly fewer parameters compared to the language model, especially when updated separately. Additionally, the method is less prone to forgetting, particularly the LLM knowledge.
We evaluated our approach on various VQA tasks, comparing it to state-of-the-art CL methods and the PEFT baselines in various few-shot CL settings. We show significant improvements in the performance the full VLM model across all settings, with very low rates of forgetting, without using any replay buffer of data from previous tasks. By selectively updating the image encoder, our method provides a robust and efficient solution for handling visual shifts. This targeted adaptation strategy avoids the need to modify the entire model, preserving existing knowledge while ensuring strong performance in new domains.
The contributions of this paper are as follows:
We propose LoRSU, a novel replay-free PEFT method tailored for FSCL.
We introduce two new VQA datasets, TSI and DALLE, created to expose the limitations of pre-trained image encoders in VLMs.
We conduct the first large-scale study of FSCL in VLMs (spanning more than 700 experiments) evaluating LoRSU across ten diverse VQA datasets and benchmarking against state-of-the-art PEFT and CL methods. LoRSU consistently outperforms all baselines.
To the best of our knowledge, this work is the first to investigate FSCL in generative VLMs, showing for the first time strong performance gains and negligible previous knowledge deterioration under the strict CL. setting with no use of previous models or examples.
2 Related Work
Continual Learning. Our work falls within the CL literature, where a model needs to be updated incrementally as new data arrive, accumulating knowledge over tasks and reducing forgetting of previously acquired information De Lange et al. (2021).
Continual Learning for Multimodal Language Models. Recent work surveys efficiency and forget- ting in continual LLM updates Wu al. (2024) and explores VQA-based adaptation with frozen vision encoders Srivastava et al. (2024) instruction tuning via expanding projection heads He et al. (2023b). Das ct al. 2024)) introduced a pseudo-rchearsal strategy for vision-language models, updating only the language projection layer. Our method adapts only the vision encoder, preserving language capabilities.
Continual Learning with Few-Shot Updates. Verwimp et al. (2023) posits that an ideal continual learning solution would enable continual correction of model's mistakes at a lower computational cost than retraining from scratch. However, most continual few-shot learning from pre-trained models focuses on classification tasks and introduces solutions that cannot scale large multimodal models. [anos al. (2023) update the vision encoder the first task only, later adapting a covariance matrix for incoming tasks. Goswami et al. (2024) calibrate the covariance matrix for new classes based semantic similarity. Zhao et al. (2024) introduce few and slow updates, proposing a transfer loss function and a cross-classification loss to mitigate catastrophic forgetting. Few-shot updates can also be viewed through the lens of model editing Sinitsin et al. (2020). MEND Mitchell et al. (2022) scales model editing to large language models by transforming the gradient obtained from fine-tuning, through a low-rank decomposition fed to auxiliary networks designed make fast, local edits to a pro-trained model, requiring a set of unrelated examples to prevent forgetting, ROME [een et al. (2022) applies causal tracing to identify layers where incorrect factual knowledge is stored, applying low-rank update. However, ROME does not scale to continual updates or non-association types of updates. [Cheng et al. (2023) studied multi-modal editing, showing negligible deterioration in multi-modal task performance when updating language models but severe forgetting when updating vision encoders. To the contrary, our method focuses on adapting the vision encoder rather than updating the factual knowledge in the LLM, yet achieving strong performance gains and negligible forgetting.
Continual Learning of Pre- Trained Image Encoders. SPT He et al. (2023)) estimates a mask of updates based on parameter sensitivity, performing low-rank or sparse updates. SPU Zhang et al. (0024)
3

---

Published in Transactions on Machine Learning Research (12/2025)
localizes updates to the first feed-forward layer of each transformer block, inspired by knowledge neuron theory Dai et al. (2021). Our approach generalizes updates to all layers, selecting relevant parameters and maintaining gradient norms, combined with LoRA on selected attention heads for adaptívity and stability, achieving SOTA performance on continual fewshot multimodal tasks.
Our work builds upon two key lines of research: structured sparse updates (SPU) (Zhang et al., 2024 and low-rank adaptation (LoRA) (Hu et al. 2021). Whilst SPU demonstrates the value of gradient traseeen-based parameter selection in the MLP layers, and LoRA shows the efficiency of low-rank updates, neither addresses the unique challenges of VLM continual learning where both attention mechanisms and feed- forward layers must be adapted jointly. LoRSU's contribution lies in the insight that these techniques should be applied differentially: structured sparsity for MLP layers and selective LoRA for attention heads, with head selection based on cross- parametrr gradient importance.
3 Low-Rank Adaptation with Structured Updates
Few-shot continual learning is highly practical and challenging scenario, where models must incrementally adapt to new tasks with limited supervision while retaining previously acquired knowledge. This setting closely mirrors real-world applications, such as interactive AT assistants and autonomous systemss, where models receive a continuous stream of novel data but only sparse supervision per update.
To address the challenge of efficiently fine-tuning largo-scale visual encoders and transformer- based models in a few-shot continuous learning setting, without causing catastrophic forgetting (i.e., degradation in performance previously learned tasks), we propose novel parameter-efficiee fine-tuning method called Low- Rank Adaplation with Structured Updales (LoRSU) illustrated in Fig. 2
LoRSU updates specific parameters within each transformer block in a resource-efficient manner, mitigating the risk of generic knowledge loss when fine-tuning for new tasks. Specifically, we selectively update a subset of parameters from the first linear layer in the MLP block of each transformer layer, as proposed in Zhang et al. (2024). Although this approach reduces the fine-tuning burden, it may limit the model flexibility as the remaining parameters in the transformer block remain fixed. To enhance flexibility, we further update the most informative attention heads based on the gradient of task-specific loss. More specifically, let a dataset D,- = (xn,Yn) / for the current. task t where x, is an image with text description Yn. We define D.) := (0) as the loss used for training the model and O E Rd is the full set of model's parameters. The standard Multi-head Self-Attention Mechanism (MSA) Vaswani et al. (2017). comprised of И dimdinnnsional heads, is defined as the concatenation of multiple self-attention (SA) blocks where q w ZT. ((0) w(()) 2T, v(t) w!() =T E RR are the query, key and value matrices, which are used to compute the self-attention outputs as follows
A ( = softmax(q ( 1t' E NNN, SA,(Z) = (T) E NRDD 1 H.
(1)
Z E NxD is the input matrix of N tokens of dimension D and ((( w and W are the query, key, and value matrices of learnable parameters for head i, respectively. The final MSA function is defined as MSA(Z)= = Concat [SA, SA, [Z)] W, E NND, W. E HDD,. Since we care to update the parameters of the heads that cause the largest changes in (O),, we compute the loss gradient with respect to the parameters of each head, and then update only those heads with the largest cumulative contribution to the loss change. Since the matrices w w(()) w are all the parameters of head i, we can define an importance score for each head by adding the squared values of their corresponding gradients Ge(²) V W Cr,
G(6 V wgo Le, and c(0)) V WWfh Li, as follows
8; = Σ (IG() 1)) + (G( 1)3 + (G (m, mm..))).
(2)
m.d
We provide a theoretical justification of (2) in the next section. We update only the top-k heads, based their importance scores {s1,...,,8t 811), I C (1,..., H3, to be updated the current task. Nevertheless,
4

---

Published in Transactions on Machine Learning Research (12/2025)
Transformer Encoder
p% Trainable Parameters
Lx
Win
Fc2-MLP *
Frozen g
Gradient on Target data
Fc1-MLP
Finetuned
(
Norm#
LoRA Adapter
Selected
hn
RS
Heads
Multi-Head Attention
Prenre Weight
WAmn
I
Norm
Figure 2: LoRSU mechanism: After computing the gradient VeC,(O) over the target dataset at time t, LoRSU picks small number of attention heads and small number of parameters from the first linear layer of the MLP module in the transformer block based on the magnitude of the gradients of WWa.n Ce and Vw.. Lt, respectively. Computational efficioncy is ensured by introducing LoRA adapters to the attention weight matrices.
the number of parameters remains high due to the large weight matrices. Therefore, we parametrize the original weights using LoRA Hu et al. (2021) to further reduce the computational burden. The matrices w w() W!" E I are now defined
w
w + 1 B(0 a (g,
(3)
Finally, to ensure that we only update ( ( ! ViE we use binary mask on the gradient vector with respect to all parameters of all attention heads. We keep the projection matrix W, frozen. We note that most modern implementations of transformer blocks concatenate the three attention weight matrices W. WR, Wo into one and thus we only need to apply LoRA once this concatenated matrix.
Regarding the first linear layer in the MLP module, Wia RRdDD, mask the gradients of Wia only the most important parameters for the current task to be updated, i.e. use the following biased gradient update.
Wwn Ce = Mia @ V Cr.
(4)
where Mea E (0, Dx zero-one mask that built by choosing a proportion of the largest squared values of Vwa.fa La in similar manner as in Zhang al. (2024) and O the Hadamard product.
Theoretical justification. The importance scores in can be derived from the following constrained (binary) optimization problen!
Ipo Vw C(ooll
G (1, d), 40L, = Vi
p' = arg max (0.))))
s.t. [w
(5)
E=1
G
and C= C= cts s Hel Ae, ||lll SC,
where 0g is the vector of the pretrained parameters before using D, for fine-tuning the model. The groups of parameters 4 correspond to the parameters of a specific module (e.g. Self- Delf-Attention or MLP projector) we aim learn, hence the constraint of mutually exclusivencss, I I 0, between different pairs of parameter
For notational simplicity, assume single transformer block for this case.

---

Published in Transactions on Machine Learning Research (12/2025)
groups. Also note that we allowed one to choose a subset C₁ of the parameters of a specific group It which is the underpinning mechanism of LoRSU choosing attention heads and parameters of fe1. The mask p" is chosen so that the gradient norm of the masked gradients is as large as possible under the sparsity constraints. We prove in Appendix A that the indices of the nonzero values of p can be found using the importance scores
(2) and the magnitudes the gradients with respect to the fel parameters.
Connection to Fisher Information Matrix. The gradient magnitude criterion in (2) is fundamentally related to the empirical Fisher Information Matrix. The squared gradient magnitudes use correspond to the diagonal entries of the empirical Fisher, which measures parameter sensitivity to the current task. However, our approach represents a conceptually distinct and inverse paradigm compared to traditional Fisher-based regularization methods like EWC Kirkpatrar et al. (017)). Specifically, it estimates the diagonal of Fisher Information matrix for previous tasks and uses it to regularize important parameters from changing, attempting to preserve past knowlodge. On the other hand, LoRSU estimates Fisher Information (via squared gradients) for the current task and uses to select which parameters to update, allowing task-relevant parameters to adapl while freezing others. This inverse analogy makes sense precisely because we start. from strong foundation model: rather than preventing important-to-past parameters from moving, we allocate capacity for the new task by identifying and updating only paramcters with high scnsitivity to current data. As we show in our experimental evaluation, this type of approach significantly outperforms traditional Fisher-based regularization methods.
4 Experiments
We conduct a series of experiments under three different few-shot continual learning (FSCL) scttings (CL-5, CL-20, and CL-50 shots) to thoroughly investigate the performance of LoRSU based on ten VQA datasets. By adopting this paradigm, we aim to assess the adaptability and efficiency of LoRSU under constrained learning conditions, ensuring that remains both computationally feasible and effective in improving downstream performance.
4.1 Datasets
Regular VQA datasets. To capture a broad spectrum of visual shift and reasoning challenges from classfficatonn-ttlle tasks to open-ended question answering- we select four representative VQA datasets from the large pool of available benchmarks. These cover spatial reasoning, robust domain shifts, multimodal pattern recognition, and fine-grained perception in scientific imagery: VSR Liu et al. (2023), for spatial reasoning: HM Kiela et al. (2020), classification-style VQA for detecting hateful memes under strong domain shift; MMVP Tong et al. 2024). a challenging datasct assessing multimodal visual patterns with substantial distributional shifts; VisOnly Kamoi et al. (2024), for fine-grained visual perception in scientific figures.
Classffication-tt-VQQAAA datasets, We convert four classification datasets to multiple-choice VQA tasks with five answer choices: GTS (Sallkamp et al. (0012), German traffic signs; CAn Wang et al. (2024b), for testing robustness to spurious features in animal images; AIR Maji al. (2013), fine-grained aircralt dataset; ESAT Helber al. 22019) for land cover classification in satellite images.
TSI & DALLE. We introduce two novel datasets to explore domain shift deterioration indpendently from model's knowledge of present concepts: TSI Das et al. (2019), classification dataset of 10K training and 5K test images of 27 activity classes; DALLE, generated by querying DALL. F 2, with 660 images from 22 activity classes in TSI.
For FSCL, we split each dataset into 5 sets of disjoint classes /categorras and use 5/20/50 shot settings for model fine-tuning. Dataset splits detailed in Appendix Cl
4.2 Experimental Setting
Metrics. Our proposod metrics aim to reflect the desiderata outlined in the introduction: 1) maintaining the VLM's generic knowledge, 2) achieving significant performance improvements beyond zero-shot capabilities, 3) ensuring computational efficiency, and 4) avoiding replay-based methods. Standard continual learning
6

---

Published in Transactions on Machine Learning Research (12/2025)
(CL) metrics typically measure accuracy and forgetting only within the set of adapted tasks/classes, without considering model's pre-existing knowledge or broader capabilities. However, VL.Ms encapsulate extensive generic knowledge across diverse domains, making it essential to evaluate how continual adaptation affects their overall knowledge and performance.
To assess our first two desiderata, we propose two complementary metrics. First, we introduce the Target Improvement (TI) accuracy, which quantifies the knowledge accumulation capability by measuring the change in accuracy relative to the zero-shot performance the test split of each target dataset after continual fine-tuning, Positive TI values indicate improvements over the pre-trained model's generic knowledge. Second, to evaluate the retention of the VLM's broader knowledge base and potential positive backward transfer, we define the Control Change (CC) accuracy. CC computes the average change in accuracy across a set of control datasets datasets distinct from the current target task to gauge whether fine-tuning leads to forgetting or, conversely, positive transfer to unrelated tasks. Both TT and CC metrics are computed after the final contimual learning session. Finally, address how accuracy and forgetting evolve throughout continual adaptation explicitly, we include standard continual learning metrics such Average Accuracy (ACC) and Backward Transfer (B (T) [Lopez-Pazz & Ranzato 2017). Unlike Tl and CC, these traditional metrics focus exclusively on adapted tasks without considering the broader generic performance of the model on other datasets.
Implementation details. Please see Appendix B
Models. For most of our experiments, we consider the popular Vision Language Model LLaVA-v1.5 (Liul et al. 2024) that leverages frozen CLIP image encoder. Specifically, LLaVA utilises frozen Ap
L-14 Radford et al. (2021) with LLM (Vicuna-7b (Chiang et al. 2023 The two modules are connected through a two-layer MLP projector that aligns image and text features. The LLM and the MLP projector are optimized during the visual instruction tuning while CLIP remains frozen. LLa VA concatenates adjacent tokens from CLIP-IL-14 and processes them with an MLP projector as input to LLama-2 (7B-chat) (Touvron let al. 2023); the MLP projector and the language model are optimized while the image encoder remains Trozen. Finally, we also consider MiniGPTv2 (Chen et al. 2023) that uses the same L.M as LLaVA but a frozen EVA-CLLP----1 14 image encoder. We chose LLaVA for its representative architecture, as is widely adopted and uses CLIP-L-14, one of the most common vision encoders in VLMs, making our findings broadly relevani.
Baselines. We compare LoRSU to the following methods that also use the CLIP loss to fine-tune the image encoder: LN (Perez et al.] 2018 Panos et al.] 2023) is used for both few-shot and CL. Only the image encoder LayerNorm modules' parameters are optimized. F-FT is the standard fine-tuning technique where all image encoder parameters undergo gradient updates. F-EWC fine-tunes all the image encoder parameters with EWC regularization (Kirkpatrick al. 2017). LoRA (Hu et al., 2021] a popular PEFT method which parameterizes incremental updates by two low-dimensional matrices and only fine-tunes them. AdaLoRA (Zhang et al. 2023) dynamically adjusts the low-rank update budget allocation during training. SPU (Zhang ct al. 2024) is a PEFT bascline, specifically designed to tackle catastrophic forgetting in CL scenarios, that utilises structured sparsity based on gradient information to fine-tune the most significant. parameters of the fel module in the transformer block.
Table 1: Offline fine-tuning results for MiniGPTv2 and LLa under two adaptation strategies com- pared to zero-shot baseline (no FT): LLM-only tuning (TJaMA+Pj) and vision-encoder tuning. Best scores in each column are in bold. Updating the vision encoder separately leads to best gains.
4.3 Offline performance on different VLMs
In this first experiment, we compare the performance of two VLMs (LLaVA and MiniGPTv2) on TSI and DALLE. TSI data depict elderly people activities (age bias), blurred faces (blurring effect) and is cap- tured from mounted camera with relatively low resolution, yet the actions are easily recognizable to the human eye. DALLE that is composed of same activity classes under clear concept centred images (see Appendix D). First, results in Table | illustrate the visual domain shift of TST with respect to models
MiniGPTv2
LLa15-1.5
FT Method
DALLE
TSI
DALLE
TSI
No FT (Zr-shot)
83.6
62.9
91.1
53.1
LLaMA+Pj Vision-encoder FT
86.5
82.3
88.5
73.3
87.1
86.0
91.1
75.5
7

---

Published in Transactions on Machine Learning Research (12/2025)
pretraining given the significantly lower performance of TSI compared to DALLE indicating an update is indeed on TST to improve the performance. Next, compare two adaptation strategies in an offline fine-tuning setting to zero-shot (no updates), LLM-only tuning (LLaMA+Pi). Across both models, tuning only the vision encoder separately yields the largest gains. These consistent improvements strongly corroborate our claim that vision-only fine-tuning is an efficient and effective stralegy for adapting visual-language models under visual shift sctting. For all subsequent experiments, we employ only the LLaVA model.
4.4 CLIP-based Updates
Table 2: Average accuracy (ACC) (t) and backward transfer
We evaluate the performance of the Vision-Language Model (VLM) when only the image encoder is fine-tuned using the CLIP loss in CL setting. This exper- iment compares six strong CLIP-based basclines with our proposed method, LoRSU. Table 3 reports the average accu- racies of TI/CC over three runs; detailed results can be found in appendix [D] We observe that LoRSU consistently achieves superior TI scores across datasets and CL settings, underscoring its ability to enhance task-specific performance effee- tively. Furthermore, LoRSU maintains CC accuracies that take consistently small negative or even positive values, highlight- ing its capacity to preserve or slightly im- prove performance on control datasets while fine-tuning on target datasets. Even in datasets where other methods struggle (e.g., CAn, ESAT), LoRSU often performs better, maintaining positive CC scores. For instance, In ESAT (CI-50) containing challenging satellite images, LoRSU achieves the highest TI (7.0) with a positive CC (0.2), outperforming SPU (TI=5.8, CC=0.1) and all other methods.
(BWT) (1) scores (%) across different continual learning (CL) setting and fine-tuning datasets (FTD). LoRSU achieves top ACC at BWT >-1% (i.c., close to zero forgetting). The highest scores across methods are in bold.
LoRA
SPU
LoRSU
Setting
FTD
ACC
BWT
ACC
BWT
ACC
BWT
GTS
79.2
-7.1
80.8
0.5
81.1
0.4
CL-5
ESAT
73.8
-3.4
79.8
1.5
82.2
2.0
GTTS
77.2
-9.1
82.8
-0.6
83.5
-0.4
CL-20
ESAT
64.1
-18.3
82.0
2.0
82.7
0.1
GTS
79.3
-10.3
83.8
-0.7
84.7
-0.5
CL-50
ESAT
61.4
-27.8
81.2
-2.4
82.1
-0.5
CL metrics. We assess the performance of LoRSU against LoRA and SPU in terms of ACC and BWT across two out-of-domain datasets, GTS and ESAT. Since LoRA and SPU have similar number of train- able parameters as LoRSU and competitive performance in our previous experiment, we choose those for comparison. Table 2 shows that LoRSU's performs well with respect to these metrics, following similar patterns as TI and CC in Table 3 Increasing the number of shots generally improves accuracy, particularly for structured update methods (SPU and LoRSU); however, this improvement comes with increased negative backward transfer for SPU, whereas LoRSU demonstrates comparatively less forgetting. Crucially, LoRSU strikes the best balance: it leverages up to 50 shots for top-end accuracy with forgetting (negative BWT) less than 1%. Similar patterns are observed in additional datasets in the Appendix D.3, Therefore, our results demonstrate that LoRSU effectively meets all four desiderata: (1) preserving the VLM's generic knowledge, (2) achieving substantial improvements over zero-shot performance, (3) maintaining computational efficiency, and (4) eliminating the need for replay- based methods.
4.5 CLiP-based vs. Perplexity- Updates
Traditionally, LLMs and VLMs achieve impressive performance through fine-tuning with the perplexity loss. We evaluate how the CLiP-based fine-tuning methods, LoRSU and LoRA, perform compared to their perplexity-based counterparts, LoRSU-Ppl and LoRA-Ppl, respectively. Furthermore, how these methods compare to parancter-officient finc-tuning approaches when the entire VLM (LoRA-F) or only the LLM component (LoRA-L) is updated. The results in Table 4 highlight the strong and robust. performance of LoRSU and LoRSU-Ppl compared to other baseline methods in various settings. Both LoRSU and LoRSU-Ppl achieve minimal negative or even positive changes in CC, indicating reduced catastrophic forgetting and improved retention of generic knowledge compared to baselines. The use of the perplexity loss in LoRSU-Ppl demonstrratss a considerable improvement in TT accuracy over LoRSU when fine-tuned for VQA
seek to explore
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Performance comparison of LoRSU with the CLIP loss against baselines fine-tuning the image encoder using the same loss. We report the Target Improvement (TI (1)) and Control Change (CC (t)) accuracies across three different continual learning (CL) settings and five fine-tuning datasets (FTD). Greener shades indicate higher positive values, while redder shades signify lower negative values. The highest accuracies across methods for each dataset are underlined. LoRSU achieves the best TI and CC.
FT Method
LN
F-FT
F-EWC
LoRA
AdaLoRA
SPU
LoRSU
Setting FTD
TI
CC
CC
TI
CC
CC
TI
CC
TI
CC
CC
GTS
3.5
-1.5
3.7
-6.5
5.0
-1.55
0.7
-4.8
-0.9
-4.9
5.4
-0.6
6.4
-0.7
TSI
0.8
0.0
7.4
-1.1
8.5
-1.0
-0.1
-2.8
1.1
0.2
0.9
0.1
3.2
0.1
CL-5
CAn
-2.4
-0.2
-2.4
-2.2
-16.7
-9.4
-1.3
-4.6
-1.0
-0.1
-0.4
0.1
0.3
0.3
AIR
0.3
-1.6
2.0
-2.7
2.9
-2.8
1.3
-3.7
0.4
0.0
3.1
0.1
4.8
0.4
ESAT
4.2
0.6
-10.3
-1.4
-8.4
-2.1
-1.6
-0.7
1.9
0.1
4.5
0.1
6.8
0.2
GTT
5.2
-5.9
4.6
-7.3
6.7
15.6
2.5
10.5
0.2
-2.2
7.9
-1.3
8.6
-1.0
TSI
5.1
-1.9
15.3
-3.4
16.0
-32.5
8.5
-4.4
1.3
-9.6
7.8
-0.3
10.6
-0.1
CL-20
CAn
-2.4
-0.4
0.3
-2.9
0.1
-5.1
-2.3
-5.4
-3.5
-2.5
0.1
0.5
1.1
0.3
AIR
-0.2
-3.0
9.3
-1.8
10.2
-2.0
5.3
-2.7
2.7
-0.7
3.0
-0.2
5.9
-0.5
ESAT
0.9
-0.1
-24.9
-1.7
-22.0
-3.8
-1.5
-0.5
-6.8
-2.7
5.4
0.3
6.6
0.2
GTTS
4.8
-6.5
3.4
-9.8
5.3
-12.9
3.1
-11.1
1.0
-3.3
7.7
-1.5
9.7
-1.3
TSI
7.0
-3.0
17.2
-4.6
22.4
13.4
18.2
-6.3
7.9
-1.9
12.2
-0.5
19.1
-0.3
CL-50
CAn
-5.7
-3.3
-1.0
-4.9
0.6
-9.7
-0.4
-4.4
-1.8
-0.8
0.6
-0.3
1.3
-0.5
AIR
-3.9
10.0
-3.1
10.9
-3.3
-3.8
4.6
-0.9
-0.6
8.2
-0.7
ESAT
4.6
0.1
-41.4
-3.3
-38.1
-2.0
14.5
-3.6
-17.3
-2.4
5.8
0.1
7.0
0.2
datasets. For example, LoRSU-Ppl achieves 10% higher TI accuracy than LoRSU on VSR. We hypothesize that the perplexity loss acts an additional signal that optimizes the image encoder complement the frozen language model more effectively, improving the alignment between visual and textual modalities in VQA. However, we observe that LoRSU achieves a balance between lask-specific improvements and generalization, consistently demonstrating higher CC accuracy compared to LoRSU-Ppl in most datasets. Updating the LLM tend to have higher TI under VSR and HM datascts compared to pure distributional shift datasets (o.g. GTS, TSI, ESAT) indicating that updating the LLM can be less optimal under distribution shift scenarios compared to the vision encoder updates suggested in LoRSU. Lastly, although LoRA-F achieves high TT scores on many datasets, it suffers significantly from forgetting, underscoring the importance of LoRSU's structured updates in CL scenarios.
4.6 Ablation Studies
We systematically evaluate LoRSU's design choices by varying the number of tuned attention heads k, the LoRA adapter rank r, and head-selection strategies (random vs. all-heads). We provide summary of these results in Table 5 detailed results are reported in Appendix D Regarding the impact of LoRA's rank we see that performance peaks at r-64 for both GTS and TSI datasets, with graceful degradation at other values, demonstrating robustness. Using performance and efficioncy, with diminishing returns beyond this point. Finally, the results demonstrato that our gradient-based head selection (LoRSU) consistently outperforms both random selection (LoRSU-Rand) and updating all heads (LoRSU-AAH) across all CL settings, validating our structured update approach.
attention heads provides an optimal balance between
4.7 Computational Efficiency
9

---

Published in Transactions on Machine Learning Research (12/2025)
Table 4: Performance comparison between LoRSU using the CLIP loss (LoRSU) or the perplexity loss (LoRSU-Ppl) and other basclines that fine-tune only the vision encoder (LoRA, LoRA-Ppl), only the LLM (LoRA-L), both of them (LoRA-F). We report the Targel Improvement. (TI (f)) and Control Change (CC (1)) for each CL setting. f and denote classification-to- VQA and regular VQA datasets, respectively, The highest accuracies across methods for cach fine-tuning datasct (FTD) are underlined.
FT Method
Setting
FTD
LoRA-L
LoRA
LoRSU
LoRA-Ppl
LoRA-F
LoRSU-Ppl
TI
CC
TI
CC
TI
CC
TI
CC
TI
CC
TI
CC
GTS
-4.1
-0.2
0.7
-4.8
6.4
-0.7
-7.5
-3.0
-2.7
-1.8
1.6
-1.0
TSI
6.0
-0.1
-0.1
-2.8
3.2
0.1
10.9
-2.4
-8.0
-2.4
13.1
1.5
CAn
-3.3
-0.2
-1.3
-4.6
0.3
0.3
-3.5
-5.5
-4.1
-1.6
0.2
-0.2
CL-5
AIR
-1.7
0.3
1.3
-3.7
4.8
0.4
-0.7
-1.5
9.6
-1.9
5.8
-0.2
ESAT
-0.2
-0.1
-1.6
-0.7
0.2
-0.6
0.4
5.4
-0.5
0.1
VSR
16.8
-0.6
0.5
-4.0
0.4
0.2
10.2
-12.5
18.0
-00.6
10.5
-1.2
HM
7.4
-2.7
-0.4
-6.8
0.6
0.4
-1.2
-1.2
6.0
-4.5
-0.8
0.2
VisOnly 00.
-0.1
-1.1
-4.5
0.9
0.1
0.3
-0.3
0.2
-0.4
2.7
0.7
GTS
-1.4
0.1
2.5
-10.5
8.6
-1.0
-0.5
-6.4
-1.4
-0.8
3.9
-0.7
TSI
5.9
0.0
8.5
-4.4
10.6
-0.1
6.5
-1..6
2.9
-3.1
13.9
-0.6
CAn
-1.9
-0.6
1.1
0.3
-3.7
-2.1
-1.7
0.5
-1.2
CL-20
AIR
3.7
0.3
5.3
-2.7
5.9
-0.5
4.8
-3.5
16.3
-0.3
6.0
-0.3
ESAT
0.7
0.4
-11.5
-0.5
6.6
0.2
-1.2
-0.1
-4.6
-0.0
2.9
-0.1
VSR
22.2
1.0
0.4
-3.9
0.1
-0.2
19.
-0.3
23.3
-5.1
22.9
-1.6
HM
10.6
-2.2
-1.8
-5.8
0.7
0.2
10.7
-0.1
11.7
-1.4
10.9
-0.2
VisOnly
-2.3
0.7
-2.0
0.5
-1.0
1.7
0.5
GTS
-0.7
-0.3
3.1
-11.1
9.7
-1.3
-1.4
-6.7
-3.9
-2.1
-0.4
TSI
9.9
-0.0
18.2
-6.3
19.1
-0.4
-1.6
16.5
15.1
-0.7
22.0
-1.1
CAn
-1.8
-0.5
-1.8
-2.1
-1.1
1.0
-3.4
CL-50
AIR!
4.6
0.4
7.8
-3.8
8.2
-0.7
6.2
-3.1
17.9
-0.9
8.9
-0.4
ESAT
1.0
0.2
14..5
-3.6
7.0
0.2
1.7
0.2
-9.5
-0.6
-0.7
-0.5
VSR
21.9
1.0
0.4
-4.5
-0.3
20.2
-5.3
23.4
-3.6
HM
10.2
-
0.7
-4.5
0.3
0.2
12.5
-1.5
12.3
-3.7
12.2
0.2
VisOnly+
-2.4
0.6
-0.2
-6.8
0.3
-0.1
-2.0
0.7
0.2
0.3
0.1
235
Pizana (30)
In Figure 3 we assess the computational benefits of LoRSU (CLIP loss) compared to baseline methods. We focus two key metrics: trainable pa- rameters and TFLOPs. LoRSU requires 25x fewer computation resources than LoRA-F and LoRSU-Ppl, demonstrating the suitability of using CLIP loss when computational resources are limited. Unlike perplexity loss, which requires forward and backward passes through both the vision encoder and LLM, the CLIP loss operates solely on the vision encoder, significantly reducing computational overhead. This makes LoRSU more scalable, enabling efficient CL even in resource-constrained settings.
365
150
56
A1
0066
L:R4S
LaBS
Figure 3: TFlops and trainable rameters comparison.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: Ablation study on LoRSU (with CLIP loss) hyperparameters in the CL-50 setting. (Top) Effect of LoRA rank r on target dataset (TSI) accuracy. (Middle) Effect number of attention heads k. (Bottom) Comparison of head selection strategics: gradient-based (LoRSU), random (LoRSU-RRan)), and all heads (LoRSU-AAH) using GTS with ESAT as control. Results show 64 and provide optimal performance, with gradient-base based selection outperforming alternatives.
(a) LoRA Rank Ablation
FT Dataset
rank (r)
Target
DALLE
VSR
HM
MMVP
VisOnly
67.2
91.1
51.5
61.6
58.0
31.5
32
68.9
91.2
51.5
61.6
58.0
31.6
TSI
64
72.1
90.5
51.6
61.4
58.0
31.6
128
65.8
90.6
51.5
62.1
56.7
31.6
(b) Number of Attention Heads Ablation
FT Dataset
# heads
Target
DALLE
VSR
HM
MMVP
VisOnly
64.2
90.8
51.5
61.8
57.3
31.5
64.8
90.5
51.5
61.6
58.0
32.0
TSI
2
72.1
90.5
51.6
61.4
58.0
31.6
66.8
90.5
51.5
62.1
58.0
31.4
(c) Head Selection Strategy (GTS)
Method
TI (t)
CC (t)
Params (M)
LoRSU-Rand LoRSU-AAH
7.8 = 0.5
-18.1 0.8
0.36
9.1 = 0.1
-19.6 + 0.5
2.88
LoRSU
9.7 = 0.1
-14.3 0.7
0.36
11

---

Published in Transactions on Machine Learning Research (12/2025)
G18 Avg lol -288%
TSI Avg 1oU 8%
Aug 22.7%
Sooden
(a) GTS
(b) TSI
(c) CAn
Aug 23.7%%
ESAT Avg 1oU=1885%
122
(d) AIR
(e) ESAT
Figure 4: Jaccard similarity matrices of LoRSU's binary masks across CL sessions for five datasets. The heatmaps show the Intersection over Union (IoU) between the binary masks selected in different continual learning sessions. Diagonal elements are 1.0 (perfect overlap), while off-diagonal elements indicate the degree of overlap between different sessions for parameters chosen for update. We include the average IoUs for each dataset in the title of the panel.
4.8 Parameter Overlap Analysis
understand why LoRSU achieves effective plasticity-stability balance, we analyze the overlap of binary parameter masks selected across the We compute the Jaccard index (Jaccard 1901 also known as Intersection over Union (IoU), between masks from different sessions for five represcntative dalasets in the CL-50 setting with the CLIP loss. The results are illustrated in Fig. 4 whilst exact. values are given in Appendix D.1
Results show consistent parameter overlap only 15-25% across all session pairs. This small overlap provides insight into LoRSU's effectiveness. First, sufficient. separation (75-85% distinet parameters) provides plasticity for learning new tasks without interfering with previous knowledge, allowing task-specific adaptation. At the same time, moderate overlap (15-25% shared parameters) ensures the model continues leverage the foundation model's general-purpose visual features rather than fragmenting into isolated task-specific modules. This overlap also prevents catastrophic forgetting: if cach task used completely disjoint parameters, the cumulative updates across sessions would shift a large proportion of the pretrained parameters away from their foundation model values, undermining the stability and general capabilities acquired during pretraining.
This consistent parameter overlap emerges naturally from our gradient-basee selection: high-magnitude gradients identify paraneters that are both critical for the new task and exhibit moderate overlap with previous task parameters, balancing learning and retention.
12

---

Published in Transactions on Machine Learning Research (12/2025)
5 Discussion
We introduced LoRSU, a novel parameter-eficient fine-tuning method specifically designed for few-shot contimual learning scenarios with VLMs. Unlike existing approaches, LoRSU operates without relying on a replay buffer, making it uniquely suited for resource-constraineel settings. Through more than 700 experiments, we demonstrate that LoRSU satisfies all four desiderata: (1) preserving the VLM's generic knowledge, (2) attaining substantial improvements over zero-shot performance, (3) maintaining computational efficiency, and (4) eliminating the need for replay-based methods. LoRSU outperforms 12 baselines in over 80% of evaluations across 10 datasets and 3 settings, achieving the highest TI accuracies in most cases while maintaining stable or even positive CC accuracies. To the best of our knowledge, we are the first to explore few-shot continual learning of VLMs. Whilst we focus on CLIP and LLaVA due to computational constraints, our method is generic to any transformer model, and we plan to extend it to other VLMs and image encoders. Another promising direction is using a smaller LLM proxy model in perplexity-based methods like LoRSU-Ppl, which has shown strong VQA performance. This could improve scalability and LoRSU's use in resource-limited settings. Finally, LoRSU's binary mask-based structured updates ensure efficient, precise parameter updates, but scaling to larger architectures like LLMs poses challenges. Replacing binary masks with more scalable solutions for vast parameter spaces will be crucial to manage memory and processing demands, offering opportunities for further refinement.
References
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXic preprint Xiv:233300 09/78 2023.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 14155244198 2024.
Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen, and Ningyu Zhang, Can we edit multimodal large language models? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13877-13888, Singapore, December 2023. Association for Computational Linguistics. doi: v122633//112233..o/ eemepp.mmllpppp. main. org///gg//222
856. URL
main. 856/
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See httpss//vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXic:2104.08696, 2021.
Deepayan Das, Davide Talon, Massimiliano Mancini, Yiming Wang, and Elisa Ricci. One vim to keep it learning: Generation and balancing for data-free continual visual question answering. ar preprint 3rXiv:2411.02211 2024.
Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garationi, Francois Bremond, and Gianpiero Francesca. Toyota smarthome: Real-world activities of daily living. In Proceedings of the IEEE/CV international conference on computer vision, pp. 833-842, 2019.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pottern analysis and machine intelligence, (4()):3366-3333 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255,
2009. doi:
13

---

Published in Transactions on Machine Learning Research (12/2025)
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198 11201, 2024.
Dipam Goswami, Bartlomiej Twardowski, and Joost Van De Weijer. Calibrating higher-order statistics for few-shot class-ineremental learning with pre-trained vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp.
2024.
Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter- efficient fine-tuning, In Proceedings of the LEEE/CVF International Conference on Computer Vision, pp.
11825-11835, 2023a.
Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. Continual instruction tuning for large multimodal models. arXiv preprint wrXim:2311112200 2023b.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXio preprint arXiv:2106. 006608,5 2021.
Wenke Huang, Jian Liang, Xianda Guo, Yiyang Fang, Guancheng Wan, Xuankun Rong, Chi Wen, Zekun Shi, Qingyun Li, Didi Zhu, et al. Keeping yourself is important in downstream tuning multimodal large language model. arXiv preprint arXiv:2508.04 04543, 2025.
Paul Jaccard. Étude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sei Not, 37:547-579, 1901.
Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-motte in continual learning. arXiv preprint wrXiv:2210.04428 2022.
Ryo Kamoi, Yusen Zhang, Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, and Rui Zhang. Visonlyqa: Large vision language models still struggle with visual perception of geometric information. arxiv preprint 1rXiv:2412.0000 2024.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33:2611 162211126224, 2020.
Diederik P Kingma. Adam: A method for stochastic optimization. anXiv preprint arXiv:1412.6980 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, (14(13):352) 3526, 2017.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping sanguage-mage pre-training with frozen image encoders and large language models. In Internalional conference machine learning, pp. 19730-199 19742. PMLR, 2023.
Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
David Lapez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neuraa information processing systems, 30, 2017.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Loshchilov. Decoupled weight decay regularization. arXic preprint arXiv:1711.05100 2017.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, University of Oxford, 2013.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Newral Information Processing Systems, 35:17359 17372, 2022.
Eric Mitchell, Charles Lin, Antoine Bossclut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022. URL https:///peenrevie..nn / net/
f 21=00CZxeKKOUPF
Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: A strong replay-free baseline for daasmincremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18820- 18830, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
Ethan Perez, Florian Strub, Harm De Vries, Vincent. Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicap,, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In International Conference on Learning Representations, 2020. URL httts://openreviie
Shikhar Srivastava, Md Yousuf Harun, Robik Shrestha, and Christopher Kanan. Improving multimodal large language models using continual learning, arXiv preprint arXic:2410.1190 19922, 2024.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Tgel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32:323 332, 2012.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9568-9578, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xix preprint or xxXi:2307.009288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. Advances in neural information processing systems, 30, 2017.
Eli Verwimp, Rahaf Aljundi, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, et al. Continual learningg Applications and the road forward. arXiv preprint ar 2rXiv:2311.11908 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing lasion-language model's perception of the world at any resolution. Xix preprint Niv:24400 12191, 2024a.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Qizhou Wang, Yong Lin, Yonggiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. Do clips always generalize better than imagenet models? arXiv preprinl arXie:2403. 1 1497, 2024b.
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv preprint arXi::2402.01364 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-effcient fine-tuning. arXiv preprint Xrx::23003. 10512, 2023.
Wenxuan Zhang, Paul Janson, Rahaf Aljundi, and Mohamed Elhoseiny. Overcoming generic knowledge loss with selective parameter update. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pottern Recognition, pp. 24056, 2024.
Linglan Zhao, Xuerui Zhang, Ke Yan, Shouhong Ding, and Weiran Huang. Safe: Slow and fast parameter- efficient tuning for continual learning with pre-trained models. arXie preprint 1rX0v2441.002155, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. ar preprint arX6o:2344.1059,, 2023.
16

---

Published in Transactions on Machine Learning Research (12/2025)
A Proof for the optimal mask p"
Definition A.1. The operator TOP-C: R + Rd for S C' S d is defined as
(TOP-C(x))=() == In((), iso C 0, otherwise,
where (#1,..., ra)T Rd and is permutation of f1. df such that [xx()]] 2 [ for = l,...,d- d-dd-11 1, i.e. the TOP-S operator keeps only the S largest elements of in magnitude and truncates the rest to zero.
Lemma A.2. For any Rd -d--(0) (0}, 1SCdd, d, the optimal mask
Ipox|
p' arg max
s.L. |PP S SC
||||²
(0
has zeros everywhere except the C largest elements of x in magnitude.
Proof. Rewriting the optimization problem as
E S.A. Σ pi S.
max PC (0.1)
Notice that this is trivial binary knapsack problem with maximum weight capacity C' and weights equal to Hence, the maximun is attained when we pick the top C maximal xē elements.
Remark A.3.
It holds that TOP-S(x)= p x
Corollary A.4. The optimal mask in 5 has zeros everywhere except for the indices {j 30 e (I....,G), G), such that E (me(1)..... ..... (e())) where ne is the same permulation as in Definition A.1 for the set of indices It.
Proof. The result follows from the mutual exclusiveness of in the constraints of 5 and Lemma A.2 0
B Implementation Details
We describe below the implementation details of section
All the experiments are conducted single NVIDIA A100 GPU.
We have included error bars over three runs for all experiments.
We use PyTorch Paszke et. al. 2019) to implement all the algorithms.
We use Adam Kingma, 2014 as optimizer for the methods that utilize the CLIP loss for fine tuning and AdamW [Loshchllov
2017) for those ones that use the perplexity loss.
learning rate scheduler of Cosine Annealing with Warmup is employed for all methods.
For all experiments, we set the learning rate 1 x 10-5 and 2 x 10-5, for LoRSU and LoRSU-Ppl, respectively.
We set batch size to 16 for all methods that fine-tune the vision encoder through CLIP loss. We reduce the batch size to for those methods that fine-tune the vision encoder through perplexity loss those that fine-tune the LLM. This due to GPU memory limitations.
17

---

Published in Transactions on Machine Learning Research (12/2025)
All methods run for 20, 15, and 10 epochs for the CL-5, CL-10, and CL-50 settings, respectively,
For LoRA (-Ppl), we set rank r 64 while LoRA-L and LoRA-F use 8, for all experiments.
For AdaLoRA, we set the initial rank to 70 and the final average rank to 64.
The adapters of LoRA and AdaLoRA are applied to all weight matrices of each of the transformer blocks.
For SPU, use sparsity=15% for all experiments.
For LoRSU (-Ppl) we use sparsity=10% rank-64, and we pick the top-2 attention heads for all experiments.
The choice of the above hyperparameters ensures that LoRA (-Ppl), LoRA-L, LoRA-F, AdaLoRA. SPU, and LoRSU (-Ppl) have similar number of trainable parameters.
C
Datasets
Details on all datasets used in section ! are presented here.
C.1
VQA Datasets
We evaluate the performance of LoRSU on ten visual question answering (VQA) datasets falling in two broad categories:
regular VQA datasets and classification datasets converted to VQA datasets.
Regular VQA datasets. We consider four standard VQA datasets used for benchmarking VLMs' perfor- mance Duan et al. (024)): LiR et al. (2023), the Visual Spatial Reasoning corpus consists of caption-image pairs labeled as True or False, where each caption describes the spatial relation between two objects in the image. VT.Ms evaluate whether the caption accurately reflects the image. HM Kiela et al. (2020). the Hateful Memes dataset designed to detect multimodal hateful memes. MM VP Tong et al. (2024). the Multimodal Visual Patterns dataset is a challenging benchmark which has been built on images that CLIP perceives as similar despite their clear visual differences. VisOnly Kamoi et al. 2024 a novel dataset created to directly assess the visual perception abilities of VLMs in answering questions about geometric and numerical details in scientific figures. This dataset allows us to assess fine-grained visual perception in VLMs independently of other abilities, such as reasoning, making it the most challenging among the previously mentioned datasets.
Classification-to-VOO. VQA datascts. We convert four popular multi-class classification datascts into multiple- choice VQA problems, where each question has five choices, and the VLM is tasked with selecting the correct answer. These datasets are introduced as examples of scenarios where visual domain shifts are encountered, allowing us to examine the utility of updating the image encoder; a critical consideration often overlooked in many standard VQA datasets. The datasots include: GTS Sttllkamp et al. (0012). the German Traffic Sign dataset, which Zhhan el al. (2024) considered as an out-of-distribbttion dataset for CLIP pretraining: CAn Wang et al. (2024b), a recent dataset created to test CLIP's robustness with animal images containing realistic spurious features such as unexpected backgrounds; AIR Maji et al. (2013). fino-graines aircraft.
classification dataset; ESAT Helber et al. 2019 dataset of satellite images used for land cover classification.
TSI & DALLE. In addition to these existing datasets, we introduce two novel VQA datasets: TSI and DALLE, both designed to explore the effects of domain shift. For more details see sections F and C.2
We follow the common practice few-shot continual learning Panos et al. (2023) construct the sequences. We divide each datasct into sets of disjoint classes/ /casss//catcooicc and consider 5/20/50 /50 shot scttings where only
5/20/50 images per class in the current set are used for fine-tuning the model. More details on how we split. each of these datasets for the CL settings are provided in appendix C
C.2
TSI & DALLE
We start with the description of how constructed our newly introduced VQA datasets TSI and DALLE.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: The original action names of the Toyota Smarthome dataset and their corresponding captions used to create the Toyota Smarthome Images (TSI) dataset. We use X to denore the actions that are ambiguous and were not used to build the TSI dataset. The final prompt is created as " "The person in this image is { fcaption
Original Class name/ name/Action Action
Generated Caption
Clenndilesssses
washing dishes cleaning up cutting food
Cook. Cleamup
Cook. Cut
Cook.Stir
stirring the pot x cutting bread holding a bottle holding can holding a cup holding a glass cating
Cook. Usestove
Cook. Cutbread
Drink. Fromcup
Drink. Fromglass
Ett.Attable
Eat.Snack
x walking
Enter
Getup
x lying down walking
Laydown
Leave
Makecoffee. Pourggriis
using a white coffee machine using a white coffee machine
Makecoffee. Pourwater
P
Boilwatrrr
boiling water in a black kettle
making tea holding a bottle holding can holding a black kettle
Pour Frombottle
Pour. Fromcan
Pour Fromkettle
Readbook
reading a book sitting down x
Sitdown
Takepills
Uselaptop
using a laptop using tablet
Usetablet
Usetelephone
using cordless phone
Walk
walking watching TV
WatchTV
TSI. To extract images from the videos of the Toyota Smart Home dataset (TSI), we discretized each video clip into 2 frames per second and then selected the frame in the middle of the total time duration of the video clip. In Table we describe the actions that were selected and the corresponding prompt used for CLIP classification. We also note dropping few actions to avoid ambiguous classes. Note that we did not use any extra data for this CL benchmark and all the images were created from the already available videos of the datasct.
The TST dataset focuses on elderly individuals performing daily activities in domestic settings, which may not represent the full diversity of how these activities are performed across age groups, abilities, and cultural contexts, and it could potentially reflect or amplily age-related biases.
We generated images from DALL E using OpenAl python package and we used the prompt AA person {a}" where a E using a white coffee machine, eating, cutting bread, stirring the pot, holding a glass, watching TV, holding a bottle, walking, making tea, cutting food, holding cup, using a laptop, lying
DALLE.
19

---

Published in Transactions on Machine Learning Research (12/2025)
down, holding can, person holding black kettle, reading book, cleaning up, sitting down, using tablet, boiling water in black kettle, using cordless phone, washing dishes).
In Table 7 we present the average number of images per session used to update the model for each CL setting. Finally, Table g provides characteristics of the datasets used for evaluating performance.
C.3 Continual Learning Splits
For the continual learning settings of section we split all datasets into five non-overlapping continual learning (CI.) splits based on the classes/ categories of each dataset. Unless stated otherwise, we use the training split of each dataset to construct these CL splits.
GTS Stallkamp et al. (2012)
We split the 43 classes of GTS follows:
Session 1: [25, 2, 11, 1, 40, 27, 9,
Session 2: [32, 29, 20, 39, 21, 15, 23, 10, 3].
Session 3: [18, 38, 42, 14, 22, 34, 19, 33].
Session 4: [12, 26, 41, 37, 6, 13, 24].
Session 5: [30, 28,
31, 7, 28,31,
16,
TSI Das et al. (2019). We split the 27 action categories of TSI as follows:
Session 1:
[WtchhTV Laydoun, Sitdown, Pour. Fromkettle, Enter, Drink: Frombottle)
Session 2:
Eat.attable, Pour. Frombotoee,, Cook. Cleondishes, Maketea. Boilwater, Leave, Cook. Cleanup).
Session 3:
Maketta..nseetttene
Makeoofees.Puuruter, Pourwater,
Readbook, Cutbread].
Session 4:
Drink.. Fromcup, Drink. Fromglass, Usetablet, Pour. Fromoun, Usetelephone)-
Session 5:
[Walk, Cook. Makeooffee.Porrguinn Cook. Cul, Useluptop).
CAn Wang et al.
(2024b). The 45 classes of CAn split follows:
Session 1: [102,9. 9, 20, 56, 23, 30, 357, 291, 144].
Session 2: [41, 293, 42, 49, 54, 57, 70,279 20,277,300
Session 3: 771, 10, 76, 79,349, 16, 81,83, 100).
Session 4: [130, 30, 133, 150, 275, 276, 58, 277, 277.80).
Session 5: [39, 290, 37, 296, 316, 337, 89, 360, 128).
The indices of CAn correspond to those of ImageNet Deng et al. (2099) since the dataset was built based on these 45 animal classes ImageNet.
AIR Maji et al. (2013) We split the 100 aireraft types of AIR as follows:
Session 1: [23, 8, 11, 7, 48, 13, 1, 91, 94, 54, 16, 63, 52, 41, 80, 2, 47, 87, 78, 66].
Session 2: [19, 6, 24, 10, 59,30, 22, 29, 87, 43, 99. 86, 28, 34, 88, 44, 14).
Session 3: [84, 70, 4, 15, 15,21,31,
76,,,,,,,, 77, 69, 25, 98, 46, 96, 0, 72,
Session 4: [58, 92, 95, 56, 26, 89, 71, 60, 82, 39, 18, 77, 68).
Session 5: [32, 79, 12, 85,36, 17, 64, 27, 74, 68, 51, 62, 65, 33, 53, 49).
20

---

Published in Transactions on Machine Learning Research (12/2025)
ESAT Helber et al. (2019).
We split the 10 different land terrain classes of ESAT follows:
Session [0, I].
Session 2: [2. 3].
Session 3: [4, 5].
Session 4: [6, 7].
Session 5: 9).
DALLE. This dataset only used for performance evaluation (control datasct), and not finc-tuning.
VSR Liu et al. (2023) The images of this VQA dataset are labeled according to 36 different categories that describe the dominant object of the image. We create the CL splits as follows:
Session [oven, dining table, spoon, boat, cake, donut, sandwich).
Session 2:
[fire hydrant, elephant, airplane, truck, apple, hot dog, sheep).
Session 3:
[kite, baseball glove, cow, tie, scissors, toaster, tu].
Session 4:
[iicycle, banana, couch, teddy bear, bus, umbrella, bird].
Session 5: ppotted plant, bownl broccoli, bottle, knife, orange, person, pizza].
HM Kiela et al. (2020). For the hateful memes dataset, since there was not any labeling information of the images so we can spli the images in a meaningful way, we randomly split the training images into five disjoint sets to create our final CL splits.
MMVP Tong et al. (2024). This is the only dataset where no training split available and it is comprised of just 300 images. For this reason, we only used it for evaluation in our experiments in the main paper. However, for completeness, we included results in Table 27 where we fine-tune on We use 150 images for training which are equally split into five sessions and the rest of the 150 images are used for evaluation. Thus, the setting can be considered 30-shot CL setting.
VisOnly Kamoi et al. (2024). This dataset categorizes its samples into seven categories describing the nature of the geometric and numerical information in scientific figures. We created the splits as follows:
Session Geometry- Triangle.
Session 2:
Geomelry- Quadriiaeea............
Session 3: Geomelry-Lene Lenglh
Session 4: Geomelry-Anue Angle.
Session 5: [Geometry-Area, 3D-Size, AD-leg..).
D Detailed Results
D.1 Exact IoU results
We provide the exact Jaccard similarity values for the binary masks across sessions for each dataset in tables
through 13
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table 7: Average number of images per session (5 sessions in total) for each dataset used for fine-tuning.
FT Dataset
Setting
GTS
TSI
CAn
AIR
ESAT
VSR
HM
VisOnly
CL-5
43.0
27.0
45.0
100.0
10.0
100.0
100.0
7.0
CL-20
170.0
84.0
180.0
400.0
40.0
274.6
300.0
28.0
CL-50
430.0
253.8
450.0
1000.0
100.0
485.2
600.0
70.0
Table Characteristics of the datasets used for performance evaluation in section
Eval Datasets
GTS
TSI
CAn
AIR
ESAT
DALLE
VSR
HM
MMVP
VisOnly
Samples
3, 990
4, 1.98
1, 796
3, 333
17,
660
1, 1,222
2, 0.0
150
1,150
Classes
43
27
45
100
10
27
36
NaN
NaN
Table 9: Jaccard similarity matrix for the GTSRB dataset.
Session
1
2
3
4
5
1 1.000
0.225
0.200
0.183
0.209
2
0.225
1.000
0.240
0.222
0.232
3
0.200
0.240
1.000
0.216
0.219
4
0.183
0.222
0.216
1.000
0.241
5
0.209
0.232
0.219
0.241
1.000
Table 10: Jaccard similarity matrix for the TSI dataset.
Session
1
2
3
4
5
1
1.000
0.236
0.211
0.193
0.212
2
0.236
1.000
0.249
0.223
0.241
3
0.211
0.249
1.000
0.244
0.229
4
0.193
0.223
0.244
1.000
0.229
5
0.212
0.241
0.229
0.229
1.000
Table 11: Jaccard similarity matrix for the CAn dataset.
Session
1
2
3
4
5
1 1.000
0.233
0.250
0.227
0.210
2
0.233
1.000
0.227
0.252
0.201
3
0.250
0.227
1.000
0.234
0.210
4
0.227
0.252
0.234
1.000
0.212
5
0.210
0.201
0.210
0.212
1.000
22

---

Published in Transactions on Machine Learning Research (12/2025)
Table 12: Jaccard similarity matrix for the AIR dataset.
Session
1
2
3
4
5
1 1.000
0.224
0.232
0.224
0.226
2
0.224
1.000
0.244
0.250
0.242
3
0.232
0.244
1.000
0.252
0.239
4
0.224
0.250
0.252
1.000
0.231
5
0.226
0.242
0.239
0.231
1.000
Table 13: Jaccard similarity matrix for the ESAT dataset.
Session
1
2
3
4
5
1 1.000
0.157
0.153
0.170
0.173
2
0.157
1.000
0.208
0.198
0.192
3
0.153
0.208
1.000
0.195
0.194
4
0.170
0.198
0.195
1.000
0.211
5
0.173
0.192
0.194
0.211
1.000
Table 14: Accuracy scores (%) for LLaVA with the pretrained (Zr-Shot) or fine-tuned image encoder. All
baselines use GTS dataset for fine-tuning the image encoder (the LLM remains frozen) via CLIP loss. We include error bars over runs.
VQA Datasets (Ace %)
Setting Method
GTS
TSI
CAn
AIR
ESAT DALLE
VSR
HM
MMVP VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LN
7
53.62o.s
81.220.6
61.021a
58.920.9
91.1a1.
51.921.
62.721.1
59.620.2
31.820.4
F-FT
79.3+
55.1
76,8+1.a
58.8+1.0
25.6+0.9
89.2+1.
51.7+0.0
62.1+a.8
56.4+0.4
30.9+0.2
F-EWC
80.6+o.e 37.4+1.3
63.2+0.7
55.8+1.4
16.1
81.5+1.
51.8+14
61.2+0.6
53.8+0.4
31.2+0.4
CL-5
LoRA
a 52.621.4
73.320.6
56.7112
49.320.s
87.1113
51.821.2
61.3212
58.1
31.620.4
AdaLoRA
74.720. 49.720.7
79.620.
56.32o.
42.520.s
91.611.
52.020.s
60.921.2
57.120.3
31.720.2
SPU
81.0+1.4
53.741.5
82.52o.7
61.0+1.0
67.820.6
91.641.
52.020.s
62.0.1.a
58.220.a
31.620.a
LoRSU
8 53.521.a
82.410.s
60.8
66.620.9
91.5+1.
51.620.7
61.721.
59.8
31.620.2
LN
80.8 49.5
77.7
59.7
32.7
89.8
51.820.7
62.3+a.
57.5
31.2=0.2
F-FT
80.2 54.520.7
74.920.
57.221.0
23.210.7
86.74o.4
51.920.
61.621.
58.320.2
31.720.3
F-EWC
82.310
35.520.a
55.720.4
35.410.
28.720.9
72.41o.
51.620.7
60.910.
53.520.a
31.020.3
CL-20
LoRA
78.1+0.8 55.6+0.3
59.0+o.9
47.6+0.4
26.0+0.6
83.6+o.s
52.1+0.5
62.1+1.
53.7+
30.8+0.2
AdaLoRA
75.8+0.a 51.9+0.s
79.3+0.9
59.340.4
62.4
90.7+1.
51.640.5
61.1+n.6
57.7+0.2
31.7+0.2
SPU
83.520.6
53.120.a
82.220.7
60.7.o.s
62.02o.4
91.524.4
51.920.5
61.820.7
58.820.2
31.520.2
LoRSU
81.200 52.9
82.2
60.740.8
64.720.6
90.8
51.920.4
61.72a.
59.5
31.620.2
LN
80.4 50.44o.1
74.940.1
58.3
30.4
89.04o.1
51.820.0
62.02a.a
58.7
31.4=0.1
F-FT
79.0 48.9
65.0zo.2
55.0
23.520.0
86.8to.2
52.02o.1
60.8
54.9
30.7
F-EWC
80.9+a2
45.2+0.4
60.5+a.4
43.2
26.920.3
78.5+a.
52.0+0.0
58.7+a.1
52.940.o
31.7
CL-50
LoRA
78.710 50.7a0.0
62.120a
47.410.1
24.220.2
82.910.
51.720.
61.02a.2
54.320.1
30.820.0
AdaLoRA
7
50.440.0
79.02o.
37.410.1
58.32o.1
90.41o.
51.622.2
61.82n.
55.420.
31.820.1
SPU
8 53.8+0.a
81.8+02
61.1
58.8+0.0
91.0+0.
51.8+0.4
62.1+a.1
59.5+0.1
32.2+0.1
LoRSU
85,3+ 54.2+0.1
81.9+0.2
60.5+o.2
61.4.0.3
91.0+o.1
51.7+0.
62.2+a.4
58.940.1
31.8+0.1
23

---

Published in Transactions on Machine Learning Research (12/2025)
Table 15: Accuracy scores (%) for LLaVA with the pretrained (Zr-Shot) or fine-tuned image encoder. All basclines use TSI datasct for fine-tuning the image encoder (the LLM remains frozen) via CLIP loss. We include error bars runs.
VQA Datasets (Acc %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE
VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LN
75.4
53.920.6
82.621
60.0
75.920
91.1
51.721.4
61.921.
58.410.
30.910.
F-FT
73.8
60.521.1
81.620.9
59.521
70.421.
91.141.2
51.820.9
61.521.
56.910.
31.320
F-EWC
74.9+1.1
61.6+1.0
82.1+1.1
58.8+0.9
72.3+1.
89.9+1.4
51.++0.9
62.4+1.4
55.5+0.4
31.5+0.3
CL-5
LoRA
73.4+1.0 53.0=0.9
80,2+0.6
58.8+a.7
59.1+1.
90.2+1.1
51.6+1.3
61.2+14
56.7+0.4
31.7+0.4
AdaLoRA
75.620.s
54.220.6
82.621.1
60.0
75.7
91.111.2
51.620.9
62.121.0
59.51a.
31.720.2
SPU
75.4
54.021.1
83.021.
60.1
75.721.
91.3+1
51.9214
61.720.9
58.529.4
31.620.4
LoRSU
75.920.9
56.320.7
82.720.9
60.8
76.2
91.341.2
51.620.9
61.720.s
57.740.
31.220.
LN
72.9
58.220.s
78.9
56.8+o.4
6
91.4+o.
51.62o.s
62.62o.s
56.3+a.
31.3ta.2
F-FT
72.1 68.4
80.0.a.7
55.4+0.
58.8
88.4+0.3
51.840.a
62.3.o.s
56.9+a.2
31.2+a.
F-EWC
23.320.0
69.1
20.420.7
20.110.
24.210.
17.7107
51.720.7
56.920.
49.610.
31.140.
CL-20
LoRA
68.510.7
61.620.3
76.720.9
55.3107
55.610.
88.810s
51.920.3
61.420.6
59.110.
31.110.
AdaLoRA
70.3+0.b 54.4+0.4
72.4+0.s
43.6+a.8
34.6+0.7
77.0+0.3
52.2+0.9
62.6+0.4
57.0+a.1
31.9+0.a
SPU
75.5+0.7
60.9+0.s
82.3+0.4
59.2+o.s
73.7+1.0
91.2+0.7
51.7+0.s
61.8+0.9
58,2+0.
32.0+0.2
LoRSU
75.920.6
63.7
82.820.s
60.42a.
73.42a.
90.9
51.720.4
61.520.7
58.810.2
31.920.2
LN
73.0=o.2 60.1
79.6
57.7+0.4
61.320.4
89.64o.4
51.920.0
61.320.0
55.5+0.1
31.3+0.1
F-FT
72.5.0.4
70.3
78.3
53.4
50.620.2
89.1
52.320.3
61.120.2
57.1
31.720.0
F-EWC
48.0+o.a
75.5
59.5+a.4
38.8+a.1
42,6+a.
82.5+o.a
52.5+o.1
56.42o.3
55.4+a.1
31.3ta.1
CL-50
LoRA
6 71.3+0.3
76.0+0.1
56.0+a.1
44.5+0.2
88.9+0.3
51.8+0.1
60.4+0.2
56.3+a.1
31.6+a.
AdaLoRA
1 61.020.o
80.620.0
52.01o.
72.210
88.9103
51.720.2
62.02o.4
59.110.
31 31.210.1
SPU
75.420.0
65.32o.1
81.820.1
59.71a.
72.320.1
90.810.2
51.920.1
61.920.4
58.010.1
31.820.0
LoRSU
75.320.2
72.220.4
82.41o
59.71o.
72.520.
90.8103
51.720.2
61.720.4
58.510.
31.720.0
24

---

Published in Transactions on Machine Learning Research (12/2025)
Table 16: Accuracy scores (%) for LLaVA with the pretrained (Zr-Shot) or fine-tuned image encoder. All basclines use CAn datasct for fine-tuning the image encoder (the LLM remains frozen) via CLIP loss. We include error bars runs.
VQA Datasets (Acc %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE
VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LN
74.3
52.921.4
80.3
78.9107
72.4
91.1
52.020.9
61.521.
61.720a
32.120.4
F-FT
73.52m.1
50.620.9
80.320.
56.52..
63.1
91.311.
51.7a1.
61.820.
58.410.
31.320
F-EWC
65.9+1.5
39.1+
66.0+1.a
40.0+0.9
41.7+0.7
86.2+o.s
51.++1.s
59.9+1.
57.6+0.4
31.3+0.2
CL-5
LoRA
69.7+1.4 44.8+1.1
81.4+0.7
56.9+1.
50.7+1.3
92.9+1.3
52.0+1.0
61.8+1.
56.5+a.4
31.3+0.4
AdaLoRA
A
75.5 53.220.7
81.720.6
60.1107
72.0
92.110.a
51.921.4
61.821.5
59.02a.
31.920a
SPU
76.02o.9
53.220.s
82.3
60.3
75.720.9
91.3+1
51.720.s
61.5212
58.4
31.410.4
LoRSU
75.2
52.720.9
83.021.0
60.1
76.841.0
91.841.4
51.621.1
62.321.2
58.740.
31.420.4
LN
72.9
54.020.9
80.3
4
4
90.7
51.820.s
61.920.9
61.0+a.1
31.4taa
F-FT
72.9 47.9
83.0
56.9+a.9
62.7+0.9
90.6+a.b
51.940.4
61.320.4
56.5+a.2
31.5+a.
F-EWC
70.1 48.7
82.82o.
51.110s
54.810.
88.3107
51.821.0
57.020.
59.610.
31.2103
CL-20
LoRA
67.520.6
48.920.6
80.420.4
57.3109
39.7
91.110.
51.820.9
61.720.3
60.110.
31.910.
AdaLoRA
72.5+1.0 51.5+1.0
79.2+0.4
54.1+10
65.5+0.7
90.6+o.s
51.7+0.9
61.9+0.9
56.5+0.a
31.7+0.a
SPU
75.0+0.5
53.5+0.a
82.8+0.a
59.9+0.6
76.1+0.9
91.6+0.>
51.6+0.6
61.920.4
61.8+0.2
31.6+0.a
LoRSU
75.320.s
53.120.9
83.820.9
58.8
75.520.7
92.01o.3
51.920.4
62.320.6
60.42a.2
31.622.2
LN
71.1 50.4
77.020.a
57.520.
57.9
89.7
51.620.1
62.4
56.140.1
31.940.0
F-FT
7
48.920.3
81.720.0
56.220.2
47.5a0.
89.940.3
52.0ao.1
61.220.1
57.740.1
31.1+0.1
F-EWC
61.7 43.9
83.3+a.4
46.2+a.
38.9
87.5+0.1
51.840.3
55.820.3
54.7+a.1
30.7
CL-50
LoRA
66.8+0.2
47.8+0.3
82.3+0.2
55.7+a.0
52.0+o.a
91.0+o3
51.7+0.3
61.6#0.2
60.2+a.0
31.6+a.
AdaLoRA
73.520.0
49.920.1
80.920.4
55.710.
77.810.
93.1
51.520.1
61.420.
56.910.
31.620.
SPU
75.2
53.220.0
8
59.31o.
73.1
91.410.
51.720.3
61.720.1
58.510.1
31.620.
LoRSU
75.020.2
51.820.1
84.020.4
58.510
72.710.
91.9103
51.720.1
62.320.4
58.110.0
31.720.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Table 17: Accuracy scores (%) for LLaVA with the pretrained (Zr-Shot) or fine-tuned image encoder. All basclines use AIR dataset for fine-tuning the image encoder (the LLM remains frozen) via CLIP loss. We include error bars over runs.
VQA Datasets (Acc %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE
VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LN
73.4
51.321.2
80.210.6
60.721
66.9207
91.31o.
51.920.9
62.4
58.510.
30.610.2
F-FT
72.5
50.520.
79.920.9
62.41o.
60.7a1.
90.61os
51.720.9
60.921.1
58.310.
31.420.
F-EWC
74.9+1.2
52.4+0.s
71.5+1.
63.3+10
63.8+1.
90.7+1.5
51.2+0.5
61.2+0.a
58.1+0.4
31.4+0.4
CL-5
LoRA
70.9+0.9
52.7=0.o
79.0=0.7
61.7+0.5
48.8+0.7
90.6+0.a
52.0+0.9
62.5+0.8
60.0+a.a
31.1+0.2
AdaLoRA
75.021.0 53.320.s
83.720.9
60.81a.
75.2
91.741.a
51.620.s
61.620.s
56.92a.
31.920.4
SPU
76.2
53.021.3
83.020.8
63.52a.
75.3
91.521.
51.520.s
61.520.
58.1
31.520.4
LoRSU
76.2
53.4
82.5+1.
65.2+1.3
76.020.9
91.840.8
51.620.s
62.1
59.0+0.4
31.220.
LN
70.3
53.7
77.9+1.
60.2
90.6
51.7
62.8.0.7
58.1+a.1
31.8ta.
F-FT
73.0 54.1+0.s
80.3+a.9
69.7+a.b
62.7
90.0+o.4
51.940.3
61.820.4
58.9+a.1
31.4ta.
F-EWC
71.2 53.921.0
79.320.
70.621.
64.6207
89.7
51.720.4
61.520.
58.910.
31. 31.420
CL-20
LoRA
71.820.a 51.120.s
78.620.
65.7104
63.41o
89.911.
51.720.3
62.320.3
56.210.
31.520
AdaLoRA
73.4+0.s 51.6+0.0
81.2+0.9
63.1+a.6
73,8+o.s
90.8+0.5
52.1+0.4
62.7+0.8
57.7+0.2
31.2+0.1
SPU
75.7+0.4
52.2+0.7
82.0+0.a
63.4+0.9
72.6+0.6
91.7+0.a
51.8+0.6
62.2+0.5
59.0+0.2
31.4+0.2
LoRSU
75.720.9
52.620.9
81.420.7
66.32o.7
73.02o.
90.920.
51.920.s
61.820.8
56.92a.1
31.620.
LN
69.620.4
54.020.1
76.920.a
62.2.0.2
50.920.2
90.2
52.020.3
62.8
57.7.0.1
31.5+0.1
F-FT
71.2 50.3
78.3
70.4
59.940.0
90.1
51.920.1
61.820.3
57.540.1
31.3201
F-EWC
71.8 51.620.1
78.3+0.0
71.3+a2
57.6+02
90.2+o.2
51.7
61.1
57.4+a.1
31.5ta.0
CL-50
LoRA
69.8+
54.7
77.0+
68.2+a.
51.6+0.1
90.0+o.1
52.0+0.4
62.4+0.0
57.1+0.1
31.5+aa
AdaLoRA
74.2 52.020.a
82.4go.1
65.010.
72.620.
91.910.
51.720.2
60.720.1
55.610.0
31.310.0
SPU
25.2
52.220.4
82.620.
66.610.
70.02o
91.610a
51.920.2
62.02o.3
57.620.0
31.820.0
LoRSU
75.410.4
52.720.3
81.620.2
68.6103
69.72o.
91.5102
51.720.4
62.220.1
58.710.1
31.110.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Table 18: Accuracy scores (%) for LLaVA with the pretrained (Zr-Shot) or fine-tuned image encoder. All basclines use ESAT dataset for fino-tuning the image encoder (the LLM remains frozen) via CLIP loss. We include error bars runs.
VQA Datasets (Acc %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT
DALLE
VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LN
75.820.9
53.220.6
82.611
60.0
80.321.0
92.721.0
51.910.
61.7a0.
60.420.4
31.810.
F-FT
69.1
50.520.6
8
57.721.
65.810.6
91.321.
51.8107
62.01o7
58.820.3
30.410.2
F-EWC
66.3+0.9
52.1+1.4
79.3+1.
56.8+1.
67.7+1.a
90.9+0.s
51.9+1.
62.0+1.a
55.4+0.2
30.9+0.4
CL-5
LoRA
73.2+1.a
49.3=1.2
80,6+0.9
60.4+1.
74.5+0.s
92.3+1.
52.0+1.1
61.6+1.1
57.4+o.
31.4+0.3
AdaLoRA
75.9
52.4214
82.410.
60.51a.
78.0+1.
91.52a.
51.620.
61.521.3
59.0.o.4
30.9
SPU
75.8 53.2=1.4
82.821.
60.521.
80.620.9
91.5211
51.720.8
61.721.5
57.520.4
31.520.2
LoRSU
76.221.0
53.6
82.5+1.2
60.840.s
82.9+1.0
91.520.0
51.620.8
61.320.7
57.720.4
31.940.4
LN
74.5
o2.6
82.5
58.8
77.0+a.4
92.4
5L.9
62.5
58.0=o.s
31.2+0.1
F-FT
66.5
51.1=a.7
79.1+0.4
56.7+a.e
51.2+a.r
92.0+a.4
51.6+0.a
61.4+0.a
10.1
31.5+0.2
F-EWC
51.221. 69.3
60.510.
57.110.
54.1
89.7
51.910.6
60.91o7
58.420.2
31.810.2
CL-20
LoRA
71.1
50.920.s
80.311
59.410.7
64.620.7
91.1107
52.010.4
62.310.
62.320.a
31.310.
AdaLoRA
70.0=0.6
47.3=0.8
78,4+0.9
51.7+0.4
69.3+o.5
91.3+0.7
51.7+0.9
60.8+0.9
18.1+
31.6+0.1
SPU
75.6=0.0
53.1+0.a
82.8+0.9
59.9+0.s
81.5+0.6
92.3+0.4
51.940.5
61.5+0.
58.8=0.2
31.7+0.1
LoRSU
75.3
53.720.8
82.820.4
60.72a.s
82.720.7
91.620.6
51.620.4
61.520.4
58.420.2
31.410.2
LN
13.1 53.0=0.2
82,020.1
59.1
80.7
92.4+0.2
51.840.8
62.0.o.1
60.4=0.0
32.04o.b
F-FT
58.020.4
50.3
76.820.1
57.2+0.2
34.7
89.7.0.0
51.740.2
61.620.
58.120.0
31.640.
F-EWC
59.0=a.1
64.5=a.1
7
56.3+a.1
38.0+a.2
87.3+0.2
51.9+0.2
60.7
58.2=0.1
31.8+0.a
CL-50
LoRA
62.8+0.3
47.2=0.4
72.4+0.4
54.4+a.
61.6+0.4
90.2+o.a
5L.7+0.2
62.0+o.1
60.8=0.0
30.9+0.
AdaLoRA
67.220.2
49.320.a
78.810
56.910.
58.810.
89.610.
51.810.
61.910.
56.020.1
31.610.
SPU
75.1
53.420.2
82.510
60.210.
81.920.1
92.31o.
51.810.
61.620.1
57.120.1
31.910.
LoRSU
75.420.3
53.920.1
83.110
60.31a.1
83.110.1
92.110.
51.620.2
61.2
57.620.a
31.110.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Table Average accuracy (ACC) and backward transfer (BWT) scores (%) for LLaVA with the fine-tuned CLIP-I-14. Each column indicates the sctting and fino-tuning method. We inchude crror bars over runs.
FT Method
Zr-Shot
LoRA
SPU
LoRSU
Setting FTD
ACC (t) BWT (t) ACC (t)
BWT (t)
ACC (1)
BWT (t)
ACC (t) BWT (t)
GTS
75.4
0.0
79.2
7.120.s
80.820.5
0.520.s
81.120.6
0.420.7
TSI
54.0
0.0
55.5.0.9
22.5
55.520.6
0.2
57.0=0.s
0.5
CL-5
AIR
60.4
0.0
59.2-0.s
20.7
64.7
2.8+o.a
65.0
2.5+0.6
ESAT
76.4
0.0
73.8-0.9
-3.4+..0
79.8+0.6
1.5+0.7
82.2+0.7
2.0+0.6
GTS
75.4
0.0
77.5 77.2
-01
82.8=0.4
0.6
83.520.6
-.4+0.s
TSI
54.0
0.0
60.6-o.3
- 7.2+0.4
60.1
1.7
62.1
0.9
CL-20
AIR.
60.4
0.0
64.320.4
3.620.6
65.220.7
1. 1.1
65.420.3
0.920.4
ESAT
76.4
0.0
64.1=0.5
18.3+0.7
82.0=0.4
2.0+0.2
82.7
0.1+0.3
GTS
75.4
0.0
79.3-o.3
10.3
83.8=0.2
0.7
84.7
0.5
TSI
54.0
0.0
67. 67.02o.3
20.1
61.820.2
-1.920.3
67.920.2
-1.1203
CL-50
AIR
60.4
0.0
65.6-0.4
-0.1+0.3
67.1
0.5+0.2
67.7
0.7+0.3
ESAT
76.4
0.0
61.4
-2..8tu.4
81.220.3
-2.4
82.1
-0.22..2
D.2
CLiP-based Updates+
The detailed accuracies for all baselines and datasets used create Table 3 of the main paper can be found in Tables 14 through 18
D.3 Extra ACC and BWT results
In Table 19 we present results of the ACC and BWT on extra datasets plus the ones in the main paper. The results follow the same patterns in section with LoRSU demonstrating the most consistent performance in both ACC and BWT compared to the other two baselines. SPU is close to LoRSU in terms of BWT but it significantly lacks behind in ACC.
D.4 CLiP-based Perplexity-based Updates+
The detailed accuracies for all baselines and datasets used to create Table 4 of the main paper can be found We have also included results fine-tuning the model using MMVP dataset in
in Tables 20 through 24
Table 27
E
Detailed Ablation Studies
E.1
Ablation on the rank r of LoRSU
In Table 29 we investigate the effect on performance of using different ranks for LoRSU. As the rank r increases, the VQA accuracy on the target dataset slightly improves, peaking at r = 64. Beyond that, performance slightly decreases. Performance on other datasets remains relatively stable with small fluctuations.
E.2 Ablation on the number of optimal attention heads LoRSU
In Table we examine how the number of attention heads chosen to be fine-tuned affects LoRSU's performance. We notice that more attention heads marginally improve the performance of the model while more forgetting, e.g. ESAT.
the extra flexibility
28

---

Published in Transactions on Machine Learning Research (12/2025)
Table 20: Exact accuracy scores (%) for each baseline used to fine-tune the model on the GTS dataset under
three different continual learning (5, 10, 50 shots) settings. We include error bars over 3 runs.
VQA Datasets (Ace %)
Setting
Method
GTS
TS1
CAn
AIR
ESAT DALLE VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LoRA-L
71.5+1.
52.3=0.s
81.2+0.6
60.0+1.a
75.5+0.9
91.5+1.
51.9+1.8
61.2+1.1
57.6-=0.
32.2+0.6
LoRA
76.31os
52.621.4
7
56.721.2
4
87.1
51.821.2
61.311.
58.120.3
31.610.1
LoRSU
82.0
53.521.
82.42..
60.8
66.9
91.511.
51.622.7
61.721.4
59.820.2
31.610.2
CL-5
LoRA-Ppl
68.1
54.5=1.4
8
59.3#1.2
52.8
90.741.a
51.7=112
60.7
54.8=0.4
33.4
LoRA--
72.92o.0
54.0=0.7
8
59.62o.s
61.920.
90.3
51.920.8
60.9
58.4
31.1
LoRSU-Ppl
77.2+1.4
55.1+1.s
82.1+a.7
58.9+1.0
67.0+a.e
90.9+1.3
51.8+0.6
61.6+1.9
58.7+
30.4+o.a
LoRA-L
74.210.
52.220.9
82.1
59.621.0
75.910
91.841.0
51.620.4
62.1
59.1
31.810.2
LoRA
78.1 18.1
55.620.3
59.010.
47.620.4
26.010
83.610.
52. 62.1
53.720.3
30.810.2
LoRSU
84.2+0.9
52.9+0.6
82.2+o.6
60.7+0.0
64.7+0.6
90.8+
51.9+0.4
61.7+0.5
59.5-0.1
31.6+0.2
CL-20
LoRA-Ppl
75.1+0.9
50.4
7
56.5=0.s
40.1
89.7+0.s
51.6+0.7
57.8+0.s
54.2=0.2
31,5+0.4
LoRA-F
74.2
52.720.3
80.1
59.520.4
66.020.6
90.1
52.1
61.721.0
60.420.4
32.31o
LoRSU-Ppl
79.5.o.
56.120.s
82.1
59.820.4
66. 66.120.4
90.8
51.720.s
62.1
59.0go.s
31.5t0
LoRA-L
74.9
=1.7
81.840.2
59.8.o.3
75.8
91.54o.
52.0.0.1
61.1
57.4
31.8+0.1
LoRA
78.7+ 50.7=0.0
62.1
47.4
24.2t0.2
82.9
51.720.3
61.0
54.3=0.1
30.8
LoRSU
85.3+o.1
54.2=0.1
81.9+0.
60.5+0.2
61.4+0.
91.0+o.1
51.7+0.2
62.2
58.9=0.1
31.8+a.1
CL-50
LoRA-Ppl
74.210.1
49.420.2
76.0203
57.920.3
37.210.0
89.510.2
51.720.
57.710.1
55.620.1
29.810.1
LoRA-F
71.710.
51.720.4
80.810
58.320.0
60.910.3
90.810.1
52.1
63.31o.1
57.520.0
30.940.1
LoRSU-Ppl
82.5+o.a
55.8=0.0
82.1+0.2
59.9+0.1 65.4+0.2
91.0+o.3
51.6+0.a
61.7+0.2
62.3=0.1
32.2+0.0
Table 21: Exact accuracy scores (%) for each baseline used to fine-tune the model on the TSI dataset under three different continual learning (5, 10, 50 shots) settings. We include crror bars over 3 runs.
VQA Datasets (Acc %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE
VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LoRA-L
76.041.5
59.120.6
82.740.
60.7
75.920.9
91.541.
51.520.0
63.6
54.120.4
31.240.4
LoRA
73.4+1.
53.020.9
80.2
58.8
59.1+1.
90.2
51.6
61.2
56.7=0.4
31.7+0.4
LoRSU
75.9+0.a
56.3=0.7
82.7+a.9
60.8+1.0
76.2+14
91.3+1.
51.6+0.9
61.7+0.s
57.7=0.s
31.2+o.3
CL-5
LoRA-Ppl
75.021.0
64.020.6
82.811.
58.421.0
60.810
88.711.
51.621.
61.511.
55.020.4
32.210.4
LoRA-F
75.310.
45.121
82.51o
57.221.5
73.211.0
83.941.a
53.820.9
64.321.3
45.620.3
30.940.4
LoRSU-Ppl
76.1+11
66.2+1.0
83.9+11
66.1+0.o
76.1+12
91.1+1.4
52.0=0.o
64.4+1.4
60.8
31.1+0.4
LoRA-L
76.1 59.02o.6
82.41o.
60.8
75.710.
91.31o.7
51.520.
63.941.
55.420.3
30.810.
LoRA
68.52a.7
61.620.3
76.720.
55.3
55.620.
88.81o.
51.920.
61.42os
59.1
31.120.
LoRSU
75.940.6
63.720.4
82.84o.
60.4
73.4+0.6
90.940.6
51.720.
61.54o.7
58.820.2
31.9402
CL-20
LoRA-Ppl
62.1
59.620.s
71.920.
48.320.7
42.5+1.
75.810.s
51.620.6
49.02o.5
49.7
32.410.
LoRA-F
76.1
56.0=o.s
82,8+a.
58.2
67.7+0.9
87.5
51.6+0.a
64.4+o.s
40.3=0.4
31.2+a.
LoRSU-Ppl
76.4+o.r
67.0=0.4
83.0+a.r
57.4+0.4
74.0+o.a
88.
51.8+0.6
63.6+o.5
57.6=a.2
30.8+a.a
LoRA-L
76.4+0.2 63.0+0.2
81.9+0.2
60.5+0.2
75,6 75,6+0.2
91.1+o.2
51.7+0.2
64.1+o.a
55.6+0.2
30.9+a.0
LoRA
66.1+ 71.3+0.
76.0+o.1
56.0+o.1
44.5+0.2
88,9+o.3
51.8+0.1 60.4+0.2
56.3+
31.6+0.1
LoRSU
75.3+o
72.2+0.4
82.4+0.a
59.7+
72.5+0.a
90.8+o.3
51.7+0.2
61.7+0.4
58.5+0.1
31.7+0.0
CL-50
LoRA-Ppl
46.32oa
51.520.3
63.42a1
40.1
41.310.
73.920.2
51.7200
49.5go.3
40.220.1
32.740.1
LoRA-F
74.0202
68.220.1
81.610a
59.22o.o
75.1102
88.5402
56.820.1
65.02o.3
50.820.1
30.41a1
LoRSU-Ppl
75.840.2
75.120.a
82.140a
56.020.4
74.210.4
86.040.4
52.02o.o
63.2 63.2+o.0
58.120.1
30.240.1
29

---

Published in Transactions on Machine Learning Research (12/2025)
Table 22: Exact accuracy scores (%) for each baseline used to fine-tune the model on the CAn dataset under three different continual learning (5, 10, 50 shots) settings. We include error bars over 3 runs.
VQA Datasets (Ace %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LoRA-L
75.5+1.4
53.1=0.s
79.4+14
59.2+0.6
75.2+0.9
91.++.1
52.4+1.8
60.2+1.1
57.7=0.8
32.1+0.3
LoRA
69.711.4
14.821.1
81.4107
56.921.
50.721.
92.941.3
52.021.
61.811.
56.520.4
31.310.
LoRSU
75.220.s
52.720.9
83.021.
60.1
76.811.
91.8114
51.621.1
62.3
58.720.3
31.41o.
CL-5
LoRA-Ppl
65.8+1.1
50.7=0.6
79.2.0.5
48.4
63.0+1.2
86.741.a
51.8210
57.2+1.4
52.5=0.a
32.4+0.4
LoRA--
7
52.220.7
78.620.
50.9
7
91.341.0
54.720..
62.2
58.020.5
31.320.
LoRSU-Ppl
74.6+0.9
51.3+1.4
82.9+12
58.4+1.2
77.7+1.
91.8+1.s
51.5+1.1
64.7+1.4
56.5+a.6
29.8+a.a
LoRA-L
73.621.0
52.220.9
80.82o.
56.7.0.4
74.710.8
91.710.5
52.220.6
60.9
59.1
31.910.4
LoRA
67.510.
48.920.6
80.410
57.320.9
39.710.4
91.1106
51.820.9
61.720.
60.1
31.910.
LoRSU
75.3+0.a
53.1+0.9
83.8+0.9
58.8+1.0
75.5+0.7
92.0+o.3
51.9+0.4
62.3+o.6
60.4-0.2
31.6+02
CL-20
LoRA-Ppl
65.6+0.5
47.0=0.7
79.0+0.4
46.0=0.6
58.9+0.8
82.5+0.s
51.9=0.7
43.9+1.
52.5=0.4
30.4+0.3
LoRA-F
69.4
54.920.4
80.62a.
50.4
72.020.
91.220.5
51.920.
61.321.
57.0=0.3
31.610.
LoRSU-Ppl
72.4
19.220.4
83.220.7
56.420.9
75.510.8
91.810.
51.620.5
61.02o.
57.7
31.62a.
LoRA-L
1
51.620.2
80.9
56.940.1
74.920.2
91.340.3
51.7=0.2
61.24o.a
58.0=0.1
32.4+0.1
LoRA
66.8
47.8
82.3+o.2
55.7=
52.0+e.a
91.0+o.3
51.720.3
61.6to.2
60.2=0.0
31.6ta.1
LoRSU
75.0+o.
51.8=0.1
84.0+a4
58.5+0.2
72.7+0.a
91.9+0.3
51.7+0.1
62.3+o.4
58.1+
31.7+0.1
CL-50
LoRA-Ppl
56.21o4
36.420.0
80.9101
48.520.
54.110.3
78.1
53.620.4
62.31o.3
48.420.1
32.410.1
LoRA-F
69.21o
52.020.2
80.6101
53.720.a
74.410.
90.710.2
51.820.4
66.520.0
58.720.
31.410.
LoRSU-Ppl
74.9+0.4 49.7=0.4
83.7+0.0
42.5=0.4
74.9+0.2
91.2+o.3
51.2+0.a
52.2+0.4
58.5=0.a
32.3+0.2
Table 23: Exact accuracy scores (%) for each baseline used to fine-tune the model on the AIR dataset under three different continual learning (5, 10, 50 shots) settings. We include error bars over 3 runs.
VQA Datasets (Acc %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LoRA-L
75.640.7
54.420.s
81.8411
58.720.9
75.7
92.0+1.4
51.620.0
61.020.6
59.1
32.240.5
LoRA
70.9
52.7=0.s
79.0+a.7
61.7
48.8
90.6+o.a
52.02a.9
62.5
60.0
31.1+0.2
LoRSU
76.2+o.a
53.4=1.4
82.5+1.0
65.2+1.s
76.0+0.9
91.8+o.a
51.6+0.s
62.1
59.0=a.4
31.2+o.
CL-5
LoRA-Ppl
74.910.
54.221.2
79.1
59.720.9
68.510.
90.811.3
51.820.7
62.01o.7
55.120.s
31.110.
LoRA-F
72.31o
50.621.
78.741.
70.021.3
64.420.
90.910.6
54.921
57.721.
62.02o.6
32.210.8
LoRSU-Ppl
75.6+1.
54.6+1.2
79.8+1.
66.2+0.6
76.4+1.
90.6+1.3
51.7+1.a
60.1+o.p
58.8=0.4
31.1+0.4
LoRA-L
75.410. 53.620.4
82.2
61.121.
75.710.
92.210.3
51.520.3
61.52o.
58.920.2
31.910.
LoRA
71.820. 51.120.s
78.620.
65.720.4
63.4208
89.94
51.720.
62.32o.
56.2
31.520.2
LoRSU
75.740.
52.620.0
81.4
66.3
73.02o.8
90.946.s
51.920.8
61,840.s
56.920.1
31.640.
CL-20
LoRA-Ppl
72.1
48.020.s
72.740.4
65.221.
65.1
90.410.a
51.820.6
61.52o.s
55.8
31.740.
LoRA-F
74.5+o.a
53.0=o.s
82,0+a.
76.7
74.9+0.9
91.1+0.3
52.4+0.
59.3+o.s
59.6=a.4
31.3+a.
LoRSU-Ppl
7
55.5+0.s
78.7+0.a
66.4+o.a
75.7+
91.6+1.
51.5+0.a
59.8+0.5
58.1+0.
31.2+a.4
LoRA-L
75.6+0.2
53.8+0.1
83.5+0.1
65.0+o.0
75.7+0.1
92.0+o.a
51.8+0.2
61.1+o.1
58.7+
32.3+0.0
LoRA
69.8+o.a
54.7+0.0
77.0+0.a
68.2+o.s
51.6+0.1
90.0+o.1
52.0+0.4
62.4+o.a
57.1+0.1
31.5+a.1
LoRSU
75.4+0.4
52.7+0.s
81.6+0.2
68.6+0.3
69.7+
91.5+02
51.7+0.4
62.2+o.1
58.7+0.1
31.1+0.
CL-50
LoRA-Ppl
74.40a1
50.920.4
76.8102
66.620.s
65.420
91.310.
51.620.
57.2
53.720.1
31.520.
LoRA-F
74.620.
53.220.2
78.320.
71.4102
91.4iaa
52.920.4
60.0
57.420.
31.1102
LoRSU-Ppl
75.1+0.2
54.5.0..
78.020.
69.32o.1
75.740.1
91.5+0.1
51.720.
61.5.0.1
58.220.0
30.810.0
30

---

Published in Transactions on Machine Learning Research (12/2025)
Table 24: Exact accuracy scores (%) for each baseline used to fine-tune the model on the ESAT dataset under three different continual learning (5, 10, 50 shots) settings. We include error bars over runs.
VQA Datasets (Ace %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LoRA-L
7
52.2-1.4
82.8+o.6
60.6+1.5
75.9+1.
91.7+0.9
51.5+0.6
60.2+o.s
57.6==.8
31.6+0.4
LoRA
73.2
19.321.2
80.61a
60.421.1
74.520.
92.311.
52.021.1
61.6111
57.420.4
31,410
LoRSU
76.2:1.0
53.621.1
82.5
60.82o.s
82.911.
91.52o.
51. 51.620.9
6l.32o.
57.7=0.4
31.91a.1
CL-5
LoRA-Ppl
76.0
52.6#1.0
82.6*1.
60.4
75.520.9
91.941.0
51.820.9
62.8+o.s
59.0=0.4
31.640.
LoRA--
74.3
51.521.4
81.1
60.3
81.521.
90.8
51.921.2
61.921.
57.7
31.320.
LoRSU-Ppl
75.6+1.4
52.3+0.6
82.0+12
60.5+1.0
79.8+1.1
92.3+o.5
51.8+12
62.2+1.4
57.7+0.4
31.3+a.4
LoRA-L
7 52.4
8
60.821.0
76.810.3
91.310.5
51.720.
60.4
61.520.3
31.610.
LoRA
7 50.920.s
80.311.
29.4
64.610.
91.110.7
52.02o.4
62.320.6
62.32n.2
31.310.1
LoRSU
75.3+1.0
53.7+0.s
82.8+o.4
60.7+0.s
82.7+0.7
91.6+0.a
51.6+0.4
61.5+0.4
58.4-0.2
31.4+02
CL-20
LoRA-Ppl
75.5+0.0
51.6=0.7
82,0+o.4
59.3=0.6
74.9+0.a
91.6+0.5
51.7+0.6
62.8+0.5
57.0=0.1
32.1+0.1
LoRA-F
7
52.721.
81.61aa
59.4...
71.520.
91.02o.
51.720.7
63.4
58.920.2
31.010.2
LoRSU-Ppl
74.1
52.020.9
82.51o.
59.82o.s
79.020.7
51.820.9
61.820.4
58.720.4
31.62a.
LoRA-L
75.610.2
53.0
82.7+0.a
60.6.0.3
77.1
91.540.2
51.7=0.1
60.740.0
59.8=0.1
31.4+0
LoRA
62.8+o.3
47.2=0.4
72.4+a.4
54.4
61.62o.4
90.2+o.3
51.720.2
62.0
60.8=a.0
30.9
LoRSU
75.4+0.3
53.9
83.1+0.2
60.3+o.1
83.1+0.1
92.1+o.1
51.6+0.2
61.2+o.a
57.6=0.0
31.1+a.
CL-50
LoRA-Ppl
74.910.
51.720.3
81.910
59.8
77.810
92.1
51.820.2
62.910.3
59.420.2
31.910.1
LoRA-F
7
51.820.3
81.210.
58.1
66.610.
90.710.
51.620.1
63.7
58.42o.
30.51..
LoRSU-Ppl
72.9+0.1 51.1=0.s
81.3+0.4
59.4+0.4 75.4+0.2
91.6+0.2
51.7+0.1
62.7+0.4
57.5=a.1
32.1+0.0
Table 25: Exact accuracy scores (%) for each baseline used to fine-tune the model on the VSR dataset under three different continual learning (5, 10, 50 shots) settings. We include crror bars over 3 runs.
VQA Datasets (Acc %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LoRA-L
7
59.921.4
81.0411
56.220.5
66.8
90.1+1.a
68.3a1.1
65.0a1.4
57.620.3
32.520.4
LoRA
72.6+1.
49.5=1.a
78.2+a.s
57.5.1.s
55.0ta.9
88.8
52.0=1.0
61.9t1.
59.7
30.4+a.a
LoRSU
75.6+0.7
52.2=1.4
82,2+a.e
10.1
77.9+0.8
91.1+1.1
51.9+1.s
62.2
58.4=0.s
31.7+0.
CL-5
LoRA-Ppl
65.81o7
48.720.s
65.411.
33.811.4
48.810.
81.711.2
61.720.
56.210.7
43.620.2
32.810.4
LoRA-F
76.01o.
64.52o.s
81.211.3
57.620.6
69.7
89.4
69.521.
12.8
30.3
13.010.a
LoRSU-Ppl
73.6+0.7
57.5+1.1
80.3+11
57.8+1.
73.1+1a
90.7+1.1
62.0+1.s
57.4+0.6
57.9=0.6
30.3+0.4
LoRA-L
77.110.
54.720.9
84.510s
61.420.s
75.510.7
90.910.a
73.720.3
64.52o.
56.920.2
32.610-1
LoRA
72.620.7
54.520.9
76.620s
57.4
57.320.
87.91o.
51.920.7
59.02o.5
57.620.2
31.320.
LoRSU
74.940.6
54.620.5
82.1
58.520.7
75.520.
91.640.5
51.620.6
62.4
57.520.2
30.940
CL-20
LoRA-Ppl
74.920.4
62.220.4
82.41oa
58.2.0.7
7
89.0
71.020.8
64.8
55.820.a
28.640.2
LoRA-F
75.4+o.5
60.6
80.9+a.
56.6+0.9
63.1
88.2
74.8+0.s
+8.7
50.1=0.4
20.2+0.2
LoRSU-Ppl
72.6+0.a
52.7=0.s
81.6+0.a
60.3+o.s
69.4+
89.6+0.5
74.4+0.9
62.5+o.s
57. 57.1+
29.7+a.4
LoRA-L
77.2+o. 56.5+0.1
84.5+a.0
61.4+0.2
76.4+0.2
91.5+o.3
73.4+o.1
65.3+o.2
54.4+0.1
31.5+0.1
LoRA
73.4+o.1
53.8+0.0
74.6+0.4
56.7+o.1
56.2+o.1
87.0+o.
51.9+0.0
59.2+o.2
57.6+0.1
30.8+a.0
LoRSU
75.3+o.1
54.7+0.1
81.6+0.
58.3+0.2
75.7+0.1
91.4+0.4
53.8+0.2
62.1
57.3+
30.8+a.0
CL-50
LoRA-Ppl
71.740.
48.720.1
75.1102
16.320.4
64.620.
87.910.2
71.720.4
61.920.2
55.120.1
30.920.0
LoRA-F
76.310.
64.220.2
84.5104
58.1
6
90.110.
72.520.3
64.640.1
61.42..
30.620.
LoRSU-Ppl
72.1+0.2
49.8
74.840.
57.620.o
71.020.4
88.240.1
74.9
58.3+0.2
55.42n.a
30.040.2
31

---

Published in Transactions on Machine Learning Research (12/2025)
Table 26: Exact accuracy scores (%) for cach bascline used finc-tune the model on the HM datasct under three different continual learning (5, 10, 50 shots) settings. We include error bars 3 runs.
VQA Datasets (Ace %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LoRA-L
76.521.
51.521.
8
60.520.8
75.7
90.910.a
51.620.9
68.610.7
34.420.5
31.110.
LoRA
68.81o.
47.021.
7
51.721.
54.120.6
89.1
52.2
60.8
54.720.6
30.510.
LoRSU
75.7+12
54.1+1.1
82.9+a.6
60.7+1.0
76.3+11
92.2+o.a
51.5+0.9
61.8+12
58.1+0.2
31.9+a.6
CL-5
LoRA-Ppl
76.2+o.a
48.4+1.
82,5+12
57.2+0.9
72.8+0.9
90.9+o.9
51.8+10
60.0+1.0
56.4+0.4
33.1+a.4
LoRA-F
71.8111
17.820.a
79.9
57.621.0
63.2111
90.111.0
48.020.7
67.210.a
49.02o.3
31.510.
LoRSU-Ppl
76.621.
51.721.3
83.621.4
60.3
75.220.8
90.8
51.721.
60.4
60.7
31.220.2
LoRA-L
75.1
50.5
82.1
59.3
65.1
91.820.
51.920.s
71.82..
52.8
31.720.
LoRA
68.1 46.82o.
7
26.4.o.s
49.62
87.320.s
51.720.4
49.4
59.7
31.4taa
LoRSU
76.1
53.0
82.7
60.4
75.7
92.1
51.820.s
61.9
58.420.2
31.5+a
CL-20
LoRA-Ppl
77.01o.
52.02o.4
8
63.62o.7
73.420.
90.51o.a
53. 13.1
71.920.7
54.120.2
31.1
LoRA-F
75.610.4
50.920.s
80.610s
60.820.6
71.210.7
90.910.7
52.220.7
72.910.7
53.620.a
31.610.
LoRSU-Ppl
76.1+o.a
49.8+0.9
83.5+o.9
59.8+o.o
76.1+a.9
91.0+o.9
51.7+0.6 72.1+0.4
59.5-a.2
30.5+a.4
LoRA-L
75.810.2
49.520.3
83.410.
59.920.3
71.110.3
89.910.3
51.720.1
71.420.2
48.720.1
31.110.0
LoRA
72.7+0.a
47.1=0.2
72.6+0.2
56.7=0.s
60.4+0.1
89.740.3
51.9+0.1
61.9+0.1
57.1=0.2
31.1+0.0
LoRSU
75.3+0.a
53.2=0.1
8
60.5=0.1
74.9+0.1
92.2+0.1
51.620.2
61.5+0.0
58.9=0.1
31.3+0.0
CL-50
LoRA-Ppl
76.640.2
49.3=0.4
81.940.
60.3
72.7+0.2
89.846.3
52.520.2
73.7
52.7=0.0
30.9+0.1
LoRA-F
74.1
52.020.s
80.620.2
57.0z0.1
63.520.3
88.7
53.020.4
73.5
46.02o.1
31.8ta.0
LoRSU-Ppl
76.0+o.1
50.4
83.4to.1
60.620.4
76.4+0.1
91.4to.2
51.920.1
73.4
59.8
32.0+a.1
Table 27: Exact accuracy scores (%) for each baseline used to fine-tune the model on the MMVP dataset. under three different continual learning (5, 10, 50 shots) settings. We include error bars over runs.
VQA Datasets (Acc %)
Sctting
Method
GTS
TSI
CAn
AIR
ESAT
DALLE
VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
LoRA-L
75.5
52.8
82.0
60.5
76.0
91.5
51.5
63.6
57.7
30.6
LoRA-Ppl
75.5
53.6
83.0
60.3
75.6
91.1
51.5
63.1
60.7
31.7
CL
LoRA-F
75.2
52.9
81.3
60.5
74.3
90.4
51.6
63.6
60.0
31.4
LoRSU-Ppl
75.1
52.0
81.2
57.4
75.2
90.2
51.7
63.9
60.3
30.8
32

---

Published in Transactions on Machine Learning Research (12/2025)
Table 28: Exact accuracy scores (%) for each baseline used to fine-tune the model the VisOnly dataset under three different continual learning (5, 10, 50 shots) settings. We include error bars over 3 runs.
VQA Datasets (Acc %)
Setting
Method
GTS
TSI
CAn
AIR
ESAT DALLE VSR
HM
MMVP
VisOnly
Zr-Shot
75.6
53.1
82.7
60.4
91.1
51.5
61.2
58.0
31.3
LoRA-L
76.5+12
51.920.7
82.4+14
60.5.1.a
76.1+1.
91.5+o.a
51.6+0.9
60.3+1.0
57.6=a.3
31.3+a.4
LoRA
70.9+1.4
52.1+1.
77.5+1a
55.6+0.6
52.6+0.8
89.3+o.a
51.7+0.s
61.7+0.r
56.9+a.6
30.9+a.6
LoRSU
75.9+o.r
53.1+0.s
82.5+o.r
60.4+1.0
76.1+16
91.9+o.a
51.5+1.a
61.3+1.2
58.9-0.4
31.5+a.2
CL-5
LoRA-Ppl
7
50.721.1
82.210.
61.021.
73.410.
91.7a1.3
52.121a
59.311.
58.0go.2
35.010.
LoRA-F
76.02a.
51.1214
82.911.1
59.920.7
71.211.2
91.711.
51.621.
60.8
58.420.2
34.910.1
LoRSU-Ppl
76.2+1.1
53.0=0.9
8
61.3#1.4
76.6*0.8
92.3+0.5
52.0=1.0
61.6+0.7
60.7=0.s
32.0+0.6
LoRA-L
77.8 53.02o.
62.1
7
91.620.4
52.420.
61.22o.
55.620.3
32.51.
LoRA
73.3 49.320.4
77.9.0.
56.420.6
17.7
91.220s
51.820.8
61.52oa
57.020.3
32.8ta.1
LoRSU
75.740.5
53.320.7
82.02o.:
60.0
76.1
91.940.9
51.720.6
61.620.3
58.220.3
31.520.4
CL-20
LoRA-Ppl
78.010.4
52.820.4
8
60.9
74.310
91.510.7
51.920.5
61.710.7
56.020.a
32.810.
LoRA-F
77.4+o.a
51.7+0.9
83.7+a.8
59.7+
73.9+0.9
91.2+o.5
53.4+0.4
62.0+o.9
56.9-a.4
31.0+a.a
LoRSU-Ppl
76.7+0.5
53.7+0.4
83.8+a.8
61.4+0.3
75.5+0.6
91.2+o.a
51.8+0..
61.9+o.9
59.6+0.4
31.3+a.2
LoRA-L
76.4+0.4 54.5+0.3
84.1+0.3
61.3+0.0
73.9+0.1
91.++o.1
51,9+0.3
62.8+o.1
55.4+0.0
32.1+0.1
LoRA
70.0+o.1 46.8=0.0
70,5+
51.0=0.2
50.9+0.0
88.
52.0=o.a
61.2+0.3
57.8=0.2
31.7+0.1
LoRSU
75.6+0.4
53.1=0.1
81.7+0.3
58.2+0.1
75.3+0.2
91.8+0.3
51.7+0.1
62.1
58.3=0.1
31.9+0.0
CL-50
LoRA-Ppl
76.92a.
54.620.1
84.1203
60.520.2
74.910.4
91.2203
51.820.
62.52o.3
56.020.1
33.02a.0
LoRA-F
77.110.
53.020.2
83.92a.4
60.920.1
73.1
92.2
51.920.2
61.4
58.02o.0
32.51a.1
LoRSU-Ppl
76.110.
51.520.2
81.620.1
60.2
75.620
92.210.3
52.020.2
61.21o.3
58.320.0
33.540.1
Table 29: Ablation study over the effect of the rank r used by LoRSU to fine-tune the image encoder, CLIP-I-14. We report the VQA accuracies of the last session in the 50-shot CL setting. The accuracies on the target datasct are in red color. For this experiment, we use two attention heads to finc-tune with LoRSU.
VQA Datasets (Acc %)
FT Dataset rank (r)
GTS
TSI
CAn
AIR
ESAT DALLE
VSR
HM
MMVP
VisOnly
8
83.0
53.2
81.3
60.9
61.0
91.2
51.5
61.6
60.0
31.6
16
83.9
53.4
81.5
60.2
54.0
91.4
51.5
62.1
60.7
31.6
32
84.8
53.1
81.9
60.5
58.0
90.6
51.6
61.8
58.7
31.5
GTS
64
84.9
53.2
81.3
60.7
61.7
90.9
51.5
61.9
59.3
31.3
128
84.3
53.2
81.8
60.6
56.8
91.5
51.6
61.8
58.7
31.2
256
84.5
53.1
81.5
61.1
51.5
90.3
51.6
62.0
58.7
31.6
8
75.2
67.2
82.0
59.2
71.6
91.1
51.5
61.6
58.0
31.5
16
75.4
68.0
82.3
59.1
71.0
90.6
51.6
61.6
56.7
31.2
32
74.9
68.9
81.8
59.3
70.1
91.2
51.5
61.6
58.0
31.6
TSI
64
75.3
72.1
82.0
59.3
72.3
90.5
51.6
61.4
58.0
31.6
128
75.1
65.8
81.7
59.0
70.0
90.6
51.5
62.1
56.7
31.6
256
75.4
66.4
82.3
59.6
72.0
91.2
51.5
62.1
56.7
31.5
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
33

---

Published in Transactions on Machine Learning Research (12/2025)
Table 30: Ablation study over the effect of the number of attention heads used by LoRSU fine-tune the image encoder. We report the VQA accuracies of the last session in the 50-shot CL setting. The accuracies the larget dataset are in red color. For this experiment, we use r = 64 for the rank of LoRSU.
VQA Datasets (Ace %)
FT Dataset
# heads
GTS
TSI
CAn
AIR
ESAT
DALLE
VSR
HM
MMVP
VisOnly
0
83.1
52.7
82.2
60.8
60.6
91.1
51.6
61.7
59.3
31.6
1
83.9
53.8
82.0
60.7
55.4
91.2
51.6
61.6
60.0
31.8
2
84.9
53.2
81.3
60.7
61.7
90.9
51.5
61.9
59.3
31.3
GTS
4
84.7
53.5
81.0
60.5
60.5
90.6
51.5
61.8
58.7
31.5
8
84.9
52.9
81.2
60.5
58.8
90.5
51.5
61.6
59.3
31.5
16
85.0
53.1
81.3
60.0
59.2
90.6
51.5
61.6
56.7
31.3
0
75.1
64.2
82.1
59.3
72.2
90.8
51.5
61.8
57.3
31.5
1
75.3
64.8
81.9
59.5
74.0
90.5
51.5
61.6
58.0
32.0
2
75.3
72.1
82.0
59.3
72.3
90.5
51.6
61.4
58.0
31.6
TSI
4
74.9
66.8
82.2
58.9
74.0
90.5
51.5
62.1
58.0
31.4
74.7
67.4
81.7
59.1
71.5
91.2
51.5
62.2
58.0
31.7
16
75.3
65.2
81.8
59.9
69.1
90.5
51.5
61.6
58.0
31.3
Zr-Shot
75.6
53.1
82.7
60.4
76.1
91.1
51.5
61.2
58.0
31.3
Table 31: Robustness comparison of LoRSU with respect to the number of training epochs. We consider LoRSU, LoRSU-Rand where the k attention heads are chosen randomly and LoRSU-AAH where all the attention heads are chosen for fine tuning. We use 50 shots on the GTS for cach method and we report the Target Improvement (TT) on this dataset and the Control Change (CC) using only ESAT as a control dataset. We include error bars over runs.
LoRSU-Rand
LoRSU-AAH
LoRSU
# Epochs
TI (t)
CC (t)
TI (t)
CC (t)
TI (t)
CC (t)
5.210s
-11.121
6.120.
-11.6207
5.620.
-9.7208
7.640.8
-15.020.0
9.320.4
-15.620.6
8.620.3
-12.6t0G
10
7.8+as
-18.1=0.
9.1+0.1
-19.6+a.5
9.7+0.1
-14.3+a.
20
5.9+a.6
-20.0=0.7
8.1+0.1
-21.5+0.0
7,4+0.
-15.7+0.6
Table 32: Comparison of the importance of choosing small subset of attention heads. The GTS dataset is used for fine- fine-buning. We include error bars over 3 runs. The highest accuracies across methods are underlined.
Setting Scores LoRSU-Rand LoRSU-AAH LoRSU
TI (t)
4.1 40.4
5.9 20.5
6.4 +1.3
CL-5
CC (t)
-1.0 +0.6
-0.9 +0.3
-0.7 +0.6
TI (t)
6.2 40.6
7.5 #0.6
8.6 40.9
CL-20
CC (t)
-1.4 +0.3
-0.7 40.4
-1.0 #0.6
TI (t)
7.8
9.1
9.7
CL-50
CC (t)
-1.7 10.2
-0.9
-1.3 20.1
34

---

Published in Transactions on Machine Learning Research (12/2025)
(a) TSI
(b) DALLE
Figure 5: Instances of the Use Laptop' action.
E.3 Robustness the Choice of Attention Heads
We show in Table 31 that LoRSU's mechanism of choosing the most important attention heads provides clear advantage in terms of robustness over the other two LoRSU's variants, LoRSU-Rand and LoRSU-AAH. We can see that TI and CC decline in lower rate compared to that of LoRSU-RAnd and LoRSU-AAH, as we increase the number of training epochs.. As expected, LoRSU-Rand appears to be the least robust. method since the random choice of the attention heads constitute more unstable.
F TSI vs. DALLE
In Figures through g we present examples images from TSI and DALLE for different actions. In general, we observe that TSI comprised of natural, unposed images of semior individuals performing daily tasks, reflecting real-life scenarios. The images are broader, showing the surrounding environment, which crucial for context. On the other hand, DALLE images idealized stylized images. The focus narrower, with emphasis the object of the action (e.g. tablet, glass, etc.).
G Limitations
LoRSU is highly efficient, but it comes with few key caveats. First, date LoRSU has only been evaluated CLIP-baseds encoders within LLaVA; extending it to other VLM architectures and image encoders remains future work, as does integrating smaller LLM proxies to reduce compute further. Finally, because it relies on binary masks to isolate updates, scaling LoRSU's masking strategy to much larger parameter spaces (e.g., full LLMs) poses challenges, and more scalable masking or parameter-sclection mechanisms will be needed to apply beyond vision encoders.
35

---

Published in Transactions on Machine Learning Research (12/2025)
(a) TSI
(b) DALLE
Figure 6: Instances of the 'Watching TV action.
(a) TSI
(b) DALLE
Figure Instances of the Use Tablet' action.
(a) TSI
(b) DALLE
Figure 8: Instances of the Use telephone' action.
36


Reviews:
[{"review_id": "jjKuokLz2n", "reviewer": "Reviewer_3hPB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce Low-Rank adaptation with Structured Updates (LoRSU), a new technique for fine-tuning large models. Its key innovation is a \"surgical\" approach that identifies and updates only the most critical parameters within a model's vision encoder—specifically, the most important attention heads and MLP layer parameters—based on their gradient signals. This is combined with LoRA to ensure high efficiency.\n\n\nThey created two new visual question-answering (VQA) datasets, TSI and DALLE, specifically designed to test a model's robustness to \"visual domain shift\" (i.e., how well it handles images that look different from its original training data).\n\nThis work presents the first large-scale exploration of Few-Shot Continual Learning (FSCL) for generative visual-language models. They conducted an extensive evaluation (over 700 experiments) across ten datasets, comparing LoRSU against a wide range of state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This is supported by a large-scale empirical study (over 700 experiments) across ten diverse VQA datasets and three few-shot settings (CL-5, CL-20, CL-50). The results in Tables 2, 3, and 4 consistently show LoRSU achieving superior Target Improvement (TI) while maintaining Control Change (CC) close to zero, outperforming 12 strong baselines.", "audience_yn": "Yes", "audience_explanation": "The CL community would be interested in a novel, replay-free method that demonstrates strong performance and reduced forgetting, especially in a challenging few-shot setting.\n\n Researchers working on PEFT methods (like LoRA, AdaLoRA) would be interested in a new, structured update approach that combines gradient-based parameter selection with low-rank adaptation.", "requested_changes": "While ablation studies are mentioned, key results (e.g., the impact of the number of heads k, the LoRA rank r, and the comparison to random/all-head selection) should be summarized in the main text with a small table or figure. This is crucial for validating the design choices of LoRSU and should not be relegated entirely to the appendix.", "broader_impact_concerns": "The newly introduced TSI dataset, which \"depict elderly people activities,\" could potentially contain or amplify societal biases related to age. The authors should add a statement acknowledging this possibility and describing steps taken to document and mitigate such biases in the dataset.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jPAGqkDc1g", "reviewer": "Reviewer_NZmk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an efficient method LoRSU for few shot continual learning for Vision Language Models (VLMs) that improves downstream performance.\n\nStrengths:\n- Improves efficiency and downstream task performance of VLMs.\n- The method is replay-free that is does not use the same example multiple times.\n- Introduces novel Visual QA datasets to expose issues with pre-trained image encoders in VLMs.\n- Provides theoretical justification of the importance score for attention heads.\n- Measures both gain of new abilities in VLMs (TI) and retention of prior capabilities (CC).\n\nWeaknesses:\n- Some lack of clarification in experimental choices.\n- Lack of clarification on the novelty of TSI dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- Clearly shows efficiency gain using TFlops and number of trainable parameters.\n- Shows gains in downstream tasks using ACC and BWT on various VQA datasets.\n- Shows improvement on new data using TI and retention of old capabilities using CC.", "audience_yn": "Yes", "audience_explanation": "It presents an efficient method for few shot continual learning for VLMs with theoretical reasoning and shows improvements through experimental results in both downstream performance and reduction in Tflops and trainable parameters.", "requested_changes": "- Clarify if the TSI dataset is novel as the authors also provide a reference Das et al. (2019).\n- Clarify is there is any reason to go forward with only Llava model.\n- Clarify the differences of datasets used in Table 2,3 and 4. Why not show the results for all datasets in all experiments?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b3OoMRhRUg", "reviewer": "Reviewer_7192", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces LoRSU (Low-Rank Adaptation with Structured Updates), a parameter-efficient fine-tuning (PEFT) method for few-shot continual learning (FSCL) in Vision-Language Models (VLMs). The primary goal is to adapt the VLM's vision encoder to new tasks from a few examples, without replaying old data and without catastrophically forgetting previously learned knowledge. The key idea is to selectively update a small subset of parameters in the vision encoder, specifically by applying structured sparse updates to MLP layers and low-rank adaptation (LoRA) to the most sensitive attention heads, identified via gradient magnitudes.\n\nKey Strengths:\n\nThe paper tackles a highly relevant and practical problem: efficiently adapting large VLMs in dynamic, real-world settings where data is scarce.\nThe experimental evaluation is extensive, spanning over 700 experiments across 10 VQA datasets and multiple FSCL settings. The proposed method, LoRSU, demonstrates consistently strong empirical performance, outperforming numerous baselines in both target task improvement and knowledge retention (low forgetting).\nThe introduction of new VQA datasets (TSI, DALLE) to specifically probe the limitations of pretrained encoders is a valuable contribution to the community.\n\nKey Weaknesses:\n\nThe conceptual novelty of the method feels somewhat limited. LoRSU appears to be a well-engineered combination of existing techniques—namely structured sparsity similar to SPU and selective low-rank adaptation—rather than a fundamentally new approach.\nThe theoretical justification for the method is superficial. While it proves that the chosen parameter selection strategy is optimal for a defined objective, it fails to provide a deeper theoretical link explaining why this objective leads to effective continual learning (e.g., reduced forgetting).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper's claims of superior scalability, efficiency, and accuracy are well-supported by a comprehensive and rigorous set of experiments. The authors compare LoRSU against a wide array of relevant baselines, including full fine-tuning, regularization-based methods (EWC), and various PEFTs (LoRA, AdaLoRA, SPU). The results, presented in Tables 2, 3, and 4, clearly show that LoRSU consistently achieves a better trade-off between target task improvement (TI) and forgetting on control datasets (CC) across different few-shot settings (5, 20, 50 shots). The computational efficiency gains reported in Figure 3 are also convincing.", "audience_yn": "Yes", "audience_explanation": "N/A", "requested_changes": "To strengthen the paper and secure a recommendation for acceptance, I believe the following changes are critical:\n\n(Critical) Deepen the Theoretical Analysis: The current theoretical justification is insufficient. The authors should move beyond simply proving that their selection criterion optimizes their chosen objective. I strongly suggest they provide a more profound analysis that connects their method to the core challenges of few-shot continual learning.\n\nSuggestion: Can the authors link the gradient magnitude criterion to the Fisher Information of the parameters? High-magnitude gradients might correspond to parameters with low Fisher Information for previous tasks, making them \"safer\" to update.\nSuggestion: Could a connection be made to work on neural network pruning or the lottery ticket hypothesis? Perhaps the selected parameters form a \"subnetwork\" that is critical for the new task but less entangled with prior knowledge.\nSuggestion: Explore the geometry of the loss landscape. Does this method guide the update in a direction that lies in a \"flat\" basin for old tasks while moving towards a sharp minimum for the new one?\nA more substantive theoretical discussion that provides intuition and a principled reason for why the LoRSU update rule works for FSCL would transform the paper from a purely empirical one to one with significant conceptual depth.\n(Critical) Re-evaluate and Clarify the Novelty: The authors should be more precise in positioning their contribution. Instead of presenting LoRSU as an entirely new paradigm, it would be more accurate and intellectually honest to frame it as a novel and effective synthesis of existing PEFTs.\n\nSuggestion: In the Introduction and Related Work sections, explicitly state that LoRSU integrates principles from structured sparsity (like SPU) and low-rank adaptation (LoRA).\nSuggestion: The core of the novelty could then be articulated as the insight that selectively applying these methods to different components of a transformer block (MLP vs. Attention) and using a gradient-based importance score for head selection is key to achieving a superior plasticity-stability balance in VLMs. This reframing does not diminish the paper's value but makes its claims more accurate.\nIn summary, the paper presents an empirically powerful method for a very important problem. However, its limited conceptual novelty and weak theoretical grounding prevent it from being a top-tier contribution in its current form. Addressing the requested changes, particularly by providing a deeper theoretical analysis, would significantly strengthen the submission.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)