Progressive LoRA for Multimodal Continual Instruction Tuning
Authors: Yahan Yu, Duzhen Zhang, Yong Ren, Xuanle Zhao, Xiuyi Chen, Chenhui Chu
Accepted at: acl25

Progressive LoRA for Multimodal Continual Instruction Tuning

Yahan Yu1, Duzhen Zhang2*, Yong Ren3, Xuanle Zhao3, Xiuyi Chen3, Chenhui Chu1
1Kyoto University, Japan
2Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
3Institute of Automation, Chinese Academy of Sciences, China
yahan@nlp.ist.i.kyoto-u.ac.jp, duzhen.zhang@mbzuai.ac.ae,
chu@i.kyoto-u.ac.jp

Abstract
Multimodal Continual Instruction Tuning (MCIT) empowers Multimodal Large Language Models (MLLMs) to adapt to ever-evolving requirements without continuous costly retraining. However, MCIT faces challenges in mitigating Catastrophic Forgetting (CF) and enhancing Knowledge Transfer (KT). Existing works combine Mixture-of-Expert (MoE) and LoRA to address these. However, using a fixed number of shared LoRA blocks across tasks can lead to the overwriting of acquired knowledge, making MLLMs harder to handle CF and KT. Therefore, we propose the Progressive LoRA framework (ProgLoRA), which contains a progressive LoRA pool and trains a new LoRA block for each incremental task to reduce knowledge interference. Specifically, ProgLoRA has two key mechanisms: task-aware allocation for effectively leveraging acquired knowledge at current task and task recall for realigning the model with learned tasks. Additionally, considering different application scenarios, we design a static ProgLoRA for the more idealized basic setting and a dynamic ProgLoRA for the more realistic challenging setting. Experiments on the latest MCIT benchmark demonstrate that ProgLoRA outperforms existing approaches.1

1 Introduction
Multimodal Large Language Models (MLLMs) (Zheng et al., 2024b; Dai et al., 2023) have demonstrated remarkable capabilities across a wide range of vision-language tasks (Mishra et al., 2019; Goyal et al., 2017), showcasing their potential to handle diverse scenarios. To support joint training across multiple tasks, instruction tuning (Liu et al., 2023a; Shen et al., 2024) is commonly used. In real-world applications, MLLMs are often required to continuously follow new instructions to keep pace with advancing knowledge. Despite this demand, most existing MLLMs are static, restricting their ability to address emerging tasks. Considering the computation cost and efficiency, retraining MLLMs from scratch for every new instruction is impractical. Therefore, existing works (Zheng et al., 2024a) conceptualize this challenge within Multimodal Continual Instruction Tuning (MCIT) (He et al., 2023), which is designed to incrementally instruction-tune MLLMs while maintaining high performance on previously tuned tasks.

However, MCIT faces two challenges: mitigating Catastrophic Forgetting (CF), which involves preserving the model’s previously acquired knowledge after learning new tasks; enhancing knowledge transfer (KT), which encourages models to leverage acquired knowledge to improve the learning of new tasks. To address this challenge, the Mixture-of-Experts LoRA (MoELoRA) (Liu et al., 2023b; Chen et al., 2024) leverages a set of special-

![A comparison between MoELoRA and our ProgLoRA. In LoRA blocks, higher transparency represents a lower allocation weight. MoELoRA updates the same set of blocks and allocates them by a simple router. In contrast, ProgLoRA updates only newly added block in current task and assigns blocks by task relevance.](page_1016_384_384_384.png)

Figure 1: A comparison between MoELoRA and our ProgLoRA. In LoRA blocks, higher transparency represents a lower allocation weight. MoELoRA updates the same set of blocks and allocates them by a simple router. In contrast, ProgLoRA updates only newly added block in current task and assigns blocks by task relevance.

1Our code is available at https://github.com/ku-nlp/ProgLoRA.
ized LoRA blocks designed to capture task-specific knowledge from sequential tasks. Nevertheless, a limitation is that MoELoRA shares the same set of LoRA blocks across different tasks as shown in Fig. 1 (a). As a result, acquired task-specific knowledge is easily overwritten by the knowledge of subsequent incremental tasks, which is more challenging to mitigate CF and enhance KT.

To further mitigate CF and enhance KT, we propose the Progressive LoRA framework (ProgLoRA). To prevent the disruption of acquired knowledge due to model parameter updates, ProgLoRA incorporates a progressive LoRA pool with multiple LoRA blocks. As shown in Fig. 1 (b), in each incremental step, a newly added LoRA block is trained to capture task-specific knowledge, while all previously tuned LoRA blocks are frozen. ProgLoRA isolates task-specific knowledge in independent LoRA blocks, preventing overwriting during updates. Meanwhile, frozen blocks still contribute to the training of other tasks. Specifically, ProgLoRA comprises two main components: task-aware allocation and task recall. Even if the acquired knowledge within the LoRA pool originates from tasks entirely unrelated to the current one, it may still enhance the learning of the current task. Therefore, we propose task-aware allocation, encouraging the model to effectively select and fuse LoRA blocks. Additionally, since the model contains trainable parameters beyond the LoRA pool, we introduce task recall to align with previous tasks, aiming to mitigate CF.

Considering different application backgrounds, we propose two kinds of implementations: ProgLoRA (static) for the ideal basic setting and ProgLoRA (dynamic) for a practical, challenging setting. The basic setting can assess task information associated with the input sample during training and testing stages, while challenging setting can only assess during training. Thus, in ProgLoRA (static), we propose static allocation weights in task-aware allocation and simple replay in task recall, which are both based on precomputed task similarity scores. In ProgLoRA (dynamic), as for the task-aware allocation, we assign a key to each LoRA block and calculate dynamic allocation weights with LoRA keys for adaptively fusing LoRA blocks. For the task recall, we apply a Kullback-Leibler (KL) divergence loss to constrain the allocation weights of previous samples in the current step more similar to the ones in previous steps, where the previous samples are the same set as in the ones of ProgLoRA (static).

Our contributions can be summarized as follows:

1. We propose ProgLoRA, where knowledge from different tasks is stored in separate LoRA blocks, thereby minimizing task interference.

2. We design the task-aware allocation to select and fuse LoRA blocks, aiming to enhance KT by effectively using acquired knowledge. We also propose the task recall to constrain the model updating and further mitigate CF. Considering basic and challenging settings, we design ProgLoRA (static) and ProgLoRA (dynamic), respectively.

3. Comprehensive experiments on LLaVA-1.5 using the latest MCIT benchmark demonstrate that ProgLoRA outperforms MoELoRA.

2 Related Works

2.1 MLLMs
MLLMs extend LLMs’ (Chung et al., 2022; Bai et al., 2023; Touvron et al., 2023; Li et al., 2025) capabilities to process visual and textual inputs. By combining the advanced reasoning skills of LLMs with rich visual representations from visual backbones, MLLMs achieve refined multi-modal reasoning and deep content understanding abilities (Zhang et al., 2024; Zhao et al., 2025b,a). LLaVA (Liu et al., 2023a), NExt-GPT (Wu et al., 2023), and MiniGPT-v2 (Chen et al., 2023) adopt a linear projection layer to connect a frozen LLM with a visual encoder for multimodal alignment. Meanwhile, models like InstructBLIP (Dai et al., 2023) and BLIP-2 (Li et al., 2023) trained the Q-Former projector to bridge the gap between modalities. Additionally, instruction tuning is leveraged to make models follow human instructions. These developments highlight the rapid evolution of MLLMs in addressing reasoning and comprehension tasks across modalities.

2.2 MCIT
MLLMs are necessary to be continuously updated to equip new abilities and keep pace with the rapidly evolving landscape of human knowledge (Wu et al., 2024b; Zheng et al., 2024b; Shi et al., 2024). To solve this, MCIT has emerged as a key solution, enabling models to integrate new data incrementally without the expensive process of retraining from scratch. Recent efforts have intro-
duced new MCIT benchmarks designed to incrementally fine-tune MLLMs on multiple tasks (He et al., 2023; Chen et al., 2024). Facing the major obstacle of mitigating CF and enhancing KT, where knowledge from earlier tasks interrupts with the new ones, some methods such as MoELoRA have been proposed. These methods aim to preserve instruction alignment across tasks by aggregating LoRA experts (Chen et al., 2024). Despite these efforts, such methods often fall short, as the shared expert design can result in knowledge interference, ultimately degrading model performance over time.

3 Preliminary

3.1 Problem Definition
Continual learning (Rypeś et al., 2024; Zhang et al., 2023; Zheng et al., 2025; Dong et al., 2022, 2024) aims to tackle challenges from sequentially evolving tasks, avoiding the high cost of full re-training. MCIT (Chen et al., 2024) is one of the continual learning approaches that allows MLLMs to adapt to new datasets in a step-by-step sequence through instruction tuning. The goal of MCIT is to sequentially train a model \( \mathcal{M} \) on the stream of tasks while ensuring strong performance across all previously encountered tasks. The problem is structured as a sequential stream of tasks, represented as \( \{\mathcal{T}_1, \mathcal{T}_2, ..., \mathcal{T}_N\} \), where \( N \) represents the total number of tasks. Each task \( \mathcal{T}_i \) is associated with its own dataset \( \mathcal{D}_i \). Notably, datasets from different tasks are diverse and cover knowledge from multiple domains without restriction. In each dataset \( \mathcal{D}_i \), it consists of \( M_i \) data pairs \( (X_{ij}^v, X_{ij}^q, X_{ij}^a, M_{ij}) \), where \( X^v, X^q \) and \( X^a \) denote image tokens, instruction tokens, and answer tokens, respectively.

3.2 LoRA
LoRA (Hu et al., 2022; Wu et al., 2024a; Wang et al., 2023, 2024a,b) is a parameter-efficient fine-tuning technique designed to adapt pre-trained LLMs to new tasks without modifying full set of parameters. Given an intermediate representation \( x \) from preceding attention layer, the forward pass in LLMs is defined as:

\[
h = W x + \frac{\alpha}{r} B A x,
\]

where \( W \in \mathbb{R}^{d_{in} \times d_{out}} \) denotes pre-trained weight in LLMs, \( A \in \mathbb{R}^{r \times d_{out}} \) and \( B \in \mathbb{R}^{d_{in} \times r} \) denote trainable low-rank matrices with rank \( r \), and \( \alpha \) denotes hyperparameter to finetune effect of \( r \).

In MCIT, Chen et al. (2024) adopt the MoELoRA (Dou et al., 2023; Liu et al., 2023b) to mitigate CF, utilizing a set of specialized experts to capture task-specific knowledge with a fixed expert pool \( \{E_1(\cdot), E_2(\cdot), ..., E_K(\cdot)\} \) and a router \( R(\cdot) \) (Fedus et al., 2022; Zhu et al., 2024). The MoELoRA layer computes output as follows:

\[
h = W_F x + \frac{\alpha}{r} \sum R(x)[k] E_k(x)
\]
\[
R(x) = Softmax(W_R x[0]), \quad E_k(x) = B_k A_k x
\]

where \( W_F \in \mathbb{R}^{d_{in} \times d_{out}} \) denotes parameters of the FF layer. The model employs \( K \) experts and each is characterized by two trainable low-rank matrices, \( A_k \in \mathbb{R}^{d_{in} \times \frac{r}{K}} \) and \( B_k \in \mathbb{R}^{\frac{r}{K} \times d_{out}} \). These matrices are with a reduced rank of \( \frac{r}{K} \), ensuring that overall trainable parameters remains equivalent to a single LoRA setup, thereby maintaining computational efficiency. Meanwhile, \( R(\cdot) \) takes the first token of \( x \) as input and uses trainable \( W_R \in \mathbb{R}^{d_{in} \times K} \) to generate a probability distribution of each expert.

4 Approach

4.1 Overview
Our ProgLoRA, as shown in Fig. 2, is a novel framework to address CF and KT in a progressive and task-aware manner. We set a progressive LoRA pool \( \{BLK_1, BLK_2, ..., BLK_N\} \), where \( BLK_i(x) = B A x \) represents \( i \)-th LoRA block. ProgLoRA comprises two components:

Task-aware Allocation The LoRA blocks \( BLK_p \) trained on previous tasks \( \mathcal{T}_p \) (\( p \in [1, i) \)) are frozen during the training of the current task \( \mathcal{T}_i \) to preserve the acquired knowledge. This knowledge, even though it comes from different tasks than the current one, can also be effectively utilized to promote KT. Thus, we design allocation weights \( w \) to select and fuse LoRA blocks in LoRA pool, where each block stores task-specific knowledge.

Task Recall Freezing previous \( BLK_p \) intuitively mitigates CF. However, MLLMs often include other trainable components (e.g., cross-modal projection layers) that remain active across tasks, which creates a vulnerability of CF. Thus, we introduce task recall.

Specifically, our method has two versions: (1) For the basic setting, where task information (i.e., task ID \( i \)) is available during training and testing, we propose ProgLoRA (static) in Fig. 2 (a) with fixed allocation weights. (2) For the challenging
setting, where task ID is only available during training, we introduce ProgLoRA (dynamic) in Fig. 2 (b), which enables the adaptive allocation of LoRA blocks.

4.2 ProgLoRA (static)

The basic setting represents an idealized scenario. In this context, our goal is to leverage a lightweight model.

Task-aware Allocation Prior works (Nikandrou et al., 2022) have optimized continual learning methods by leveraging similarity between \( \mathcal{T}_p \) and \( \mathcal{T}_i \). Higher similarity facilitates the retention of acquired knowledge from \( \mathcal{T}_p \) while enabling efficient learning of new knowledge in \( \mathcal{T}_i \). Inspired by these, we propose static allocation weights \( w^s \) based on task similarity. When the similarity between \( \mathcal{T}_p \) and \( \mathcal{T}_i \) is higher, the weight \( w^s_{p,i} \) assigned to \( BLK_p \) increases. The formulas are as follows:

\[
\theta_{BLK_i^s} = w^s_{i,i} \theta_{BLK_i} + \sum_{p=1}^{i-1} w^s_{p,i} \theta_{BLK_p},
\]
\[
w^s_{p,i} = \gamma(e^v_p, e^v_i) \cdot \gamma(e^q_p, e^q_i) \cdot \gamma(e^a_p, e^a_i)
\]

where \( \gamma \) denotes cosine similarity measure, \( \theta \) denotes the parameters, \( BLK_i^s \) denotes the fused LoRA block. \( e^v_i, e^q_i \) and \( e^a_i \) denote pre-computed average image, instruction and answer embeddings of \( \mathcal{D}_i \). Note that \( w^s_{i,i} = 1 \). The training loss for allocation in \( \mathcal{T}_i \) is:

\[
\mathcal{L}_{A,i}^s = - \sum_j^{M_i^{\text{train}}} \log P(X_{i,j}^a | X_{i,j}^v, X_{i,j}^q; \theta_{BLK_i^s}, \theta_{M_i}),
\]

where \( \theta_{M_i} \) denotes parameters of MLLM backbone. This static method ensures that more relevant acquired knowledge is better leveraged to facilitate the learning of new knowledge.

Task Recall Given that ProgLoRA (static) maintains its simplicity by not introducing any additional trainable parameters beyond the LoRA pool, we propose a replay-based task recall to keep this resource-efficient way. This approach leverages a small amount of replayed data from \( \mathcal{T}_p \) based on task similarity to align the model. Specifically, we replay \( (1 - w^s_{p,i}) \phi \) samples from \( \mathcal{T}_p \) when learning \( \mathcal{T}_i \), where \( \phi \) denotes base value of replay samples. \( \mathcal{T}_p \) that is less similar to \( \mathcal{T}_i \) will have more replay samples, thereby enhancing the recall of acquired knowledge. The training loss \( \mathcal{L}_{R,i}^s \) for task recall in \( \mathcal{T}_i \) is the same as the calculation process in Eq. (4), while the input data is the replay ones.

The final loss is defined as \( \mathcal{L}_i^s = \mathcal{L}_{A,i}^s + \lambda \mathcal{L}_{R,i}^s \), where \( \lambda \) is the loss coefficient. During the inference phase, for a test sample originating from \( \mathcal{T}_i \), we directly execute using the fused LoRA block \( BLK_i^s \).

![The overall structure of ProgLoRA. Considering different application scenarios, our method has two versions: ProgLoRA (static) in (a) for basic setting and ProgLoRA (dynamic) in (b) for more challenging setting.](page_32_68_1397_563.png)

Figure 2: The overall structure of ProgLoRA. Considering different application scenarios, our method has two versions: ProgLoRA (static) in (a) for basic setting and ProgLoRA (dynamic) in (b) for more challenging setting.
4.3 ProgLORA (dynamic)

In challenging settings where task IDs are unavailable during the testing phase, it becomes necessary to incorporate a mechanism within the model that can adaptively allocate and fuse LoRA blocks. Therefore, we propose ProgLORA (dynamic) to extend the static method.

Task-aware Allocation We design dynamic allocation weights to fuse LoRA blocks. Specifically, at the training stage of \( \mathcal{T}_i \), a key vector \( key_n \) is assigned to each LoRA block \( BLK_n \) (\( n \in [1, i] \)), and instance-level weights are computed. This ensures that the allocation process is block-agnostic and can be compatible with various adaptation techniques (e.g., LoRA). The dynamic allocation process begins when the \( X_{i,j}^q \) is passed through layers preceding the LoRA pool of the LLM backbone and obtains embeddings \( x_{i,j}^q \). For better alignment with the key vector spaces, we first fed \( x_{i,j}^q \) into a projector \( Proj_i \) (we omit the \( q \) and \( j \) for simplicity):

\[
h_i = \mathrm{LN}(W_2 \varphi(W_1 x_{i,[max]} + b_1) + b_2),
\]

where \( W_1 \in \mathbb{R}^{d_p \times d_m} \) and \( W_2 \in \mathbb{R}^{d_m \times d_p} \) denote trainable parameters, \( b_1 \) and \( b_2 \) denote bias, \( \varphi(\cdot) \) and LN(\cdot) denote SiLU function and Layer Norm to ensure the reliability of the training process. Note that max-pool operation is applied to \( x_i \) along the token length dimension to align it with the dimension of the \( key_n \in \mathbb{R}^{d_m} \). Then, dynamic allocation weights \( w_{n,i}^d \) is processed with aligned \( h_i \) as:

\[
w_{n,i}^d = Softmax(h_i key_n / \alpha_{tem}),
\]

where \( \alpha_{tem} \) represents a temperature factor to constrain the confidence of \( w_{n,i}^d \), which makes the model less sensitive to small differences and leads to more balanced weight distributions to improve robustness. Similar to Eq. (3), the LoRA blocks are fused to \( BLK_i^d \) as follows:

\[
\theta_{BLK_i^d} = \sum_{n=1}^i w_{n,i}^d \theta_{BLK_n}
\]

To update the model, the calculation of training loss \( \mathcal{L}_{A,i}^d \) is similar to that of \( \mathcal{L}_{A,i}^s \) in Eq. (4), but it incorporates \( \theta_{BLK_i^d}, \theta_{key}, \theta_{Proj_i} \), and \( \theta_M \) into the optimization process.

Task Recall As tasks are sequentially trained, the \( Proj \) in Eq. (5) is updated continuously. To ensure that inputs from \( \mathcal{T}_p \) still accurately perform the dynamic allocation weights and identify the specific fusion of LoRA blocks, we introduce the KL-divergence-based task recall. This is achieved by using samples from \( \mathcal{T}_p \); however, unlike in ProgLORA (static), we don’t perform full LLM replay on these samples. Instead, they are solely used to generate pseudo dynamic allocation weights \( \hat{w}^d \) for the recall of \( Proj_i \) at the training of \( \mathcal{T}_i \). To help \( Proj_i \) accurately recall the correct \( w_p^d = \{w_{1,p}^d, ..., w_{i,p}^d\} \) for samples \( \mathcal{T}_p \), each previous sample is fed into layers in Eq. (5) and (6) to get \( \hat{w}_i^d = \{\hat{w}_{1,i}^d, ..., \hat{w}_{i,i}^d\} \) at the training of \( \mathcal{T}_i \). To ensure that \( \hat{w}_i^d \) aligns with its intended value \( w_p^d \), we minimize the recall loss \( \mathcal{L}_{R,i}^d \):

\[
\mathcal{L}_{R,i}^d = \sum_{p=1}^{i-1} \sum_{r=1}^{N_p^{recall}} \mathrm{KL}(\hat{w}_i^d || w_p^d),
\]

where \( N_p^{recall} \) denotes the number of samples from \( \mathcal{T}_p \) and is kept consistent with the replay sample size used in the ProgLORA (static). \( \mathrm{KL}(\cdot) \) denotes a KL divergence loss. Note that \( w_p^d \) are padded with zeros to match the sequence length of \( \hat{w}_i^d \).

Finally, we also optimize the model in a multi-task learning manner:

\[
\mathcal{L}_i^d = \mathcal{L}_{A,i}^d + \lambda \mathcal{L}_{R,i}^d
\]

During the inference phase, we dynamically allocate all available LoRA blocks to the testing samples in an adaptive manner by Eq. (5)-(7).

5 Experimental Settings

5.1 Datasets

We follow the setup outlined in the MCIT benchmark, CoIN (Chen et al., 2024), which includes 8 multimodal datasets covering a diverse range of tasks. Datasets in CoIN benchmark are: (Lu et al., 2022), TextVQA (Singh et al., 2019), ImageNet (Deng et al., 2009), GQA (Hudson and Manning, 2019), VizWiz (Gurari et al., 2018), Grounding (Kazemzadeh et al., 2014; Mao et al., 2016), VQAv2 (Goyal et al., 2017), and OCR-VQA (Mishra et al., 2019). The datasets are processed in the same sequential order as experiments in Chen et al. (2024). Detailed statistics for the 8 datasets in the CoIN benchmark (Chen et al., 2024) are presented in Appendix A.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="8">Accuracy on Each Task</th>
    <th colspan="4">Overall Results</th>
  </tr>
  <tr>
    <th>ScienceQA</th>
    <th>TextVQA</th>
    <th>ImageNet</th>
    <th>GQA</th>
    <th>ViQWiz</th>
    <th>Grounding</th>
    <th>VQAv2</th>
    <th>OCR-VQA</th>
    <th>ACC↑</th>
    <th>MAA↑</th>
    <th>BWT↑</th>
    <th>FWT↑</th>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>49.91</td>
    <td>2.88</td>
    <td>0.33</td>
    <td>2.08</td>
    <td>0.90</td>
    <td>0.00</td>
    <td>0.68</td>
    <td>0.17</td>
    <td>-</td>
    <td>7.12</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Multi-task</td>
    <td>56.77</td>
    <td>49.35</td>
    <td>95.55</td>
    <td>56.65</td>
    <td>53.90</td>
    <td>30.09</td>
    <td>59.50</td>
    <td>55.65</td>
    <td>-</td>
    <td>57.18</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>82.45</td>
    <td>49.99</td>
    <td>96.05</td>
    <td>56.40</td>
    <td>55.45</td>
    <td>31.27</td>
    <td>62.20</td>
    <td>57.08</td>
    <td>28.74</td>
    <td>32.97</td>
    <td>-32.62</td>
    <td>-4.82</td>
  </tr>
  <tr>
    <td>LwF*</td>
    <td>21.26</td>
    <td>28.74</td>
    <td>10.25</td>
    <td>36.78</td>
    <td>32.45</td>
    <td>0.83</td>
    <td>42.50</td>
    <td>57.08</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LwF*</td>
    <td>81.36</td>
    <td>50.59</td>
    <td>96.84</td>
    <td>51.98</td>
    <td>48.19</td>
    <td>25.13</td>
    <td>41.30</td>
    <td>64.12</td>
    <td>30.41</td>
    <td>34.95</td>
    <td>-27.03</td>
    <td>-8.75</td>
  </tr>
  <tr>
    <td>EWC*</td>
    <td>26.78</td>
    <td>37.52</td>
    <td>12.64</td>
    <td>35.18</td>
    <td>25.24</td>
    <td>2.87</td>
    <td>38.92</td>
    <td>64.12</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>EWC*</td>
    <td>82.81</td>
    <td>51.76</td>
    <td>96.80</td>
    <td>46.19</td>
    <td>48.68</td>
    <td>26.82</td>
    <td><b>66.37</b></td>
    <td>63.46</td>
    <td>32.90</td>
    <td>36.93</td>
    <td>-27.46</td>
    <td>-5.81</td>
  </tr>
  <tr>
    <td>MoELoRA</td>
    <td>30.33</td>
    <td>36.08</td>
    <td>11.62</td>
    <td>35.75</td>
    <td>37.50</td>
    <td>3.48</td>
    <td>44.98</td>
    <td>63.46</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MoELoRA</td>
    <td>75.78</td>
    <td>51.73</td>
    <td>96.70</td>
    <td>59.42</td>
    <td>58.88</td>
    <td>37.50</td>
    <td>64.22</td>
    <td>60.08</td>
    <td>37.13</td>
    <td>42.76</td>
    <td>-25.91</td>
    <td>-4.01</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td>63.09</td>
    <td>38.63</td>
    <td>10.50</td>
    <td>37.38</td>
    <td>43.62</td>
    <td>0.59</td>
    <td>43.15</td>
    <td>60.08</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td>76.27</td>
    <td>60.78</td>
    <td>97.32</td>
    <td>61.27</td>
    <td>60.16</td>
    <td>39.35</td>
    <td>65.83</td>
    <td><b>64.44</b></td>
    <td>59.09</td>
    <td>62.38</td>
    <td>-6.59</td>
    <td>1.37</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td><b>74.84</b></td>
    <td><b>51.83</b></td>
    <td><b>83.90</b></td>
    <td><b>49.93</b></td>
    <td><b>53.87</b></td>
    <td><b>31.19</b></td>
    <td><b>62.71</b></td>
    <td><b>64.44</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 1: Main results on the LLaVA-1.5-7B model. For accuracy of each task of MCIT methods, the first row denotes the results for each task evaluated after its tuning (\( i.e., P_{i,i} \) (\( i \in [1, N] \))) with the best performance highlighted in red, while the second row shows each task’s results after tuning the final task (\( i.e., P_{N,i} \)) with the best ones in blue. For overall results, the bold highlights the best performance. * represents results from our re-implementation, while others are cited from CoIN (Chen et al., 2024).

5.2 Compared Methods

We compare ProgLoRA with zero-shot, multi-task and MCIT methods (LoRA (Hu et al., 2022), MoELoRA (Chen et al., 2024), LwF (Li and Hoiem, 2017), and EWC (Kirkpatrick et al., 2017)). Details are shown in Appendix B.

5.3 Evaluation Metrics

For metrics, we use Average Accuracy (ACC), Mean Average Accuracy (MAA), Backward Transfer (BWT), and Forward Transfer (FWT). \( ACC = \frac{1}{N} \sum_{i=1}^{N} P_{N,i} \), where \( P_{N,i} \) is the performance on \( i \)-th task after training the final task \( T_N \). \( MAA = \frac{1}{N} \sum_{i=1}^{N} (\frac{1}{T} \sum_{k=1}^{T} P_{i,k}) \), where \( P_{i,k} \) is the performance on \( T_k \) after training \( T_i \). \( BWT = \frac{1}{N} \sum_{i=1}^{N} (P_{N,i} - P_{i,i}) \), where \( P_{i,i} \) is the performance on \( T_i \) after training on \( T_i \). \( FWT = \frac{1}{N} \sum_{i=1}^{N} (P_{i,i} - P_{0,i}) \), where \( P_{0,i} \) represents the performance of training \( T_i \) in isolation. Among these metrics, ACC reflects the model’s ability to promote KT, MAA measures accuracy across all tasks to focus on stability, BWT assesses the model’s capacity to mitigate CF, and FWT measures the model’s capacity to enhance KT. More details are shown in Appendices C.

5.4 Implementation

We also follow the same implementation settings as Chen et al. (2024) and utilize LLaVA-1.5-7B (Liu et al., 2023a) as the backbone. For all the experiments, LoRA (Hu et al., 2022) is integrated into the LLMs. The vision encoder and LLM remain frozen, while only the projection layer and LoRA are updated. For a fair comparison, for a single LoRA block in ProgLoRA, rank \( r \) is set to 16, \( \alpha \) is kept consistent with MoELoRA, with the base value of replay amount \( \phi \) set to 200 and loss coefficient \( \lambda \) set to 2.0. For the calculation of task similarity, \( e_i^v, e_i^l \) and \( e_i^a \) denote pre-computed average image, instruction and answer embeddings of \( D_i \), respectively. These embeddings are obtained using BERT (Kenton and Toutanova, 2019) to encode texts and ViT (Dosovitskiy et al., 2020) to encode images. All experiments are conducted on 4 NVIDIA A100 GPUs, each with 80GB of memory.

6 Results and Analysis

6.1 Main Results

Results are summarized in Table 1. Considering that MCIT baselines don’t require task information during testing, we compare only ProgLoRA (dynamic) with them. For the overall results, our method significantly outperforms all baselines, which illustrates the designed task-aware allocation and task recall can solve problems of CF and KT. For accuracy on each task, our method consistently achieves the best performance on both \( P_{i,i} \) and \( P_{N,i} \), where \( P_{i,i} \) denotes performance on \( T_i \) after its tuning and \( P_{N,i} \) denotes performance on \( T_i \) after tuning on the last task \( T_N \). Meanwhile, it’s worth noting that while baselines may perform better on \( P_{i,i} \), this doesn’t indicate a weakness in ProgLoRA. This is because the overall \( r \) of ProgLoRA is always less than the baselines before training the last
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="8">Accuracy on Each Task</th>
    <th colspan="4">Overall Results</th>
  </tr>
  <tr>
    <th>ScienceQA</th>
    <th>TextVQA</th>
    <th>ImageNet</th>
    <th>GQA</th>
    <th>VizWiz</th>
    <th>Grounding</th>
    <th>VQAv2</th>
    <th>OCR-VQA</th>
    <th>ACC↑</th>
    <th>MAA↑</th>
    <th>BWT↑</th>
    <th>FWT↑</th>
  </tr>
  <tr>
    <td>ProgLoRA (static)</td>
    <td><b>76.27</b></td>
    <td>60.07</td>
    <td>92.65</td>
    <td>58.45</td>
    <td>57.33</td>
    <td>17.68</td>
    <td>64.45</td>
    <td>63.54</td>
    <td>58.73</td>
    <td><b>66.44</b></td>
    <td>-2.57</td>
    <td>1.06</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td><b>76.27</b></td>
    <td><b>60.78</b></td>
    <td><b>97.32</b></td>
    <td><b>61.27</b></td>
    <td><b>60.16</b></td>
    <td><b>39.35</b></td>
    <td><b>65.83</b></td>
    <td><b>64.44</b></td>
    <td>59.09</td>
    <td>62.38</td>
    <td>-6.59</td>
    <td>1.37</td>
  </tr>
</table>

Table 2: Comparision of ProgLoRA (static) and ProgLoRA (dynamic) on LLaVA-1.5-7B. Meaning of the highlight marker is the same as in Table 1.

![Three line plots showing the effect of replay amount, model scale, and LoRA rank on accuracy](page_184_370_1080_273.png)

Figure 3: Step-wise comparison of ProgLoRA (dynamic) with different (a) replay amount, (b) model scale, and (c) LoRA rank.

<table>
  <tr>
    <th>Variant</th>
    <th>ACC↑</th>
    <th>MAA↑</th>
    <th>BWT↑</th>
    <th>FWT↑</th>
  </tr>
  <tr>
    <td>ProgLoRA (static)</td>
    <td>58.73</td>
    <td><b>66.44</b></td>
    <td>-2.57</td>
    <td>1.06</td>
  </tr>
  <tr>
    <td>w/ allocation weight \( w^s = 1 \)</td>
    <td>54.60</td>
    <td>63.43</td>
    <td>-2.62</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>w/o task recall</td>
    <td>56.06</td>
    <td>65.06</td>
    <td>-4.84</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>w/ random task recall</td>
    <td>57.62</td>
    <td>66.00</td>
    <td>-2.76</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td><b>59.09</b></td>
    <td><b>62.38</b></td>
    <td><b>-6.59</b></td>
    <td><b>1.37</b></td>
  </tr>
  <tr>
    <td>w/o task recall</td>
    <td>57.52</td>
    <td>62.12</td>
    <td>-8.07</td>
    <td>1.08</td>
  </tr>
</table>

Table 3: The ablation study of ProgLoRA on the LLaVA-1.5-7B backbone. Bold highlights the best performance.

task, making the slightly lower results reasonable. We also observed that ProgLoRA (dynamic) outperformed the multi-task setting in terms of MAA, demonstrating that our method effectively facilitates knowledge transfer.

We also compare the ProgLoRA (static) results separately with the dynamic ones to demonstrate the impact of different allocation and recall strategies within the same progressive LoRA pool, which is shown in Table 2. **ProgLoRA (static) achieves higher MAA, BWT and most \( P_{N,i} \)**, demonstrating its advantage in mitigating CF. We think that it operates under idealized setting, where pre-determined allocation weights are applied in the testing stage based on known task details. As a result, its performance after tuning different tasks remains stable. Additionally, for each task, the higher \( P_{i,i} \) and FWT only occur with ProgLoRA (dynamic), for the reason that **ProgLoRA (static) is lack of flexibility**, highlighting the advantages of dynamic task allocation in improving KT.

6.2 Ablation Study

As shown in Table 3, we analyzed each component. For ProgLoRA (static), KT is hindered when setting \( w^s = 1 \), indicating the necessity of leveraging task similarity for LoRA block allocation when the allocation weight is fixed. Additionally, removing task recall or replacing it with random sampling of \( \phi \) samples from each previous task increases CF, highlighting the importance of task recall. For ProgLoRA (dynamic), removing task recall leads to a significantly greater occurrence of CF, underscoring the critical role of this component in enabling adaptive LoRA block allocation.

6.3 More Explorations

**Effect of Replay Amount** We show the step-wise results on ProgLoRA (dynamic) with different replay amount \( \phi \) after training on the final task (OCR-VQA) in Fig. 3 (a), where the backbone is LLaVA-1.5-7B. More results are shown in Appendix F. The results showed a very slight improvement with the increase in \( \phi \), but it can be observed that tasks later in the task sequence (such as OCR-VQA) are slightly affected as the \( \phi \) increases. Meanwhile, it’s preferable to use as little replay data as possible, because storing and replaying large amounts of data can be computationally
User:
Select the better answer.
What property do these two objects have in common?
A. sour
B. blue
Answer with the option’s letter from the given choices directly.
MoELoRA: B X   ProgLoRA (static): A ✓   ProgLoRA (dynamic): A ✓

User:
Fish live underwater. They have fins, not limbs. Fish are cold-blooded. The body temperature of cold-blooded animals depends on their environment. A Banggai cardinalfish is an example of a fish. Select the fish below.
A. water buffalo
B. poison dart frog
C. great white shark
Answer with the option’s letter from the given choices directly.
MoELoRA: A X   ProgLoRA (static): A X   ProgLoRA (dynamic): D X

User:
Who wrote this book? When the provided information is insufficient, respond with 'Unanswerable'. Answer each question using a single word or phrase.
MoELoRA: BSAVAX
ProgLoRA (static): Ian Ramsey ✓
ProgLoRA (dynamic): Ian Ramsey ✓

User:
Is this book related to Children's Books? When the provided information is insufficient, respond with 'Unanswerable'. Answer each question using a single word or phrase.
MoELoRA: No X
ProgLoRA (static): No X
ProgLoRA (dynamic): Yes ✓

(a) Example from ScienceQA   (b) Example from OCR-VQA

Figure 4: Comparison between MoELoRA and ProgLoRA on cases after training on the final task.

![Visualization of allocation weights in ProgLoRA (dynamic)](page_328_670_795_312.png)

Figure 5: Visualization of allocation weights in ProgLoRA (dynamic).

expensive and time-consuming in real-world scenarios. Therefore, \( \phi = 200 \) is appropriate.

Effect of Model Scale We choose LLaVA-1.5-13B as the new backbone to evaluate ProgLoRA (dynamic). We show the step-wise results after training on the final task in Fig. 3 (b). We observed that larger models tend to offer better capabilities for mitigating CF and promoting KT. We analyse the reason that, with an increase in model size, the model retains a larger optimization space for learning more new knowledge.

Effect of LoRA Settings We investigate the influence of LoRA rank \( r \). Step-wise results on ProgLoRA (dynamic) after training on the final task are provided in Fig. 3 (c), where MLLM backbone is LLaVA-1.5-7B. More results are shown in Appendix G. We observed that performance improves as \( r \) increases, demonstrating that more trainable parameters enhances model’s ability to acquire new multimodal knowledge. Additionally, CF also decreases, because the additional parameters provide the model with sufficient optimization space to learn more multimodal knowledge.

We also conduct further explorations, including results on Other MLLM Backbone (Appendix D), analysing the effect of Task Order (Appendix H), effect of Instruction Diversity (Appendix I), and discussion on Efficiency (Appendix J).

6.4 Visualization of Allocation Weights

We show the allocation weights on ProgLoRA (dynamic) after completing the final task in Fig. 5, where MLLM backbone is LLaVA-1.5-7B. We observe that task-specific LoRA block associated with the input samples occupies the largest proportion. This demonstrates that our designed Task-aware Allocation can automatically identify the most relevant task-specific knowledge. Compared to static weights provided in Appendix K, the partial consistency between dynamic weights and their corresponding parts proves that task similarity is effective in improving the performance of MCIT, but it cannot be fully relied upon. Designing dynamic allocation weights is necessary.

6.5 Case Study

We provide some case studies in Fig. 4. For the top case of Fig. 4 (a), after fine-tuning the final task, ProgLoRA still provides the correct answer for the first task, while MoELoRA doesn’t. This highlights ProgLoRA’s advantage to mitigate CF. At the same time, we also demonstrate potential weaknesses in ProgLoRA’s reasoning capability. For the bottom case of Fig. 4 (a), we present a failed case where ProgLoRA provides an incorrect answer when handling scenarios with extensive contextual informa-
tion and complex options. Similarly, for the top case of Fig. 4 (b), ProgLoRA successfully generates the correct answer for the final task, whereas MoELoRA fails to do so, demonstrating the edge of ProgLoRA in leveraging acquired knowledge to promote KT. The bottom one shows a failed case of ProgLoRA (static), illustrating limitations of static method when dealing with obscure information. More case studies are shown in Appendix L.

7 Conclusion

In this paper, we proposed ProgLoRA with a progressive LoRA pool that mitigates task interference by isolating knowledge in separate LoRA blocks. We design task-aware allocation and task recall to mitigate CF and promote KT. Experiments show that ProgLoRA outperforms MoELoRA and LoRA across two MLLM backbones on CoIN benchmark, offering a more efficient and stable MCIT approach. Future work could explore how to achieve a progressive LoRA pool with fewer parameters.

Acknowledgments

We express our gratitude to the anonymous reviewers for their valuable and insightful comments. This work was supported by JST BOOST, Grant Number JPMJBS2407.

Limitations

While ProgLoRA offers significant improvements in mitigating catastrophic forgetting and enhancing knowledge transfer, it does have certain limitations. First, our work was confined to the CoIN benchmark. To further validate the effectiveness of MCIT methods, future work could focus on creating more comprehensive and challenging benchmarks that encompass a wider array of tasks and domains.

Secondly, our method is influenced by the task order to some extent. We believe that maintaining robustness across different task orders is a valuable research direction. For instance, further enhancing the ability to transfer knowledge could allow the model to not only learn new knowledge but also revisit and fill gaps in previously learned knowledge.

Finally, We have considered that the total parameters of the model will increase as new tasks are added. Since the additional parameters introduced by our method at each task come from one LoRA block, which is small compared to the total number of model parameters, the impact of the increasing number of tasks on the model’s parameter size can be neglected in our work. Thus, currently, we are primarily focused on how to best mitigate forgetting and promote knowledge transfer within a task sequence of a certain length for MLLM. For future work, this is an important and interesting direction to explore how to achieve a more efficient MCIT model in the context of an infinite number of new tasks. Some methods, like the fusion of similar block fusion and block pruning, can be used to address this problem.

Ethics Statement

In addressing the ethical aspects of our work, we would like to provide the following clarifications: (1) This research does not make use of any sensitive or private data, and no tasks were performed that involve sensitive information. (2) All experiments were conducted using publicly accessible datasets, which have been sourced from reputable and established scientific research. (3) We have provided comprehensive details on the dataset statistics and the hyperparameter configurations employed in our methodology, ensuring full transparency and alignment with the reported results. (4) In the interest of fostering reproducibility and encouraging further research, we plan to make our code publicly available on GitHub upon the publication of this work.

References

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Owen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

Cheng Chen, Junchen Zhu, Xu Luo, Heng Tao Shen, Jingkuan Song, and Lianli Gao. 2024. CoIN: A benchmark of continual instruction tuning for multimodal large language models. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In Thirty-seventh Conference on Neural Information Processing Systems.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee.

Jiahua Dong, Hongliu Li, Yang Cong, Gan Sun, Yulun Zhang, and Luc Van Gool. 2024. No one left behind: Real-world federated class-incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2054–2070.

Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. 2022. Federated class-incremental learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.

Shihuan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhua Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. 2023. Loramo: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. arXiv preprint arXiv:2312.09979, 4(7).

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.

Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617.

Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. 2023. Continual instruction tuning for large multi-modal models. arXiv preprint arXiv:2311.16206.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations.

Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, pages 4171–4186.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 19730–19742.

Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947.

Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. 2025. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual Instruction Tuning. In Thirty-seventh Conference on Neural Information Processing Systems.

Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. 2023b. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. arXiv preprint arXiv:2310.18339.

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20.

Anand Mishra, Shashank Shekhar, Ajeeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE.

Mayina Nikandrou, Lu Yu, Alessandro Sugiia, Ioannis Konstas, and Verena Rieser. 2022. Task formulation matters when learning continually: A case study in visual question answering. arXiv preprint arXiv:2210.00044.

Grzegorz Rypeś, Sebastian Cygert, Valeriya Khan, Tomasz Trzcinski, Bartosz Zielinski, and Bartlomiej Twardowski. 2024. Divide and not forget: Ensemble of selectively trained experts in continual learning. arXiv preprint arXiv:2401.10191.

Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang. 2024. Multimodal instruction tuning with conditional mixture of lora. arXiv preprint arXiv:2402.15896.

Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. 2024. Continual learning of large language models: A comprehensive survey. arXiv preprint arXiv:2404.16789.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Alldert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Huiyi Wang, Haodong Lu, Lina Yao, and Dong Gong. 2024a. Self-expansion of pre-trained models with mixture of adapters for continual learning. arXiv preprint arXiv:2403.18886.

Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, and Hinrich Schütze. 2024b. Rehearsal-free modular and compositional continual learning for language models. arXiv preprint arXiv:2404.00790.

Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2023. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.14152.

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519.

Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong. 2024a. Mixture-of-subspaces in low-rank adaptation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7880–7899, Miami, Florida, USA. Association for Computational Linguistics.

Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. 2024b. Continual Learning for Large Language Models: A Survey. arXiv preprint arXiv:2402.01364.

Duzhen Zhang, Wei Cong, Jiahua Dong, Yahah Yu, Xiuyi Chen, Yonggang Zhang, and Zhen Fang. 2023. Continual named entity recognition without catastrophic forgetting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8186–8197.

Duzhen Zhang, Yahah Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. 2024. Mm-llms: Recent advances in multimodal large language models. In Findings of the Association for Computational Linguistics ACL 2024.

Xuanle Zhao, Xuexin Liu, Haoyue Yang, Xianzhen Luo, Fanhu Zeng, Jianling Li, Qi Shi, and Chi Chen. 2025a. Chartedit: How far are mlms from automating chart analysis? evaluating mlms’ capability via chart editing. arXiv preprint arXiv:2505.11935.

Xuanle Zhao, Xianzhen Luo, Qi Shi, Chi Chen, Shuo Wang, Wanxiang Che, Zhiyuan Liu, and Maosong Sun. 2025b. Chartcoder: Advancing multimodal large language model for chart-to-code generation. arXiv preprint arXiv:2501.06598.

Junhao Zheng, Qianli Ma, Zhen Liu, Binquan Wu, and Huawen Feng. 2024a. Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer. arXiv preprint arXiv:2401.09181.

Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma. 2024b. Towards Lifelong Learning of Large Language Models: A Survey. arXiv preprint arXiv:2406.06391.

Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, and Qianli Ma. 2025. Lifelong learning of large language model based agents: A roadmap. arXiv preprint arXiv:2501.07278.

Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. 2024. Llama-moe: Building mixture-of-experts from llama with continual pre-training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15913–15923.
<table>
  <tr>
    <th>Task</th>
    <th>Dataset</th>
    <th>Instruction</th>
    <th>Train Number</th>
    <th>Test Number</th>
  </tr>
  <tr>
    <td rowspan="2">Grounding</td>
    <td>RefCOCO<br>RefCOCO+<br>RefCOCOg</td>
    <td>Please provide the bounding box coordinate of the region this sentence describes: &lt;description&gt;</td>
    <td>55k</td>
    <td>31k</td>
  </tr>
  <tr>
    <td></td>
    <td>What is the object in the image?</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Classification</td>
    <td>ImageNet</td>
    <td>Answer the question using a single word or phrase</td>
    <td>129k</td>
    <td>5k</td>
  </tr>
  <tr>
    <td>Image Question Answering (IQA)</td>
    <td>VQAv2</td>
    <td>Answer the question using a single word or phrase</td>
    <td>82k</td>
    <td>107k</td>
  </tr>
  <tr>
    <td>Knowledge Grounded IQA</td>
    <td>ScienceQA</td>
    <td>Answer with the option’s letter from the given choices directly</td>
    <td>12k</td>
    <td>4k</td>
  </tr>
  <tr>
    <td>Reading Comprehension IQA</td>
    <td>TextVQA</td>
    <td>Answer the question using a single word or phrase</td>
    <td>34k</td>
    <td>5k</td>
  </tr>
  <tr>
    <td>Visual Reasoning IQA</td>
    <td>GQA</td>
    <td>Answer the question using a single word or phrase</td>
    <td>72k</td>
    <td>1k</td>
  </tr>
  <tr>
    <td>Blind People IQA</td>
    <td>VizWiz</td>
    <td>Answer the question using a single word or phrase</td>
    <td>20k</td>
    <td>8k</td>
  </tr>
  <tr>
    <td>OCR IQA</td>
    <td>OCR-VQA</td>
    <td>Answer the question using a single word or phrase</td>
    <td>165k</td>
    <td>100k</td>
  </tr>
</table>

Table 4: The statistic of multimodal datasets in CoIN (Chen et al., 2024).

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="8">Accuracy on Each Task</th>
    <th colspan="4">Overall Results</th>
  </tr>
  <tr>
    <th>ScienceQA</th>
    <th>TextVQA</th>
    <th>ImageNet</th>
    <th>GQA</th>
    <th>VizWiz</th>
    <th>Grounding</th>
    <th>VQAv2</th>
    <th>OCR-VQA</th>
    <th>ACC↑</th>
    <th>MAA↑</th>
    <th>BWT↑</th>
    <th>FWT↑</th>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>64.56</td>
    <td>48.15</td>
    <td>11.82</td>
    <td>44.50</td>
    <td>9.57</td>
    <td>0.00</td>
    <td>64.10</td>
    <td>27.50</td>
    <td>33.78</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td><b>67.69</b></td>
    <td><b>66.36</b></td>
    <td><b>53.70</b></td>
    <td>59.30</td>
    <td>36.38</td>
    <td><b>63.10</b></td>
    <td>71.00</td>
    <td>47.80</td>
    <td>41.23</td>
    <td>43.35</td>
    <td>-16.94</td>
    <td>-2.87</td>
  </tr>
  <tr>
    <td>MoELORA*</td>
    <td>31.05</td>
    <td>42.45</td>
    <td>29.57</td>
    <td>55.57</td>
    <td>15.30</td>
    <td><b>40.33</b></td>
    <td>67.75</td>
    <td>47.80</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td><b>73.99</b></td>
    <td>50.91</td>
    <td>29.84</td>
    <td><b>60.03</b></td>
    <td>39.44</td>
    <td>3.93</td>
    <td><b>73.44</b></td>
    <td>55.10</td>
    <td>42.08</td>
    <td>46.83</td>
    <td>-6.26</td>
    <td>-1.15</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td>54.66</td>
    <td>48.82</td>
    <td>29.15</td>
    <td><b>56.31</b></td>
    <td>35.97</td>
    <td>1.30</td>
    <td>55.33</td>
    <td>55.10</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td>68.33</td>
    <td>56.17</td>
    <td>31.98</td>
    <td>57.38</td>
    <td><b>40.63</b></td>
    <td>8.46</td>
    <td>68.78</td>
    <td><b>57.13</b></td>
    <td><b>47.09</b></td>
    <td><b>52.07</b></td>
    <td><b>-1.51</b></td>
    <td><b>0.23</b></td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td><b>68.07</b></td>
    <td><b>53.08</b></td>
    <td><b>30.22</b></td>
    <td>53.16</td>
    <td><b>45.22</b></td>
    <td>1.26</td>
    <td><b>68.58</b></td>
    <td><b>57.13</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 5: Main results on Qwen-VL model. For accuracy of each task of MCIT methods, the first row denotes the results for each task evaluated after its tuning (i.e., \( P_{i,i} \) (\( i \in [1, N] \))) with the best performance highlighted in red, while the second row shows each task’s results after tuning the final task (i.e., \( P_{N,i} \)) with the best ones in blue. For overall results, the bold highlights the best performance. * represents results from our re-implementation, while others are cited from CoIN (Chen et al., 2024).

A Datasets

Detailed statistics for the 8 datasets in the CoIN benchmark (Chen et al., 2024) are presented in Table 4.

B Compared Methods

We evaluate the ProgLoRA by comparing with these methods: (1) Zero-shot: Evaluating each task directly using pre-trained MLLMs without any fine-tuning; (2) LoRA (Hu et al., 2022): Sequentially updating knowledge through two low-rank matrices while keeping the pre-trained MLLM parameters intact; (3) MoELORA (Chen et al., 2024): Leveraging multiple independent yet identical LoRA blocks to capture task-specific knowledge across sequential tasks, achieving state-of-the-art performance on the CoIN benchmark; (4) LwF (Li and Hoiem, 2017): Restricting the shared representation layer to remain close to its original state before acquiring the new task; (5) EWC (Kirkpatrick et al., 2017): Finetuning the entire model with a regularization loss that restricts parameter updates to avoid disrupting previously learned tasks.

Note that zero-shot approach primarily reflects the reasoning capability of the MLLM backbone on the CoIN benchmark without task-specific adaptation, serving as a lower bound for MCIT methods. we aim for ProgLoRA’s overall results to surpass zero-shot.
<table>
  <tr>
    <th>Task</th>
    <th>ScienceQA</th>
    <th>TextVQA</th>
    <th>ImageNet</th>
    <th>GQA</th>
    <th>VizWiz</th>
    <th>Grounding</th>
    <th>VQAv2</th>
    <th>OCR-VQA</th>
  </tr>
  <tr>
    <td>ScienceQA</td>
    <td>75.78</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>31.40</td>
    <td>51.73</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ImageNet</td>
    <td>40.98</td>
    <td>13.15</td>
    <td>96.70</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GQA</td>
    <td>37.42</td>
    <td>35.53</td>
    <td>6.62</td>
    <td>59.42</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VizWiz</td>
    <td>47.15</td>
    <td>41.69</td>
    <td>5.53</td>
    <td>45.27</td>
    <td>58.88</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Grounding</td>
    <td>59.06</td>
    <td>13.97</td>
    <td>0.25</td>
    <td>32.62</td>
    <td>30.40</td>
    <td>37.50</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VQAv2</td>
    <td>33.27</td>
    <td>50.48</td>
    <td>8.15</td>
    <td>51.75</td>
    <td>28.52</td>
    <td>0.97</td>
    <td>64.22</td>
    <td></td>
  </tr>
  <tr>
    <td>OCR-VQA</td>
    <td>63.09</td>
    <td>38.63</td>
    <td>10.50</td>
    <td>37.38</td>
    <td>43.62</td>
    <td>0.59</td>
    <td>43.15</td>
    <td>60.08</td>
  </tr>
</table>

Table 6: Detailed results of LLaVA-1.5-7B MoELOra in CoIN, cited from (Chen et al., 2024).

<table>
  <tr>
    <th>Task</th>
    <th>ScienceQA</th>
    <th>TextVQA</th>
    <th>ImageNet</th>
    <th>GQA</th>
    <th>VizWiz</th>
    <th>Grounding</th>
    <th>VQAv2</th>
    <th>OCR-VQA</th>
  </tr>
  <tr>
    <td>ScienceQA</td>
    <td>76.27</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>75.72</td>
    <td>60.07</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ImageNet</td>
    <td>74.95</td>
    <td>58.56</td>
    <td>92.65</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GQA</td>
    <td>74.60</td>
    <td>56.08</td>
    <td>88.86</td>
    <td>58.45</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VizWiz</td>
    <td>75.58</td>
    <td>57.38</td>
    <td>89.67</td>
    <td>54.66</td>
    <td>57.33</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Grounding</td>
    <td>74.53</td>
    <td>56.66</td>
    <td>90.20</td>
    <td>54.67</td>
    <td>56.01</td>
    <td>17.68</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VQAv2</td>
    <td>73.58</td>
    <td>57.02</td>
    <td>92.24</td>
    <td>52.45</td>
    <td>56.43</td>
    <td>13.28</td>
    <td>64.45</td>
    <td></td>
  </tr>
  <tr>
    <td>OCR-VQA</td>
    <td>72.70</td>
    <td>57.08</td>
    <td>91.20</td>
    <td>53.12</td>
    <td>56.04</td>
    <td>12.54</td>
    <td>63.64</td>
    <td>63.54</td>
  </tr>
</table>

Table 7: Detailed results of LLaVA-1.5-7B ProgLoRA (static) in CoIN.

C Evaluation Metrics

We use four metrics: Average Accuracy (ACC) to evaluate after training on the final task, Mean Average Accuracy (MAA) to evaluate throughout training process, Backward Transfer (BWT) to quantify the extent of CF, and Forward Transfer (FWT) to evaluate the extent of KT.

We evaluate outputs of MLLMs by performing word-by-word comparisons with ground truths. Given the diverse output formats across tasks, we employ task-specific evaluation metrics.

In line with the CoIN benchmark (Chen et al., 2024), we evaluate performance on the Image Question Answering task, which includes VQAv2, ScienceQA, TextVQA, GQA, VizWiz, and OCR-VQA, by measuring the accuracy of predicted answers against the ground truth, following the method used in LLaVA (Liu et al., 2023a). For classification tasks, we compare the predicted labels with the actual labels. In the referring expression comprehension (grounding) task, we use the Intersection over Union (IoU) metric, which is commonly used for this purpose, to evaluate prediction accuracy. A prediction is considered correct if the IoU between the predicted and ground-truth bounding boxes is greater than 0.5.

D Results on Other MLLM Backbone

We also evaluate our ProgLoRA on the Qwen-VL-7B (Bai et al., 2023) model. Results are shown in Table 5. For the overall results, our method significantly outperforms all baselines, which illustrates the designed task-aware allocation and task recall can solve problems of CF and KT. For accuracy on each task, while the \( P_{i,i} \) for some tasks is not as good as the baselines, it still maintains a leading position in \( P_{N,i} \) for most tasks. This reflects that although Qwen-VL may be more sensitive to the allocation of LoRA blocks as both MoELOra and ProgLoRA exhibit lower \( P_{i,i} \), our method can still effectively reduce interference and forgetting among tasks.

E Detailed Main Results

Detailed results of MoELOra and ProgLoRA on LLaVA-1.5-7B are shown in Tables 6, 7, and 8.
<table>
  <tr>
    <th>Task</th>
    <th>ScienceQA</th>
    <th>TextVQA</th>
    <th>ImageNet</th>
    <th>GQA</th>
    <th>VizWiz</th>
    <th>Grounding</th>
    <th>VQAv2</th>
    <th>OCR-VQA</th>
  </tr>
  <tr>
    <td>ScienceQA</td>
    <td>76.27</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>59.06</td>
    <td>60.78</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ImageNet</td>
    <td>70.92</td>
    <td>52.57</td>
    <td>97.32</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GQA</td>
    <td>51.56</td>
    <td>50.33</td>
    <td>79.68</td>
    <td>61.27</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VizWiz</td>
    <td>65.62</td>
    <td>50.79</td>
    <td>81.17</td>
    <td>48.99</td>
    <td>60.16</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Grounding</td>
    <td>40.08</td>
    <td>47.53</td>
    <td>77.75</td>
    <td>50.61</td>
    <td>48.30</td>
    <td>39.35</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VQAv2</td>
    <td>76.90</td>
    <td>56.79</td>
    <td>73.90</td>
    <td>53.64</td>
    <td>40.63</td>
    <td>35.96</td>
    <td>65.83</td>
    <td></td>
  </tr>
  <tr>
    <td>OCR-VQA</td>
    <td>74.84</td>
    <td>51.83</td>
    <td>83.90</td>
    <td>49.93</td>
    <td>53.87</td>
    <td>31.19</td>
    <td>62.71</td>
    <td>64.44</td>
  </tr>
</table>

Table 8: Detailed results of LLaVA-1.5-7B ProgLoRA (dynamic) in CoIN.

<table>
  <tr>
    <th>Method</th>
    <th>ACC↑</th>
    <th>MAA↑</th>
    <th>BWT↑</th>
    <th>FWT↑</th>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic) w/ \( \phi = 200 \)</td>
    <td>59.09</td>
    <td>62.38</td>
    <td>-6.59</td>
    <td>1.37</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic) w/ \( \phi = 500 \)</td>
    <td>59.14</td>
    <td>62.74</td>
    <td>-6.47</td>
    <td>1.40</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic) w/ \( \phi = 1000 \)</td>
    <td><b>59.66</b></td>
    <td><b>63.23</b></td>
    <td><b>-6.21</b></td>
    <td><b>1.38</b></td>
  </tr>
</table>

Table 9: Overall Results of ProgLoRA (dynamic) with different replay amount. Bold highlights the best performance.

<table>
  <tr>
    <th>Method</th>
    <th>ACC↑</th>
    <th>MAA↑</th>
    <th>BWT↑</th>
    <th>FWT↑</th>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic) w/ \( r = 16 \)</td>
    <td>59.09</td>
    <td>62.38</td>
    <td>-6.59</td>
    <td>1.37</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic) w/ \( r = 64 \)</td>
    <td>61.04</td>
    <td>64.88</td>
    <td>-6.23</td>
    <td>1.39</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic) w/ \( r = 128 \)</td>
    <td><b>62.63</b></td>
    <td><b>65.01</b></td>
    <td><b>-6.09</b></td>
    <td><b>1.43</b></td>
  </tr>
</table>

Table 10: Overall Results of ProgLoRA (dynamic) with different LoRA setting. Bold highlights the best performance.

F Effect of Replay Amount

We provide comparison of ProgLoRA (dynamic) with different replay amount \( \phi \) as shown in Table 9.

G Effect of LoRA Settings

We provide comparison of ProgLoRA (dynamic) with different LoRA Rank \( r \) as shown in Table 10.

H Effect of Task Order

We compared the alphabet order of tasks as shown in Table 11, which proves that changing the order significantly impacts performance of ProgLoRA. This is because the knowledge acquired from earlier tasks can either facilitate or hinder the learning of subsequent tasks in our proposed LoRA pool. The overall performance in the alphabet order drops, which we attribute to the decline in performance on the GQA and Grounding. In contrast, with the original sequential order, since ScienceQA has few training samples, even a single LoRA block is sufficient for effective learning, providing a solid foundation for subsequent tasks. Therefore, the knowledge acquired early in sequential order, due to its higher quality, enables better KT. However, since GQA and Grounding have larger datasets and there is no prior knowledge available for assistance at the start of the alphabet order, their performance is poorer. These results demonstrate that the performance improvement of ProgLoRA is attributed to the effective utilization of acquired knowledge. It’s worth noting that GQA shows significant improvement after completing all tasks in alphabet order, further validating that the proposed dynamic allocation can efficiently allocate LoRA blocks.

Moreover, the task order with the worst performance (Alphabet order) still outperforms other comparison methods (such as MoELoRA) in terms of ACC, MAA, BWT, and FWT. This indicates that, while our method is influenced by task order, its performance does not degrade to a level where it is no longer SOTA. It provides evidence from another perspective of the superiority of our method in mitigating CF and promoting KT.

I Effect of Instruction Diversity

We investigated the sensitivity of ProgLoRA to different instruction templates because tasks in CoIN share overlapping instructions as shown in Table 12. We followed Diverse and 10Type instruction templates in Chen et al. (2024). The list of instruction templates for each task is shown in Table 14. Original: Certain tasks in the CoIN benchmark (Chen et al., 2024) use similar instructions. Diverse: Assigning distinct and specific instruction templates to each task. 10Type: Randomly selecting an instruction template from a pool of 10 unique templates for each task.

Although ACC is similar to those of the original type, the use of Diverse and 10Type templates both increase MAA by exposing it to a wider variety of instructions. Additionally, both
<table>
  <tr>
    <th rowspan="2">Order</th>
    <th colspan="8">Accuracy on Each Task</th>
    <th colspan="4">Overall Results</th>
  </tr>
  <tr>
    <th>ScienceQA</th>
    <th>TextVQA</th>
    <th>ImageNet</th>
    <th>GQA</th>
    <th>VizWiz</th>
    <th>Grounding</th>
    <th>VQAv2</th>
    <th>OCR-VQA</th>
    <th>ACC↑</th>
    <th>MAA↑</th>
    <th>BWT↑</th>
    <th>FWT↑</th>
  </tr>
  <tr>
    <td>Sequential</td>
    <td>76.27</td>
    <td>60.78</td>
    <td>97.32</td>
    <td>61.27</td>
    <td>60.16</td>
    <td>39.35</td>
    <td>65.83</td>
    <td>64.44</td>
    <td><b>59.09</b></td>
    <td><b>62.38</b></td>
    <td><b>-6.59</b></td>
    <td><b>1.37</b></td>
  </tr>
  <tr>
    <td></td>
    <td>74.84</td>
    <td>51.83</td>
    <td>83.90</td>
    <td>49.93</td>
    <td>53.87</td>
    <td>31.19</td>
    <td>62.71</td>
    <td>64.44</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Alphabet</td>
    <td>39.23</td>
    <td>8.11</td>
    <td>90.75</td>
    <td>56.98</td>
    <td>69.37</td>
    <td>56.52</td>
    <td>60.96</td>
    <td>69.95</td>
    <td>49.51</td>
    <td>42.85</td>
    <td>-6.96</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td></td>
    <td>57.03</td>
    <td>4.65</td>
    <td>66.32</td>
    <td>42.42</td>
    <td>68.96</td>
    <td>54.62</td>
    <td>32.18</td>
    <td>69.95</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 11: The results of LLaVA-1.5-7B ProgLoRA (dynamic) about different task orders. Bold highlights the best performance in overall results.

<table>
  <tr>
    <th rowspan="2">Instruction Type</th>
    <th colspan="8">Accuracy on Each Task</th>
    <th colspan="4">Overall Results</th>
  </tr>
  <tr>
    <th>ScienceQA</th>
    <th>TextVQA</th>
    <th>ImageNet</th>
    <th>GQA</th>
    <th>VizWiz</th>
    <th>Grounding</th>
    <th>VQAv2</th>
    <th>OCR-VQA</th>
    <th>ACC↑</th>
    <th>MAA↑</th>
    <th>BWT↑</th>
    <th>FWT↑</th>
  </tr>
  <tr>
    <td>Original</td>
    <td>76.27</td>
    <td>60.78</td>
    <td>97.32</td>
    <td><b>61.27</b></td>
    <td><b>60.16</b></td>
    <td><b>39.35</b></td>
    <td>65.83</td>
    <td>64.44</td>
    <td>59.09</td>
    <td>62.38</td>
    <td>-6.59</td>
    <td>1.37</td>
  </tr>
  <tr>
    <td></td>
    <td>74.84</td>
    <td>51.83</td>
    <td>83.90</td>
    <td>49.93</td>
    <td><b>53.87</b></td>
    <td>31.19</td>
    <td>62.71</td>
    <td>64.44</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Diverse</td>
    <td>76.27</td>
    <td><b>60.95</b></td>
    <td>97.69</td>
    <td>59.88</td>
    <td>58.87</td>
    <td>37.03</td>
    <td>67.34</td>
    <td><b>66.49</b></td>
    <td>59.76</td>
    <td>62.72</td>
    <td>-5.80</td>
    <td>1.32</td>
  </tr>
  <tr>
    <td>10Type</td>
    <td><b>75.50</b></td>
    <td>55.44</td>
    <td>82.37</td>
    <td><b>53.57</b></td>
    <td>52.38</td>
    <td>29.34</td>
    <td>63.01</td>
    <td><b>66.49</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td><b>76.85</b></td>
    <td>60.11</td>
    <td><b>97.95</b></td>
    <td>60.32</td>
    <td>57.21</td>
    <td>38.15</td>
    <td><b>68.93</b></td>
    <td>65.23</td>
    <td><b>59.97</b></td>
    <td><b>63.74</b></td>
    <td><b>-5.62</b></td>
    <td><b>1.41</b></td>
  </tr>
  <tr>
    <td></td>
    <td>75.35</td>
    <td><b>56.26</b></td>
    <td><b>85.62</b></td>
    <td>52.36</td>
    <td>49.14</td>
    <td><b>31.70</b></td>
    <td><b>64.10</b></td>
    <td>65.23</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 12: The results of LLaVA-1.5-7B ProgLoRA (dynamic) about different instruction templates. Meaning of the highlight marker is the same as in Table 1.

<table>
  <tr>
    <th>Method</th>
    <th>Trainable Param.↓</th>
    <th>ACC↑</th>
    <th>MAA↑</th>
    <th>BWT↑</th>
    <th>FWT↑</th>
  </tr>
  <tr>
    <td>MoELORA</td>
    <td>320M</td>
    <td>37.13</td>
    <td>42.76</td>
    <td>-25.91</td>
    <td>-4.01</td>
  </tr>
  <tr>
    <td>ProgLoRA (dynamic)</td>
    <td>42M</td>
    <td><b>59.09</b></td>
    <td><b>62.38</b></td>
    <td><b>-6.59</b></td>
    <td><b>1.37</b></td>
  </tr>
</table>

Table 13: Efficiency comparison of ProgLoRA and MoELORA. Bold highlights the best performance.

Diverse and 10Type templates lead to an increase in BWT because instruction diversity helps mitigate instruction-following degradation by enhancing adaptability. Although changing the template helps improve performance, the variation in results within ProgLoRA remains below 1.5%, and it consistently outperforms the baselines. This shows that ProgLoRA has low sensitivity to different instructions and can reliably extract task-specific knowledge from them.

J Discussion on Efficiency

We compare the amount of trainable parameters of ProgLoRA (dynamic) with MoELORA on LLaVA-1.5-7B, with results presented in Table 13. In ProgLoRA, trainable parameters in LoRA pool amount to 42M (0.25% of total parameters), while MoELORA has 320M trainable parameters (2.06% of total parameters). Therefore, our method significantly reduces the number of trainable parameters, and with the arrival of new tasks, the number of trainable parameters also doesn’t increase.

![Visualization of allocation weights in ProgLoRA (static)](page_1012_1342_388_388.png)

Figure 6: Visualization of allocation weights in ProgLoRA (static).

K Visualization of Allocation Weights

We show the allocation weights after completing the final task (OCR-VQA) in Fig. 6, where the MLLM backbone is LLaVA-1.5-7B and the method is ProgLoRA (static).

L More Cases

Fig. 7 and 8 show more cases from ScienceQA and OCR-VQA after training on the final task.
<table>
  <tr>
    <th>Task</th>
    <th>Original</th>
    <th>Diverse</th>
    <th>10Type</th>
  </tr>
  <tr>
    <td>ScienceQA</td>
    <td>Answer with the option’s letter from the given choices directly</td>
    <td>Answer with the option’s letter from the given choices directly</td>
    <td>Answer with the option’s letter from the given choices directly<br>Select the correct answer from the given choices and respond with the letter of the chosen option<br>Determine the correct option from the provided choices and reply with its corresponding letter<br>Pick the correct answer from the listed options and provide the letter of the selected option<br>Identify the correct choice from the options below and respond with the letter of the correct option<br>From the given choices, choose the correct answer and respond with the letter of that choice<br>Choose the right answer from the options and respond with its letter<br>Select the correct answer from the provided options and reply with the letter associated with it<br>From the given choices, select the correct answer and reply with the letter of the chosen option<br>Identify the correct option from the choices provided and respond with the letter of the correct option<br>From the given choices, the correct answer is [letter] and the corresponding letter of the correct option</td>
  </tr>
  <tr>
    <td>Grounding</td>
    <td>Please provide the bounding box coordinate of the region this sentence describes</td>
    <td>Please provide the bounding box coordinate of the region this sentence describes</td>
    <td>Extract and provide the bounding box coordinates based on the region described in the sentence<br>Please provide the bounding box coordinate for the region mentioned in the sentence<br>Please provide the bounding box coordinate that matches to the region described in the sentence<br>Give the bounding box coordinates as described in the sentence<br>Determine and provide the bounding box coordinates based on the description in the sentence<br>Identity and provide the coordinates of the bounding box described in the sentence<br>Provide the coordinates for the bounding box based on the region described in the sentence<br>Extract and provide the bounding box coordinates as described in the sentence<br>Identity and give the coordinates of the bounding box as described in the sentence<br>Respond to the question with a single word or a short phrase<br>Respond to the question using only one word or a concise phrase<br>Answer the question with a single word or a brief phrase<br>Respond to the question with just one word or a brief phrase<br>Provide your answer in the form of a single word or a concise phrase<br>Respond to the question with just one word or a brief phrase<br>Answer the question using a single word or a concise phrase<br>Provide your response using only one word or a short phrase<br>Respond to the question with a single word or a brief phrase<br>Answer the question with one word or a short phrase<br>Answer your answer in a single word or a short, descriptive phrase<br>Provide your answer using a single word or a brief phrase<br>Describe the content of the image using only one word or a concise phrase<br>Respond to the question with a single word or a short, descriptive phrase<br>Classify the image content using only one word or a brief phrase<br>Give your answer in the form of a single word or a concise phrase<br>Use a single word or a short phrase to categorize the image content<br>Express your answer with one word or a short, descriptive phrase<br>Summarize your response in a single word or a brief phrase<br>Use one word or a short phrase to classify the content of the image</td>
  </tr>
  <tr>
    <td>GQA</td>
    <td>Answer the question using a single word or phrase</td>
    <td>Respond to the question briefly, using only one word or a phrase</td>
    <td>Answer the question using a single word or a concise phrase<br>Provide your response using only one word or a short phrase<br>Respond to the question with a single word or a brief phrase<br>Answer the question with one word or a short phrase<br>Answer your answer in a single word or a short, descriptive phrase<br>Provide your answer using a single word or a brief phrase<br>Describe the content of the image using only one word or a concise phrase<br>Respond to the question with a single word or a short, descriptive phrase<br>Classify the image content using only one word or a brief phrase<br>Give your answer in the form of a single word or a concise phrase<br>Use a single word or a short phrase to categorize the image content<br>Express your answer with one word or a short, descriptive phrase<br>Summarize your response in a single word or a brief phrase<br>Use one word or a short phrase to classify the content of the image</td>
  </tr>
  <tr>
    <td>ImageNet</td>
    <td>Answer the question using a single word or phrase</td>
    <td>Express your answer in a single word or a short, descriptive phrase</td>
    <td>Answer the question using a single word or a concise phrase<br>Provide your response using only one word or a short phrase<br>Respond to the question with a single word or a brief phrase<br>Answer the question with one word or a short phrase<br>Answer your answer in a single word or a short, descriptive phrase<br>Provide your answer using a single word or a brief phrase<br>Describe the content of the image using only one word or a concise phrase<br>Respond to the question with a single word or a short, descriptive phrase<br>Classify the image content using only one word or a brief phrase<br>Give your answer in the form of a single word or a concise phrase<br>Use a single word or a short phrase to categorize the image content<br>Express your answer with one word or a short, descriptive phrase<br>Summarize your response in a single word or a brief phrase<br>Use one word or a short phrase to classify the content of the image</td>
  </tr>
  <tr>
    <td>OCR-VQA</td>
    <td>Answer the question using a single word or phrase</td>
    <td>Condense your answer for each question into a single word or concise phrase</td>
    <td>Answer with the option’s letter from the given choices directly<br>Select the correct answer from the given choices and respond with the letter of the chosen option<br>Determine the correct option from the provided choices and reply with its corresponding letter<br>Pick the correct answer from the listed options and provide the letter of the selected option<br>Identify the correct choice from the options below and respond with the letter of the correct option<br>Choose the right answer from the options and respond with its letter<br>Select the correct answer from the provided options and reply with the letter associated with it<br>From the given choices, select the correct answer and reply with the letter of the chosen option<br>Identify the correct option from the choices provided and respond with the letter of the correct option<br>From the given choices, pick the correct answer and respond by indicating the letter of the correct option<br>Answer the question with just one word or a brief phrase</td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>Answer the question using a single word or phrase</td>
    <td>Capture the essence of your response in a single word or a concise phrase</td>
    <td>Answer the question using only one word or a short, descriptive phrase<br>Provide your answer in the form of a single word or a brief phrase<br>Use a single word or a short phrase to respond to the question<br>Summarize your response in one word or a concise phrase<br>Respond to the question with a single word or a brief phrase<br>Provide your answer in one word or a short, descriptive phrase<br>Answer the question with a single word or a brief, descriptive phrase<br>Capture the essence of your response in one word or a short phrase<br>Capture the essence of your response in a single word or a concise phrase</td>
  </tr>
  <tr>
    <td>VizWiz</td>
    <td>Answer the question using a single word or phrase</td>
    <td>Provide a succinct response with a single word or phrase</td>
    <td>Answer the question using only one word or a concise phrase<br>Respond to the question with a single word or a brief phrase<br>Provide your answer using just one word or a short phrase<br>Respond with one word or a concise phrase<br>Answer the question using only one word or a brief phrase<br>Use a single word or a short phrase to answer the question<br>Provide your answer in the form of one word or a brief phrase<br>Reply to the question using one word or a concise phrase<br>Answer with a single word or a short phrase</td>
  </tr>
  <tr>
    <td>VQAv2</td>
    <td>Answer the question using a single word or phrase</td>
    <td>Answer the question using a single word or phrase</td>
    <td>Answer the question using only one word or a concise phrase<br>Answer the question using just one word or a concise phrase<br>Provide your answer using only one word or a brief phrase<br>Respond to the question with a single word or a short phrase Use a single word or phrase to answer the question<br>Provide an answer using only one word or a brief phrase<br>Answer the question succinctly with one word or a brief phrase<br>Answer the question with just one word or a short phrase<br>Respond to the question using a single word or a concise phrase</td>
  </tr>
</table>

Table 14: The list of different instructions template for each task (Chen et al., 2024).
User:
Which material is this path made of?
A. plastic
B. brick
Answer with the option's letter from the given choices directly.

MoELoRA: plastic ✗   ProgLoRA (static): A ✗   ProgLoRA (dynamic): B ✔

User:
Are the bubbles in soda a solid, a liquid, or a gas?
A. a gas
B. a solid
C. a liquid
Answer with the option's letter from the given choices directly.

MoELoRA: C ✗   ProgLoRA (static): A ✔   ProgLoRA (dynamic): A ✔

User:
This organism is Asimina triloba. It is a member of the plant kingdom. Asimina triloba is commonly called the pawpaw. Pawpaw trees grow in the southeastern part of the United States. They have large, sweet fruit. The fruit is sometimes called a prairie banana.
Does Asimina triloba have cells that have a nucleus?
A. Yes
B. no
Answer with the option's letter from the given choices directly.

MoELoRA: no ✗   ProgLoRA (static): A ✔   ProgLoRA (dynamic): A ✔

User:
Use the graph to answer the question below.
Which month is the hottest on average in Cape Town?
A. April, May, and November
B. December, January, February, and March
C. August, September, October, and November
Answer with the option's letter from the given choices directly.

MoELoRA: B ✔   ProgLoRA (static): A ✗   ProgLoRA (dynamic): C ✗

User:
Look at the models of molecules below. Select the elementary substance.
A. Fluoromethanol
B. tetraphosphorus
C. methane
Answer with the option's letter from the given choices directly.

MoELoRA: C ✗   ProgLoRA (static): A ✗   ProgLoRA (dynamic): C ✗

Figure 7: Comparison between MoELoRA and ProgLoRA on cases from ScienceQA after training on the final task.
User:
What is the title of this book?
When the provided information is insufficient, respond with 'Unanswerable'.
Answer each question using a single word or phrase.

MoELoRA: The Conflict Resolution Training Program \( \times \)
ProgLoRA (static): The Conflict Resolution Training Program: Leader's Manual \( \checkmark \)
ProgLoRA (dynamic): The Conflict Resolution Training Program: Leader's Manual \( \checkmark \)

User:
Is this a comics book?
When the provided information is insufficient, respond with 'Unanswerable'.
Answer each question using a single word or phrase.

MoELoRA: No \( \times \)
ProgLoRA (static): Yes \( \checkmark \)
ProgLoRA (dynamic): Yes \( \checkmark \)

User:
Is this book related to Politics & Social Sciences?
When the provided information is insufficient, respond with 'Unanswerable'.
Answer each question using a single word or phrase.

MoELoRA: No \( \times \)
ProgLoRA (static): No \( \times \)
ProgLoRA (dynamic): Yes \( \checkmark \)

User:
Who is the author of this book?
When the provided information is insufficient, respond with 'Unanswerable'.
Answer each question using a single word or phrase.

MoELoRA: Bartholomew de Las Casas \( \times \)
ProgLoRA (static): Bartolome de las Casas \( \checkmark \)
ProgLoRA (dynamic): Bartolome de las Casas \( \checkmark \)

User:
Who wrote this book?
When the provided information is insufficient, respond with 'Unanswerable'.
Answer each question using a single word or phrase.

MoELoRA: Ettienne Wenger \( \times \)
ProgLoRA (static): Ettiene Wenger \( \times \)
ProgLoRA (dynamic): Ettiene Wenger \( \times \)

Figure 8: Comparison between MoELoRA and ProgLoRA on cases from OCR-VQA after training on the final task.

Reviewer paper summaries: This paper explores the application of Progressive LoRA in enhancing the performance of MLLMs. The authors propose a two-phase training framework to address challenges such as catastrophic forgetting and knowledge transfer in continual learning scenarios. The experimental results show that ProgLoRA outperforms traditional multi-task approaches in various metrics, demonstrating its effectiveness in enhancing model performance and reducing task interference.

This paper introduces ProgLoRA, a novel framework designed for Multimodal Continual Instruction Tuning (MCIT), which enables Multimodal Large Language Models (MLLMs) to incrementally adapt to new tasks without the need for costly retraining.

ProgLoRA ensures task-specific knowledge is isolated, thereby reducing task interference and mitigating CF. The framework introduces two key mechanisms:

Task-aware Allocation: Dynamically selects and fuses relevant LoRA blocks, enabling better KT by leveraging previously acquired knowledge.
Task Recall: Realigns the model with past tasks by replaying data or aligning allocation weights to further mitigate CF.
Experiments were conducted on the CoIN benchmark, which includes 8 diverse multimodal datasets, using two MLLM backbones (LLaVA-1.5 and Qwen-VL). ProgLoRA consistently outperformed MoELoRA, LoRA, and baseline methods .
Summary of strengths: The proposed Progressive LoRA framework introduces an approach to handling catastrophic forgetting and improving knowledge transfer in continual learning settings. This innovation could have significant implications for advancing MLLMs.
The empirical results demonstrate that ProgLoRA outperforms existing methods, indicating its superior capability in managing multiple tasks efficiently.

1. The paper propose a new architecture. ProgLoRA introduces a progressive LoRA pool to isolate task-specific knowledge, minimizing Catastrophic Forgetting (CF). It employs Task-aware Allocation for leveraging prior knowledge and Task Recall to realign with past tasks, addressing both CF and Knowledge Transfer (KT) effectively.

2. Despite using additional LoRA blocks for each task, the framework maintains parameter efficiency by keeping the blocks low-rank. This ensures scalability and computational feasibility compared to existing methods like MoELoRA.

3. ProgLoRA consistently outperformed MoELoRA, LoRA, and baseline methods. on the CoIN benchmark. The paper also includes detailed ablation studies that analyze the contributions of task-aware allocation, task recall, and replay strategies. These studies validate the effectiveness of the proposed components and highlight their individual impacts on performance.

4. The author discusses the differences between continual learning and multi-task learning, noting that multi-task fine-tuning typically serves as a theoretical upper bound for continual fine-tuning. Their proposed ProgLoRA even outperforms traditional multi-task learning in certain scenarios, demonstrating strong experimental results.
Summary of weaknesses: The authors' proposed method, ProgLoRA, demonstrates limited novelty as similar approaches [1] have been previously published. Additionally, the experimental section requires further strengthening and lacks thoroughness, for example, (1) the author utilized LLaVA-1.5-7B and Qwen-VL-7B as the backbones and more fundamental models should be experimented with. (2) whether ProgLoRA always has an advantage over MoELoRA at different numbers of tasks in the training phase.
Ablation studies are missing empirical proof that the approach requires the step-wise approach of freezing components and also in the order in which the freezing is done matters

[1] Yufei Ma, Zihan Liang, Huangyu Dai, et al. MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning. EMNLP2024.

The author has solved all my previous problems.
Comments, suggestions and typos: None

No
Overall assessment scores: 3.5

4