Large Continual Instruction Assistant
Authors: Jingyang Qiao
zhizhong zhang
Xin Tan
Yanyun Qu
Shouhong Ding
Yuan Xie
Venue: ICML 2025 poster (2025)
Keywords: Catastrophic Forgetting
Continual Learning
Instruction Tuning
Exponential Moving Average
Large Foundation Models
Primary area: general_machine_learning->everything_else
TL;DR: A general continual instruction tuning framework

Abstract: Continual Instruction Tuning (CIT) is adopted to continually instruct Large Models to follow human intent data by data. It is observed that existing gradient update would heavily destroy the performance on previous datasets during CIT process. Instead, Exponential Moving Average (EMA), owns the ability to trace previous parameters, which can aid in decreasing forgetting. Nonetheless, its stable balance weight fails to deal with the ever-changing datasets, leading to the out-of-balance between plasticity and stability. In this paper, we propose a general continual instruction tuning framework to address the challenge. Starting from the trade-off prerequisite and EMA update, we propose the plasticity and stability ideal condition. Based on Taylor expansion in the loss function, we find the optimal balance weight can be automatically determined by the gradients and learned parameters. Therefore, we propose a stable-plasticity balanced coefficient to avoid knowledge interference. Based on the semantic similarity of the instructions, we can determine whether to retrain or expand the training parameters and allocate the most suitable parameters for the testing instances. Extensive experiments across multiple continual instruction tuning benchmarks demonstrate that our approach not only enhances anti-forgetting capabilities but also significantly improves overall continual tuning performance. Our code is available at https://github.com/JingyangQiao/CoIN.

Large Continual Instruction Assistant
Jingyang Qiao 12 Zhizhong Zhang'
Xin Tan
Yanyun Qu
Shouhong Ding Yuan Xie
Abstract
ation (Li et al., 2023; Zhu et al., 2023; Achiam et al., 2023; Touvron et al., 2023a). It generally exists in two stages: large scale pre-training and instruction-tuning Instruction tuning is extremely significant due to guiding LFMs (Chen et al., 2024b; Zheng et al., 2023; Touvron et al., 2023b) in following human intent and aligning different modalities (Li et al., 2024; Liu et al., 2023), which seems to be an essential technique enhance the capabilities and controllability of LFMs e.g. LLMs, MILMSs (Jiang al., 2024).
Continual Instruction Tuning (CIT) is adopted to continually instruct Large Models to follow human intent data by data. It is observed that existing gradient update would heavily destroy the performance on previous datasets during CIT process. Instead, Exponential Moving Average (EMA), owns the ability to trace previous pa- rameters, which can aid in decreasing forgetting. Nonetheless, its stable balance weight fails deal with the ever-changing datasets, leading to the between plasticity and stability. In this paper, propose general continual instruc- tion tuning framework to address the challenge. Starting from the trade-off prerequisite and EMA update, we propose the plasticity and stability ideal condition. Based on Taylor expansion in the loss function, find the optimal balance weight can be automatically determined by the gradients and learned parameters. Therefore, we propose a stable-plasticity balanced coefficient to avoid knowledge interference. Based on the semantic similarity of the instructions, we can de- termine whether to retrain or expand the training parameters and allocate the most suitable param- eters for the testing instances. Extensive experi- ments across multiple continual instruction tuning benchmarks demonstrate that our approach not only enhances foti-forgetting capabilities but also significantly improves overall continual tuning performance. Our code is available at https: / //github. wwt.hu...oooooeeeeeeeeeeeeeeeeee httpaauc...ooiigggggggao////////
As knowledge continuously evolves with the development of human society, new instructions are constantly generated, e.g. the emergence of new concepts or disciplines. How to enable existing LFMs to assimilate novel instructions and undergo self-evolution becomes the key challenge (Zhu et al., 2024). To accommodate the new instructions, the most effective strategy is incorporating both old and new instructions for joint training. However, even such relatively lightweight fine-tuning is unaffordable. Furthermore, di- rectly fine-tum te-inunnn these new instructions would destroy the pre-training knowledge, e.g. catastrophic forgetting (Good- fellow et al., 2013; Li et al., 2019; Nguyen al., 2019).
Continual Instruction Tuning (CIT) is proposed address this challenge (Zhang et al., 2023; He et al., 2023; Chen et al., 2024a). Taking MILMS as a case study, previous methods, EProj (He et al., 2023), FwT(Zheng et al., 2024), and CoIN (Chen et al., 2024a), utilize the model expansion framework by continually adding new branches for the novel instructions, which therefore has less impact on the old knowledge. However, they suffer from memory explosion and high computational cost problems. On the other hand, continually full fine-tuning (FFT) downstream datasets with single branch architecture would destroy the pre-trained parameters, and greatly reduce the zero-shot generalization performance of MLLMs (Zhai et al., 2023).
1. Introduction
Considering the essential mechanism parameter update, discover that the gradient update might not be satisfac- tory choice for CIT. First of all, we find that the gradient in- evitably drives the update of parameters toward the optimiza- tion of the new dataset, which causes forgetting. Instead, Exponential Moving Average (EMA) adopts a weighted summation between old and new model parameters, which enjoys the natural advantage of keeping old knowledge. However, it faces the challenge of balancing old and new
Large Foundation Modals (LFMs) have demonstrated re- markable capabilities in multi-task understanding and gener-
'Est China Normal University 'Shanghai Innovation Institute Shhnnhaai AI Laboratory "Xiamen University TTencett YouTu Lab. Correspondence to: Zhizhong Zhang <zzzhang www.eeeessscc.....o.m Yanyun Qu <yyqu@ muu.cdu.nn>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Large Continual Instruction Assistant
Tarpp
2009
(a) Baseline Ours on LLaVA-7B (h) Baseline u.s. Ours 1AV-333 (c) Baseline v.s. Ours Qwen- VL.
Figure 1. Radar chart of comparisons on Final Accuracy (higher is better) between baseline (LoRA Fine-Tune) and ours.
Vional Reasening Task
the number of groups across various instruction templates, is approximately 50% of the number of tasks. Therefore, it can be regarded limited model-expansion method.
Qualion:
Instraction: Answer question msing single word or phease.
Quastion: the pircher wcaring a
Intrrction: Answer the question using single word or phease.
In order to validate our method, for MLLMs, we adopt LL (Liu et al., 2024a) and Qwen-VL (Bai et al., 2023) as the backbone and insert the efficient-tuning param- eters: LoRA (Hu et al., 2022) in the LLM. We continually fine-tune on multimodal instruction datasets and verify per- formances of the tuned model. Furthermore, we also imple- ment our method with the LM-adapted version of T5-small on the NLP continual instruction tuning tasks (Raffel et al., 2020; Zhang et al., 2023). Experimental results consistently demonstrate excellent forgeottrrttnn and continual tuning performance of our method. In summary, the contributions of this paper are as follows:
DRUPALCON
OCR Taken Recognition Tack
Quasiion: small white spell?
Inttaction: Answer question msing single weed or phease.
Yisaal Grounding Task
Indraction: Plaves provide the baunding hox coondirote of the region this sercence describes:
Objuct baiier.
Figure 2. Instruction reuse in multimodal instruction datasets.
knowledge in continual tuning, as a fixed EMA weight can- not adapt to the continuously evolving datasets, e.g. from location identification task to OCR token recognition task or from sentiment analysis task to question answering task. To determine this balance weight, it seems that the gradient actually represents the magnitude or discrepancy between the model and the new instructions.
A Alleviating Catastrophic Forgetting in CIT. Through rig- orous deduction, we propose dynamical EMA weight- adjusting method of reducing catastrophic forgetting in con- tinual instruction tuning.
Generalized Application and Limited Tuning Costs. Our method is model-agnostic and can be easily applied a wide range of CIT methods. Additionally, it costs limited tuning resources, due instruction grouping strategy.
In this paper. we propose a general continual instruction tuning framework to address the catastrophic forgetting in CIT. We use Taylor expansion of loss function formulate the stability and plasticity ideal equations with the EMA update. By employing the Lagrange multiplier method to solve the equations, we provide a theoretical derivation for dynamically updating the weight of EMA in each training iteration, aiming to achieve the optimal balance of stability and plasticity. Furthermore, for different tasks, the instruc- tions may be similar or even identical, as shown in Figure 2. We refer to this phenomenon instruction reuse. Therefore, construct codebook composed of historical instructions and group them according to their semantic similarity. Each group corresponds to set of trainable parameters. During the training phase, we select the trainable parameters corre- sponding to the most semantically similar group with the current instruction and perform CIT. It is worth noting that
State-of-The-Ar Performance. To the best of our knowl- edge, our method shows the best comprehensive continual tuning performance, especially significant enhancements in the performance of the LoRA Fine-tuning baseline (as shown in Figure 1).
2. Related Work
Large Foundation Models. LFMs own strong reasoning and complex contextual understanding capabilities, which can generate text sequences according to the input images and texts. With a frozen large language model and visual tower, BLIP2 (Li al., 2023) bridged the gap between image and text modality through Qformer structure. In- spired by the notable reasoning power of instruction tuning
2

---

Large Continual Instruction Assistant
in LLMs, egg, LLaMA (Touvron et al., 2023a) and GPT (Floridi & Chiriatti, 2020; Achiam al., 2023), LLaVA (Liu et al., 2024b) and MiniGPT4 (Zhu et al., 2023) adopted instruction tuning with only training a linear projection layer to align the image-text modalities. Recently, VA-. (Liu et al., 2024a) and MiniGPT5 (Zheng et al., 2023) re- fined the instructions and improved the performance across a wider range of multi- modaiit datasets.
In Appendix A, we imply that 0 is a weighted sum of (i E (1,1),, and the summation weight is a product of 3 in different iterations. Each update can contribute to the EMA parameters by reviewing the previous parameters, which can make it have excellent stability. While for the traditional gradient update, the gradient only carries novel information, without reviewing the previous one. We further discuss that the performance of the EMA method is greatly affected by the summation weight. A stable EMA weight cannot be applied to flexible and various instruction datasets (see Appendix F). Thus, we are motivated to propose a dynamical update method for EMA weight.
Continual Instruction Tuning Work: (Zhang et al., 2023) first proposed the continual instruction tuning with LLMs. After that, TRACE, another continual instruction tuning benchmark, is designed to evaluate the general ability, in- struction following, and safety for LLMs (Wang et al., 2023). Following them, (He et al., 2023) and (Chen et al., 2024a) designed distinct cross-modality continual instruction tun- ing benchmarks with InstructBLIP and LLaVA, respectively. Specifically, (He et al., 2023) adopted unique linear projec- tion layers for each tuning dataset and utilized key-query driven mechanism to infer the dataset identifier. (Chen et al., 2024a) employed the MOELoRA paradigm, assigning dif- ferent LoRA weights to different datasets based on expert knowledge, Nevertheless, they still faced the issue of forget- ting due to repeated training (Yang et al., 2023).
4. Method
4.1. Proposition of Our Method
Primarily, we propose the following equations simultane- ously achieve the optimal ideal state with EMA update.
Proposition 4.1. (Ideal State). Given an LFM with con- tinual instruction tuning, with its parameters @ and EMA parameters 0", after training on the iteration i, we can de- scribe the ideal new knowledge transferring and the ideal old knowledge protecting as:
3. Preliminary
{ [(0) = =(),
(2)
02 = =0
Continual Instruction Tuning Definition: CIT (Zhang et al., 2023) is defined to leverage LFMs to continually instruction-tune on new datasets without costly re-training. Compared with traditional continual learning, CIT differs in that it pays more attention effectively leveraging the rich natural language instructions to prevent catastrophic forgetting and encourage knowledge transfer. Additionally, each task is an independent dataset. CIT can be described as a set of datasets Teqq = (4,,..,,t by that arrives sequentially. Note that datasets in the stream can be any type and are not restricted to specific categories or domains. Each dataset t2 E Tay has natural language instruction 1", training set D' train and test set D The goal of CIT is to learn a single model f from Taz sequentially.
The first equation of Eq.(2) represents ensuring the model performance on the new dataset with no change of training loss (ideal new knowledge transferring), inspired by the Optimal Brain Surgeon framework (Hassibi et al., 1993; Le- Cun et al., 1989; Frantar Alistarh, 2022; Molchanov et al., 2022). The second equation of (2) represents preserving the model performance on old datasets with no change of model parameters (ideal old knowledge protecting).
From Proposition we can further find that the starting point is modellindependent. Thus our method can be extended to more LFMs, and parameter-efficien tuning paradigms, even more continual learning scenes.
Discussion:
The EMA update has two kinds of parameters 1, one is parameters 0 updating normally with gradient, and the other is EMA parameters 0* updating as:
Brief Review of The EMA Update Policy:
4.2. Self-Adaption Dynamical Update Method
In order to find a dynamical update 3,, and realize Proposi- tion.4.1, start from a Taylor expansion of the loss func- tion L around the individual parameter 0, 2. Our basis is that the gradient, which represents the discrepancy between the parameters and the new knowledge, is generated by the loss function.
0: = BB00 + (1 (1- B,)00,
(L)
where 3r is the EMA weight, t and t is the training iteration. According to (q)(1), performance the current training iteration of 0 is worse than 0, because it only transfers a portion of the new knowledge from 0, to 0;.
0,) (0) ["(O. (B -1))++0(00-0))
C(#) -
2
(3)
Without specific illustration, parameters in this paper refer to trainable parameters.
FFor simplification, we omit the x, in C
3

---

Large Continual Instruction Assistant
To further introduce og in the Taylor expansion, we replace 0 with or, and have:
By solving these equations, we obtain one feasible solu- tion for 3, which minimizes the objective function while satisfying the constraint:
C" (0)) 0.).. (4)
[(0)) - C(o.) - (0))) - (07
2
[)())) +1
Be 1 (f)-8)-))
(12)
Notice that we have omitted the high-order infinitesimal term of O 0.)3- Additionally, we introduce the re- laxation factor A and have the stability constraint that Or 0 + A0. Here, our intuition is from the perspec- tive of EMA parameters update. The relaxation factor A denotes the newly assimilated model parameters. Moving the left item to the right of the equation, we have:
Please refer to Appendix C for the detailed deduction.
Discussion: Based on (q.(1 we can discover that the optimal weight is basically related to the new gradient L and the old parameters 07 proving that the obtained EMA weight 3t can make the trade-off between stability and plas- ticity. Detailed deduction about high-dimensional matrix O is presented in Appendix P.
A 0- 0: = O.
(5)
Start from the stability constraint, combined with (I), we can obtain:
4.3. Two Approximate Optimizations
3 1 Be B: -1
3
6-- 8 8
A0
A6.
(6)
In (q2() the calculation of (0,) involves the inverse of the Hessian matrix, which needs to obtain second-order par- tial derivatives. However, the above calculation is complex, leading to expensive memory and time-consuming, let alone for LLM, which further increases the training burden. Thus, how to approximately express the Hessian matrix without a complex calculation process becomes a challenge and urgently needs to be solved.
Please kindly refer to Appendix B for detailed demonstra- tions. In order to achieve the ideal new knowledge transfer- ring and the ideal old knowledge protecting, we minimize the difference between [(0]) and C(O.). Of and 1- Merg- ing the two minimal situations, we have unified optimal objective function and set up the following minimization problem:
Approximate Optimization Step I: Considering that the Hessian matrix is obtained by partially deriving the gradi- ents, we can approximate the derivative with the quotient of the differentiation. Here we recognize each iteration as a period of parameter update and further simplify the denom- inator. As a result, we have the following approximation equation to estimate the (0.))).
min (C(0))- - -((t)) + 07 -0;-1; s.1.00 + 07-1- - =;
(7)
To further consider the constrained minimization problem, we use the method of Lagrange multipliers, which combines the objective function with the constraint by incorporating the Lagrange multiplier X.
F [(])) [(O)) + 07 - X(D0 - (8)
(O,))) = OC' ('(O,) - (0(---)) O. 0₂-1
(13)
00;
From (1), we can transfer the s.t. equation as:
where the ('t))) represents gradients in the current iteration, and the L'(8,-1) denotes gradients in the last iteration.
AO + 0 - 0- = 20 + 07-1 - 13,07-1 1- + -
= ~0 + (1 -:) 0t).
(9)
O, in (12) represents the individual parameter, which causes the EMA weight Br to be individual parameter-wise. However, the training parameters / always own a high di- mension, leading to huge computational cost. Thus, we propose set the union parameter-wise 8r, egg. one B, for one module layer, and utilize the following method to approximate the Br.
After that, substitute (56)),), (5), (6) and (9) into (57).
= C'(..) Be - 20 + (0))
3,
3r
3e - A0) 50
2
+ NA10 + (1 (,) - -.)]].
(10)
Taking the derivative of the Lagrangian concerning 3r we set it to zero and determine the direction in which the La- grangian is stationary. This condition is essential for finding the optimal solution.
Approximate Optimization Step II: Assuming that the EMA weight of each individual parameter in the same mod- ule layer would not change a lot. Here, we introduce LI- Norm and further approximate the B3 as:
OF
3e (B. - 1)3
(C'( + 1] 0.-1) (0. - - L C'(()_)))
(3: 1)2 A0
L" (0.))02
B
(14)
03,
- -1(0011 - 0²) = 0.
(11)
i represents the whole layer parameters, we have a coarse
4

---

Large Continual Instruction Assistant
MLLM Model Structure
Step Match Initialize
Step Clear Buffer
Yes
Previous Init
Copy
Remove
Q-1 (0,-1)
Vision Encoder
Projection Layer
Iustruction
Buffer
Match
Random Init
Copy
0"
(00,.)
Codebook & Pool
Remove
0 L (0,-1)
Large Language Model
Step Record Buffer
Step 3: EMA Update
Step S: Save Checkpoints
up
PLM Module W Rand
O₁
(0,)
Add
L(00+1 8: (0. OL (0.)
Mateli
down
LoRA Insertion
0₂
((a))
Buffer
Gup
If
Training Param
Frozen Parsm
0,
((,)
Add
8.0% + 8.30
Match
Appena
Figure 3. Overview of the proposed method (Taking MLLMs as an example). It is mainly divided into five steps. Step 1: Match with the codebook and initialize the training parameters. Step 2: Record the gradients and training parameters task l. Step 3: Calculate the EMA weight and update the EMA parameters. Step 4: Clear the former records at task t-1. Step 5: Save the EMA parameters and instructions.
approximation that 0--1= -0;=0 0;-11
discover that:
maximum similarity in. If not, we create new set of train- ing parameters. 4. After training, if we reuse the instruction, we overwrite the parameters checkpoint and append the un- repeated training instructions into the group. If not, we both restart new parameters checkpoint and instruction group.
C'(0,)+1 +1 |>1.
B1 ('(O.) - (':0--))
(15)
Considering that the 3, E (0, 1), thus we adopt the Be 0.99 when Br exceeds the range of (0, 1). where the constant value 0.99 is empirically obtained from experiments.
Discussion: We validate three kinds of instructions and the detailed grouping results can be found in Appendix
Discussion: The motivation for adopting L1-Norm to ap- proximate the 3e is L1-Norm occupies few computation loads compared to other normalization methods. Detailed implementation of our method please refer to Appendix E.
4.5. Overview of Our Method
Our method consists of five steps, as shown in Figure 3. Before training, the initial step is match instructions of the current task with instructions in the codebook. If we find similar instruction group, we utilize the corresponding parameters checkpoint to initialize parameters @ and EMA parameters 6". Otherwise, we randomly initialize parame- ters f and EMA parameters 0* After the training of iteration 4, gradients (0)).) and model parameters Og would be saved and involved the EMA weight calculation process the iteration t 1. Based on (14), we can obtain the adaptation EMA weight 3, With the EMA weight 83, can update the EMA parameters from 07_ to 0ž based on Eq.(1). After the training of iteration f, to reduce the mem- ory burden, we will clear the saved gradients L' C'(()-1) and model parameters 0, After training in each downstream dataset, we only save the EMA parameters 6* and update the instruction codebook. For the detailed algorithm process please kindly refer to Appendix Q.
4.4. Instruction Grouping Strategy
Inspired by I.2P and Eproj (Wang et al., 2022b; He et al., 2023) 3, we design the following strategy to decide whether to retrain or expand the training parameters. We primar- ily build an instruction codebook and training pool. In- structions are grouped in the codebook and each group is associated with a set of training parameters in the pool.
1. Before training new task, we extract the instructions of the entire task and utilize the Term Frequency-Invert Inverse Doc- ument Frequency (TF-IDF) model (Christian et al., 2016) to convert the instruction text into token vectors. Since the TF-IDF a lightweight machine learning model, its com- putational cost is negligible (Detailed introduction can be found in Appendix H). 2. Based on the cosine similarity function, we calculate the similarities between the training instruction tokens and each token of saved instruction in the codebook. 3. If the maximum similarity is larger than the threshold e, we retrain the training parameters corre- sponding the group which is the saved instruction of the
5. Experiments
5.1. Experimental Setup
Implementation: We adopt LLaVA-1.5/Qwen VL (Liu et al., 2024b; Bai et al., 2023) as our backbone with inserted
DDtaaieed comparisons can found Appendix
5

---

Large Continual Instruction Assistant
Table 1. Avg.ACC, Forgetting, and NcW.ACC, performance comparisons between ours and baselines on LLaVA-7B.
Method
Venue
Datasets
Metries Forgenting(()
SclencsQA
TexVVQA
ImageNNtt
GQA
VizWiz
Grounding
VQAV2
OCR-VQA
Avg.ACCC())
New,ACCLt)
Zeru-shot
19.91
2.88
0.33
2.08
0.90
0.00
0.68
0.17
7.12
LoRA Fine-Tuneekie et 2022)
ICLR
21.26
28.74
10.25
36.78
32.45
0.83
42.50
57.08
28.74
57.29
61.36
MoELooRACChen et 2024x)
ArXiv 24
58.92
38.59
8.85
37.10
44.25
2.45
41.40
$5.35
35.86
25.71
58.36
L.RLI & Holem, 2017)
1RAMF16
63.14
39.60
8,00
34.83
14.53
2.48
40.67
62.35
33.31
22.32
52.58
EWCoKikkpuricke al., 2017)
17
8.18
61.02
20.51
MT12hu et al., 2024)
ICML'24
79.63
55,47
35.64
58.70
44.37
32.20
62.21
61.59
53.73
14,03
66.00
PGP 2024ga)
14
K5.17
$6.85
32.26
61.74
49.63
32.74
65.74
62.20
55,77
12.94
67.09
ErrajtHe et al., 2023)
ArXi
78.51
57.53
92.35
44,67
36.59
63.74
57.00
60.79
5,42
Ours
83.38
59,96
97.07
60.31
48.11
39.61
65.85
62,80
64.64
1.93
66,33
Multi-Task
56.77
49.35
95.55
56.65
53.90
30.00
59.50
55.65
57.18
Table 2. Avg.ACC, Forgetting, and New.ACC performance comparisons between ours and baselines on LLaVA-13B.
Method
Venue
ScienceQA
Tex:VQA
ImageNet
GQA
Datasets
VieWaz
Grounding
VQAv2
OCR-VQA
Avg.ACC(1)
Forgetings () New.ACO(t)
Metries
LaRA Tunetitu 2022)
ICLR'22
60.03
41.19
31.03
2,60
46.33
68
32.90
MT12ha et al., 2024)
JCML/24
80.43
00.72
46.70
00.35
49.19
33.16
63.74
65.44
57.47
11.26
67.32
PGPoQiao al., ko)
2CLR221
82.50
60.61
19.15
62.53
19.43
37.37
65.57
65.82
59.13
10.11
67.96
EProj(He et 2023)
ArXiv23
77.65
58.93
92.31
38.27
33.77
64.39
5.84
Ours
83.94
61.40
97.05
62.61
43,99
39.72
66.29
65.78
65.10
2.31
67.12
LoRA (Hu et al., 2022) in the LLM side. For all methods, including ours and other baselines, during the continual in- struction tuning, we freeze the vision encoder and LLM, with only training the projection layer and LoRA. We fol- low the datasets and tuning orders of the CoIN benchmark (Chen et al., 2024a). including ScienceQA (Lu et al., 2022). VAX (Singh et al., 2019), ImageNet (Deng et al., 2009). GQA (Hudson & Manning, 2019), VizWiz (Gurari et al., 2018), 8), Grounding (Kazemzadeh et al., 2014; Mao et al., 2016), VQAv2 (Goyal et al., 2017), and OCR-VQA (Mishra et al., 2019). Experiments on continual instruction tuning with Large Language Models are shown in Appendix M.
and EProj (He et al., 2023) can perform well both in sta- bility and plasticity, while the Avg.ACC still needs to be improved. is highlighted that our method owns the high- est Avg.ACC, the lowest Forgetting and the comparatively higher New.ACC among these methods, which shows that our method can achieve the best trade-off between plasticity and stability.
Furthermore, based on LLaVA-13B, we compare the con- tinual instruction tuning performance in Table 2 on larger MILM. We observe that our method can improve the AvC.ACC (+3.68), the New.ACC (+0.59) and reduce the Forgetting (-3.53) compared with the best of other methods (EProj method), highlighting its superior anti-forgetting and continual tuning capabilities. When compared with other SOTA methods, e.g. PGP (Qiao et al., 2024a), MT (Zhu et al., 2024), and EProj (He et al., 2023), it is also notewor- thy that our method owns the highest Avg.ACC, the lowest Forgetting and the comparatively higher New.ACC among these methods, which shows that our method can achieve the best trade-off between plasticity and stability in larger MILM and lays the foundation for the application of our method stronger MLLM with more parameters.
Compared Methods: We compare our method against nine methods including (1) zero-shot and (2) LoRA Fine-Tune
(Hu et al., 2022) (3) MoELoRA (Chen et al., 2024a); (4) LWF (Li & Hoiem, 2017); (5) EWC (Kirkpatrick et al.,
2017); (6) EProj (He et al., 2023); (7) MT (Zhu et al., 2024);
(8) PGP (Qiao et al., 2024a); (9) Multi-Taste The detailed descriptions of each method can be found in Appendix K.
Evaluation Metries: We follow the most popular protocols for evaluation (Wang et al., 2022b;a; Smith et al., 2023; Qiao et al., 2024b), which are Average Accuracy (Simplified as Avg.Acc), Forgetting, and New Accuracy (Simplified as New.Acc). Please refer to Appendix L for more details.
Additionally, based on Qwen-VI. we extend our evaluation to a wider range of MLLMs and compare the CIT perfor- mance in Table for more comprehensive studies. The experimental results show that our method can also be ef- fective for Qwen-VL, significantly improving its continual instruction tuning and anti-forge ability, which illus- trates that our method owns generalization ability and can be applied to more MLLMs.
5.2. Continual Instruction Tuning Results
Comparison to SOTA: Based on LLaVA-7B, we com- pare the performance in Table 1. We observe that our method can improve the best of other methods (FProj method) by +3.85@ AVC.ACC, -3.49 Forgetting, and +0.79@ NCW.ACC, demonstrating its excellent anti- forgetting and continual tuning ability. Although methods like LWF (Li & Hoiem, 2017) and EWC (Kirkpatrick et al., 2017) can resist forgetting, their plasticity is greatly influ- enced. PGP (Qiao et al., 2024a), MT (Zhu et al., 2024),
5.3. Robust Performance
To further validate the robustness of the proposed method, we adopt two robustness evaluation experiments with varied tuning task orders and distinct instruction types. (I). For the
6

---

Large Continual Instruction Assistant
Table Avg.ACC, Forgetting, and NeW.ACC, performance comparisons between ours and baselines on Qwen-VI.
Datasets
Metrics
Method
SelesceQA
TeuVQQA
ImageNet
GQA
Grounding
VQJv2
OCR-VQA
Avg.ACCr(t
Forgeting(()
New.ACO(t)
LoRA Fine-Tune Tune-Tuntllu et al., 2022)
ICLR222
31,05
12.45
29.57
55.57
15.30
40.33
67.75
47.80
41.23
19.36
58.17
EWC (Kirkparrick 2017)
PNAS'17
64.30
$8.67
44.04
48.04
41.76
52.46
DiPoQiao al., 202
ICLR'4
66.12
41.33
32.16
19.83
36.05
24.22
58.60
13.96
11.07
5.90
48.30
Ours
66.52
59.44
53.56
57.81
39,57
47.44
70.36
50,44
55.64
1.62
56.19
Bi Diteeen kihee
22:90
17.17
20,60
48.11
10
07.61
4298
Reverse Alphileet
Chradda
V0443
OERMOA
Figure 5. Column chart of comparisons of final results on each task between distinct instructions.
Figure 4. Radar chart of comparisons of final results on each task between varied tuning orders.
(2) For distinct instruction types experiment, we employ three types of instructions (Detailed information please refer to Appendix O). The evaluation metrics performance shown in Table 5. We can find that although New.ACC and Forgetting have a small range of fluctuations, but for Avg.ACC, as a comprehensive performance metric, its range of variation is almost invisible. We further research the Final Accuracy of each dataset in various types of instructions, and the results shown in Figure 5. The Final Accuracy of the same dataset in each type looks very close to each other, which also proves the robustness of our method.
varied tuning task orders experiment, we set three different tuning orders (For detailed information please refer Ap- pendix N). The final results of each task are shown in Figure 4 and the evaluation metrics performance
shown in Table
4. We can find that although changing the order of tasks inevitably impacts the results of each task, the overall situ- ation tends to stabilize with no significant fluctuations, as suggested by the similar distributions in Figure 4. In Table 4, because the knowledge acquired from previous tasks can either benefit hinder subsequent training, New.ACC and Forgetting have small range of fluctuations. Avg.ACC, comprehensive performance metric, its range of variation also exists caused by the changing of New.ACC and Forget- ting. Notice here the fluctuations are affected by the intrinsic attribute of tuning sequence, which has no relationship with the specific method.
Based on the above observations, we conclude that the fluc- tuations in New.ACC and Forgetting are caused by changes in instruction type and dataset tuning order. However, our method has strong robustness, which can maintain the Avg.ACC a stable level in each type of training strategy.
5.4. Analysis of Examples
Table Avg.ACC, Forgetting, and NeW.ACC performance com- parisons between varied tuning orders.
In Figure 6, we show three testing instances with outputs from distinet methods and observe that method can revise errors caused by continual instruction tuning. For the GQA instance (as shown in the left part), our method can keep the geographic location recognition capability of MLLMS after tuning. LoRA Fine-Tume seems to lose such ability and mistakes the chair for table. Although MT and EProj can recognize the object, their outputs are still not aligned with the ground truth, i.e. the first letter is not capitalized and the plural form is used. For the ImageNet instance (as shown in the middle part), our method can pre- serve the fine- fine-grained knowledge of MLLMs after tuning, In contrast, LoRA Fine-Tune can only recognize the coarse attribute of the object. MT even appears the hallucination. For the Grounding instance (as shown in the right part),
Metries Forgetting()) New.ACC(t)
Order
Avg.ACC(t)
Origin
64.64
1.93
66.33
Reverse
62.84
2,25
64.81
Alphabet
61.49
2.75
63.90
Table 5. Avg,ACC, Forgetting, and NeW.ACC performance com- parisons between distinct training instructions.
Metries Forgetting())
Type
Avg.ACC(()
New.ACC(1)
Origin
64.64
1.93
66.33
Diverse
64.36
0.45
64.76
10Type
62.92
2.86
65.41
7

---

Large Continual Instruction Assistant
Madel ofter continnaliniing rening inst GQA
Madelaller costional tening trd Imogrhet
Madel after continsal maving fnt Grozsding
trage)
Image)
(Iqpu Image))
Waulk box the lase?
[Ogeel
kascaet roddie storectues
Dipua Queikes]
Chair
[Grondd Trush)
[0-11 63yu22]
Inmeetee]
Anewer using
Insmettee)
Anewer
Like
Dog
E.aKA
(0,1,.00,000 100,0.1,,00000
PMT Ouppsl]
Pit Ouppal]
Pig dag
[MI Chepall
[MA 6AM)
[Ehg Outpal
chare
Oktze]
Poneosiandog
00.6.33..55.330
[ou Osgua]]
Chair
[Cur Osipaa]
Pesscranion
[Dur Outpx)
Figure 6. Visualization of multimodal continual instruction tuning examples, comparison between LoRA, MT, EProj, and ours.
Table 6. Ablation study results for each proposed component.
Metries Forgetting())
Method
Avg.ACC(t)
New.ACC(1)
LoRA Bascline
28.74
37.29
61.36
Stable EMA (3 = 0.99)
48.09
16.24
62.30
Dynamical EMA
55.33
7.04
61.49
Dynamical EMA + Instruction Grouping
64.64
1.93
66.33
our method can protect the visual grounding ability in real scenarios. It is obvious that the loU between output and ground truth is the largest compared to other meth- ods. Additionally, we are surprised to find that our method can spontancously suppress the occurrence of hallucina- tions in the continual instruction tuning. (Zhai et al., 2023) deem that the hallucination in large models related to the catastrophic forgetting in continual tuning. The above view consistent with multiple rounds of dialogue results, which are shown in Appendix D.
improvements. In summary, compared the original LoRA Fine-Tune method, our method greatly enhances the met- ries of model in Avg.ACC (+35.90), New.ACC (+4.97), and reduces Forgetting (-35.36), achieving an optimal balance between plasticity and stability.
6. Conclusion
To enable LFMs to possess the ability of continual instruc- tion tuning and further resist forgetting, propose gen- eral continual instruction tuning framework. Combined with the exponential moving average, the proposed method can protect previous knowledge and incorporate new knowledge at the same time. By solving a set of equations based on the Lagrange multiplier method, we obtain the self-adaption weight of EMA in each update process. Subsequently. two compensation mechanisms further introduced to alle- viate the computational costs. Additionally, based on the instruction grouping strategy, we can retrain the parameters of semantic-sinilat instructions and limitedly expand the parameters of semantic-irrelevant instructions. the testing phase, we also utilize the strategy to match the most suitable parameters for the instances. Experiments on MLLMs and LLMS show that approach not only owns excellent anti- forgetting but also well continual tuning performance. Due to computational resource constraints, our current focus is primarily on the image and text continual instruction tuning, In the future, we aim to extend our method to continual instruction tuning benchmarks with more modalities and more continual tuning scenarios.
5.5. Ablation Study
To validate the efficiency of each component in the proposed method, starting from the LoRA Fine-Tune baseline, we incrementally add the component and compare the continual instruction tuning performances, Results are shown in Table The experimental results demonstrate that each proposed component efficient in enhaneing accuracy and reducing forgetting of MLL.Ms
In order demonstrate the effectiveness of the proposed dynamic EMA update method, compare it with the fixed EMA weight method, where the fixed EMA weight is set to 0.99, as suggested by the experiments. We can observe that, compared to the fixed EMA weight method, the dynamic EMA update method significantly improves the anti-forge anti-forget ability (-9.20@ Forgetting) and contin- ual instruction tuning ability (+7.24@ ACC. Moreover, with the aid of the instruction grouping strategy, both forget- ting resistance (-5.11 @ Forgetting) and continual instruction tuning ability (+9.31 @ Avg.ACC) show further significant
8

---

Large Continual Instruction Assistant
Acknowledgements
Floridi, L. and Chiriatti, M. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-60 2020.
This work is supported by the National Natural Sci- ence Foundation of China (No.6222260,, No.62176092,
No. 2o.U23320343, No.62476090), Natural Science Foun- dation of Shanghai (232R1420400), Shanghai Sailing Program (23YF1410500) CCF-Tencent RAGR20240122 Young Elite Scientists Sponsorship Program by CAST
Frantar, E. and Alistarh, D. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
(YESS20240780).
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic for- getting in gradient-based neural networks. arXiv preprint arXiv: 2013.
Impact Statement
The potential of our method to enhance LFMs' continual instruction tuning performance and mitigate catastrophic forgetting. 2ts relevance to real-world applications, such as lifelong Al assistants and LLM continual evolution. 3.The datasets utilized in this paper are derived from publicly ac- cessible sources. Rigorous data processing procedures have been implemented to ensure the exclusion of any person- ally sensitive information, in full compliance with relevant privacy protection and ethical standards.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Gurari, D., Li, Q. Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 2023.
Hassibi, B., Stork, D. G., and Wolff, G. J. Optimal brain surgeon and general network pruning. In IEEE interna- tional conference on nearal networks, pp. 293-299. IEEE, 1993.
Bai, J., Bai, S., Yang. S., Wang. S., Tan, S., Wang. P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308. 12966, 1(2): 3, 2023.
He, J., Guo, H., Tang, M., and Wang, J. Continual instruc- tion tuning for large multimodal models. arXiv preprint arXiv: 23N1::331116622 161206, 2023.
Chen, C., Zhu, J., Luo, X., Shen, H., Song, J., and Gao, L. Coin: A benchmark of continual instruction tuning for multimodel large language models. Advances in Neu- ral Information Processing Systems, 37:57817-57840, 2024a.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S.,, Wang, L., and Chen, W. Lora: Low-rank adaptation of large lan- guage models. In International Conference on Learning Representations, 2022.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019.
Chen, D., Liu, J., Dai, W., and Wang, B. Visual instruc- tion tuning with polite flamingo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17745-17753, 2024b.
Christian, H., Agus, M. P., and Suhartono, D. Single docu- ment automatic text summarization using term frequency- inverse document frequency (1f-idf). ComTech: Com- puter, Mathematics and Engineering Applications, 7(4): 285-294, 2016
Jiang, Y., Yan, X., Ji, G.-P., Fu, K., Sun, M., Xiong, H., Fan, D.-P., and Khan, F. Efffectivnness assessment of recent large vision vision-language models. Visual Intelligence, 2(1): 2024.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. Referitgame: Referring to objects in photographs of natu- ral scenes. In Proceedings of the 2014 conference on em- pirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Deng. J., Dong, W., Socher, R., L..J., Li, and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.

---

Large Continual Instruction Assistant
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., and Trrabka-BBawwinske. A. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 14(13):3521-3526 2017.
analysis and recognition (ICDAR), pp. 947-952, IEEE, 2019.
Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. Pruning convolutional neural networks for resource effi- cient inference. In International Conference on Learning Representations, 2022.
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Nguyen, C. V., Achille, A., Lam, M., Hassner, T., Ma- hadevan, V., and Soatto, S. Toward understanding catas- trophic forgetting in continual learning. arXiv preprint arXiv: vrriiv/908..0 1908809081.1.. 2019.
Li, C., Gan, Z.., Yang, Z., Yang, J., Li, L.,, Wang. L., and Gao, Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends in Computer Graphics and Vision, 16(1-2)11-214 2024.
Qiao, J., Tan, X., Chen, C., Qu, Y., Peng. Y., and Xie, Y. Prompt gradient projection for continual learning. In The Twelfth International Conference on Learning Representations, 2024a.
Ji Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19744 PMLR, 2023.
Qiao, J., Zhang, Z., Tan, X., Qu, Y., Zhang, W., and Xie, Y. Gradient projection for parameter-efficien continual learning. arXiv preprint arXiv:2405. 10.05.133300 13383, 2024b.
Li, X., Zhou, Y., Wu, T., Socher, R., and Xiong, C. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International con- ference on muchine learning, pp. 3925-3934. PMLR, 2019.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Li, and Hoiem, D. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-247) 2017.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8317-8326, 2019.
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Aligning large multi-modal model with robust instruction tuning. CORR, 2023.
Liu, H., Li, C., Li, Y., and Lee, J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Smith, J. S., Karlinsky, L.., Gutta, V., Cassanne-Bonnllee P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-promne Toda-prompt: Continual decomposed attention- prompting for rehearsal-free continual learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 11909-11919, 2023.
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. Llama: Open and efficient foundation lan- guage models. arXiv preprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 11:2507-25211 2022.
230.1131,, 2023a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. Llama Open foundation and fine-tuned chat models. arXiv preprint mtXiv:2307.09266 2023b.
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A. L., and Murphy, K. Generation and comprehension unam- biguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11-00, 2016.
Wang, X., Zhang, Y., Chen, T., Gao, S., Jin, S., Yang, X., Xi, Z., Zheng, R., Zou, Y., Gui, T., et Trace: A comprehen- sive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762. 2023 2023.
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Ocr-vqa: Visual question answering by reading text in images. In 20/9 international conference on document
Wang. Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y,, Ren, X., Su, G., Perot, V., and Dy, J. Dualprompt:
10

---

Large Continual Instruction Assistant
Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-645 Springer, 2022a.
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, and Pfister, T. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 139- 149, 2022b.
Yang. Y., Cui, Z., Xu, J., Zhong. C., Zheng, W.-S., and Wang, R. Continual learning with bayesian model based fixed pre-trained feature extractor. Visual Intelli- gence, 1(1):,, 2023.
Zhai, Y., Tong. S., Li. X., Cai, M., Qu, Q.. Lee, Y. and Ma, Y. Investigating the catastrophic forgetting in multimodal large language models. In NeurlPS 2023 Workshop Instruction Tuning and Instruction Following, 2023.
Zhang, Z., Fang, M., Chen, L., and Rad, M. N. Citb: A benchmark for continual instruction tuning. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023.
Zheng, Ma, Q., Liu, Z.. Wu, B., and Feng, H. Be- yond forgettng::::::: Multimodal continual instruction tuning with positive forward transfer. arXiv preprint arXiv:2401.09181, 2024.
Zheng, K., He, X., and Wang, X. Minigpt-5: Interleaved vision-and aoan--nd-lamma and-language generation via generative vokens. CORR, 2023.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhaneing vision-language understanding with advanced large language models. In The Twelfth International Conference Learning Representations, 2023.
Zhu, D., Sun, Z.,, Li, Z.., Shen, T., Yan, K., Ding, S., Wu, C., and Kuang, K. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. In Forty-first International Conference on Machine Learn- ing, 2024.

---

Large Continual Instruction Assistant
A. Decomposition of EMA Update
In the EMA update, exists two kinds of parameters, normally parameters 0 and EMA parameters 0%. At iteration 01 is
updated according to Eq.(1) as:
0 = 3100 + -1-3
(16)
Then at iteration 2, by replacing (q.( 82 is updated as:
0 = =-0 + (1 32)02 - 321 + (1 B:)81] + 32)02
= -2.3.00 + 32(1 81)0, + + -11-22)02.
(17)
After that, at iteration 3, by replacing Eq.(1 03 is updated as:
0g 8302 (1 B3)03 0 + 32 (1 -1-11112-0000000 32)02] B3)03
BB833300 3352 81)01 + 33(1 -2)02 + (1 33)00.
(18)
Observing the equation form, based on the method of summarization and induction, we have the following assumption for iteration n 1:
n-1
6--
II
B: 0 + Σ (1- B.)
II
3, 8:
(19)
i=1
2=1
j=++1
Finally, at iteration n, by replacing (1.( 0% is updated as:
n 0, = B
n-
7-1
8 Oi
Σ
(1 3.)
II j=i+1
By 0.] + - -n
1=1
i=1
06
Σ
(1 (1)
Bg 0: + (1 Bn,)
t=1
j=++1
n
B: 0
Σ (1 3a)
By A j=i+1
II
(20)
It can be found that (20) also has the same form as Eq.(1 which means that the assumption is established. Due to utilizing 6g to initialize 6% EMA parameters 0rr can be represented by normally parameter as:
07
II 1=
B. G, + Σ (1
B.)
II B A
(21)
1=1
1m
B. Proof of relationship between 0, 0r and Ae
From s.t. constraint, we have:
A0 0; 01-1
(22)
0-11 = 0g A0.
(23)
Replace 07-1 with 07- - A0 in Eq.(1):
0r = B, A0) + (1 3.)0c.
(24)
Rearrange the above equation and have:
0: 0, 3, 0.) B.D0,
(25)
(1 (0)
0.) B.D0.
(26)
Finally, we can achieve that:
3t A0
B. A0.
0: 0
(27)
3
3r
12

---

Large Continual Instruction Assistant
C. Be Solving Process
With introducing (2) and (6), we can represent 07-1 - 02 as:
3:
A0
- - 0. = 0-- A 8,
(28)
3:
B -
Taking the derivative of the Lagrangian to A and setting it to zero as Eq.(1I), we have:
OF 8 (8.) + (3 - 1)2
8²
+1
(29)
dA0
(B - 1)
Further, substitute (28) and Eq. (29) into (11( and have:
1
3t
8ec
0
A0
L" (0,) -1-
(0,)))
(3: - 1)2
(3 1)3
(3. 1)
L (6 - 11(07-1- -Os)
(30)
(Be 1)2
1
Be
(0
(0:) + (B. 1)2
('()))
(3. - 1)2
(B, 1)3
18
AO
C" (0))
+
(31)
(B, 1)3
3r -
0 -1+ Be
-3t + 32
L" (0,) + A
C'(0, A0 (0,
(32)
(3. 1)²
(3, 1)3
B.-1'
3r
AB
('(0)) +
(0) A02 +
(33)
(3. 1)
(B, 11)
B. -
= A0| (B. - 1) (0.) + 3r C" (0:) +
(34)
(3. 1)2
1.
By observation, we can find one solution that A0- O, which means that 07 07-1 and Br 1. Obviously, it not the global optimal solution due to the lack of updates to EMA parameters.
Then, we can find another solution through the following equation:
3r
= (B. 1) -)
c((,) + (Be 1)2
C" (0.)
+
(35)
3 1
Due to the situation that 3r - 1 has been discussed, we can remove unlimited:
Be
0 L'(O.)
(3--1) 1) (0,) (8))11((00))00+++ +1
(36)
From Eq,(2 can get:
AO (0/_1 - 0.) - 1).
(37)
Substitute (37) into E36(3
0 = C'(0)) + 1 Oz) + 1.
(38)
Finally, we obtain another solution for 3, that:
L'(O.) +1
3, = (0. - 0 ((()))
(39)
13

---

Large Continual Instruction Assistant
C.1. Discussions of 33 Satisty the equation
According to Eq.( have already proved:
(07-1 02) = A0
(40)
31 -
(07-1 (3: 1) = AO.
(41)
Thus, we can achieve the i.e. constraint with the solution as:
A + 1-111 0r; (f (c) 1) + ff 0 (0) 0.) 0g 0
(0 0,) 0, [3,0 (1 3.)0,]
0. + 3.0g 0. 1 0131 0, B,0,-1
(42)
D. Cases of Multiple Rounds of Dialogue
In this section, we test the zero-shot performance of MLLMs continually fine-tuned with our method multiple rounds of dialogue tasks. Images and questions are from (Liu et al., 2024b). To have a comparison, we also test the zero-shot performance of MILMS continually fine-tuned with the baseline on multiple rounds of dialogue tasks.
Visual input example, Funny Image:
llava-lora (baseline)
Human: What is the name of the man in the photo?
Assistant: the man in the photo is named clon musk
Human: Where is the interesting aspect of this picture?
Assistant: the interesting aspect of this picture is the man holding dog
Human: What Elon Musk holding his hand?
Assistant: clon musk holding dog
Human: What kind of dog is clon's hand?
Assistant: puppy
llav-ddema (ours)
Human: What is the name of the man in the photo?
Assistant: Elon Musk
Human: Where is the interesting aspect of this picture?
Assistant: Elon Musk's face
Human: What Elon Musk holding his hand?
Assistant: puppy
Human: What kind of puppy in Elon's hand?
Assistant: Shiba Inu
Figure 7. Case of multiple rounds of dialogue, red marks the rough and low-quality answers generated by the baseline, and blue marks that method can generate fine-grained and high-quality answers. We helieve that this skill is learned from the downstream datasets. Besides that, we also discover that generated answers of our method are more concise and to the point, directly addressing the problem.
14

---

Large Continual Instruction Assistant
Visual input example, Extreme Ironing:
llava-lora (baseline)
Human:
you describe image?
Assistant: image, is standing on the back of ycllow truck, holding clothestine with clothes hanging on He appears washing clothes in the middle of busy street. The truck driving down road, and there are other vehicles around, including taxi car.
There are also several traffic lights visible in the scene, indicating that the area is likely busy intersection. The man focused on his task, despite the bustling environment around him.
Human: Do you think there is anything that goes against human intuition?
Assistant: Yes, the image of man washing clothes in the middle busy street goes against human intuition. Typically, people would expect to see someone washing clothes in more private or secluded area, such as a backyard or luundromat, However, in this case, the man hus chosen to wash clothes in the middle of a busy strect, which is an unusual and unconventional location for such an activity.
llaa-dema (ours)
Human:
Can you describe image?
Assistant: image, man standing on folding chair back of a yellow cab, ironing clothes. taxi driving down a busy city street, surrounded by other vehicles such as cur and a truck. The man appears to focused on task, possibly trying to dry clothes quickly
moving vehicle.
Human: Do you there is anything that goes against human intuition? Assistant: Yes, the image of man ironing clothes in back of
is unusual and goes against human intuition. It not typical secnario one would expect to scc, as ironing clothes is usually done in more comfortable and controlled environment, such as home or laundry room. The mun's decision to iron clothes in the back of a taxi cab suggests that he is trying to save time or that he might be in hurry.
Figure 8. Case II of multiple rounds of dialogue, red marks the hallucinations existing in the sentences generated by the baseline, and blue marks our method can overcome the occurrence of hallucinations in relevant statements and generate correct answers. Besides that, we also discover that our method can provide reasonable explanations for some unusual hehaviors.
15

---

Large Continual Instruction Assistant
Visual input exumple, Conking Recipe:
llava-lora (baseline)
Human: What are the meals that can cook with these?
refrigerator, you can prepare a varioty of meals. example, you can make fruit using strawberrics bludbaries. You can also use the carrots to prepare lecalilty vegetable slit-liy, soup, or salad. The milk can be used to make smoothic, sauce, creamy soup. The eggs grilled choese sandwich, or cheess-based sauce. The yogurt can be used to make a yoguri parfait, a dip, sauce. The butler can be used to cook. used make dessent topping for other dishes. Humam: would like to the fruit salad. Show me the recipe.
Assistant: With the items
used make fried egg, omelette,
baked dish. Additionally, you
the cheese make pizza,
bake dishes,
postries. ice
Assistant: Sure, here's simple recipe fruit salad: Ingredients:
cup strawberries, bulled and sliced cup bluehenies, rinsed and drained
1/2 cup raspherries, rinsed and drained
1/2 zup grapes, seedless
1/2 cup cantaloupe, diced
1/4 cup honey or sugar (optional)
1:4 cup lemon juice (optional)
1/4 cup chopped fresh mint leaves (optional) Instructions:
In large bowl. combine straw berrisss, blueberries, raspberries, grapes, and cantaloupe.
If d'desired, add honey
taste.
desired, lemon juice the howl for a tangy flavor.
Gently mix the fruit together.
with chopped fresh leaves, desired.
Enjoy your fruit salad!
Hava-dema (ours)
Human: What the meals that with these?
Assistant: With the variety of fruits, vegetables, and milk available in the refrigerntor, you can poepare of healthy and delicions meals. Some ideas include:
I. Fruit salad: Combinc the straberries, blucburris, and other fruits available in the refrigerator. You can add squeces of lemon juice and a
sprinkle of sugar simple, refreshing dessert.
2. Vegetable stir-fry: Chop up the carrots and any other vegetables available the refrigerator. Heat some oil in a and sti--fyy the vegetables chicken, beef) and flavorful sauce.
with your choice protein (e.g.
3. Smoothie: Blend the milk with the blucberrios, strawberics, and any other fruits available in the reffigerater. You can also add some ice and a
scoop protein powder for nutritious and deficious smoothie.
4. Vegetable soup: Use the carrots and any other vegetables available the refrigerator to make vegetable soup. You can add some milk or vegetable broth added flavor nutrition.
5. Omelette: Whisk eggs with milk and season with salt and pepper, Cook the omelette in pan and add the carrots and any other vegetables
available refrigerator
nutritious
filling breakfist.
These are just few examples of the many meals you can prepare using the ingradients available the reftigerntor.
Human: would like to the fruit salad. Show me the recipe.
Assistant: Sure, here's a simple recipe for fruit salad using the ingredients available in the refrigerator:
Ingredients:
Strawberrees
Blucherries
Carrots (optional)
Milk
Lemon juice (optional)
Sugar (optional)
Instrutions::
Wash the berriss and blucherries tboroughly.
Chop straw/errres
bite-sized pieces.
Wash the blueberries and chop them inlo bite-sized pieces.
If you'd
add camrots, wash and chop them well.
In large howl, combine the chopped strawberries, blueherries, and carross (if using).
over thuit mixture If you'd like to add little sweethess, squeeze a lemon wedge over the fruit mixture and sprinkle with sugar. immediately or cefrigerate a hours allow the flavors to together. Enjoy your delicious and refreshing fruit salad!
Pour
stir gently.
Figure 9. Case III of multiple rounds of dialogue, red marks the hallucinations existing in the sentences generated by the baseline, which have not existed in the image. Blue marks that our method can generate answers that are clearer and more aligned with human reading habits.
16

---

Large Continual Instruction Assistant
Visual input example, Webpage Making:
My Joke Website
[ Funny Joke]
[push to reveal purtolnne
llava-lora (baseline)
Human: Pleuse turn this mock-up into an interactive website using html/js, and replace the joke with real joke.
Assistant: I'm sorry, but I am not able to create an interactive website using html/js. However, I can provide you with real joke:
Why was the math book sad?
Because had too many problems.
llava-dema (ours)
Human: Please turn this mock-up into interactive website using html/js, and replace the joke with real joke.
Assistant: Sure, can help you with that! Here's simple HTML/JS mock-up joke website: "hhml
<!DOCTYPE html>
<html>
<head>
<tilleMMy Joke Website</tile>>
</head>
<body>
<h1>My Joke Website<th>>
<h2=-uunyy Joke</h2>
sp>Push to reveal punchline=/p>
<button "ttiiid""eeeelllmcciiee""""""""
Punchlines/button>
Sp id- "puncchinnn""">ipp
<criptt>
function revealPunchlinet)
Ceaumenn.getFlementtee "itprmmeeetllddmmmmliiiiiiiiiiir Munchline"),nnerHHTMM. "Why did the tomato tum red?":
strpppp
<bdyy>
</tml>
This simple mock-up the joke website, which includes a button reveal the punchline. The punchline stored in the 'punchlinc" clement, which hidden by default. When the button clicked, the revealPunchline() function is culled, which the text of the ppunchline' element to "Why did the tomato turn red?".
Figure 10. Case IV of multiple rounds of dialogue, after continually fine-tuning, the baseline has forgotten the pre-trained knowledge and the ability write HTML/IS code. While our method can protect the previous knowledge and still retain ability webpage code writing after continually fine-tuning.
17

---

Large Continual Instruction Assistant
E. Detailed Implementation
Based on Eq,(14), we continue to further simplify it as:
(c,) + (0, 0(-1)
Bg
(0, - (1-1)) L (0(
((.))) 1] - + 0;- 0l-1)
(0, -0-1)((0((0)
C'(0r-1)
C'(6.) +1
10z-1 07-1l(('(0.) +1]
('(0.) ('(0--1)
(0, 02-1)(f((0.)
L' L'(0--))l
f'(0.) + (0 (((00))1
10 -0-1-01-1lc' 07-1ll('(0)) +1]
('(O,) ('((1-1)
(0. - - (
-
|1+ 1+ (0-1)
10--1-07 (7-1)(((00,) + +1]
(43)
L'(O.) L' (0--1)
(0. 0: (f. ('(O)-)))
Additionally, by observation in experiments, find that | [' + 11l < (())())) L'(6,-1)l leading to:
1+ 1+ || 0 ('(O,) - ('(
(44)
Therefore, Eq.( could be transferred
10-1 - (())) + ~ 1-
0 (('()²) +
8. ~ ||
(45)
(6, - 10-00-1)<'(()) C'(6,-1) (0: (0,-0:-1)( () - L' c'(0-_))
The above is our final result, and we approximate 3r using (45) in implementation.
F. Influence of EMA summation weight
We conduct toy experiment to discuss that distinct stable EMA weights can greatly influence the continual instruction tuning results. We fine-tune on six tasks, including ScienceQA, TextVQA, GQA, VizWiz, VQAv2, and OCRVQA, based on the LLaVA-BB backbone. To have comparison, utilize three EMA weights and the results are shown in Table 7.
Table Continual Instruction Tuning Performance Distinct EMA Weights
EMA Weight
ScienceQA
TexiVQA
GQA
VizWiz
VQAv2
OCRVQA
Avg.ACC
Forgetting
New.ACC
76.37
62.13
62.20
58.60
67.24
63.91
B 0.990
60.76
5.17
65.08
73.69
54.67
59.60
47.26
65,45
63.91
76.28
62.23
61.07
58.00
67.20
63.48
β 0.991
58.83
7.06
64.71
72.88
52.29
58.18
41.84
64.29
63.48
76.47
61.94
61.91
58.09
67.46
65.54
8 0.992
60.69
5,46
65.24
72.88
55.40
59.44
45.40
65.46
65.54
From Table further discover that the performance of the EMA method is greatly affected by the summation weight, and the results of continual instruction tuning obtained under different EMA weights vary significantly.
G. Comparisons Between Our Method With Others
Notice that existing methods e.g. L2P (Wang et al., 2022b) and EProj (He et al., 2023) also own the training parameters (Prompt or Linear Projection Layer) selection mechanism. which are similar to our instruction grouping and parameters expansion strategy. Thus, in this section, we analyze and illustrate the differences between our method and theirs.
18

---

Large Continual Instruction Assistant
L2P owns fixed prompt pool, which consists of key-prompt pairs. During the training and testing stage, L2P adopts a frozen pre-trained Vision-Transforman
extract the image embedding and match it with the keys by using cosine similarity. After matching, it would find the prompts with the top-K similarity. Notice here, besides the training cost of prompt parameters, key parameters also need to be optimized in the training iteration.
EProj proposes a novel task-similarity model expansion, which decides whether to retrain or expand the image-to-text projection layer by measuring the cosine similarity between the current task and history tasks. To be specific, it would collect the mean embeddings of image e(v), instruction e(t). and output e(o) of the entire dataset to comprise the task embeddings. in MLLMs respectively, During the training stage, it also extra trains key parameters to match with the input task embeddings aiming to retrieve the corresponding projection layer at the testing time.
The corresponding embeddings are encoded by the BERT model and the frozen
While for our method, (I), It does not add any extra parameters e.g. key parameters, during the training time. Because we utilize the TF-IDF machine learning model, could save the computation load and time without training extra parameters. (2). During the testing stage, we also utilize the TF-IDF machine learning model to extract the instruction token. Compared to L2P and EProj, which need heavy LLMs to embed the input embeddings, our method small and lightweight. (3). In our codebook, it saves the previous instruction texts ("KB). While for L2P and EProj, they save the key parameters ("MB). Thus, we can observe that our method would occupy less memory space.
H. Term Frequency-Invers Document Frequency
Term Frequency-Invee Inverse Document Frequency (TF-IDF) (Christian et al., 2016) is a machine learning method commonly used in natural language processing and information retrieval to evaluate the importance of word in a document relative to corpus. It combines two components: Term Frequency (TF) and Inverse Document Frequency (IDF). Term Frequency
measures how often term appears in document:
Number of occurrences of term in document d'
TF d)
(46)
Total number of terms in document d Inverse Document Frequency reduces the weight of terms that in many documents, emphasizing terms in fewer documents:
[D]
IDF D) = log 1+ ((d E D: te dHl'
(47)
where DD is total number of documents in the corpus and [[d E D E d)| is number of documents containing term t.
The TF-IDF score is calculated by multiplying TF and IDF:
TF-IDF d, D) = TF(t,d) D).
(48)
TF-IDF is effective for capturing term importance and extracting features for text classification, making it a simple yet powerful tool in text processing. In our code, we utilize the TfidVVectorizer class in sklearn library to tokenize the instruction texts into numerical vector.
Instruction Grouping Results
In this section, we summarize the details of three kinds of instruction templates, as shown in Table 9. The instruction grouping results are shown in Table 10. It can be observed that in the three kinds of instruction templates, our strategy divides the instructions of the eight tasks into 4 groups (both for Origin and 10Type), 5 groups (for Diverse), respectively, which can be deemed as a limited expansion compared to the total number of tasks.
J. Training Details
In the implementation of our method, the codebase is based on CoIN (Chen et al., 2024a). The inserted LoRA in each module layer of LLM has a rank of 128. For each fine-tuning dataset, the training epoch is set to 1, and the initial learning rate and weight decay are configured at 2 and O. The max length of input text is fitted as 2048. Additionally, we adopt gradient checkpoint strategy and mixed precision mode of TF32 and BF16. Furthermore, we also utilize the ZeRO stage: 0 mode of DeepSpeed for training.
19

---

Large Continual Instruction Assistant
K. Compared Methods
LoRA Fine-Tune (Hu et al., 2022) prepends LoRA parameter efficient tuning paradigm into LLM. In the training stage, only trains the linear projector and LoRA parameters, with frozen vision encoder and LLM: MoELoRA (Chen al., 2024a) is based on the LoRA, and the number of experts for each MoE layer is set 2; LWF (Li & Hoiem, 2017) calculates the results of the new dataset samples on both the old and new models. After that, it calculates the distillation loss and adds it to the loss function as regularization penalty term. EWC (Kirkpatrick et al., 2017) considers the change of the training parameters and proposes the specific parameters changing loss as a regularization penalty. PGP (Qiao et al., 2024a) introduces a gradient projection method for efficient parameters, and changes the gradient direction orthogonal to the previous feature subspace. E-Proj (He et al., 2023) is the representative SOTA dynamic modal method, which expands the visual projection layer with the task increasing. MT (Zhu et al., 2024) is the representative SOTA regularization method, which compensates for changes in trainable parameters.
L. Evaluation Metrics
It is worth noting that our judgment of whether the prediction results are correct or not is strictly based on the direct comparison between outputs of MILMs and ground truths, which is defined as Truth Alignment in (Chen et al., 2024a).
Therefore, our judgment criteria would be more stringent.
Average Accuracy (Avg.ACC) is used for averaging the test accuracy of all datasets, which represents the comprehensive performance of continual tuning.
Forgetting (FOR) is utilized to indicate the test accuracy reduction of past datasets after learning the new dataset, which denotes the stability performance.
New Accuracy (New.ACC) is employed to average the test accuracy of new datasets, which means the plasticity performance.
Overall, Average Accuracy, Forgetting, and New Accuracy are generally defined as:
T
Average Accuracy = T
AT.
(49)
i=1
T-1
1
Forgetting
max max(A),))ek.TT - AT,
(50)
T 1
T
1
New Accuracy T
Σ
Aci:
(51)
i=1
where T is the number of datasets, Ar, is the accuracy of i-th dataset on the model trained after T-th dataset, Aj. is the accuracy of i-th dataset on the model trained after j-th dataset, and Ar, is the accuracy of i-th dataset on the model trained after i-th dataset.
M. Continual Instruction Tuning Results on Large Language Model
Dataset. We use the InstrDialog Stream dataset from (Zhang et al., 2023), which consists of 4 tasks from dialogue state tracking, 11 tasks from dialogue generation, and 4 tasks from intent identification, resulting in a total of 19 dialogue tasks.
Model. We use the instruction-tuume T5 model from (Zhang et al., 2023), which has learned to understand some instructions and can act as a good starting point to conduct subsequent learning-
Comparison Methods. (1). L2 and EWC regularization methods, both of which leverage a Fisher information matrix to mitigate forgetting by regularizing the loss function, thereby penalizing changes to the erucial parameters of previous tasks. (2). Replay method, which stores random instances from each task in memory buffer, trains the model jointly on both the new task data and the stored instances. (3). AGEM introduces constraints to ensure that parameter updates do not lead to increased loss on previously learned tasks. These losses are calculated using samples stored memory. (4). AdapterCL remains the pre-trained model frozen, and trains an independent Adapter for each task. (5). For ours, we freeze the pre-trained model and train a set of Adapters. Specifically, we utilize the instruction grouping strategy to decide whether
20

---

Large Continual Instruction Assistant
Table Performance of different methods on the InstrDialog dataset. Means and standard deviations are reported as eangp. means zero-shot performance.
InstrDialog FWT
Method
AR
BWT
FT-no-init
29.62.1
8.0
-10.82.3
AdapterCL
8.10.1
9.40.7
-21.90.9
Init
22.5t
FT-init
35.70.2
18.50.7
-4.60.2
L2
35.60.1
17.50.5
-3.11.2
EWC
34.50.6
16.80.4
-6.81.5
AGEM (10)
33.2u.4
19.10.1
-7.31.0
AGEM (50)
34.90.9
18.11.0
-6.00.9
Replay (10)
38.40.7
23.70.0
- -1.30.5
Replay (50)
40.40.0
22.90.1
1.61.2
Ours
39.80.5
21.30.2
0.80.3
Multi
42.10.6
to retrain or expand the Adapter according to the current task instructions and adopt dynamical EMA update when retraining the Adapter. Notice here, besides AdapterCL and ours, which freeze the LLM and only train the Adapter, others adopt the full fine-tuning (FFT) mode.
Metrics. Following (Zhang et al., 2023), we use the following metrics to measure the continual tuning performance. Let aj, represents the ROUGE-L score obtained by the model on the test set of task t, after has been trained on task ty. We define the following metries:
Average ROUGE-L (AR), evaluates the model's 's average performance across all tasks after completing the training on the task ty:
T
AR
Σ
OT
(52)
T
Forward Transfer (FWT), evaluates the extent to which knowledge from previous tasks facilitates learning new task:
FWT
1.11.
(53)
T
Backward Transfer (BWT), evaluates the impact that continually learning on subsequent tasks has on previous tasks:
1 T-1 T
BWT
Σ 1=1
(ar,- - aj,s).
(54)
is worth noting that a positive BWT value indicates that the performance of previous tasks can be improved by subsequent tasks, whereas a negative value signifies knowledge forgetting.
From Table 8, we can observe that (1). Our method surpasses all other non-replay methods, as well the replay method with memory size of 10, and the AGEM method in terms of the AR metric. While it is only slightly behind the replay method with memory size of 50 (39.8 vs. 40.4). These results demonstrate its strong continual instruction tuning performance, with achieving results comparable to those of replay methods (note that our method is non-replay method).
(2) For the BWT metric, all other non-replay methods own negative values. However, because of the EMA-based ability to review the old knowledge, our method aligns with the replay method with a memory size of 50 (positive BWT values), indicating excellent anti-forgetting capability.
21

---

Large Continual Instruction Assistant
(3) In terms of the FWT metric, since our method achieves optimal balance between stability and plasticity, its performance is highly competitive compared to other methods.
N. Three Types of Tuning Order Sequences
In order to validate the robustness of our method, we adopt the following three types of tuning order.
1). Origin tuning order: ScienceQA, TexIVQA, ImageNet, GQA, VizWiz, Grounding, VQAv2, OCR-VQA.
2). Reverse tuning order: OCR-VQA, VQAv2, Grounding, VizWiz, GQA, ImageNet, TextVQA, ScienceQA.
3). Alphaber tuning order: GQA, Grounding, ImageNet, OCR-VQA, ScienceQA, TextVQA, VizWiz, VQAv2.
O. Three Types of Instruction Templates
In order to validate the robustness of our method, we adopt the following three types of training instructions. For detailed instruction templates please kindly refer to Table
1). Original instruction type: Each task owns only one instruction, and several tasks share the same instructions.
2). Diverse instruction type: Each task owns only one instruction, and different tasks are tailored to distinct instructions.
3). 10Type instruction type: Each task owns around ten instructions, and several tasks share similar instructions.
P. Deduction of high-dimensional O
For high-dimensional ON, we have the following Taylor expansion results:
(06) T (0))
0.) H (
((0) = C(O.)
(0)
(0- 0,)
(55)
To further introduce 0; in the Taylor expansion, we replace with 0;, and have:
(0C)
(0; - θ,) + (0; -0,- H
[(0)) - L(O,)
2 (0; - -
(56)
(0
6=:;
Notice that we have omitted the high-order infinitesimal term of 0(e -,) In this way, can obtain the following minimum optimization goal:
ac a
F'
-(-) - - (0; - (0; θ,) + 07_1) + 7 - 07). (57)
0=0;
The following deduction is similar to the process f, finally, we have:
0L T 00
B. AOT HA0 +
AO
(58)
(Bt - 1)
θ= (B. - 1)2
Br 8--1
Considering that AO = (0;_1 - (3, - 1), we can obtain:
JLY T
0
(0 0,) + -,) (
(59)
oe
1==
Obviously, 3r equals
0C T
B, =
+
(60)
J0
10=0;
can be discovered that (60) has the same form as (1.). Thus, we can draw the conclusion that our method can be expanded to high-dimensional O,.
22

---

Large Continual Instruction Assistant
Table 9. The list of instructions each task.
Original
Divers
10Type
the option's
given choices dimetly
Select the conect nnvwer from the gives thoiees and respod with the
the choscn epriee
Deteriin the conect sption from the pros slad chosees and reply with its concoparaling letter Pick the corrctt auseer from the lsted optonn and jeuvide the letter of the selected option
Answer with the option's given choices directly
the option's
Identty
given choice the options below respond with letter the option
ScienceQA
given
corrct answer respond
letter
choice
choices direcily
Choose the rightt answer from the options and respond with lener
Select the correct answer from the proviced optoos and reply with the leser associated with From the gaven choices, select the correct answer and reply with the lettrr of the chosen option Iéeenify the
option from the choices proviced and respond with lettrr of correct option From givon choioss, pick
respond indicarnng
option
Ansser the question with juet one word of Srief phrase
Use one ward or couuise plaise to repoai the quesiion
Answer using only one wood or short descrigtive phrise
Provide
form of
word beiet
the question using single word or phrase
Capture essence your repprrs in single wond or conciie phrvv
Use single word Summorize your response in onr word concise phmne
phrase respond
TeaVQA
Respod to the quedioo using a single werd or a brief plaase Provide anseer is word or shect, descrijtive plaase Answer the question with single werd brief descrptive phrase Captons
your responae Capture the easence your responic is single werd concise please
plraee
Summarize the obgect in the image in single word or brief phease
Provide the object the Give the object Express object
single word leief plass
image the form single descrptise phone
concise phrase
Amsweer object the image using single word parase
Expies the abject in the image in a single wond ara shoet, deseripive plesse
klntify the type coment the image using one werd concive phrase
NetggNNt
Respaid the objactt is the image with single wend or a shoet, deseripiive plass Deseribe the costent of the image using one word concise please
Express object the phrase categorize image content
short, descriptive phrase
word
Classily image content using only one weed or plaase
Use one word shoet perase to classify the conten of the image
Requrd to the queeics with single word shott plesse Respond question using one word
Answer the question
single word brief please Reqund with word shoet planae
Ausees the quesiio using single parase
Resporal to the questias briefly, using only one word phrase
Provide your miswer in the foom of single word or concise plxase Respond to the quesioon with just one weed brief perase Ansver the
GQA
using single
phrase
your response using one
Resond to quesion with single wonl ara brief phone
Regord to the queeiio wsing just one word or concive plrase Answer the question with one word or short phrase
Answer question
word
Respond the question using only one
concise phrase
Respond to the question with single word or beief phrose Provide your answer miing just one word or shoet phrnse Respand with one ward or concise phrase Answer question
Answer the quesiloa using single phrasee
Provide a succiieet response with single
VieWic
brief phrase
phrase
single
answer question
Provide your anvoer the form one werd ar brief phrise Reply the questio using one word or concise planse
Anseer with single word shert pluis
belef
the quesiion
Ideatiy provide bounding box cocrcinates that match description given
Exrrct and provide the bounding box cocedimate based the region described in the sentence Plaase prowide the hourding hox cosedinsse of the region this sentence deseribes
Find and provide the hounding hox enoedinttes fir the rapios metriened the senience
provide the bounding coondimate region this sentence deserives.
the bounding
Provide the continatss the boundingg hos coneepena
region descrited
Grounding
bot countinate region this semtence describes
bounding
Determing and provide hounding has commlinates losed on deseriytion Mlciify and grovide the condinates of the bounding box deseribed the scotence
seitence
Provide the coondinates for the bounding box based on the region deseribed the sentence
Extrcc and provide the coordinates the bounding described the
Ideatfy give the coordinates
bounding bot described by sentence
Anveer the question bong nngle wond pirase
Ansore the quesiio with single word or brief plane Use word or a short plaase respord to the question the questios using just
the questies uning a single word or parase
quodioo
concise answer to question using only one or boief phrase
VQAY2
word or phrase
using a single
Provide an niswer using only one word or brief plaase
Respond questinn a single
shott phaasc Use single word or phaase to answer the question
Auswer the question succinclly with one word or brief phrase
Answer the question with Respond
word short plaase
using single word concice pirase
"Keepenn ti queiifur
tingle word
pfirisi
Anewer the quosiio using a single wnr or concise plosse
Provade you resporne using only one ward shnnt plaase Use word brief Reply question using word concise phrase Use single word or shoet phrase answer the question
question
the question using single word or phrase
Condense your answer toch question into single word or concise phrose
OCR-YQA
Use singlo word or phrase to answer the quention Provide answer using only une word or brief pirase
Provide
the question
or phaase
Respend question using single Anower the quosion using a single ward or plrne
phrase
23

---

Large Continual Instruction Assistant
thm to codd de hendle cuga tu cour en p.e borth moo
Coorr Cocmaae poe bo Aoch did bor conpe hodd in Soceatee fulor coaee
That and antion de hes woalh the car diou to mo ansend Be Bractee
Brewcen proder cereiattiinnn laaeii the wor les decieeceee we
Clan de foood for coodl mo be cemm
Roverre moto spae
Pour Poorpok
Oar to and for in gr for of aougl westh coree dlenn
Mend th sor draen tor mepre rap sortte com Bhee
Daith to of math evay wodlr cre bo Bloee
Liaa regle wet an Bluu on orpytto lere cree Uicta maaic abo atac the come 1eaut
Table instruction grouping results.
Aggas Barboc
See medee mes wagg sigh ward
Buggnal quuee ME MU west SOOCE poaee
Vi HUUL mODE dun BROC DEE QUEEE
Dooil aad ln mod an aem pick
rn th ofd Ford AOR de Ros
Mener Souuraa So quttan
Benal soou tiie gora a www.ewewaeeeeeeeee.
rigltioaa prom aeout
bo sand
te ueroz cor ind intutt th brac du caron (pan
Seeds We evercc che or coouee ho coltnn fnpppe path Bo enter By camo meee
thee dig you che ceno eenc and Ail Bileer or be Bhocc comooo
Chano Rortae doeer or fooped di boop
Acve beeeet feed
Kouer with boryg of weee fowm bo poreec dondddedd
Orgg
Dioee
24

---

Large Continual Instruction Assistant
Q. Algorithm
Algorithm Dynamical EMA Updating and Instruction Grouping
Input: Pre-trained LFMs fifm, number of datasets D, number of iterations T, training set {(r,, 7, y 1==1) i=1/t=1 rate 7/, loss function Lz, matching threshold
learning
Output: training parameters pool (fim) - instruction codebook {/')
initialize: (fim)i=1 {/))=-- for d= I, ..., D do
1. Collect the instructions of the current task L. match with I' in codebook, and obtain the maximum cosine similarity
if then
Initialize the current training parameters ften and EMA parameters fimm from the matching parameters.
else end
Initialize the current training parameters ftom from the last EMA parameters firm
for epoch = do
for: .... T do
3. Draw mini-batch B {(x), I v())
for (xy, I w1) in B do
if 2 then
4. Prepend the finn into the fifm and obtain prediction û. ((are( It)).
5. Calculate per batch loss Lg by accumulating (y () and update ferm with optimizer.
6. Record farn and the corresponding gradients L' at iteration
7. Calculate EMA weight 3e according to (q.)4
8.
Update fürm by (1).
9. Clear Jirn and L' at iteration / 1.
else
4. Prepend the fon into the Jifm and obtain prediction û, Sism((are [t))
5. Calculate per batch loss LB by accumulating (v. ú) and update fiom with optimizer.
end
end
end
end
s (then
10. Cover matching parameters in the pool with firm and update the codebook with 4e-
else
Append firm into the pool and update the codebook with Ic-
end
end
25


Reviews:
[{"review_id": "ilUqHWTzks", "reviewer": "Reviewer_cghr", "summary": "This paper presents a novel approach to Continual Instruction Tuning for vision-language models, addressing the problem of catastrophic forgetting. The proposed method is built on Exponential Moving Average (EMA)-based updates, dynamically adjusting weight balance based on Taylor expansion and Lagrange multiplier optimization. Additionally, the paper introduces an instruction grouping strategy to minimize redundancy and improve parameter efficiency. The experimental results demonstrate that the method significantly reduces forgetting and enhances continual instruction tuning performance across various vision-language instruction tuning benchmarks.\n\n## update after rebuttal\nMy original assessment was already supportive, so I will maintain it.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer cghr, thanks for your valuable suggestions. Here are our responses:\n\n**Response 1 (Additional Experiments)**: To address your concerns, we set two diverse task settings: Conventional **Image Continual Classification** and **NLP Continual Classification** to further validate the effectiveness of our method.\n\n(1). We choose two **Prompt Continual Learning (PCL)** methods based on ViT backbone, namely L2P [1] and DualPrompt [2]. The **Image Continual Classification Benchmark** is dividing a whole image classification dataset into several splits and each split is seen as a task. We freeze the ViT backbone and only train the prompts. Based on the original L2P and DualPrompt baselines, we integrate our dynamical EMA update method. The continual learning dataset is 10-Split-CIFAR100 with **Class Incremental Learning (CIL)** setting. The experimental results are shown as:\n\n|**Method**|**Avg.ACC**|**Forgetting**|\n|:-----------------:|:---------:|:------------:|\n|L2P|83.77|6.63|\n|**L2P-Ours**|**86.47**|**4.83**|\n|DualPrompt|86.50|5.77|\n|**DualPrompt-Ours**|**88.47**|**3.19**|\n\nIn conclusion, our method exhibits superior Avg. ACC and enhanced anti-forgetting performance compared to the original L2P and DualPrompt.\n\n(2). We implement our method on the **NLP Continual Classification Benchmark** (MRCC->SST-2->Sentiment140) with **Task Incremental Learning (TIL)** setting, based on the LLM (Mistral 7B [3]). The **NLP Continual Classification** is selecting multiple NLP classification datasets and each dataset is seen as a task. We freeze the LLM and insert trainable LoRA paradigm into each layer of the LLM respectively. For baseline comparisons, we adopt the original LoRA and one state-of-the-art continual learning method, CurLoRA [4]. The experimental results are shown as:\n\n|**Method**|**Avg.ACC**|**Forgetting**|\n|:--------: |:---------:|:------------:|\n|LoRA|60.00|19.00|\n|CurLoRA|82.00|0.00|\n|**Ours**|**88.00**|**0.00**|\n\nIt can be seen that our method exhibits comparable stability while achieving superior plasticity in LLM continual tuning tasks compared to the state-of-the-art method in [4]. Notably, both CurLoRA and our method attain zero forgetting due to the simple TIL setting (knowing the task identifier in the testing stage).\n\nThe above experiments demonstrate that **our method can perform well across diverse task settings** and greatly improve the baselines’ performance. These findings further indicate that our method has strong generalization capabilities.\n\n[1] Learning to prompt for continual learning.\n\n[2] Dualprompt: Complementary prompting for rehearsal-free continual learning.\n\n[3] Mistral 7b.\n\n[4] Curlora: Stable llm continual fine-tuning and catastrophic forgetting mitigation.\n\n**Response 2 (Impact Statement)**: Thanks for your kindly reminding. We apologize for the absence of the impact statement. In the revised version, we have added an Impact Statement section to discuss how our approach contributes to continual instruction tuning in LFMs. Specifically, we emphasize that:\n\n1. This paper presents work whose goal is to advance the field of continual instruction tuning and mitigate catastrophic forgetting in LFMs.\n\n2. Its relevance to real-world applications, such as lifelong AI assistants and MLLM continual evolution.\n\nWe sincerely hope this addition can provide a clearer perspective on the significance and practical implications of our work.\n\n**Response 3 (Multi-Task)**: (1). The multi-task results presented in Table 1 of our paper are directly from the original CoIN paper (Table 2 in [5]). To maintain the authority of the experimental results, we choose to cite the multi-task results from the original report as a reference. (2). The authors of CoIN found that the performance of multi-task is not always higher than that of LoRA continual learning (shown in the following Table, copied from [5]). They explain that “unlike traditional continual learning, where the multi-task model often serves as the upper bound, in CoIN, the performance of the multi-task model is not the best due to the influence of task gaps”. Notably, this phenomenon is consistent with our results. (3). Our idea is that the theoretical upper bound could be understood as the ideal conditions with zero-forgetting, which is represented by the New.ACC metric (The result of training on the single task). While how to choose the upper-bound in continual instruction tuning is still a valuable research topic. \n\n|**Method**|**Backbone**|**Avg.ACC**|\n|:--------:|:--------:|:---------:|\n|LoRA|QWen|41.23|\n|Multi-Task|QWen|41.87|\n|LoRA|MiniGPT|25.45|\n|Multi-Task|MiniGPT|21.50|\n\n[5] Coin: A benchmark of continual instruction tuning for multimodel large language model.\n\nWe sincerely appreciate your careful review of our paper. Your valuable suggestions have greatly enhanced the quality of our manuscript by pointing out areas that required improvement. We are truly grateful for the time and effort you dedicated to improving our work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper improves widely used EMA-based methods for mitigating forgetting via a dynamic weighting strategy based on theoretical analysis.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The paper presents a strong theoretical basis for its EMA-weighted update strategy, providing clear mathematical derivations.\n\nWeaknesses:\n1. The proposed method provides a general improvement to EMA-based approaches and is not limited to vision-language tasks. However, experiments are conducted solely on vision-language tasks. Additional experiments across diverse task settings would better validate the method's effectiveness.\n2. No impact statement is found.", "other_comments_or_suggestions": "I don’t have additional comments.", "questions_for_authors": "1. Can the authors provide more details about the multi-task setup listed in Table 1 and explain why its performance is unexpectedly low, given that the multi-task setting typically serves as a theoretical upper bound for continual learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gO6CtNlxXs", "reviewer": "Reviewer_pe71", "summary": "This paper addresses an important catastrophic forgetting challenge and proposes a solution in the continual instruction tuning. Based on the ideal conditions of balancing plasticity and stability, meanwhile combined with the Exponential Moving Average (EMA) update, the authors adopt the optimization method to obtain the dynamic EMA weight through gradients (new knowledge) and parameters (old knowledge). The dynamic EMA weight successfully generalizes the model to the new dataset while still retaining knowledge on the old dataset during the continual tuning process. Furthermore, the authors propose a new model-expansion method with instruction grouping strategy based on the phenomenon of instruction reuse, which enables lightweight and limited model expansion on complex and variable datasets. The proposed method could be extended to various Large Foundation Models (LFMs), including MLLMs (LLaVA-7B, LLaVA-13B), LLMs (T5). The authors conduct comprehensive experiments on Multimodal Continual Instruction Tuning (MCIT) benchmark, which consists of Visual Question Answering, Image Classification, OCR, Knowledge Grounding, Reading Comprehension, Visual Reasoning, Visual Grounding etc. The experimental results demonstrate that this method outperforms the previous baselines in terms of Forgetting and Avg.ACC evaluation metrics and achieves the new SOTA results.\n\n## update after rebuttal\nThanks to the authors for addressing my concerns and providing additional results. I will keep my score.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer pe71, thanks for your valuable suggestions. Here are our responses:\n\n**Response 1 (Resource Consumption)**: We compare the time consumption between the dynamic EMA update and the fixed EMA method under same experimental settings (adopting LLaVA-7B). We measure the time consumption for each task on 4-NVIDIA H100 GPUs. The results are shown as:\n\n|Method|ScienceQA|TextVQA|ImageNet|GQA|VizWiz|Grounding|VQAv2|OCRVQA|\n|:---------:|:-------:|:-----:|:------:|:-----:|:----:|:-------:|:----:|:-----:|\n|Dynamic EMA|6 min|25 min|69 min|108 min|12 min|80 min|80 min|112 min|\n|Fixed EMA|6 min|23 min|62 min|100 min|11 min|76 min|75 min|102 min|\n\nWe observe that our dynamic EMA update method requires a total of 492 minutes for training across eight tasks. In comparison, the fixed EMA method consumes 455 minutes, indicating that **our method only incurs an 8% increase in training time (37 minutes)**. However, as demonstrated in our experiments, the dynamic EMA update method significantly outperforms the fixed EMA method.\n\nAdditionally, we compare the memory consumption of our method with that of approaches that do not use the instruction grouping strategy. Specifically, our method introduces two additional components: one for instruction groups and another for the corresponding LoRA parameters of different instruction groups. Compared to the LoRA parameters, which occupy approximately 1 GB of memory, the saving load for instruction groups—comprising small sentences of natural language text—is negligible, due to requiring only around 1 KB. For simplicity, we define a complete set of LoRA parameters inserted into the LLM as a unit of 1. Under this definition, the method that does not use the instruction grouping strategy shares a single set of LoRA across all tasks, resulting in a quantified memory saving factor of ×1. Our method, using Origin Instruction Type as an example, extends the LoRA parameters across four groups, leading to a quantified memory saving factor of ×4. Given that LoRA is a type of Parameter-Efficient Fine-Tunings (PEFTs), even with a fourfold increase in storage, the total memory load remains smaller compared to the substantial storage demands of LLMs, which often exceed tens of GB [1].\n\nIn summary, while adopting the dynamic EMA update and instruction grouping strategy introduces additional time and memory consumption, these costs remain manageable and are significantly outweighed by the substantial performance improvements they promote.\n\n[1] Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.\n\n**Response 2 (Fixed EMA)**: Please kindly refer to the **Response 1** for the same question of Reviewer 9qib.\n\n**Response 3 (Instruction Grouping Results on LLM)**: Please kindly refer to the **Response 6** for the same question of Reviewer 4GQq.\n\n**Response 4 (Reference)**: We have incorporated this reference into the Related Work section and discussed it as “After that, TRACE, another continual instruction tuning benchmark is designed to evaluate the general ability, instruction following and safety for LLMs”.\n\n**Response 5 (Terminology)**: We have added clearer explanation and significance of the “plasticity-stability balance” in the Introduction section as “the trade-off dilemma for models to balance the new task learning and old task storing, which is the core of continual instruction tuning”. Besides that, we have checked the whole manuscript to revise other terminologies that have not been explained well.\n\n**Response 6 (Expression)**: We have changed \"knowledge confusion\" in line 31 to \"knowledge interference\", which is a more precise and widely accepted term. Besides that, we have checked the whole manuscript to revise other unsuitable phrases.\n\n**Response 7 (Dynamic EMA)**: We have summarized the values of EMA weights at each iteration on TextQA dataset with LLaVA-7B backbone and Origin instruction type. Due to the rebuttal space limitation, we provide the dynamic EMA weight in the first 10 epochs for the trainable parameters in the first three layers.\n\n|Layers|Epoch 1|Epoch 2|Epoch 3|Epoch 4|Epoch 5|Epoch 6|Epoch 7|Epoch 8 |Epoch 9|Epoch 10|\n|:----:|:-----:|:-----:|:------:|:-----:|:-----:|:------:|:-----:|:------:|:-----:|:------:|\n|1|0.03|0.65|0.85|0.91|0.93|**-0.28**|0.97|0.95|0.97|0.94|\n|2|0.41|0.74|**-0.07**|0.93|0.95|0.97|0.98|0.98|0.94|0.95|\n|3|0.75|0.88|0.84|0.96|0.97|0.98|0.96|**-0.33**|0.96|0.92|\n\nAs we can see that the dynamic EMA weight can really exceed the range of 0-1, which demonstrates our theoretical deduction.\n\nThank you for taking the time to review our manuscript and provide such detailed and constructive suggestion. Your insights have significantly improved our paper, making it much stronger overall. We have carefully considered all your suggestions and made the necessary revisions to ensure the research is more precise and accessible. We deeply appreciate your contribution to enhancing the quality of our work.", "claims_and_evidence": "The paper provides clear and convincing evidence to support the claims made in submission.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "The proofs of the deduction process of the optimal EMA weight and two approximate processes are thoroughly checked. Additionally, the complete proof process and further optimization process in the supplementary materials are also been carefully examined. All processes are accurate and convincing.", "experimental_designs_or_analyses": "In 5. Experiments section, the authors demonstrated the advantages of the proposed method by comparing to different baselines on various MLLMs, such as LLaVA-7B, LLaVA-13B, and Qwen VL, with common metrics Forgetting and Avg.ACC. The conducted experiments are soundness, convincingly showing the method's effectiveness.\n\nAlthough the experimental results are impressive and thorough, further modifications are suggested to improve the content of the paper. Especially with regard to the time and memory consumption of model training, it is recommended to further compare the time consumption caused by adopting the dynamic EMA update with the fixed EMA method, and the memory saving with the method that does not use instruction grouping strategy. Further discussing the above comparisons would enhance the understanding of the method.\n\nThe ablation studies in Table 6 are insufficient as they only compares with the fixed EMA weight of 0.99. Introducing more fixed EMA weight would reduce randomness and increase the credibility.", "relation_to_broader_scientific_literature": "Compared with other continual instruction tuning methods, this paper makes key contributions in the context of LLMs", "essential_references_not_discussed": "The authors are suggested to include more continual instruction tuning references for LLM in the Related Work section, such as TRACE: A comprehensive Benchmark for Continual Learning in Large Language Models", "other_strengths_and_weaknesses": "# Strengths\n\nNovelty: The paper proposes a novel framework to address the continual instruction tuning problem of Large Foundation Models, and its effectiveness has been validated through extensive experiments, demonstrating strong anti-forgetting ability and continual instruction tuning effects. By combining the balance conditions of stability and plasticity with traditional EMA method, the authors derive the optimal solution from an optimization method perspective. This paper offers new ideas and insights for the continual tuning of large models, which is highly inspiring\n\nThe paper conducts experiments across various large foundation models and continual instruction tuning benchmarks. These experiments show that the proposed method demonstrates excellent generalizability ability and can be transferred to much more continual instruction tuning scenarios.\n\nThe approach presented in this paper is independent of both model architecture and dataset. Therefore, this method can be easily applied to a wider range of models in practical scenarios, and further solves the catastrophic forgetting problem that exists among them.\n\nThis paper provides the optimal EMA weight to achieve the plasticity-stability balance from an optimization perspective using the Lagrange multiplier method. Furthermore, by utilizing two optimization methods, the computational load and complexity of the approach are further reduced. The mathematical proofs in the paper are rigorous and reliable, providing a strong theoretical guarantee for the method.\n\n# Weaknesses\nExperiments: Lack of more comparative experiments (See Experimental Designs Or Analyses\n) and more supplementary experiments (See Supplementary Material).\n\nReferences: Literature introduction on LLM continual instruction tuning is insufficient (See Essential References Not Discussed).", "other_comments_or_suggestions": "Some terminologies have not been explained well yet. Provide some explanations and significance of plasticity-stability balance in the introduction, which could increase the readability of the paper.\n\nSome phrasing is not very appropriate, such as \"knowledge confusion\" in line 31 can be changed to \"knowledge interference\".", "questions_for_authors": "The authors demonstrated in the paper that the dynamic EMA weight may exceed the range of 0-1 (in line 244-246). Therefore, providing the values of EMA weights at each iteration on a specific dataset (such as TextQA) would provide more profound insights.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SB0NhmFmeG", "reviewer": "Reviewer_9qib", "summary": "This paper proposes a novel framework to address the catastrophic forgetting in Continual Instruction Tuning (CIT). Starting from the trade-off between the plasticity and stability, the paper introduces an optimal balance weight of Exponential Moving Average (EMA), determined automatically by gradients and learned parameters. Additionally, the framework uses semantic similarity of instructions to decide whether to retrain or expand the model’s training parameters, and allocate the most suitable parameters to testing instances. Extensive experiments across multiple CIT benchmarks (e.g. CoIN, InstrDialog) and Large Foundation Models (LFMs, e.g. LLaVA, QWen-VL, T5) demonstrate that the proposed method not only reduces forgetting but also significantly improves overall continual tuning performance.\n\n\n## update after rebuttal\n\nThe authors have adequately addressed my concerns. My overall recommendation remains accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 9qib, thanks for your valuable suggestions. Here are our responses:\n\n**Response 1 (Fixed EMA)**: To address your concern, we have conducted ablation studies with more fixed EMA weights, *i.e.* 0.993, 0.996 0.999 (as the EWA weight is usually set in [0.990, 0.999]) [1]. The results are shown as:\n\n|EMA Weight|Avg.ACC|Forgetting|New.ACC|\n|:--------------------------------:|:-----:|:--------:|:-----:|\n|0.990|48.09|16.24|62.30|\n|0.993|48.78|17.89|64.44|\n|0.996|51.14|15.56|64.76|\n|0.999|51.81|0.85|49.44|\n|Dynamical EMA|55.33|7.04|61.49|\n|Dynamical EMA+Instruction Grouping|64.64|1.93|66.33|\n\nFrom the Table, our proposed dynamic EMA method consistently achieves the best performance across all three metrics compared to certain fixed EMA weights (e.g., from 0.990 to 0.996). When compared to a fixed EWA weight of 0.999, our method demonstrates superior Avg.ACC and New.ACC but exhibits inferior resistance to forgetting. It is important to clarify that, as described by Equation (1) in our manuscript, a higher EMA weight generally results in lower Forgetting but also lower New.ACC. However, the lower Forgetting does not always indicate better overall performance due to the trade-off between plasticity and stability. In extreme cases, the Avg.ACC metric may be significantly impacted by a reduced New.ACC. As shown in the Table, under the 0.999 EMA weight setting, the New.ACC metric only reaches 49.44, which is considerably lower than the New.ACC values of other methods (others all exceeding 60). In summary, the low Forgetting in the 0.999 EMA weight setting comes at the cost of significantly reduced New.ACC.\n\n**Response 2 (QWen Baselines)**: Please kindly refer to the **Response 1** for the same question of Reviewer 4GQq.\n\n**Response 3 (Table Titles)**: We recognize that some table titles, including Table 6, may be too brief. To improve clarity, we have revised the titles of these tables to include more detailed explanations. For example, we have updated the title of Table 6 to “Ablation study results for each proposed components”.\n\n**Response 4 (L1-Norm)**: We have replaced L1-Norm in our method with L2-Norm and reconducted the whole experiment based on the LLaVA-7B architecture with the Origin instruction type. The results are shown as:\n\n|Normalization|Avg.ACC|Forgetting|New.ACC|\n|:-----------:|:-----:|:--------:|:-----:|\n|L1|64.64|1.93|66.33|\n|L2|61.17|4.08|64.73|\n\nWe can see that the L1-Norm consistently outperforms L2-Norm, regardless of Avg.ACC, Forgetting, or New.ACC metrics, which further enhances our motivations of adopting L1-Norm approximation.\n\n**Response 5 (Phrase)**: We have replaced \"baseline\" with \"LoRA Fine-tuning baseline\" to specify which baseline is being referred to, making the context clearer for readers. The updated sentence likes: \"especially significant enhancements in the performance of the LoRA Fine-tuning baseline (as shown in Figure 1).\"\n\n**Response 6 (Avg.ACC Results For Instruction Template)**: As you have pointed out, this phenomenon is closely related to the type of instruction template. As we presented in the Table 9, the 10Type instruction type owns 10 different instruction templates for each task, which require the MLLM to memorize a larger amount of information. Consequently, the challenge of mitigating forgetting is harder compared to the Origin and Diverse instruction types, which utilize only a single instruction template for each task. This conclusion is further supported by the Forgetting values reported in Table 5 (Origin: 1.93, Diverse: 0.45, 10Type: 2.86). Therefore, due to the increased forgetting, the Avg.ACC of the 10Type instruction type is slightly lower than those observed of the Origin and Diverse instruction types.\n\n**Response 7 (Forgetting Metric)**: In fact, both the Forgetting and BWT metrics can be used to measure catastrophic forgetting. The key distinction lies in their typical applications: the BWT metric is more commonly employed in Task Incremental Learning (TIL), whereas the Forgetting metric is often used in Class Incremental Learning (CIL) [1,2]. In our paper, the choice of evaluation metric is guided by the need to ensure consistency with the benchmarks utilized [3,4]. Therefore, we follow their original metric setting in our evaluation.\n\n[1] Trgp: Trust region gradient projection for continual learning.\n\n[2] Learning to prompt for continual learning.\n\n[3] Coin: A benchmark of continual instruction tuning for multimodel large language model.\n\n[4] Citb: A benchmark for continual instruction tuning.\n\nWe would like to express our sincere thanks for your thorough review and valuable suggestions. Your thoughtful comments have been crucial in identifying areas for improvement, allowing us to refine and enhance the quality of our paper. Thank you again for your effort and dedication in helping us improve our work.", "claims_and_evidence": "Yes, all the claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense.", "theoretical_claims": "I have checked the correctness of all proofs for theoretical claims, including the optimal EMA weight deduction (Eq.(2)-Eq.(15) in the paper and Eq.(28)-Eq.(39) in the supplementary material) and the EMA expansion process (Eq.(16)-Eq.(21) in the supplementary material), etc.", "experimental_designs_or_analyses": "I have checked the soundness and validity of the experimental designs and analyses. The following two issues can be considered to further improve the paper’s quality.\n1. In Table 6, the paper presents an ablation study results. However, it only compares results using a fixed weight of 0.99 to show the superiority of proposed Dynamical EMA method. I suggest adding additional results with other fixed weights and providing more evaluations.\n2. In Table 3, compared baselines conducted on the QWen-VL architecture are insufficient, adding more state-of-the-art method with stronger baselines like PGP could be employed for further studies of the method.", "relation_to_broader_scientific_literature": "Compared to broader scientific literature on large model continual instruction tuning (e.g., EProj [1], CoIN [2]), the main contribution of this paper lies in 1. For the first time, this paper addresses the forgetting problem in continual instruction tuning from the perspective of the trade-off between plasticity and stability in Large Foundation Models, with providing strong and convincing theoretical proof, which offers insightful ideas. 2. This paper proposes a dynamic exponential moving average update strategy to balance the trade-off between plasticity and stability, achieving significant and surprising performance improvements. Additionally, the method can be applicable to various Large Foundation Model architectures, including LLMs and MLLMs (e.g., LLaVA, QWen-VL), demonstrating strong generalization. 3. Based on the phenomenon of instruction reuse and lightweight TF-IDF model, the paper achieves limited model expansion. Compared to other model expansion methods like L2P [3] and EProj [1], it effectively improves training efficiency and reduces memory usage, providing further insights into instruction for continual instruction tuning.\n\n[1] Continual instruction tuning for large multimodal models  \n[2] Coin: A benchmark of continual instruction tuning for multimodel large language model  \n[3] Learning to prompt for continual learning", "essential_references_not_discussed": "The paper has discussed almost all essential references.", "other_strengths_and_weaknesses": "**Strengths**\n\n1.\tThis paper presents extensive experiments across multiple Large Foundation Models, and several benchmark datasets, as well as natural language processing tasks. The experimental results demonstrate that the proposed method outperforms existing state-of-the-art methods in terms of anti-forgetting and continual tuning performance.\n2.\tThis paper proposes a method for dynamically adjusting EMA weights, theoretically deriving the optimal EMA weights to balance stability and plasticity in the continual learning process. Compared to traditional fixed EMA weight methods, the dynamic adjustment mechanism better adapts to continuously changing datasets and significantly mitigates catastrophic forgetting.\n3.\tThis paper proposes a semantic similarity-based instruction grouping strategy that clusters similar instructions using the TF-IDF model and assigns a trainable LoRA set to each instruction group. Compared to existing model expansion methods, this strategy not only limits the size of expansion parameters but also reduces computational costs.\n\n**Weaknesses**\n\n1. Some tables’ title (e.g., Table 6) are too brief; it is recommended to add more detailed explanatory sentences for clarity.\n2. The paper introduces the L1-Norm to approximate $\\beta_t$. However, in line 248-251, the authors only state that the motivation for adopting L1-Norm to approximate the $\\beta_t$ is L1-Norm occupies a few computation loads compared to other normalization methods. Providing experimental comparisons with other representative Normalization method (e.g. L2-Norm) would be more more convincing.", "other_comments_or_suggestions": "1. In line 96-97, it has “especially significant enhancements in the performance of the baseline (as shown in Figure 1)”. Here, the author may clarify the “baseline” to a more detailed “LoRA Fine-tuning baseline”.", "questions_for_authors": "1. In Table 5, it can be observed that the Avg.ACC results for Origin and Diverse instruction types are slightly higher than those for the 10Type instruction type. What might explain this discrepancy? Is it related to the type of instruction template?\n2. In Table 1-3, the authors utilize the Forgetting as the evaluation metric. However, in Table 8, they switch to the BWT metric. Why do the authors choose different evaluation metrics to assess the anti-forgetting ability of MLLMs and LLMs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KEAQxrnZ9f", "reviewer": "Reviewer_4GQq", "summary": "To address catastrophic forgetting in Large Foundation Models (LFMs), this paper proposes a general continual instruction tuning framework. It introduces a dynamic exponential moving average update method to preserve prior knowledge while assimilating new information. For realizing the balance of stability and plasticity in LFMs, this paper derives a self-adaptive EMA weight for each update process. Furthermore, an instruction grouping strategy allows for retraining parameters of semantically similar instructions and selectively expanding those of semantically divergent ones. Experiments on distinct MLLMs and LLMs, including multimodal continual instruction benchmark and language continual instruction benchmark, all demonstrate the framework's strong resistance to forgetting and excellent continual instruction tuning performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 4GQq, thanks for your valuable suggestions. Here are our responses:\n\n**Response 1 (QWen Baselines)**: We newly implemented another two strong baselines: EProj and EWC on the QWen-VL architecture. The results are presented as:\n\n|Method|Venue|ScienceQA|TextVQA|ImageNet|GQA|VizWiz|Grounding|VQAv2|OCRVQA|Avg.ACC|Forgetting|New.ACC|\n|:------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:--------:|:-------:|\n|LoRA|ICLR’22|31.05|42.45|29.57|55.57|15.30|40.33|67.75|47.80|41.23|19.36|58.17|\n|EWC|PNAS’17|64.30|58.67|44.04|57.73|38.16|48.04|66.98|41.76|52.46|8.68|50.67|\n|PGP|ICLR’24|66.42|41.33|32.16|49.83|36.05|24.22|58.60|43.96|44.07|5.90|48.30|\n|EProj|ArXiv’24|63.19|59.28|62.96|55.51|40.69|41.20|66.89|45.30|54.38|2.52|56.59|\n|**Ours**|**-**|**66.52**|**59.44**|**53.56**|**57.81**|**39.57**|**47.44**|**70.36**|**50.44**|**55.64**|**1.62**|**56.19**|\n\nThe results demonstrate that our method still owns the best performance, surpassing other state-of-the-art approaches.\n\n**Response 2 (Resource Consumption)**: Please kindly refer to the **Response 1** for the same question of Reviewer pe71.\n\n**Response 3 (Phrase)**: In the revised version, we have modified the sentence in lines 412-414 as follows: \"Additionally, we are surprised to find that our method can spontaneously suppress the occurrence of hallucinations in continual instruction tuning.\" We also conducted a thorough revision of the paper to identify and eliminate other redundant expressions, ensuring the writing remains clear and precise.\n\n**Response 4 (Terminology)**: We have revised lines 90-92 as follows: \"Our method is model-agnostic and can be easily applied to a wide range of CIT methods.\" Additionally, we carefully reviewed the manuscript to ensure consistent and accurate use of technical terminology throughout the paper.\n\n**Response 5 (Expression)**: To address this, we have clarified the distinction between the \"forgetting metric\" and \"catastrophic forgetting\" by adding the explanation that, in our paper, \"Forgetting\" (with an uppercase \"F\") refers to the \"forgetting metric,\" whereas \"forgetting\" (with a lowercase \"f\") denotes the \"catastrophic forgetting.”\n\n**Response 6 (Instruction Grouping Results on LLM)**: To address your concern, we have presented the instruction grouping results on LLM as follows.\n\n**Group 1**\n\npubmedqa_classification = ... Output '1' if the passage has a defininte objective/aim/goal and output '0' if the passage does not have a definite objective/aim/goal …\n\ndstc3_classification = ... output the price range the user if looking for which can take one of four values: Cheap, Moderate, Expensive and Don't Care …\n\nair_dialogue_classification = ... select the goal of the conversation …\n\ndeal_or_no_dialog_classification = ... answer 'Yes' if both participants agree to the deal, otherwise answer 'No' …\n\ncraigslist_bargains_classification = ... classify the text into one of the labels from the two possible outputs - 'accepted'/'rejected' …\n\n**Group 2**\n\nmutual_multi_turn_dialogue = ... choose the most reasonable option …\n\npersonachat_choose_next = ... Choose one and answer with the text …\n\n**Group 3**\n\ncirca_answer_generation = ... generate an answer that is relevant to the question …\n\nconvai3_sentence_generation = ... read the input, then generate a valid prediction of the user's response to the computer's clarifying question …\n\nair_dialogue_sentence_generation = ... find the answer of the previous dialogue …\n\ndiplomacy_text_generation = ... generate the next message …\n\ncuriosity_dialogs_answer_generation = ... find the dialogue that is basically a response given to a question or an aspect of the user ...\n\nsmcalflow_sentence_generation = ... identify what will be users' command for that reply ...\n\nmulti_woz_user_utterance_generation = ... Generate a language query such that it leads to this reply ...\n\npersonachat_generate_next = ... generate the next utterance in a given dialogue ...\n\ndstc3_answer_generation = ... answer the given question based on the information present in the dialogue ...\n\nconvai3_sentence_generation = ... generate a prediction of what the requester is actually trying to do ...\n\nsmcalflow_sentence_generation = ... return what will be Agent's response/reply for that particular user's command or question ...\n\n**Group 4**\n\nstorycommonsense_motiv_text_generation = ... write the character's motivation by doing a specific job, which is given in the sentence …\n\nIt can be seen that **instructions for 19 tasks are categorized into 4 groups**, which reflects the limited model expansion property of our method.\n\nThanks for your detailed and constructive review. Your review has significantly contributed to the refinement of our work, offering new insights and helping us address important aspects that were previously overlooked. We have made extensive revisions based on your suggestions, resulting in a clearer and more robust paper.", "claims_and_evidence": "All the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and used evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "I checked the correctness of proofs as followings:\n1. Equation (3) to Equation (12) and Equation (28) to Equation (39) are based on Taylor expansion and Lagrange multiplier method, deriving the optimal solution for EMA weights.\n2. Equation (13) to Equation (15) and Equation (43) to Equation (45) employ several well-founded mathematical approximations, which obtain the highly accurate approximate solution, with effectively reducing both complexity and computational cost.", "experimental_designs_or_analyses": "Although the experimental designs in this paper are comprehensive and the analyses are sound, the authors may further enhance the completeness of the experiment from the following perspectives:\n1. In Table 3, considering to include additional state-of-the-art methods would allow for a more comprehensive evaluation of the proposed method on the QWen-VL architecture.\n2. Compared to the method with stable EMA weight, the additional dynamic adjustment time required by the proposed self-adaptive EMA weight remains unknown. It is recommended that the authors compare training time on a specific dataset with stable/dynamic EMA update methods to help readers better assess the efficiency of the proposed method.", "relation_to_broader_scientific_literature": "The paper effectively cites research in related fields, particularly the latest developments in continual instruction tuning for Large Foundation Models. In contrast to the related scientific literature, the primary contributions of this paper are as follows:\n1. Compared to the other former methods, this paper examines the catastrophic forgetting issue in continual instruction tuning from the perspective of the plasticity-stability trade-off in Large Foundation Models, which is the core of continual learning. Thorough rigorous mathematical derivation, the paper presents compelling results.\n2. By exploring and exploiting the phenomenon of instruction reuse, this paper employs lightweight machine learning algorithm (TF-IDF) to cluster semantically similar instructions, enabling limited model expansion. In comparison to other model expansion techniques like L2P [1] and EProj [2], this approach significantly improves training efficiency while reducing computational and memory overhead.\n\n[1] Learning to prompt for continual learning\n[2] Continual instruction tuning for large multimodal models", "essential_references_not_discussed": "This paper has already included all the essential references.", "other_strengths_and_weaknesses": "Strengths\n+ The experimental results in this paper demonstrate significant improvements in both anti-forgetting and continual tuning performance, especially on the LLaVA-7B and LLaVA-13B models, which outperform existing state-of-the-art methods.\n+ The instruction grouping strategy utilizes the TF-IDF model for instruction matching, avoiding complex embedding calculations, reducing the computational burden and computational costs.\n+ The proposed method exhibits good robustness and generality, making it applicable to various continual instruction tuning scenarios and large foundation models.\n\nWeaknesses\n- This paper contains several redundant phrases. Such as in line 412-414, \"Additionally, we are surprised to find that our method can spontaneously suppress the occurrence of hallucinations appearing in the continual instruction tuning,\", the phrase 'appearing in' is redundant and should be revised to: \"Additionally, we are surprised to find that our method can spontaneously suppress the occurrence of hallucinations in the continual instruction tuning.\".\n- This paper contains terminology issues. Such as in line 90-92, \"Our method is model-free and can be easily applied to a wide range of CIT methods.\" should be modified as \"Our method is model-agnostic and can be easily applied to a wide range of CIT methods.\", which would be more accurate.", "other_comments_or_suggestions": "The use of some terms in the paper is not consistent enough, such as \"forgetting\" sometimes referring to \"catastrophic forging\" and sometimes referring to \"forging metric\". The authors are suggested to distinguish between these terms for clarity.", "questions_for_authors": "This paper only provides the instruction reuse results on MLLM architectures (Table 10). While the instruction reuse results on LLM are unknown.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received unanimously positive reviews from the reviewers. The proposed solution is simple yet effective and is validated with several strong experimental results. Most of weaknesses pointed out by the reviewers were on suggesting more experimental results as well as some writing issues. AC recommends the authors to include the additional experimental results in the final version to solidify the paper. The recommendation is Accept.
Decision: Accept (poster)