Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Authors: Wenke Huang
Jian Liang
Zekun Shi
Didi Zhu
Guancheng Wan
He Li
Bo Du
Dacheng Tao
Mang Ye
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Large Language Models
Fine-Tuning
Catastrophic Forgetting
Primary area: deep_learning->foundation_models

Abstract: Multimodal Large Language Model (MLLM) has demonstrated strong generalization capabilities across diverse distributions and tasks, largely due to extensive pre-training datasets. Fine-tuning MLLM has become a common practice to improve performance on specific downstream tasks. However, during fine-tuning, MLLM often faces the risk of forgetting knowledge acquired during pre-training, which can result in a decline in generalization abilities. To balance the trade-off between generalization and specialization, we propose measuring the parameter importance for both pre-trained and fine-tuning distributions, based on frozen pre-trained weight magnitude and accumulated fine-tuning gradient values. We further apply an importance-aware weight allocation strategy, selectively updating relatively important parameters for downstream tasks. We conduct empirical evaluations on both image captioning and visual question-answering tasks using various MLLM architectures. The comprehensive experimental analysis demonstrates the effectiveness of the proposed solution, highlighting the efficiency of the crucial modules in enhancing downstream specialization performance while mitigating generalization degradation in MLLM Fine-Tuning.

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Wenke Huang" !1
Jian Liang" Zekun Shi!
Didi Zhu Guancheng Wan
He Li
Bo Du
Dacheng Tao
Mang Ye 1
Abstract
Parameter Importance Difference
Visual Encoder
Connector
4.32
Multimodal Large Language Model (MLLM) has demonstrated strong generalization capabilities across diverse distributions and tasks, largely due extensive pre-training datasets. Fine-tuning MLLM has become common practice to im- prove performance specific downstream tasks. However, during fine-tuning, MLLM often faces the risk of forgetting knowledge acquired dur- ing pre-training, which can result decline in generalization abilities. To balance the trade-off between generalization and specialization, pro- pose measuring the parameter importance for both pre-trained and fine-tuning distributions, based on frozen pre-traie pre-trained weight magnitude and accumu- lated fine-tuning gradient values. We further apply an importance-awarr weight allocation strategy, selectively updating relatively important parame- ters for downstream tasks. We conduct empirical evaluations on both image captioning and visual question-answering tasks using various MLLM architectures. The comprehensive experimental analysis demonstrates the effectiveness of the pro- posed solution, highlighting the efficiency the crucial modules enhancing downstream special- ization performance while mitigating generaliza- tion degradation in MILM Fine-Tuning.
X
t2 Briefly describe content of image.
Connector
presents high parameter importance difference with unseen dowvssteammdlitriiittion distribution.
OKVQA
Flicke88
LLM
1:57 1388
Gap
Large Language Model
A man wearing hat with bland dreadlacks jugglingg under a blue
12,39
OKVQA
Flickr30k
Figure I: Background and Motivation. Fine-tuning Multimodal Large Language Model (MLLM) on downstream tasks typically involves training (+)) the connector and LLM modules, and freez- ing the visual encoder. We reveal higher parameter im- portance difference (PID) on unseen downstream distributions, egg. Flickr30k, compared to seen upstream distribution, e.g.. OKVQA. PID cos((ut [I [gl)) We utilize the absolute value of the pre-trained weight [w"| and fine-tuning gradients [g] rep- resent the upstream and downstream parameter importance.
demonstrated impressive competency in various vision- understanding tasks (Liu et al., 2023b; Dai et al., 2023; Lin al., 2023a; Liu al., 2023a). MLLM generally fol- lows the paradigm to fuse the pre-trained vision encoder (Radford et al., 2021; Dosovitskiy et al., 2021) into the repre- sentation space of the Large Language Models (LLM), e,g. LLaMA (Touvron et al., 2023) and Vicuna (Chiang et al., 2023), via the connector module (Dai et al., 2023; Liu et al., 2023b; Luo et al., 2024). Considering that Multimodal Large Language Model is optimized on huge-scale and various-type multimodality instruction-follooiing datasets (Lin al., 2014; Singh et al., 2019; Mishra et al., 2019), it brings powerful generalization ability on different related tasks. Despite this, MILMM still performs poorly on down- stream datasets (Lin et al., 2023b; Luo et al., 2023; Zhai et al., 2024; Tang et al., 2024; Lu et al., 2024).
1. Introduction
Recent years have witnessed remarkable progress in Mul- timodal Large Language Model (MLLM), which have
"qqual contribution National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan Uni- versity, Wuhan. China DDepartment of Computer Science and Tech- nology. Zhejiang University. Hangzhou, China "Nayyang Tech- nological University, Singapore. Correspondence to: Mang Ye <yemang whu..du..n.., Bo Du <duboo@ whu.edu.en>>
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
The common practice to fine-tune foundation models

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
on specific tasks to enhance task performance align the model behavior with human expectations (Zhou al., 2024; Han et al., 2024). Specifically, existing solutions normally freeze the visual encoder, focusing solely on connector lay- ers and the I.M component (Su et al., 2023; Cha et al., 2024; Lin et al., 2024). Thus, during the fine-tuning stage, the MLLM gains specialization ability to achieve excep- tional performance on the fine-tuning task. However, since the small fine-tuning dataset does not have sufficient cover- age of the distribution as well as tasks, the fine-tuned model can potentially lose its generality which is acquired through pre-training stage. The effect of deteriorating the model previous generic knowledge upon new learning is a well- documented challenge, referred as catastrophic forgetting (Ratcliff, 1990; McCloskey & Cohen, 1989; French, 1999; Luo et al., 2023; Zhai et al., 2024). Consequently, a pivotal question raises: How to enhance the specialization ability on target tasks while maintaining generalization knowledge for Multimodal Large Language Model Fine-Tuning? ?
is that gradients trajectories directly provides intensity in- formation of the learning signal imposed on each parameter element for optimization objective (Fisher, 1922; Pascanu & Bengio, 2013; Kirkpatrick et al., 2017; Lee et al., 2019; Mirzadeh et al., 2020; Sanh et al., 2020; Song et al., 2024). By comparing the parameter importance rank, we can differ- entiate between generic and task-specific parameters, As il- lustrated in Fig. 1. tuning on the unseen fine-tuning datasets, e.g., Flickr30k, exhibits more pronounced parameter im- portance discrepancy than the seen OK distributions. This observation further supports our motivation to miti- gate catastrophic forgetting in MLLM by considering pa- rameter importance discrepancy. Driven by question ID), we then propose the Importance Selection Mask (ISM) to selectively consolidate or optimize candidate parameters for the target distribution. Specifically, during the network backward pass, we identify and consolidate parameters that exhibit relatively higher importance for general task knowl- edge, while optimizing the remaining elements to enhance task-specific performance. For thorough examination, we conduct experiments on two representative MLLM: VILA (Lin et al., 2023a) and LLaVA (Liu et al., 2023b;a). We fine-tune on two major tasks image-captioning and visual question answering and evaluate the generic knowledge on the pre-trained seen datasets (Goyal et al., 2017; Marino et al., 2019; Hudson & Manning, 2019; Singh et al., 2019).
In our work, we propose a simple yet effective method, Specialization via Importance Discrepancy Evaluation for Refinement, abbreviated SPIDER to simultaneously accommodate task-oriented properties and maintain task- generic knowledge to alleviate the catastrophic forgetting phenomenon. Own to the werrparameterizzd characteris- ties of the deep neural network (Guo et al., 2017; Lakshmi- narayanan et al., 2017), we notice that not all parameters contribute equally to fit the target distribution, which has confirmed soundness in the relative researches (Li et al., 2017: Frankle & Carbin, 2019; Liu et al., 2019; Sung al., 2021; Yin et al., 2023; Sun et al., 2023b). Consequently, we assert that pre-training and fine-tuning distributions ex-
The main contributions are summarized as follows: We focus on addressing the catastrophic forgetting prob- lem in fine-tuning Multimodal Large Language Model (MLLM) for downstream tasks. We reveal that due to the distribution shift between upstream and downstream pat- terns, parameters exhibit varying degrees of importance.
We introduce SPIDER, which measures generalization and specialization based on the behavior of frozen weights and updating gradients. Our method identifies relatively important elements for the downstream task and conduct critical-aware weight allocation on candidate parameters. By selecting and ranking these elements, our approach offers novel solution to effectively tackle the generaliza- tion and specialization dilemma in MILMM
hibit distinct parameter importance degrees for representing generic and specialized knowledge. In line with this, we are motivated to selectively update relatively important pa- rameters for the downstream task while preserving the remaining for generalization and specialization ability.
Driven by this motivation, the above problem becomes more fundamental: I): How to measure parameter importance for generic and specialized knowledge? II): How to selectively maintain and optimize during downstream learning. To respond to question I), we introduce the Importance Dis- crepancy Measurement (IDM) to quantify the parameter importance degree towards the generalized and specialized knowledge. With respect to the prior generic information, we take inspiration from (Han et al., 2015; Frankle & Carbin, 2019; Sun et al., 2023b) and argue that weight magnitude positive correlates with the prediction tendency. Thus, we adopt the MLLM pretrained weight magnitude to mea- sure the parameter importance towards the generalization attitude. Towards the posterior specialized behavior, we uti- lize current gradient norm to pinpoint regions that learn crucial downstream knowledge. The rationale behind this
We conduct a comprehensive analysis on four downstream datasets: Flickr30k (Young et al., 2014), COCO- Copotcaaation (Lin et al., 2014), Science (Lu et al., 2022), and IconQA (Lu et al., 2021), using VILA (Lin et al., 2023a) and I. (Liu et al., 2023b). Along with series of ablation studies, the promising results empirically vali- date the effectiveness of SPIDER in improving fine-tuning performance and mitigating generalization forgetting.
2. Related Works
2.1. Multimodal Large Language Models
With the impressive success of Large Language Models (I.LM), such as GPT (Radford et al., 2019; Brown et al.,
2

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
2020; OpenAI, 2023), LLaMA (Touvron et al., 2023), Vi- cuna (Chiang et al., 2023), PaL.M (Chowdhery al., 2022; Anil et al., 2023), growing interest has been aroused in building end-to-end Multimodal Large Language Model (MILM), e.g., Flamingo (Alayrac et al., 2022), BLIP-2 (Li et al., 2022a; 2023a), InstructBLIP (Dai et al., 2023), QWen- VL (Bai et al., 2023), VLaVA (Liu et al., 2023b;a;
Table I: Limitation for different anti-Forgttting MLLM methods: Additive Parameter Learning (Add.), Reparameterization Tuning (Repara.), Regularization based Optimization (Reg.), and Partial- based Updating (Part.). Refer to Sec. 2.2 for details.
Limitation
Add.
Repara. Reg.
Part. Ours
Specity Architecture
X
Modify Optimization
X
Require Hyper- Parameter
x
?). VILA (Lin al., 2023a). Existing MLLM solutions nor- mally follow to utilize the visual extractor (Radford et al., 2021; Dosovitskiy et al., 2021) to encode visual features and utilize the connector module to project visual tokens into word embedding space of the LLM, i.e., treating visual input as the foreign language (Wang et al., 2023c). Then, the visual and textual tokens are concatenated and fed into the LLM. The LLM is used to accomplish various vision- language tasks an auto-regressive manner. For example, the famous MLLM work, LLaVA (Liu et al., 2023b) adopts a linear projection layer to connect the visual encoder and the LLM (Chiang et al., 2023; Touvron et al., 2023). De- spite their effectiveness, existing works primarily emphasize the generalization ability across various tasks, resulting in the constrained performance on specific downstream target tasks. Therefore, is an intuitive solution fine-tune the MLLM in order to enhance the particular task performance.
input sequence. ii) Reparameterization Tuning (Hu et al., 2022: Zhang et al., 2023; Wang et al., 2023a; Hao et al.. 2024; Liu et al., 2024; Bi et al., 2025; Liang et al., 2025) also introduce new learnable parameters during the training stage, which are then integrated into the original MLLM through reparameterization during inference. For instance, LoRA (Hu et al., 2022) assumes that the changes in lin- ear model weights follow a low-rank behavior. Despite the certain advantages, these two research streams intro- duce additional parameters into the pre-trale pre-trained model and disrupt the original architecture, leading to increased com- putational costs and presents restricted architecture compat- ibility. iii) Regularization-buset Optimization (Kirkpatrick et al., 2017; Zenke et al., 2017: Xuhong et al., 2018; Ritter et al., 2018; Buzzega et al., 2020; Li et al., 2020; Panigrahi et al., 2023) introduce the loss constraints to preserve the previously learned knowledge. Several studies add regu- larization terms to the loss functions to penalize parame- ter changes and mitigate catastrophic forgetting. However, aforementioned solutions require to modify the loss func- tion and thus conflict with personalized fine-tuning loss design. iv) Partial-bases Updating (Li et al., 2022b; Ansell et al., 2022; Li al., 2023c; Yu et al., 2024; Zhang et al., 2024c;b; Zhu et al., 2024; Lu et al., 2024) focuses on modi- fying subset of mwwsstrraam-eeeeaann parameters, making it architecture-agnoss loss objective. For instance, GPS (Zhang et al., 2024c) and SPU (Zhang et al., 2024b) perform sparse updates based on gradient signals, while DARE (Yu et al., 2024) and Tailor (Zhu et al., 2024) operate on delta parameters. However, previous methods struggle to retain generic knowledge and their performance is highly sensitive to predefined selec- tion thresholds. In our research, recognizing the distinct characteristics of deep neural networks, we argue that pa- rameters exhibit differing importance distributions between pre-training and tie-tuning phases. Therefore, we measure parameter importance in a self-driven manner, selectively updating those with relatively higher importance for down- stream tasks while preserving the generalization capability.
2.2. Catastrophic Forgetting in Multimodal Large Language Model Fine-Tuning
Commonly optimized on downstream tasks (De Boer et al., 2005), deep neural network is empirically proved to suffer from the catastrophic forgetring problem (Ratcliff, 1990; McCloskey & Cohen, 1989; French, 1999; Luo et al., 2023; Kirkpatrick et al., 2017; Zhai et al., 2024), a significant issue where models forget previously learned information when exposed to new data. In the context of MLLM, this re- sults in catastrophic forgetting of generic knowledge, which severely impairs the model transferability across previously learned datasets. Therefore, balancing the ability to fit down- stream tasks while maintaining generalization becomes a crucial challenge for Multimodal Large Language Model. Existing methods could be roughly divided into four cat- egories (Lin et al., 2023b; Han et al., 2024; Xin et al., 2024; Huang et al., 2025). i) Additive Parameter Learn- ing (Houlsby et al., 2019; Yi-Lin Sung, 2022; Lester et al., 2021; Liu et al., 2022; Zhou et al., 2022b;a; Zhang et al., 2022) primarily focus on strategically incorporating addi- tional trainable parameters within the architecture. For ex- ample, adapter (Houlsby et al., 2019; Zhang et al., 2022; Li et al., 2023b; Gao et al.. 2023; Sun et al., 2023a) typically consist of multi-layer perceptions and residual connections (He et al., 2016) that combine pre-trained features with updated ones. Additionally, prompt (Lester et al., 2021; Zhou et al., 2022b;a; Zang et al., 2022; Shu et al., 2022: Wang et al., 2022) directly appends adjustable vectors to the
and orthogonal to the downstream
3. Methodology
3.1. Preliminary
Given the Multimodal Large Language Model (MLLM) ar- chitecture, the MLLM model (0) typically includes three parts: visual encoder f, e.g., VIT (Dosovitskiy et al., 2021),
3

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
LLM (g), e.g., Vicuna (Chiang et al., 2023) and LLaMA (Touvron et al., 2023), and the connector module yp (Liu et al., 2023b; Dai et al., 2023; Liu et al., 2023a; Lin et al., 2023a). For a query instance, the input consists of both a visual image x" and a textual instruction x'. The come- sponding label is a language response y. First, we extract the visual features: z" = f(x"), and then apply the trainable projection (p to convert z" into language embedding tokens, h" = =...". And textual token as h' Tokenize(x*) ). Next, we combine both visual and textual tokens and pass them into the L.M module y to generate the language output g = "(h",,"") Following previous MLLM fine-tuning works and benchmarks (Zhou et al., 2024; Zhu et al., 2024), we select and fine-tune partial trainable parameter module w from the MLLM model to adapt to the downstream task T with distribution (D"). Normally, learnable modules connector module (()) and candidate LLM (g) blocks as w- = (P. This MILM optimization follows: arg min af & (9((s(h") D'.)).
analysis. We further rescale into the bounded range for comparison via Eq. (2c). Thus, Z[u] is within (0, 1)
Importance for Specialization Knowledge. With respect to fine-tuning the downstream task, we aim to identify which parameters are more relevant to the specific task at hand. We argue that the gradient signal acts an effective evaluation metric as the following formulation:
Oc ("((e(""")
8[u]
(3)
where d(u] denotes the gradient of the loss function with respect to the parameter w|v)] evaluated at the query sample. The intuition is that parameters with larger gradient values correspond to directions where the loss function changes most rapidly, facilitating efficient gradient descent during fine-tuning. Thus, we derive the specialization parameter importance G. The formulation is quantified as follows: G[v] = Norm( E (-oo, oo),
(4)
glo] = Sigmoid(((o))) E 1).
(1)
Notably, due to the stochastic nature of sampling, 8|u] ex- hibits instability in reflecting parameter importance, thereby introducing considerable uncertainty in estimating special- ized knowledge. To address this issue, we draw inspiration from the momentum mechanism (He et al., 2020; Chen et al., 2020; Chen et al., 2021) and iteratively accumulate sample gradients using a momentum update with a coefficient of 0.85 to reduce the impact of this uncertainty.
3.2. Specialization via Importance Discrepancy Evaluation for Refinement
To enhance downstream efficiency while preserving generic knowledge in MILM, we assess parameter importance across pre-training and fine-tuning distributions, selec- tively updating downstream critical elements, including two components: Importance Discrepancy Measurement (IDM 3.c.3.2.1) for ranking parameter importance, and Impor- tance Selection Mask (ISM Sec. 3.2.2) for selective updates.
Therefore, we evaluate parameter importance for both gener- alization and specialization by utilizing pre-training weights and fine-tuning gradient information. To ensure bal- anced assessment, we apply the consistent normalization and rescaling methods to these two metrics.
3.2.1. IMPORTANCE DISCREPANCY MEASUREMENT
Importance for Generalization Knowledge. Generic knowledge embedded in MLLM provides bases for strong performance in various domains and quick transfer to dif- ferent tasks; when directly fine-tuning on newly received tasks with no regard to preserving its pre-existing, MI.LM faces the catastrophic forgetting on the generalization ability. Thus, with respect to the generalization knowledge, we take inspiration from the magnitude pruning (Han et al., 2015) and weight magnitude represents how much the parame- ter contributes to the model prediction (Frankle & Carbin, 2019). Thus, in our work, we directly utilize the weight mag- nitude (Han et al., 2015; Frankle & Carbin, 2019; Sun et al., 2023b) for pre-trained parameters w" to rank the generaliza- tion parameter importance I as the following formulation.
3.2.2. IMPORTANCE SELECTION MASK
After localizing the parameter importance for both general- ization (I) and specialized knowledge (9) during the fine- tuning stage, we only update the selected parameters while keeping the remaining pre- pre-trained model parameters frozen. Thus, the straightforward approach is to treat the relatively important elements for the downstream task as candidate parameters for updates. Thus, we define the updating mask M as the following formulation:
Mpu = { 1, G(u) > I[u].
(5)
0, else.
П-'| = w [w]]
Absolute,
(2a)
When M [u] = 1, the query parameter is selected as updat- ing candidate. We denote the current MLLM model
П[u] Mean(I) Std(I)
Normalization,
(2b)
We utilize the frozen pre-trained parameter w" to reweight the current model, thereby restoring the original pre-trained knowledge conditions as follows:
[[u] e
Rescale.
(2c)
In this context, the notation [e] reresents the th component value of given tensor vector. The role of the above form
(6) However, this operation introduces no variance in the candi- date parameter updates. Morcover, due to its normalization property, the aforementioned solution can be seen as mask-
w C M + w" ② (1 - MM.
is threefold. Eq. (2a) computes the weight magnitude, and
Eq. (2b) is applied to eliminate the effect of dimensional
4

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine- Fine-Tuning
ing fifty percent of the parameter updates, which still results in the degradation of generalization performance. We argue that for the selected parameters, assigning higher weights to those exhibiting a greater discrepancy in importance, and conversely lower weights to less significant parameters. Thus, we propose the Importance Selection Mask (ISM) to reconstruct the aggregation weight in Eq. (5) as:
M Eq.
Eq. (2c)
Big.
Glu) )
g[v] > I[v].
M [0]
(7)
g[o] (4)
O,
else.
(a) SPIDER
(b) DARE (Yu at., 2024)
Figure 2: Conceptual Comparison. (a) SPIDER iteratively mea-
Furthermore, rescale the aggregation weights based the mean behavior of the selected elements, while restrict- ing the upper bound to 1. This can be considered as the following rescale operation strategy:
sures the parameter importance discrepancy to construct the update mask which protects generation and squeezes specialization infor- mation on the selected elements. (b) DARE combines the learned elements with the pre-trained and further rescale the candidate ones. ! means frozen pre-trained elements. denotes current learning parameters. represents completed learned ones.
Minl min(1, Meant NZ SO
Glu]
[[u]
M [u]
(8)
0,
else.
Based on the above Importance Selection Mask (ISM), we rewrite the Eq. (6) to update the current model and plot the algorithm description in Algorithm 1 and Fig. 2.
and specialization ability during the fine-tuning process.
Relative Model Merging Works. Large Model Merging promising paradigm that integrates task-specific models into unified model capable of simultaneously handling di- verse downstream tasks (Zheng et al., 2025). Existing meth- ods typically represent each task expert using downstream- optimized models and rely on various importance metrics to select specific elements for merging (Tang et al., 2024; Zhu et al., 2024; Yu et al., 2024; Lu et al., 2024; Du et al., 2024; Huang et al., 2024a; Zhu et al., 2025). However, post- merging operations often disrupt the originally optimized parameter space, limiting the specialized capabilities for in- dividual tasks. In our work, we select relative specialization importance elements during the training process, which is compatible with integrates into the training trajectory and squeeze specialized capabilities into target parameter space.
3.3. Discussion and Limitation
Related Parameter Signal Investigations. Generally speaking, parameter signals could be revealed in two as- pects: magnitude (Han et al., 2015; Frankle & Carbin, 2019;
Sun et al., 2023b) and gradient (Fisher, 1922; Pascanu & Bengio, 2013; Kirkpatrick et al., 2017; Lee et al., 2019; Mirzadeh et al., 2020; Sanh et al., 2020). The weight magni- tude represents how much the parameter contributes to the prediction. The gradient reveals the information intensity during optimization. Thus, magnitude and gradient acts as parameter importance metries to select target elements, which has incurred huge research interest in broad fields, such as network pruning (Han et al., 2015; Frankle & Carbin, 2019; Zhang et al., 2021; Li et al., 2022b), domain general- ization (Rame et al., 2022; Wang et al., 2023b; Zhu et al., 2023a), federated learning (Sung et al., 2021; Matena & Raffel, 2022), and malicious defense (Han et al., 2023; Zhu et al., 2023c; Huang et al., 2023; Zhu et al., 2023b; Huang et al., 2024b). Existing explorations focus on training a net- work from scratch and face no requirement to preserve the previously learned knowledge, thus entangling the magni- tude and gradient information to select the crucial elements for the target task. However, pre-trained Multimodal Large Language Model (MLLM) models have inherent general- ization knowledge, as evidenced by the capacity to execute diverse tasks without fine-tuning (Liu et al., 2023b;a; Zhang et al., 2024a). Thus, maintaining the generalization ability and enhancing the downstream specialization ability during the fine-tuning stage acts as a crucial task for MLLM. In our work, we utilize the pre-trained parameter magnitude (I in Eq. (2a)) and optimizing parameter gradient (6 in Eq. (3)) to respectively reveal the parameter importance metrics for the generalization and specialization abilities. We select relative downstream- kernel elements to balance the generalization
Concept Difference. Existing methods mitigate MLLM forgetting. such as DARE (Yu et al., 2024) and Tailor (Zhu et al., 2024), primarily focus on selectively updating and rescaling optimized parameters using random selection and the Hessian matrix (Fisher, 1922; Sung et al., 2021; Tang et al., 2024; Frantar & Alistarh, 2023). However, these post-combination operations can conflict with optimization strategies that aim to adjust all trainable elements for down- stream performance and is sensitive with the changing scale (Ilarcoo et al., 2023; Yu et al., 2024). Our approach evaluates parameter importance for both generalization and special- ization objectives during the tuning stage. This enables us to selectively update parameters relevant to downstream tasks while preserving others, effectively performing an informa- tion extrusion (Frankle & Carbin, 2019; Chen et al., 2021; Zhang et al., 2021; Bai et al., 2022) to reduce conflicts be- tween pre-training and fine-tuning knowledge, We illustrate the concept difference in Fig. 2.
Limitation. SPIDER leverages both previously pre-trained and current fine-tuning knowledge to select relevant down- stream important elements and re-weight the candidate pa-
5

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Table 2: Computation Complexity, Learnable Ratio and Perfor- mance Comparison. Accuracies are derived from Flickr30k based on VILA. O denotes the complexity degree. Refer to Sec. 3.3.
evaluate MLLMS generalization across different reasoning types and practical scenarios. We follow (Zhou et al., 2024) resource setting and randomly sample 10k samples from the training set of each dataset.
Metries
PT
1.2.Reg
Gootting
FTl
DARE
Taillrr
SPINER
Complexity
((MM O(2x IM ) (((M) ((MM) ((MM)
0,3x
M))
100
50%
10%
50%
Performance
55.16
18.33
49.58
59.52
53.66
51.56
66.61
Counterparts. We focus on exploring model-agnostic MLLM fine-tuning methods and mainly compare with the Regularization-ben based Optimization and Partial-based Updating solutions as follows: Full Fine-Tuning (Full FT) [arXiv'OS] (De Boer et al., 2005), L.- 12-Reeularization (L22Reg) 1PNAS'17] (Kirkpatrick et al., 2017), Grafting [C] (Panigrahi et al., 2023), Half Fine- Sine-Tuning (Half FT) [arXiv' 244] (Hui et al., 2024), DARE [ICML'24] (Yu et al., 2024), and Tailor [ICML'24] (Zhu et al., 2024).
rameters, thereby ensuring generalization while pressing the fine-tuning optimization pathway. However, ours fails in certain cireumstances.
(i) We rank parameter importance based on the pre-trained weights and the current gradient matrix, which incurs additional memory usage. However, this increase is linear relative to the scale of learnable pa- rameters, with a resource complexity of (3 x M.) We further plot the comparison in Tab. 2. (ii) Our method evaluates parameter importance by jointly considering pre- training and fine-tuming distributions, enabling the selection of task-relevant parameters that balance generalization and specialization. When the downstream task closely aligns with the upstream distribution, only minimal updates are required. In such cases, small distribution shifts result in an acceptable level of generalization loss, thereby effectively managing the trade-off between preserving broad capabili- ties and adapting to specific tasks.
Implementation Details. We follow the official code- base to conduct the fine-tuning procedure. The learning rate Ir in LLaVA (Liu et al., 2023b) is 2e 4 for LLM and 2e - 5 for visual projector. For VILA (Lin et al., 2023a), we uniformly set the learning rate to le 1e-44 4. The training epoch is E = 5. The training batch size B set 16. The fine-tuning block for LLM is the last L = layers. All experiments are conducted on 8 NVIDIA 4090 GPUs, each with 24GB memory. Due to limited computation resources, we utilize LLa for LLaVA and VILAI.5-3B for VILA.
4. Experiments
Evaluation Metries. To evaluate the performance of Multi- modal Large Language Model (MLLM) in both generaliza- tion and specialization aspects, we consider two key metrics: Source Performance (AA) and Target Performance (AT). Let 2 = { | represent the set of pre-training datasets and T denote the fine-tuning target dataset. Thus, we derive the following evalation metrics forms:
4.1. Experimental Setup
Architecture and Datasets. Adhering to the Multimodal Large Language Model paradigm, we evaluate the effec- tiveness of our methods using two popular models as the foundations for our experiments: LLaVA (Liu et al., 2023b) and VILA (Lin et al., 2023a). We categorize the datasets into two groups: pre-training (seen) and fine-tuning (un- seen) datasets to respectively measure the generalization and specialization ability. The pre-training datasets consist of those used in the training process; accordingly, we as- sess the learned generalization ability on OKVQA (Marino al., 2019), TextVQA (Singh al., 2019), GQA (Hudson & Manning, 2019), and OCRVQA (Mishra et al., 2019). For fine-tuning tasks, we consider four downstream datasets: Flickr30k (Young et al., 2014), COCO-Capation (Lin et al., 2014), IconQA (Lu et al., 2021), ScienceQA (Lu et al., 2022)', which respectively associate with image caption and visual reasoning views. To be precise, OKVQA, TextVQA GQA, and OCRVQA are obviously mentioned as the train- ing datasets in the pre-training stage, making them appropri- ate benchmarks to evaluate multimodal large language mod- els (MLLMs) generalization across diverse tasks. OKVQA examines external knowledge and common-sense reasoning, TextVQA and OCRVQA test understanding of embedded textual information, and GQA assesses compositional and logical reasoning. Together, these datasets comprehensively
AC),(U)) A" ATc.
(9)
Ace. denotes the accuracy metric. We use the CIDEr met- ric evaluate performance on the Flickr30k (Young et al.. 2014) and COCO-Capation (Lin et al., 2014) datasets. For simplicity, we apply the same notation throughout. To eval- uate effectiveness in mitigating catastrophic forgetting in MILM, we use the H-Average metric (H) and O-Average metric (O) (Zhu et al., 2024). The H-Average and O- Average metrics measure the harmonic and arithmetic mean of generalization (AA5) and specialization (AAT):
H. 2 A x AA
A A O= AS + +AT
(10)
2
4.2. Diagnostic Analysis
We ablation on Flickr30k and IconQA for in-depth analysis.
Candidate Parameter Selection Metrics. Selecting can- didate parameters plays a crucial role in mitigating catas- trophic forgetting of general knowledge while enhancing
hatt:://gghaab.ccmmmmaaaaiiillli hhttss//ggtabb.com///vaavlll.
6

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Table Ablation Analysis Candidate Parameters Selection. The Randy denotes randomly selects elements with y ratio. I, and Gy respectively denotes choose proportion of elements via magnitude and gradient. 7 is set as 50%. Please see Sec. 4.2.
Zera-shot A baby is playing with a byy while cot sits on the floor
Full FT A boby and cat sitting wooden floor
Ours: A baby plays with a toy while cat watches.
Flickr30k
CoCO-Capation
Zero-shot: A black and white dog jumping hurdle in a field. Full FT: dog, junps over bazs a dig competition. Ours: A black and white dog jumps over barrier.
Metric
A
A"
H
.A
.A"
H
Full
47.04
66.68
55.16
48.20
102.07
65.48
MP 6
I. when e satisfy [Metri]]
Response Output on Flickr30k. Text prompt Write a short description for the image. Full FT better follow the instructions than Zero-shol, but Full FT introduces hallucination (e.g., "at a dog competition"), while Zero- shot lacks task details. Please refer to Sec. 4.2.
Figure 3:
Ablation Comparison
Rand.
51.68
70.15
59.52
50.16
106.18
68.13
I, Eq. (2c)
52.47
70.28
60.08
51.07
106.10
68.95
G, Eq. (4)
47.02
67.60
55.46
47.99
105.33
65.93
000 > Ze
Bq. (5)
52.68
73.43
61.35
51.70
111.29
70.60
Table 4: Ablative Study of Key Modules for SPIDER. Incorporate sole Importance Discrepancy Measurement (IDM) can be regarded as Eq. (5). Considering both IDM and ISM, this is viewed as
Zero-shat
L2-Reg
Half DARE
Tailor
Grafting
Eq. (8), For detailed discussion, please refer to Sec. 4.2.
Flickr30k
COcO-Capation A²
BOHAS
IDM ISM
.8
AT
H
Aa
H
Zero- shot
61.39
55.43
58.26
61.39
107.64
78.19
Full Fr
47.04
66.68
35.16
48.20
To2.07
65.48
52.68
73.43
61.35
51.70
111.29
70.60
55,40
83,49
66.61
55.94
122.74
76.86
specialized behavior. In Tab. 3, we examine the impact of different mask updating strategies. Specifically, weight mag- nitude and gradient value serve as two effective metrics for assessing parameter importance relative to the current distri- bution. I. and G.y denote the selection of 7 proportion of elements based small pre-trained weight magnitude and large downstream gradient value, respectively. Three key observations emerge: I Selecting partial parameters is an effective solution for balancing generalization and special- ization. Solely considering the pre- vee-training distribution (I,) emphasizes generalization-relan ieantly limits the downstream adaptation. Exclusively incorporating fine-tuning importance (G.) undermines gen- eralization and hinders specialization. As demonstrated in Tab. 3, we select relative downstream important elements (G[u)= > Z[e] in Eq. (5)). which effectively preserves gen- eralization while ensuring specialization performance for Multimodal Large Language Model (MLLM).
THEROA
COA
COOO CoNC-Coprtinn Copetion
VILA
LLaVA
Figure 4: Visualization Comparison. Radar charts plots fine- tuning methods results across four pre-trained source datasets and target datasets, Le., Flickr30k and COCO-Capation. Our method achieves a better generalization and specialization trade-off.
elements but signif-
likely due to the forgetting of generalized knowledge. Our method effectively achieves satisfying results.
4.3. Comparison to State-of-the-Arts
Key Component Analysis. In Tab, 4, we begin by validat- ing the significance of our proposed components through their incremental integration. The first row displays the BASELINE result, representing a simple Full Fine- Tuning (Full FT) approach using standard cross-entropy loss. As demonstrated, the combination of Importance Discrepancy Measurement (IDM) and Importance Selection Mask (ISM) yields the best performance in both generalization and spe- cialization. This finding supports our motivation evaluate parameter significance across pre- pre-training and fine-tuning distributions, while selectively updating candidate param- eters for the downstream task. Additionally, we plot the response outputs in Fig. 3, revealing that Zero-shot fails to adhere to the instruction style and lacks a detailed descrip- tion. In contrast, naive Full FT introduces hallucinations,
Quantitative Results. We compare our SPIDER against related approaches image-captioning (CAP) and visual question-ansseringg (VQA) tasks. Due to the architectural complexity and task differences, we limit the VQA evalu- ation to VILAI.5-3B. As shown in Tabs. 5 and 6, several key observations can be made: The larger the task gap between the fine-tuning and pre-training distributions, the more severe the generalization-spccalizzaaoo trade-offs in the MLLM fine-tuning process. Full-FT directly updates all parameters adapt to the downstream distribution. How- ever, specialized downstream applications typically have limited data compared to large-scale pre-training datasets. Thus, Full-FT often leads to increased overfitting, especially when training for more epochs, resulting in performance degradation on downstream tasks. Notably, regularization-

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Table 5: Comparison with the slate-of-the-ar Multimodal Large Language Model (MLLM) Fine-Tuning Solutions on the image caption task: Flickr30k and COCO-Capation datasets based on VILA and LLaVA architectures. We mark the Best in bold across different tuning methods. means improved accuracy compared with Full FT. Please refer to Sec. 4.3 for relative explanations.
Flickr30kk
Coco-Capation
Methods
OKVQA
TeaVQA
OCRVQA
OQA
AA
AT
1
O
OKVQA TeaVQA OCRVQA
GQA
A
AT
76
Fise- Time
VILA orchitecture
Zero-shot
55.60
60.30
68.20
61.47
66.399
55.45
58.26
58.41
55.60
60:30
68.20
61.47
61.39
187.64
78.19
84.52
Pall
37.99
45.17
53.85
51.14
47.04
06.68
55.16
36.36
57.36
42.96
35.85
56.63
46.20
T02.07
65,48
15.14
L2- L2-Reg
31.59
25.89
47.20
49.48
39,29
62.77
18.33
51.03
33.98
11.67
52.55
50.25
41.61
99.81
61.67
72.23
Grafting
35.66
31.60
47.40
47.67
40.58
63.71
49.58
52.15
33.12
52,60
49.77
43.64
99.84
71.74
Half FT
44.15
48.71
60.90
52.97
51.68
70.15
59.52
60.92
41,41
47.47
57.80
53.94
50.16
106.
68.13
78.17
DARE
38.38
39.69
52,05
51.33
45.36
65.67
$3.66
55.52
36.73
43.34
56.5
51.33
46.98
100.70
64.86
73.84
Taikor
38.30
11.88
53.35
51.38
17,00
65.00
51.56
56.00
37.84
43.51
55,70
50,96
47,00
102.41
64.41
74.72
SPIDER
47.11
53.38
65,55
55.57
555.40
N3.49
66.61-11.8 45
69.45
46.65
54.94
65:55
56.63
55.94
122.74
76,86
-1
89.34
Fine- Tine with ZEaVA andhitertuue
Zevo-hot
58.00
58.25
66.20
61.93
61.10
25.31
35.79
43.20
58,00
58.25
66.20
61.93
61.10
110.52
78.69
85.81
Full FT
45.59
47.09
57.65
56.94
51.82
61.55 51.58
56.28
36.70
41.01
42.89
57.75
53.67
48.83
92.01
63.80
70.42
FT
48.96
49:47
60.80
56.81
54.01
62.91
58.12
58.46
37,62
41.14
60.00
45.96
46.18
79.91
58.53
63.05
DARE
44.82
48.01
58.75
57.04
52.16
56.73
57.17
39.60
56,00
46.64
90.82
Tailor
44,50
46.32
59.00
57.14
51.74
61.27
56,10
56.51
41.35
40.85
58.45
54.87
48.88
90.94
63.58
69.91
SPIDER
55.81
53.67
63.95
57.04
57.62
79.84
60.93
68.73
12
05
49.50
15.33
65.00
58.14
54.49
114.74
73.89
20.09
84.62
110.20
Table 6: Comparison with the of Multimodal Large Language Model (MLLM) Fine- Tuning Solutions on the the visual question answering task: IconQA and ScienceQA datasets based on the VILA architecture. Please details in Sec. 4.3.
IconQA
Science AA
Methods
OKVQA
TesiVQA
OCRVQA
GQA
_A8
H
OKVQA
TexvQQ
OCRVQA
GQA
Zero-shot
$5.60
60.30
68.
61.47
39
19:93
50.09
40.66
$5.60
60:30
68.20
61
61 39
69 69.89
65.37
64.64
Full FT
34.51
38.02
46.10
47.05
41.42
87.05
56.13
64.24
47.15
50.38
57.20
51.58
52.20
75.75
61.82
63.99
12.Reg
21.69
25.89
35.20
37.09
29:97
86.40
44.50
58.18
43.65
48.13
$1.80
50.42
48.50
76.40
59,33
62.45
Grafting
22.56
31.60
40:25
37.84
33.09
87.18
47.97
60.13
45.65
50,71
54.35
$3.94
51,16
76.00
61.16
63.58
Halt FT
43.36
48.71
55.25
$3.03
50.09
88.19
63,89
69.14
$2.07
54.47
60.55
$7.52
56.15
76.77
64.86
66.46
DARE
36.88
39.69
45.15
48.09
42.45
88.11
57,30
65.28
47.61
50.39
57.55
55.08
52.66
77.46
62.69
65.06
Tailor
37.99
41.42
48. 10.10
47.57
43.77
88.16
58.50
65.97
48.22
50.98
57.90
53.04
52.54
75.97
62.12
64.25
SPIDER
48.34
53.38
63.35
56.12
55.30
84.07
66.71
69.68 ys
54.13
57,33
65.60
60.07
59.28
75.97
66.60-a.33
67.63
m2 Lajer L.per
1744
<1ay
10
60.0
HARPY
DANK
(a) VILA for S
(b) VILA for fAT AAT
(a) Generalization (A5)
(b) Specialization (AT)
Figure 6: Comparison on More Fine-Tuning Layer T. on
Flickr30k dataset with the VII.A architecture. See Sec. 4.3.
1534
8922
K7.25
5U12
CLIT
41.00
51.44
35.25
5016
54.06
5012
(c) LLaVA for A
(d) LLaVA for AT
53.05
50.25
Tider
DARE
Talle
Figure 5: Comparison on Large Fine-Tuning Epochs. E varies
(a) VILA for 7
(b) LLaVA for H
from (5 rounds to 10 rounds) on Flickr30k. See See. 4.3.
Figure 7: Comparison Large Training Batch Size B on Flickr30k dataset with VILA and LLaVA. Refer to Sec. 4.3.
based optimization approaches typically offer limited per- formance improvements, as controlling parameter stiffness to regulate the extent of LLM updates remains a challenging task. Moreover, in partial-update methodologies, directly combining updated parameters with pre-trained ones often leads to performance fluctuation, largely due to the influ- ence of delta parameters scale. In contrast, both Half FT and our proposed method guide the LLM to tne-tune on se- lected parameters, demonstrating competitive performance
across various experiments. Additionally, our approach selectively targets relatively important parameters for down- stream tasks, yielding better performance compared to the random selection strategy employed in Half FT. We further plot the radar visualization in Fig. 4 to highlight the perfor- mance advantages of ours compared to other approaches.
Performance on More Tuning Epochs F., We investigate
8

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
the impact of extending fine-tuning epochs E from 5 to 10 rounds, shown in Fig. 5. 'The results highlight several key findings: (i) Extending fine-tuning epochs intensifies the pre- trainini knowledge forgetting phenomenon across different architecture scales. (ii) Smaller architectures, such as VILA-I.5-3B, encounter more severe parameter conflicts, where a decline in generalization ability results degraded specialization performance. (iii) Larger architectures, such as LLaVA-115577B which possess higher parameter redun- dancy, maintain more stable specialization ability despite extended tuning epochs. (iv) Ours achieves robust perfor- mance across various architectures and training duration.
ale Innovation Research Fund Project. The supercomputing system at the Supercomputing Center of Wuhan University supported the numerical calculations in this paper. Dr Tao's research is supported by NTU RSR and Start Up Grants. Besides, we thank Zekun Shi for his valuable collaboration. This work would not have been possible without him.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Performance on More LLM Tuning Layer L. We evalu- ate the effect of tuning block layers I. from to 5, as shown in Fig. 6. Existing methods show a slight improvement in general performance but significantly reduce target domain performance. SPIDER effectively maintains both general- ization and specialization across various tuning layers.
References
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y.. Lene, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E.,, Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M. a., Barreira, R., Vinyals, O., Zisser- man, A., and Simonyan, K. Flamingo: a visual language model for few-shot learning. In NeurlPS, volume 35, pp. 23716-23736, 2022.
Performance on Large Training Batch B. We further conduct the experiments on the large training batch B: 32 in Fig. The results show that setting a higher training batch benefits both generalization and specialization ability across different counterparts. Notably, our method SPIDER constantly achieves the best performance results.
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
5. Conclusion
In conclusion, we address the catastrophic forgetting in fine- tuning Multimodal Large Language Model (MLLM). We introduce Specialization via Importance Discrepancy Eval- uation for Refinement (SPIDER assess parameter importance for both generalization and spe- cialization, focusing on identifying elements and performing critical-aware updates on selected parameters. Our method enjoys third advantages: First, No Architecture Dependency: SPIDER functions without spe- cific model architecture, which presents high transferability across different architectures. Second, No fine-tuning Pat- tern Conflict: we conduct partial parameter updates, main- taining compatibility with various optimization fuucciinns Third, No Hyper-Paramend Configuration: leveraging pa- rameter importance discrepancies requires no additional pyper-parameters, enhancing fine-tuning effectiveness. SPI- DER has been validated on fruitful scenarios, highlighting the potential for broader applications.
Ansell, A., Ponti, E. M., Korhonen, A., and Vulié, I. Com- posable sparse fine-tuning for cross-lingual transfer. In ACL, 2022.
novel approach to
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Bai, Y., Wang, H., TAO, Z., Li, K., and Fu, Y. Dual lottery ticket hypothesis. In ICI.R, 2022.
Bi, J., Wang, Y., Chen, H., Xiao, X., Hecker, A., Tresp, V., and Ma, Y. Llava steering: Visual instruction tun- ing with 500x fewer parameters through modality linear pnresentaation-sterinn In ACL, 2025.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In NeurlPS, pp. 1877-1901, 2020.
Acknowledgement
This work is supported by National Natural Science Foun- dation of China under Grant (622251133 13, 62361 166629, 62176188, 623B2080), the National Key Research and Development Program of China (2023YFC2705000, 2024YFC3308400), and the Wuhan University Undergradu-
Buzzega, P., Boschini, M., Porrello, A., Abati, D., and
Calderara, S. Dark experience for general continual leam- ing: a strong, simple baseline. In NeurlPS, pp. 15920- 15930, 2020.
9

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Cha, J., Kang, W., Mun, J., and Roh, B. Honeybee: Locality- enhanced projector for multimodal Ilm. In CVPR, pp- 13817-13827, 2024.
French, R. M. Catastrophic forgetting in connectionist works. Trends in cognitive sciences, pp. 128-135, 1999.
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision-languen models with feature adapters. I/CV, 2023.
Chen, T., Sui, Y. Chen, X., Zhang, A., and Wang, Z. A unified lottery ticket hypothesis for graph neural networks. In ICML, pp. 1695-1706. PMLR, 2021.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the in VQA matter: Elevating the role of image understanding in Visual Question An- swering. CVPR, 2017.
Chen, X., Fan, H., Girshick, R., and He, K. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. JCML, pp. 1321-1330, 2017.
Chen X., Xie*, S., and He, K. An empirical study of training self superviee vision transformers. In ICCV, 2021.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. In NeurlPS, 2015.
Chiang, W.-L.,, Li, Z., Lin, Z. Sheng, Y., Wu, Z., Zhang, H., Zheng. L., Zhuang. S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Han, S., Park, S., Wu, F., Kim, S., Zhu, B., Xie, X., and Cha, M. Towards attack-tolerant federated learning via critical parameter analysis. In ICCV, 2023.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.00 20te,,2231,,00000
Han, Z. Gao, C., Liu, J., Zhang, S. Q., al. Parameter- efficient fine-tuning for large models: A comprehensive survey, arXiv preprint 24Xv:2440.11660 2024.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, 2023.
Hao, Y., Cao, Y., and Mou, L. Flora: Low-rank adapters are secretly gradient compressors. ICML, 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
De Boer, P.-T., Kroese, D. P., Mannor, S., and Rubinstein, R. Y. A tutorial on the cross-entropy method. Ann. Oper. Res,, pp. 19-67, 2005.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-97336 2020.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficciet transfer learning for nlp. In ICML, pp. 2790-2799, 2019.
Du, G., Lee, J., Li, Jiang, R., Guo, Y., Yu, S., Liu, H., Goh, S. K., Tang, H.-K., He, D., et al. Parameter competition balancing for model merging. In NeurIPS, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, 7., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Fisher, R. A. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of mathematical or physical character, 222(594-604)309-33
Huang, C., Liu, Q.. Lin, Y., Pang, T., Du, C., and Lin, M. Lorahub: Efficient cross-task generalization via dynamic lora composition. In COLM, 2024a.
1922.
Huang, S., Li, Y., Chen, C., Shi, L., and Gao, Y. Multi- metrics adaptively identifies backdoors in federated learn- ing. In ICCV, pp. 4652-4662, 2023.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in ohe-shot. In ICML, pp. 10323-10337.. PMLR, 2023.
Huang, W., Ye, M., Shi, Z., Du, B., and Tao, D. Fisher calibration for backdoor-robust heterogeneous federated learning. In ECCV, 2024b.
10

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Huang, W. Liang, J., Guo, X., Fang, Y., Wan, G., Rong, X., Wen, C., Shi, 7.,, Li, Q. Zhu, D., al. Keeping yourself is important in downstream tuning multimodal large language model. arXiv preprint arXiv:2503.0453 2025.
Li, Y., Luo, F., Tan, C., Wang, M., Huang, S., Li, S., and Bai, J. Parameter-effiient sparsity for large language models fine-tuning. In IJCAI, 2022b.
Li, Y., Yu, Y., Zhang, Q. Liang, C., He, P., Chen, W., and Zhao, T. Losparse: Structured compression of large language models based on low-rank and sparse approxi- mation. In ICML, pp. 20336-20350. PMLR, 2023c.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pp. 6700-6700 2019.
Liang, J., Huang, W., Wan, G., Yang. Q. and Ye, M. Lo- rasculpt: Sculpting lora for harmonizing general and spe- cialized knowledge in multimodal large language models. CVPR, 2025.
Hui, T., Zhang, Z.,, Wang, S., Xu, W. Sun, Y., and Wu, H. Hft: Half fine-tuning for large language models. arXiv preprint arXiv:2404.1 18466, 2024.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, I.., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. ICLR, 2023.
Lin, J., Yin, H., Ping, W., Lu, Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In CVPR, 2023a.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barde Baabska-Brrwinkk, A., et al. Overcoming catastrophic forgetting in neural networks. PNAS, pp. 3521-3526, 2017.
Lin, T.-Y,, Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Lin, Y., Tan, L., Lin, H., Zheng, Z., Pi, R., Zhang, J., Diao, S., Wang, H., Zhao, H., Yao, Y.. al. Speciality vs generality: An empirical study catastrophic forget- ting in fine-tuning foundation models. arXiv preprint arXiv:2309.060 2023b.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurlPS, 2017.
Lee, N., Ajanthan, T., and Torr, P. H. Snip: Single-shot network pruning based connection sensitivity. In ICLR, 2019.
Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., Han, J., Huang, S., Zhang, Y., He, X., Li, H., and Qiao, Y. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. In ECCV, 2024.
Lester, B., AI-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
Li, H., Kaday, A., Durdanovic, L., Samet, H., and Graf, H. Pruning filters for efficient convnets. In ICLR, 2017.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In CVPR, 2023a.
Li J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping traguage-image pre-training for unified vision-language understanding and generation. In ICMI,, pp. 12888- 12900. PMLR, 2022a.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2023b.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight- decomposed low-rank adaptation. In ICML, 2024.
Li, Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-img image pre- mre-trainig with frozen image encoders and large language models. ICML, pp. 19730-19742. PMLR, 2023a.
Liu, X., Sun, T., Huang, X., and Qiu, X. Late prompt tuning: A late prompt could be better than many prompts. arXiv preprint arXiv:2210. 12992, 2022.
Li, T., Sahu, A. K., Zaheer, M.. Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. In MI.ys, 2020.
Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T. Re- thinking the value network pruning. In ICLR, 2019.
Li, X., Lian, D., Lu, Z., Bai. J., Chen, Z., and Wang, X. Graphadapter: Tuning languaee models with dual knowledge graph. In NeurIPS, volume 36, pp. 13448- 13466, 2023b.
Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., and Zhu, S.-C. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In NeurlPS, 2021.

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, 2022.
Rame, A., Dancette, C., and Cord, M. Fishr: Invariant gradient variances for out-of-distriutiion generalization. In ICML, pp. 18347-18377. PMLR, 2022.
Ratcliff, R. Connectionist models of recognition memory:
Lu, Z., Fan, C., Wei, W., Qu, X., Chen, D., and Cheng, Y. Twin-man merging:: Dynamic integration of modular expertise in model merging. In NeurIPS, 2024.
constraints imposed by learning and forgetting functions. Psychological review, pp. 285, 1990.
Ritter, H., Botev, A., and Barber, D. Online structured
Luo, G., Zhou, Y., Zhang, Y., Zheng, X., Sun, X., and Ji, R. Feast your eyes: Mixture-of-resolttion adaptation for multimodal large language models. arXiv preprint arXiv:2403.0m 030003 2024.
laplace approximations for overcoming catastrophic for- getting. In NearlPS, 2018.
Sanh, V., Wolf, T., and Rush, A. Movement pruning: Adap- tive sparsity by fine-tuning. In NeurlPS, pp. 20378 20389, 2020.
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint xrXiv:2308.08747 08747, 2023.
Shu, M., Nie, W. Huang, D.-A., Yu, Z., Goldstein, T., Anandkumar, A., and Xiao, C. Test-time prompt tuning for zero-shot generalization in lasgualanguaae models. In NeurlPS, pp. 14274-14222 14289, 2022.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok- vqa: visual question answering benchmark requiring external knowledge. In CVPR, 2019.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In CVPR, pp. 8317-8326, 2019.
Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. In NeurlPS, pp. 17703-17716, 2022.
Song, W., Li, Z., Zhang, L., hai zhao, and Du, B. Sparse is enough fine- fine-tuning pre-trained large language models. In ICML, 2024.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, pp. 109-165. Elsevier, 1989.
Su, Y., Lan, T. Li, H., Xu, J., Wang, Y., and Cai, D. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.1633
2023.
Mirzadeh, S. I., Farajtabar, M., Pascanu, R., and Ghasemzadeh, H. Understanding the role of training regimes in continual learning. In NeurlPS, pp. 7308- 7320, 2020.
Sun, J., Qin, J., Lin, Z., and Chen, C. Prompt tuning based adapter for Dision-languee model adaption. arXiv preprint arXiv:2303. 15234, 2023a.
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pp. 947-952. IEEE, 2019.
Sun, M.. Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. In ICLR, 2023b.
OpenAl. Gpt-4 technical report.
arXiv preprint
Sung, Y.L.., Nair, V., and Raffel, C. A. Training neural networks with fixed sparse masks, In NeurlPS, pp. 24193- 24205, 2021.
arXiv:2303.08874, 2023.
Panigrahi, A., Saunshi, N., Zhao, H., and Arora, Task- specific skill localization in fine-tuned language models. In ICML, pp. 27011-27033. PMLR, 2023.
Tang, Q. Yu, L., Yu, B., Lin, H., Lu, K., Lu, Y., Han, X., and Sun, L. A unified view of delta parameter edit- ing in post-trained large-scale models. arXiv preprint arXiv:2410. 13841, 2024.
Pascanu, R. and Bengio, Y. Revisiting natural gradient for deep networks. arXiv preprint arXiv:
2013.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.. et Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML., pp. 8748-8788 87688876,,14 2021.
Wang. P., Panda, R., Hennigen, L. T., Greengard, P., Kar- linsky, L.,, Feris, R., Cox, D. D., Wang, Z., and Kim, Y. Learning to grow pretrained models for efficient trans- former training. In ICLR, 2023a.
12

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
Wang, P., Zhang, Z.., Lei, Z., and Zhang, L. Sharpness-aware gradient matching for domain generalization. In CVPR, 2023b.
Zhang, Q. Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. Adaptive budget alloca- tion for parameter-effiiien fine-tuning. arXiv preprint arXiv:2303. 10512, 2023.
Wang, W. Bao, H., Dong, L.. Bjorek, J., Peng. Z., Liu, Q. Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., and Wei, F. Image as a foreign language: BEiT pre- training for vision and lisoon-angguag tasks. In CVPR, 2023c.
Zhang, R., Fang, R., Zhang, W. Gao, Li, K., Dai, J., Qiao, Y., and Li, H. Tip-adapter: Training-free clip-adapter for better vision-language modeling. In ECCV, 2022.
Zhang, W., Janson, P., Aljundi, R., and Elhoseiny, M. Over- coming generic knowledge loss with selective parameter update. In CVPR, pp. 24046-24056. 2024b.
Wang, Z.,, Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pp. 631-666-00 631- Springer, 2022.
Zhang, Z., Chen, X., Chen, T., and Wang. Z. Efficient lottery ticket finding: Less data is more. In ICMI, pp. 12380-12390. PMLR, 2021.
Xin, Y., Luo, S., Zhou, H., Du, J., Liu, X., Fan, Y., Li, Q. and Du, Y. Parameter-efficien fine-tuning for pre-trained vision models: survey. arXiv preprint arXiv:2402.02244, 2024.
Zhang, Z., Zhang, Q., Gao, Z., Zhang, R., Shutova, E., Zhou, S., and Zhang. S. Gradient-basee parameter selection for efficient fine-tuning. CVPR, pp. 28566-28577, 2024c.
Xuhong, L.,, Grandvalet, Y., and Davoine, F. Explicit induc- tive bias for transfer learning with convolutional networks. In ICML, pp. 2825-2834 2834. PMLR, 2018.
Zheng, H., Shen, L.., Tang, A., Luo, Y., Hu, H., Du, B., and Tao, D. Learn from model beyond fine-tuning" A survey, Nature Machine Intelligence, 2025.
Yi-Lin Sung, Jaemin Cho, M. B. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. In NeurlPS, 2022.
Zhou, K., Yang. J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In CVPR, 2022a.
Yin, L., Li, G., Fang, M., Shen, L., Huang. T., Wang, Z., Menkovski, V., Ma, X., Pechenizkiy, M., and Liu, S. Dynamic sparsity is channel-level sparsity learner. In NeurlPS, 2023.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning prompt for vision-languae models. IJV,, 130(9):2337- 2348, 2022b.
Zhou, X., He, J., Ke, Y., Zhu, G., Gutiérrez-Basulto, V., and Pan, J. Z. An empirical study on parameter-officient fine-tuning for multimodal large language models. In ACL, 2024.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78, 2014.
Zhu, B., Niu, Y., Han, Y., Wu, Y., and Zhang, H. Prompt- aligned gradient for prompt tuning. In ICCV, pp. 15659- 15669, 2023a.
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. Language mod- els are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024.
Zhu, C., Roos, S., and Chen, L. Y. Leadfl: client self-defense against model poisoning federated learning. In ICML, pp. 43158 431 180. PMLR, 2023b.
Zang, Y., Li, W., Zhou, K., Huang, C., and Loy, C. C. Uni- fied vision and language prompt learning, arXiv preprint arX1v:2210.07225, 2022.
Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Kuang, K., and Wu, C. Model tailor: Mitigating catastrophic for- getting in multi-modal large language models. In ICML, 2024.
Znnke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In ICML, pp. 3987-3995. PMLR, 2017.
Zhu, D., Song, Y., Shen, T., Zhao, Z., Yang, J., Zhang, M., and Wu, C. REMEDY: Recipe merging dynamics in large lision-language models. In ICLR, 2025.
Zhai, Y., Tong, S., X.,, Cai, M.. Qu, Q., Lee, J., and Ma, Y. Investigating the catastrophic forgetting in multimodal large language model fine-tuning. In CPAL, pp. 202-227. PMLR, 2024.
Zhu, M., Wei, S., Shen, L.,, Fan, Y., and Wu, B. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In /CCV, 2023c.
Zhang, J., Huang, J.. Jin, S., and Lu, S. Vision-language models for vision tasks: survey, IEEE PAMI, 2024a.
13

---

Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
APPENDIX
DARE [ICML224] (Yu al., 2024): Parameters from the fine-tuned model randomly selected and re-scaled to maintain ability generalization and specialization aspects.
A. Algorithm
provide the algorithm description in Algorithm 1. The detailed method description is in Sec. 3.2.
Tailor [ICML'22] (Zhu et al., 2024): Preserve pre-trained parameters while replacing small ratio of fine-tuned pa- rameters, i.e., 10 %. based on the salience and sensitivity analysis.
Algorithm SPIDER
Input: Tiniig Epoch E. Overall MLLM Network Train- able parameter module w, Frozen Pre-trained parameter weight w²,
Output: The optimized selected MILM model AA Generalin airaaiiiiiiiiio Knowledge Rank */
+ (w") in Eq. (2c)
for = 1, E do
for (x" x. v) E D* do
4 +(f(x")) 14'=10 Tokenize(=*)
8 (g(h", (g(()" w) via Eq. (3)
apecializatie
Knowledge Rank x/
G < (ő) in Eq. (4)
Importance Selection Mask x/
glu M
9[u] > I[u]:
M [0]
else.
1 Rescale Importance Mask Matrix M
min
Mc Mean M M M
G(u) >
[(u]
M
=
0,
clsc.
w- VLC w M + w² 2 (M- Eq. (6)
// Update Param.
end
end
B. Compared Methods
We focus on exploring model-agnostic MLLM fine- tuning methods, with primary comparison between Regularization-basee Optimization and Partial-based Updat- ing solutions, as outlined below:
Full Fine-Tuning (Full FT) [arXiv'05] (De Boer et al., 2005): Default optimize full parameters towards the down- stream task.
L2- Regulalarriztiinn (L2-Reeg [7N 17) (Kirkpatrick et al., 2017): Add an L2 regularization term with the regular- ization byper-parameter, i.e., Ic-3, to the original loss function. Thus, it focuses on keeping the fine-tuning model closer the pre-trained model, thereby mitigating forgetting.
Grafting [ICML'23] (Panigrahi et al., 2023): Localize newly acquired skills inside fine-tuned language mod- els, which could be regarded as L1 regularization with the penalty weigh, i.e., le-6.
Half Fine-Tuning (Half FT) [arXiv 24] (Hui et al., 2024): Randomly update half of the parameter blocks within each transformer layer at each iteration while freezing the other elements.
14


Reviews:
[{"review_id": "mufTgHS1eJ", "reviewer": "Reviewer_mG7E", "summary": "This manuscript addresses the catastrophic forgetting issue in fine-tuning MLLMs. It proposes SPIDER, which measures parameter importance based on pre-trained weight magnitudes and fine - tuning gradients. SPIDER uses Importance Discrepancy Measurement (IDM) to rank parameter importance and Importance Selection Mask (ISM) for selective parameter updates. Experiments on image captioning and VQA with models like LLaVA and VILA show that SPIDER effectively balances generalization and specialization, outperforming baseline methods.\n\n## update after rebuttal\n\nThe response has addressed all my concerns, and I will increase my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mG7E:\n\nWe sincerely appreciate your time and effort in reviewing our paper. Through the detailed responses outlined below, we seek to fully address your concerns and provide transparency into our proposed approach.\n\n**Q1: Essential Reference Discussion and Comparison** (Essential References Not Discussed & Other Strengths And Weaknesses & Other Comments Or Suggestions)\n\nA1: We apologize for overlooking the critical reference. Memory Aware Synapses (MAS) [1] addresses catastrophic forgetting in lifelong learning by measuring parameter importance through output sensitivity and applying a regularization penalty to prevent substantial updates to critical parameters. In our experiments, we selected OKVQA as a representative dataset to estimate parameter importance for generalization and compared MAS with different regularization hyperparameters ($\\lambda$), as detailed in the table below. The results show that MAS outperforms other regularization-based methods; however, it provides limited trade-offs between generalization and specialization in the context of multimodal large language models (MLLMs). This limitation arises because the regularization terms cannot effectively maintain stable parameter constraints in large-scale models, negatively affecting overall performance. We will expand upon these observations and their implications in our revised manuscript.\n\n*Table: Performance comparison with relative methods. We conduct experiments on the VILA architecture and tune on the Flickr30k dataset.*\n|Method |OKVQA | TextVQA | OCRVQA | GQA | $\\mathcal{A}^S$ | $\\mathcal{A}^T$| $\\mathcal{H}$| $\\mathcal{O}$ |\n|----|---|---|---|---|---|---|--|--|\n| Zero-shot | 55.60 | 60.30 | 68.20 | 61.47 | 61.39 | 55.43 | 58.26 | 58.41|\n| Full FT   | 37.99 | 45.17 | 53.85 | 51.14 | 47.04 | 66.68 | 55.16 | 56.86|\n|L2-Reg| 34.59 | 25.89 | 47.20 | 49.48 | 39.29 |  62.77 | 48.33 | 51.03 |\n| MAS ($\\lambda=0.01$)| 34.86 | 40.45 | 46.50 | 47.61 | 42.35 | 63.63 | 50.85 | 52.99\n|SPIDER (Ours)|47.11 | 53.38 | 65.55 |55.57 | **55.40** |**83.49** |**66.61**| **69.45**|\n\n[1] Memory aware synapses: Learning what (not) to forget, ECCV, 2018.\n\n**Q2: Distribution Sensitivity Effect on Parameter Importance Difference** (Essential References Not Discussed & Other Strengths And Weaknesses & Other Comments Or Suggestions)\n\nA2: Our method leverages pre-trained weights and current gradient updates to separately construct parameter importance rankings for generalization and specialization. In our first module, we identify elements relatively important for downstream tasks based on these rankings. Furthermore, we note that naively updating candidate parameters does not reflect parameter-level variance effectively (Eq. 6). Therefore, we propose the Importance Selection Mask, which dynamically allocates higher update rates to parameters exhibiting greater importance discrepancies, as defined in Eqs. 7-8 and illustrated on page 5. Consequently, our approach dynamically restricts parameter updates when the downstream task distribution closely aligns with the generalization distribution, effectively preserving generalization capability.", "claims_and_evidence": "Overall, all claims are well-supported. However, one point that confuses me is the rationale behind simply comparing I[v] and G[v] after normalization and rescaling to determine which weights should be updated. This approach lacks sufficient justification. Specifically, could this method potentially lead to updates in nearly half of the weights? Given that general knowledge from pretraining typically outweighs specialized knowledge gained through fine-tuning, might this not be somewhat unreasonable or problematic?", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The paper lacks rigorous mathematical proofs.", "experimental_designs_or_analyses": "No", "relation_to_broader_scientific_literature": "Building upon the concept of catastrophic forgetting in neural networks, this paper proposes a novel approach to better balance generalized and specialized knowledge in MLLMs during fine-tuning, effectively mitigating forgetting.", "essential_references_not_discussed": "The following paper is an important paper on overcoming the catastrophic forgetting of neural networks, but it is not cited or discussed in this paper.\n[1] Aljundi R, Babiloni F, Elhoseiny M, et al. Memory aware synapses: Learning what (not) to forget[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 139-154.", "other_strengths_and_weaknesses": "Strengths:\n1. Novel solution: Proposes SPIDER to address catastrophic forgetting in MLLM fine-tuning by measuring parameter importance, offering a new way to balance generalization knowledge and specialization knowledge.\n2. Comprehensive experiments: Tests on multiple MLLM architectures and datasets, comparing with SOTA methods. Ablation studies and various metrics validate its effectiveness.\n3. Practical features: Architecture-agnostic, compatible with different optimization functions, and requires no hyper-parameter configuration.\n\nWeaknesses\n1. Memory issue: Ranking parameters by pre-trained weights and gradients increases memory usage, which may be a problem in resource-constrained settings.\n2. Distribution sensitivity: The current approach determines parameter updates by simply comparing the normalized and rescaled values of I[v] and G[v], potentially leading to updates in nearly half of the parameters. Consequently, extensive parameter updates may occur, resulting in decreased generalization ability even when the downstream task distribution differs only slightly from the pre-trained distribution.\n3. Lack of comparison with traditional methods: Catastrophic forgetting is not limited to the field of large models. There have been many studies on this aspect in neural networks, such as ref[1]. This article lacks a comparison with this traditional method.\n\n[1] Aljundi R, Babiloni F, Elhoseiny M, et al. Memory aware synapses: Learning what (not) to forget[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 139-154.", "other_comments_or_suggestions": "1. Consider how to improve the comparison between I[v] and G[v] so that the problem of generalization perturbation caused by fine-tuning under the same distribution can be solved. Specifically, in the case of close distribution, further dynamic reduction of update parameters may be difficult to achieve in the short term. 2. Adding missing traditional references and comparative experiments to overcome catastrophic forgetting.", "questions_for_authors": "See \"Other comments or suggestions\" for details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CoBw9mJrtd", "reviewer": "Reviewer_mmMS", "summary": "This paper introduces a novel and well-structured strategy to address the persistent challenge of catastrophic forgetting that arises during the fine-tuning of Multimodal Large Language Models (MLLMs). This method leverages a meticulous analysis of parameter importance discrepancies to guide the optimization process, ensuring that previously learned knowledge is effectively retained while allowing the model to adapt and specialize for downstream tasks. By carefully balancing the trade-off between generalization and specialization, this approach enhances the model’s ability to perform robustly across diverse scenarios without compromising its adaptability to new domains. Furthermore, this strategy provides a systematic framework for mitigating knowledge degradation, ultimately improving the model’s stability and performance in real-world applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mmMS:\n\nThank you very much for your affirmation of our work, as well as the insightful concerns and questions you have raised. We have carefully considered each comment and provided responses.\n\n**Q1: Value Range Boundary and Normalization Operation Rationale** （Theoretical Claims）\nA1: Sections 2a–2c measure parameter importance for generalization by leveraging the pre-trained weight magnitudes. Specifically, step 2a computes the absolute values of pre-trained weights, yielding a range of [0,$+\\infty$). Step 2b normalizes these absolute values to a standardized range of ($-\\infty$,$+\\infty$). Finally, step 2c rescales the normalized values into the range (0,1) using a sigmoid function. Together, these steps systematically rank parameters according to their importance for generalization, facilitating a meaningful comparison with parameters important for specialization. We will provide detailed explanations and discuss the rationale behind these operations in the revised manuscript. Thanks for your advice!\n\n**Q2: Depth Analysis on Full-FT** (Experimental Designs Or Analyses & Other Strengths And Weaknesses)\n\nA2: Full-FT directly updates all parameters to adapt to the downstream distribution. However, specialized downstream applications typically have limited data compared to large-scale pre-training datasets. Thus, Full-FT often leads to increased overfitting, especially when training for more epochs, resulting in performance degradation on downstream tasks. We will include a detailed analysis of Full-FT performance in the revised manuscript. Thank you for tips!\n\n**Q3: Differences between Regularization-based Optimization and Partial-based Updating solutions** (Supplementary Material)\n\nA3: Regularization-based optimization and partial-based updating represent two distinct approaches to addressing catastrophic forgetting in multimodal large language model (MLLM) tuning. The regularization-based approach introduces stiffness regularization terms, carefully balancing generalization and specialization through controlled penalization. In contrast, partial-based updating selects only a subset of parameters to update, preserving the rest. Our work adopts the partial-based updating paradigm, measuring parameter importance for both generalization and specialization, and selectively updating parameters significant for downstream tasks. We will clearly and concisely discuss the differences compared to existing methods in our revised manuscript. Thank you for your valuable suggestion!\n\n**Q4: Large Model Merging Paper Discussion** (Essential References Not Discussed & Other Strengths And Weaknesses)\n\nA4: Large Model Merging is a promising paradigm that integrates task-specific models into a unified model capable of simultaneously handling diverse downstream tasks [1]. Existing methods typically represent each task expert using downstream-optimized models and rely on various importance metrics to select specific elements for merging [2,3,4]. However, post-merging operations often disrupt the originally optimized parameter space, limiting the specialized capabilities for individual tasks. In our work, we select relative specialization importance elements during the training process, which is compatible with integrates into the training trajectory and squeeze the specialized capabilities into the target parameter space. We will add large model merging works discussion in our final version.\n\n[1] Learn From Model Beyond Fine-Tuning: A Survey, Nature Machine Intelligence, 2024\n\n[2] A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models, arXiv, 2024\n\n[3] Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models, ICML, 2024\n\n[4] REMEDY: RECIPE MERGING DYNAMICS IN LARGE VISION-LANGUAGE MODELS, ICLR, 2025", "claims_and_evidence": "Yes. This paper follows the relative requirements.", "methods_and_evaluation_criteria": "Yes. It includes difference metrics.", "theoretical_claims": "The authors could specify the value range boundaries for sections 2a-2c and provide a rationale for selecting the sigmoid and normalization operations.", "experimental_designs_or_analyses": "Although the authors conduct experiments on multimodal large language models tuning with larger training epochs, it would be better to provide the depth analysis why full-ft deceases the downstream performance with larger epochs.", "relation_to_broader_scientific_literature": "The author leverages both pre-training and downstream knowledge to construct an importance discrepancy mask, effectively mitigating forgetting while ensuring specialization. This approach presents an intriguing solution that enhances Multimodal Large Language Models (MLLMs) by striking a balance between domain-specific adaptation and generalization capability, ultimately improving their applicability and robustness in real-world scenarios.", "essential_references_not_discussed": "It would be helpful if the authors included references related to large model merging, as they make comparisons with the Tailor and Dare methods.", "other_strengths_and_weaknesses": "This paper explores the issue of catastrophic forgetting in Multimodal Large Language Models (MLLMs) fine-tuning. The manuscript is well-written, structured, and easy to follow, with the proposed method being clearly presented and demonstrating novelty. There are several weaknesses for updates.\n-The authors should include a discussion of relative performance, particularly in comparison to Full-FT. Full-FT, while a widely adopted and straightforward fine-tuning solution, yields lower performance across various downstream tasks, as shown in the Table 4 and especially for the flickr30k and coco-caption. A more detailed analysis would provide a clearer context for evaluating propose method advantages.\n-The authors should add a detailed discussion with the foundation model merge works. Because author compare the proposed method with both Tailor and Dare.", "other_comments_or_suggestions": "Please refer to the weaknesses.", "questions_for_authors": "If the authors address the issues mentioned in the weaknesses section, I would be happy to increase the score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZtR5QneJVF", "reviewer": "Reviewer_CY1z", "summary": "In this work, it focuses on the Multimodal Large Language Models (MLLMs) fine-tuning field and reveals the catastrophic forgetting on the pre-training knowledge. Authors assess parameter importance for both generalization and specialization, focusing on identifying downstream-important elements and performing critical-aware updates on selected parameters. Through a series of experiments, SPIDER is shown to effectively enhance task-specific performance, such as in image captioning and visual question answering, while preventing the degradation of generalization. This work offers an insightful contribution to improving fine-tuning practices for large, pre-trained multimodal models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CY1z:\n\nThank you for your insightful review and valuable feedback. Below, we address your key concerns in detail, aiming to clarify and demonstrate the effectiveness of our proposed approach.\n\n\n**Q1: Discussion on Incremental Learning** (Essential References Not Discussed & Questions For Authors)\n\nA1: Incremental learning focuses on continuously learning new tasks that are distinct from previously learned tasks, with the requirement to classify all observed classes during the testing phase [1,2,3]. Direct optimization solely on the current distribution can result in catastrophic forgetting of previously acquired knowledge. Existing incremental learning methods broadly fall into three categories: parameter isolation methods [4,5], regularization-based methods [6,7], and replay-based methods [8,9]. In this work, we investigate catastrophic forgetting within the multimodal large language model (MLLM) tuning process, aiming to mitigate the degradation of generalization and enhance specialization performance.\n\n[1] Class-incremental learning: A survey, IEEE PAMI, 2024\n\n[2] A continual learning survey: Defying forgetting in classification tasks, IEEE PAMI, 2021\n\n[3] A comprehensive survey of continual learning: theory, method and application, IEEE PAMI, 2024\n\n[4] Progressive Neural Networks, arXiv, 2016\n\n[5] Dense network expansion for class incremental learning, CVPR, 2023\n\n[6] Overcoming Catastrophic Forgetting in Neural Networks, PNAS, 2017\n\n[7] Memory aware synapses: Learning what (not) to forget, ECCV, 2018\n\n[8] icarl: Incremental classifier and representation learning, CVPR, 2017\n\n[9] Co-transport for class-incremental learning, ACM MM, 2021\n\n**Q2: Downstream Dataset Introduction** (Other Strengths And Weaknesses)\n\nFor downstream evaluation, we selected datasets (**Flickr30k**, **COCO-Caption**, **IconQA**, **ScienceQA**) that exhibit limited zero-shot performance or were excluded from the pre-training stage. Flickr30k and COCO-Caption focus on text generation tasks within general domains. Specifically, **Flickr30k** emphasizes everyday activities and events, while **COCO-Caption** features more complex, diverse scenes involving multiple objects and intricate interactions. Both IconQA and ScienceQA involve multiple-choice question answering tasks. **IconQA** evaluates abstract and symbolic reasoning using visual questions based on icons and diagrams, whereas **ScienceQA** assesses multimodal scientific reasoning in educational contexts, incorporating textual and visual content from disciplines such as physics, chemistry, biology, and general science. We will incorporate a detailed introduction to these downstream datasets in our revised paper.\n\n**Q3: Writing Clarity and Accessibility** (Other Comments Or Suggestions & Questions For Authors)\n\nA3: Thank you for highlighting this. We will simplify the formulations and enhance the clarity of our writing in the revised manuscript.\n\n**Q4: Relative forgetting and increase metrics** (Questions For Authors & Other Strengths And Weaknesses)\n\nA4: Thank you for the valuable suggestion. We will incorporate relative forgetting and increase metrics into Tables 4 and 5 of the revised manuscript, explicitly reporting both generalization forgetting and specialization improvement to provide a clearer evaluation of method effectiveness.\n\nThank you once again for your constructive feedback and valuable suggestions！", "claims_and_evidence": "The claim made in the paper is clear.", "methods_and_evaluation_criteria": "The authors respectively measure the downstream and upstream performances and further utilize the upstream-downstream mean value to measure the overall performance.", "theoretical_claims": "NaN", "experimental_designs_or_analyses": "The experimental results are comprehensive. The author conducts various experiments with different relative methods in various downstream tasks.", "relation_to_broader_scientific_literature": "The method is interesting as it achieves better downstream performance via measuring the parameter importance discrepancy.", "essential_references_not_discussed": "It is encouraged to add some discussions with incremental learning, as catastrophic forgetting is also a challenging and popular problem in incremental learning.", "other_strengths_and_weaknesses": "Strengths:\nThe approach of leveraging parameter magnitude and gradient norm to identify key task-specific parameters is compelling. The authors conduct extensive experiments across multiple datasets, including comprehensive ablation studies, to validate their method.\n\nWeaknesses:\nGiven that SPIDER focuses on fine-tuning Multimodal Large Language Models (MLLMs), the authors should report both generalization and forgetting metrics, as well as specialization improvements, to provide a more thorough comparison with existing methods.\n\nAs for the downstream dataset selection, the author should plot the detailed data introduction for their response type and which field they belong to.", "other_comments_or_suggestions": "The paper is written in a relatively complex manner, which may make it difficult for readers who are not experts in this specific field to fully understand it. It would benefit from improvements in clarity and accessibility, making it easier for a broader audience to grasp the key ideas and contributions.", "questions_for_authors": "The authors should add more discussion with incremental learning as catastrophic forgetting is a popular research problem in that scope. Besides, relative forgetting and increase metrics are also curial for readers to understand your paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ntb8s2XpbG", "reviewer": "Reviewer_mGiB", "summary": "This paper addresses the issue of catastrophic forgetting in the fine-tuning of multimodal large language models (MLLMs). The proposed method, Specialization via Importance Discrepancy Evaluation for Refinement (SPIDER), introduces an innovative approach for measuring parameter importance by utilizing the magnitude of frozen pre-trained weights and fine-tuning gradients. Based on this, SPIDER applies an importance-aware weight allocation strategy to prevent the pre-trained knowledge forgetting during fine-tuning. Experimental results demonstrate that SPIDER enhances downstream specialization performance while effectively mitigating generalization degradation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mGiB:\n\nThank you for your valuable comments and constructive feedback. Below, we carefully address each of your concerns point-by-point, providing detailed explanations and additional evidence to clarify our approach.\n\n**Q1: Memory Complexity** (Other Strengths And Weaknesses)\n\nA1: For our method, we utilize the pre-trained weights and the current gradient to evaluate parameter importance for both generalization and specialization, which brings a certain memory cost increase with existing methods. However, our method achieves stable performance across different tuning scenarios, as shown in the following Table. We will add the memory complexity comparison in the final version.\n\n**Q2: Parameter Modification Scale** (Other Strengths And Weaknesses)\n\nA2: Sorry for the misunderstanding. We plot the parameter modification scale in the following Table. We will add the parameter modification scale and notation table in the final version.\n\n*Table: Comparison with relative methods. $M$ means selected parameters. Performance is the harmonic mean of generalization and specialization accuracy.*\n|Method |Full FT|L2-Reg|Grafting|Half FT|DARE|Tailor|Ours|\n|----|---|---|---|---|---|---|--|\n|Memory Complexity|$\\mathcal{O}(\\|M\\|)$|$\\mathcal{O}(2\\|M\\|)$|$\\mathcal{O}(2\\|M\\|)$|$\\mathcal{O}(\\|M\\|)$|$\\mathcal{O}(\\|M\\|)$|$\\mathcal{O}(\\|M\\|)$|$\\mathcal{O}(3\\|M\\|)$|\n|Learnable Ratio|100%|100%|100%|50%|10%|10%|50%\n|Performance (VILA on Flickr30k)|55.16|48.33|49.58|59.52|53.66|54.56| **66.61**\n\n**Q3: Font size and Layout** (Other Strengths And Weaknesses)\n\nA3: Thanks for your advice. We will adjust the font size and layout to make it more readable!\n\n**Q4: Advantage Discussion for Proposed Method** (Other Strengths And Weaknesses) \n\nA4: Our method selects and optimizes the relative downstream important elements to adapt towards the downstream specialization while maintaining the generalization ability. Thus, our method does not bring additional architecture modification and does not couple with a specific model architecture. We will update Table 1 and provide a more detailed advantages discussion for our method in the final version.  \n\n**Q5: Explanation and Contribution for Selected Datasets** (Other Strengths And Weaknesses) \n\nA5: The datasets OKVQA, TextVQA, GQA, and OCRVQA are obviously mentioned as the training datasets in the pre-training stage, making them appropriate benchmarks to evaluate multimodal large language models’ (MLLMs) generalization across diverse tasks. OKVQA examines external knowledge and common-sense reasoning, TextVQA and OCRVQA test understanding of embedded textual information, and GQA assesses compositional and logical reasoning. Together, these datasets comprehensively evaluate MLLMs’ generalization across different reasoning types and practical scenarios. We will provide a detailed explanation of the selected datasets in the paper. Thanks for your comments!\n\n**Q6: Typos** (Other Comments Or Suggestions) \n\nA6: Thanks for your advice. We will fix the typos and check the final version carefully.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. The methods are well evaluated.", "theoretical_claims": "The authors provide clear normalization boundary analysis for both the weight and gradient parts.", "experimental_designs_or_analyses": "The experimental results are comprehensive. The author conducts various experiments with different relative method in various downstream tasks.", "relation_to_broader_scientific_literature": "The author introduces a novel aspect to alleviate the forgetting during MLLM fine-tuning via both pre-trained magnitude and current gradient information.", "essential_references_not_discussed": "NaN", "other_strengths_and_weaknesses": "Pros:\n\nThe SPIDER method presented in this paper is novel in that it does not rely on additional parameters. Instead, it utilizes gradients to guide parameter updates, influencing the update direction and making the model’s forgetting mechanism interpretable. Experimental results validate its effectiveness on LLaVA and VILA. Moreover, the approach is compatible with other optimization strategies, offering both simplicity and efficiency.\n\nCons: \n- While SPIDER focuses on mitigating forgetting, its memory complexity may pose challenges for scaling. It is better to discuss the memory cost with the existing methods for better understanding.\n- The authors could plot the parameter modification scale with existing methods. For example, both dare and tailor select a partial element for updates. Besides, the notation table is also important to make the paper reading friendly.\n- The text in Table 4 is too small, and some parts are obscured. Please consider increasing the font size or adjusting the layout for better readability.\n- The meaning of “Architecture Couple” in Table 1 seems unclear. Please provide a more detailed explanation of the significance of SPIDER and its advantages in the context of “Architecture Couple”.\n- Why OKVQA, TextVQA, GQA, and OCRVQA are chosen as the datasets to evaluate the generalization ability? What do these datasets represent, and how do they contribute to assessing generalization?", "other_comments_or_suggestions": "Typos: “COCO-Capation” should be “COCO-Caption” in L124, L439, etc.", "questions_for_authors": "-Please refer to Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses catastrophic forgetting in fine-tuning Multimodal Large Language Models (MLLMs) by proposing SPIDER,  which balances generalization and specialization through parameter importance measurement. SPIDER employs frozen pre-trained weights and fine-tuning gradients to guide selective parameter updates. This work validates the approach across diverse architectures (e.g., LLaVA, VILA) and tasks (image captioning, VQA), demonstrating downstream performance improvement while mitigating generalization degradation.

This paper makes a timely contribution to MLLM fine-tuning by addressing catastrophic forgetting through a theoretically grounded and empirically validated approach. The reviewers have given uniformly positive ratings, ranging from 3 (weak accept) to 4 (accept). Based on the novelty, scalability, and comprehensive evaluation, the meta-review recommends acceptance of the submission.
Decision: Accept (poster)