An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models (2024)

Abstract: August 11-16, 2024 ©2024 Association for Computational Linguistics An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models Xiongtao Zhou* Jie He* Yuhua Ke Guangyao Zhu Víctor Gutiérrez-Basulto and Jeff Z. Pan 1 Waseda University, Japan 2 School of Informatics, University of Edinburgh, UK 3 School of Computer Science and Informatics, Cardiff University, UK alenai.tao@ruri.waseda.jp, j.he@ed.ac.uk s2484588@ed.ac.uk, zhuzgy@akane.waseda.jp gutierrezbasultov@cardiff.ac.uk, j.z.pan@ed.ac.uk Abstract

heading: 1 Introduction
text: In recent years, the landscape of multimodal learning has been transformed by the emergence of multimodal large language models (MLLMs), such as LLaVA (Liu et al., 2023b), MiniGPT4 (Zhu et al., 2024), and GPT4-Vision (OpenAI et al., 2023). MLLMs have showcased impressive competency across a spectrum of multimodal benchmarks (Fu et al., 2023; Liu et al., 2023c; Li et al., 2023b) thanks to the integrated architecture of pre-trained visual encoders, connector layers, and LLMs. This
* Equal Contribution.
architecture is usually fine-tuned through multimodal instruction-following data (Xu et al., 2023). During fine-tuning, most existing MLLMs (Cha et al., 2023; Su et al., 2023; Lin et al., 2023) typically freeze the visual encoder, focusing solely on connector layers and the LLM component. Since LLMs (e.g. LLaMA (Touvron et al., 2023) and Vicuna-v1.5 (Chiang et al., 2023)) often contain hundreds of billions of parameters, full fine-tuning (FFT) (Wang et al., 2022) is unfeasible. Consequently, the parameter-efficient fine-tuning (PEFT) (Houlsby et al., 2019; Hu et al., 2021) approach (which leverages lightweight trainable parameters and keeps the majority of parameters frozen) has been widely employed in NLP for fine-tuning LLMs with instruction or task-specific datasets (Li et al., 2023c; You et al., 2023), as they allow for significant resource savings while achieving comparable performance or even surpassing FFT (Mangrulkar et al., 2022).
In contrast to standard LLMs, MLLMs introduce additional modules: visual encoder and connector layers. During the fine-tuning process, unimodal LLMs only receive text features while MLLMs get multimodal inputs, such that connector layers are also fine-tuned, not just fine-tuning the LLM. Therefore, it is crucial to reassess the performance of fine-tuning MLLMs using various PEFT methods, exploring the impact of connector fine-tuning on the model’s performance in downstream tasks, and examining PEFT’s effects on model stability, generalization, and hallucination. In this paper we address these issues by conducting comprehensive studies on three representative MLLMs containing connector layers: LLaVA-1.5 (7B, 13B) (Liu et al., 2023a), ShareGPTv4 (7B) (Chen et al., 2023), and Qwen-VL-Chat (7B) (Bai et al., 2023b). Our study looks at various issues related to PEFT methods. Specifically, we design our study to address the following questions: (1) Is it necessary to fine-tune the connector when fine-tuning MLLMS via vari-
1 10057
ous PEFT methods on unseen and seen datasets? (2) How does the position of the PEFT module in the LLM affect the MLLM’s performance? (3) Faced with different training data scales, what differences exist in the performance of different PEFT methods? (4) How do different PEFT approaches impact the stability of the model? Is there any relationship between trainable parameters and learning rate with stability?
Our key findings can be summarized as follows:
1. Fine-tuning the connector layers usually leads to performance improvement within MLLMs. 2. More trainable parameters results in better performance on unseen datasets, while fewer trainable parameters maintains the model’s performance on seen datasets. 3. Generally, fine-tuning using large scale datasets leads to better performance, but when resources are limited, fine-tuning on mediumsize datasets is more efficient. 4. Adapter shows the best overall performance in model generalization, stability, and hallucination.
Our contributions can be summarized as follows: (1) We have assembled a standardized evaluation suite that includes seven benchmarks from the vision-and-language research community. This suite encompasses five tasks in visual question answering, one in visual reasoning, and one in image caption, along with four PEFT methods. (2) We utilized these resources to conduct in-depth experiments investigating four crucial design dimensions (cf. Fig. 1, left): 1) data scaling, 2) stability of the training process, 3) overfitting and generalization, and 4) hallucination. (3) Our empirical findings show that Adapter outperforms other PEFT methods in all aspects, followed in second place by LoRA. Furthermore, we show that fine-tuning the connector layers frequently enhances performance within MLLMs.

heading: 2 Related Work
text: Multimodal Large Language Models. Flamingo (Alayrac et al., 2022) proposes a GATED XATTN-DENSE layer to align visual and textual features, connecting the visual module and language model. LLaMA-adapter (Zhang et al., 2024) applies a projection layer to connect a visual encoder and LLaMA. It proposes adding an adapter module on LLaMA, keeping only the
adapter parameters updated during training. In contrast, LLaVA-1.5 (Liu et al., 2023a) employs two layers of MLP to connect the visual encoder and LLM. During fine-tuning, it only updates the parameters of the MLP and LLM. Subsequent works mostly build upon this approach, employing the connector layers to link a visual encoder and LLMs, and then fine-tune the model using multimodal instruction-following data (Li et al., 2023a; Hu et al., 2023a; Wang et al., 2023). Recently, there are also many work on multimodal large language models (Chen et al., 2024) from the perspective of knowledge computing (Pan et al., 2023). Parameter-Efficient Fine-Tuning. Parameterefficient fine-tuning emerges as an approach capable of achieving performance comparable to full fine-tuning while keeping the majority of parameters frozen. Prompt-based methods (Lester et al., 2021) incorporate soft prompts into the input prefixes, only updating these soft prompts. Another widely used family of methods is based on adapters (Pfeiffer et al., 2020; He et al., 2022; He and Fu, 2023), which insert adapter modules at specific positions within transformer layers and update only the parameters of these inserted modules during training. Also, in MLLMs, low-rank decomposition methods are commonly employed (Hu et al., 2021; Edalati et al., 2022). These methods involve training only the parameters in low-rank matrices, significantly reducing the number of trainable parameters.

heading: 3 PEFT Methods
text: Figure 1 illustrates the architecture of MLLMs and the location of various PEFT modules. In our experiments, all considered MLLMs consist of three components: a visual encoder, connector layers, and a LLM. The structure of the connector layers may vary depending on the specific MLLM. PEFT methods can be classified into three categories, from which we select four methods: (1) reparametrization-based tuning: LoRA, IA3 (2) adapter-based tuning: Adapter. (3) prompt-based tuning: Prefix-Tuning. LoRA. We integrate the low-rank strategy proposed by Hu et al. (2021) to adjust the network weights, facilitating the model’s handling of complex tasks with an efficient parameter footprint. The original pre-trained weight matrix W0 ∈ Rd×k is updated through low-rank decomposition using
2
Equation 1, where B ∈ Rd×r and A ∈ Rr×k.
W0 +∆W = W0 +BA (1)
This method ensures our model’s adaptability is improved without a significant increase in the parameter space. IA3. Following Liu et al. (2022), we integrate three vectors vk ∈ Rdk , vv ∈ Rdv , and vff ∈ Rdff into an attention mechanisms as:
softmax( Q(vk ⊙KT )√
dk )(vv ⊙ V ) (2)
where ⊙ represents the element-wise multiplication, and (vff ⊙ γ(W1x))W2 in the position-wise FFN layers, leveraging γ as the activation function. These formulas guide the model’s attention to be fine-tuned to prioritize relevant features, optimizing performance without significantly increasing the model’s complexity or number of parameters. Adapter. We adopt the structure proposed by Houlsby et al. (2019), which adds adapter modules to the fully-connected networks after attention and the FFN layer within the transformer layers. This can be captured as follows:
hi + f(Wdown(hi))Wup → hi (3)
where hi is the output of the previous layer, which is initially down-projected by Wdown ∈ Rd×r to a lower dimension r, and then up-projected back by Wup ∈ Rr×d to the original dimension d, f is a non-linear layer. Prefix-Tuning. We follow the approach proposed by Li and Liang (2021) to employ prefix learning by appending task-specific vector “prefixes” to the input sequence fed into the pre-trained model. We
initialize a trainable matrix Pθ with dimensions |Pidx| × dim(yi), where Pidx specifies the prefix length. This yields the following conditional formulation for each element yi of the output sequence:
yi = { Pθ[i, :] if i ∈ Pidx, LMϕ(zi, y<i) otherwise.
(4)
If i ∈ Pidx, a bidirectional encoder computes the yi. For i /∈ Pidx, yi is computed by an autoregressive neural LM as a function of yi and the past activations in its left context.

heading: 4 Experiment Setup

heading: 4.1 Datasets
text: In the current era of large-scale models, dataset contamination is a significant concern as it is challenging to ensure that the data will be used for the next training process constitutes unseen data for large language models. Therefore, we categorize the datasets into two types: Unseen datasets, comprising datasets that have not been involved in the training of any of the considered models, including (1) the ScienceQA dataset (Lu et al., 2022); (2) the Vizwiz dataset (Gurari et al., 2018); (3) the IconQA dataset (Lu et al., 2021); and (4) the Flickr30k dataset (Young et al., 2014). Seen datasets, consisting of datasets used in the training of all considered models, including (1) the OKVQA dataset (Marino et al., 2019); (2) the OCRVQA dataset (Mishra et al., 2019); and (3) the VQAv2 dataset (Goyal et al., 2017). Details about datasets can be found in App. A.
3

heading: 4.2 Implementations
text: Models. We selected LLaVA-1.5 (7B, 13B), ShareGPTv4 (7B), and Qwen-VL-Chat (7B) as the base models for our experiments. Hyperparameters. We conduct fine-tuning on the training set of each dataset separately and then test on their respective test or validation sets. All experiments were conducted with a global batch size of 128. We set the random seed of the experiment to 42. Additionally, each PEFT method was trained for three epochs on the fine-tuning dataset. For LoRA, we set its learning rate to 2e-4, the adapter’s to 5e-5, IA3’s to 2e-4, and Prefix-Tuning’s to 1e-5. More information about model and hyperparameter settings is available in App. B

heading: 5 Experimental Results

heading: 5.1 Main Results
text: Should we tune or freeze the connector when considering unseen and seen datasets? Given the increasing availability of pretraining data for MLLMs, encountering contaminated data (i.e. training data contains information that is meant to be present only in the test set) is increasingly common. Additionally, most current MLLMs tune the connector layers during fine-tuning, yet the role of the connector remains unclear. Our main experiment thus focuses on investigating the performance
of PEFT methods on both unseen and seen datasets. In our experiments, following existing work (Liu et al., 2023b; Bai et al., 2023b; Chen et al., 2023), we freeze the visual encoder and apply the PEFT module to fine-tune the LLM. For the connector layers, we experimented with both FFT and freezing. We investigate the model’s performance on certain tasks during the task-specific fine-tuning when unfreezing the visual encoder, see App. C for details about unfreezing the visual encoder. We investigated the performance of LLaVA-1.5 (7B, 13B), ShareGPT4v, and Qwen-VL-Chat on the datasets mentioned in Section 4.1.
The obtained results are presented in Table 1. One can observe that LLaVA-1.5-13B with IA3 and fine-tuned connector layers achieved the best results across all unseen and seen datasets. Most IA3 models with fine-tuned connector achieved comparable performances to LoRA and Adapter on unseen datasets, while also maximizing the model’s performance on seen datasets. Benefiting from the increased number of parameters in LLaVA-1.513B, we noticed that across various settings, the average performance on all datasets of LLaVA1.5-13B surpasses that of LLaVA-1.5-7B. The average performance of ShareGPTv4 generally surpasses (except for the IA3 method with the frozen connector) that of LLaVA-1.5-7B, because it has been fine-tuned on the higher-quality multimodal instruction-following dataset (Chen et al., 2023). Under the setting of freezing the connector layers and fine-tuning the LLM with LoRA, Qwen-VLChat achieved the best performance. We found that choosing to tune the connector layers often leads to a significant deterioration in Qwen-VL-Chat’s performance on seen datasets. Figure 2 illustrates the performance of various PEFT methods under the settings of tuning or freezing the connector layers. When fine-tuning the connector layers, LoRA, Prefix-Tuning, and IA3 all exhibit better performance than freezing the connector layers on unseen datasets. In this case, IA3 gets a 15.0% increase in the average result. On seen datasets, in most cases, the performance of freezing the connector layers and that of the remaining PEFT methods (except for a slight decrease in LoRA’s performance) is similar. Note that whether the connector layers are fine-tuned or not, the performance of the Adapter remains relatively consistent on both seen and unseen datasets. Our main findings are the following:
4
• LoRA and Adapter have the best performance on all of the unseen datasets, while IA3 and Prefix-Tuning perform the best on the OKVQA and VQAv2. More trainable parameters allows the model to better adapt to unseen datasets, while fewer trainable parameters can maintain the model’s performance on seen datasets. • For the unseen datasets, tuning the connector layers often outperforms freezing the connector layers. For the seen datasets, freezing the connector layers yields the best performance.

heading: 5.2 Module Location
text: What is the best location for the PEFT module for MLLMs? Unlike LLMs, MLLMs include additional connector layers and visual encoders. Therefore, we can not straightforwardly transfer existing results for LLMs to MLLMs. With this in mind, we directly address this issue here. To this end, we selected all VQA datasets for the location study. We choose LLaVA-1.5-7B as the base model set the random seed to 42, freeze the visual encoder, and fine-tune the connector layers and LLM. We used this setting in subsequent experiments. For LoRA and IA3, we integrate them into the model’s multi-head attention layer, MLP
layer, or both. For adapters, we placed them in the same locations. The result of Qwen-VL-Chat can be found in App. D. Note that we do not consider Prefix-Tuning as the position is fixed. Table 2 presents the results on LLaVA-1.5-7B, which suggest that despite the additional modules in MLLMs compared to LLMs, the results of Hu et al. (2023b) for fine-tuning LLMs are also valid for MLLMs.
• We observe that for LoRA and IA3, the Both setting achieved the best results. As for Adapter, inserting it only into the MLP layer yielded the best performance.

heading: 5.3 Data Scale
text: In practical applications, MLLMs often require fine-tuning on downstream datasets (Li et al., 2023c; You et al., 2023), making PEFT an efficient choice. However, the sizes of these specific task datasets may vary, leading to the question: How to select PEFT methods for datasets of different scales when training? Therefore, we investigate the performance of PEFT methods on datasets of varying scales. We followed Chen et al. (2022) resource setting, and randomly sampled 1k, 5k, and 10k data points from the training set of each dataset. We categorize 1k data points as Low-Resource, 5k
5
data points as Medium-Resource, and 10k data points as High-Resource. Note that since the training set of OKVQA contains only 9k samples, we considered the full data as high-resource. Table 3 presents the results.
Our main findings are the following:
• High-Resource will make the MLLM more powerful, while Medium-Resource will be more efficient. The performance of the four PEFT methods improves as the scale of resources grows, i.e. all achieve their best performance with high-resource. Thus, when resources are sufficient, fine-tuning on highresource datasets will yield better performance. Average performance improvement is shown in App. E.
• The unseen datasets tend to favor more resources. When fine-tuning on an unseen dataset with more data, all PEFT methods show a significant performance improvement. In contrast, as the resources of the dataset increase, we did not observe significant performance improvement on seen datasets.

heading: 5.4 Stability Analysis
text: He et al. (2021) and Chen et al. (2022) carried out experiments with different random seeds to
investigate the instability of fine-tuning LLMs using PEFT methods. Analogously, we look at such instability for MLLMs. We concentrate on the SQA (img) from the unseen datasets and OKVQA from seen datasets and select three random seeds: [seed21, seed42, seed63]. We present a stability analysis for LLaVA-1.5-7B, more analysis can be found in App. F.
The number of trainable parameters plays a crucial role in the fine-tuning process of a model. However, in the multimodal setting, the relationship between the number of trainable parameters and stability when fine-tuning with PEFT is not yet clear. With this in mind, we look at the following question: Does fewer trainable parameters lead to higher stability? We conducted an experiment under different trainable parameter conditions: on seed 21, seed 42, and seed 63, and varied the Lora Rank, Adapter Bottleneck Size, and Virtual Tokens to control the number of trainable parameters. Table 6 presents the performance of various PEFT methods and their standard deviations under different numbers of trainable parameters. IA3 is not tested since its trainable parameters cannot be modified. We draw the following conclusions:
• Adapter and LoRA exhibit drastically different levels of stability on the unseen and seen
6
datasets. Prefix-Tuning shows a strong instability on the unseen datasets. Adapter gradually stabilizes with decreasing parameters on OKVQA, but becomes unstable with fewer parameters on SQA (img). Conversely, LoRA becomes unstable with decreasing parameters on OKVQA, but stabilizes with fewer parameters on SQA (img). Prefix-Tuning exhibits stability on OKVQA, and shows a relatively stable performance with fewer parameters on SQA (img).

heading: 5.5 Overfitting and Generalization
text: How robust different PEFT methods are relative to overfitting? To address this question, we consid-
ered three datasets from the unseen datasets: SQA (img), IconQA-txt, and Vizwiz. We choose one dataset from these three datasets as the source domain, and fine-tuned LLaVA-1.5-7B with PEFT methods on each source domain for 12 epochs.
Adapter and LoRA exhibit stronger robustness. Figure 3 (and Figure 8 in App. G) shows the evaluation loss of various PEFT methods on SQA as the number of training epochs changes. We observe that when overfitting occurs, there are differences in the robustness exhibited by each PEFT method on each dataset. On SQA, LoRA, IA3, and Adapter a relatively strong robustness is demonstrated, with LoRA performing the best. Prefix-Tuning shows
7
poor robustness on SQA. App. G provides further analysis on IconQA-txt and Vizwiz.
When facing overfitting an important question is How do various PEFT methods perform in terms of generalization? And how to achieve the best generalization performance during training? With these questions in mind, we conducted the next experiment. Based on Figure 3, we identified the training step with the minimum evaluation loss for each PEFT method. We selected four overfitting points which are the closest to the minimum evaluation loss point on the source domain. Subsequently, we tested the performance of each epoch on the other two target domains, yielding the results shown in Table 4. We draw the following conclusions.
• Adapter exhibits the strongest in generalization. Figure 4 shows that when a model is fine-tuned using Prefix-Tuning its generalization performance is quite poor. Models using Adapter consistently exhibit a good generalization performance regardless of the situation, while the generalization performance of a model finetuned with Prefix-Tuning is consistently negative. Models using LoRA and IA3 show fluctuation in generalization performance. • IA3, Adapter, and Prefix-Tuning show the best generalization performance at the first overfitting epoch. From the results of Table 4, we can
find that Adapters, IA3, and Prefix-Tuning, all achieve the best average generalization performance at the first overfitting epoch. In general, the model’s generalization weakens as overfitting intensifies. However, LoRA achieves the best model generalization performance at the third overfitting epoch, indicating that models fine-tuned with LoRA exhibit the best generalization when overfitting reaches a certain level, gradually weakening afterwards.

heading: 5.6 Hallucination
text: The hallucination problem in LLMs has been widely acknowledged (Ji et al., 2023; Gudibande et al., 2024). Since MLLMs are built upon LLMs, this problem is also present in them. Zhai et al. (2023) found that further fine-tuning with multimodal instruction-following data leads to hallucinations. Therefore, we aim to investigate the following question: Which PEFT method results in fewer hallucinations during fine-tuning? We select IconQA-txt as the source domain and the Flickr30k dataset as the target domain to assess the out-ofdomain hallucinations of models fine-tuned with various PEFT methods. App. I.1 elaborates on how we evaluated the model’s hallucination.
MMHAL-Bench (Sun et al., 2023) is used to evaluate the hallucinations induced by finetuning LLaVA-1.5-7B with four PEFT methods on Flickr30k and IconQA-txt, yielding the results presented in Table 5. The results show that Adapter consistently achieved the highest Avg Score and the
8
lowest Hallucination Rate across both fine-tuning tasks. This is consistent with our manual evaluation results.
Adapter demonstrates potential for addressing hallucinations in MLLMs. The results are illustrated in Table 7. We observe that Adapter achieves the lowest average hallucination rate across four epochs, at only 13.3%. It can also be found that other PEFT methods tend to produce more hallucinations after further fine-tuning, especially PrefixTuning, which generates an additional 24% of hallucinations from epoch 3 to epoch 12. In contrast, with further fine-tuning, Adapter reduced the number of hallucinations produced. In line with previous studies (Wang et al., 2023), we attribute this phenomenon to the new parameters in the Adapter method, which provides a new module to adapt to downstream datasets while keeping the base model’s original weights.

heading: 6 Conclusion
text: We conducted an extensive investigation on four PEFT methods applied to MLLMs across different multimodal tasks. By fine-tuning different MLLMs in a uniform way and conducting thorough hyperparameter optimization, we benchmarked the performance of these methods. Our findings indicate that Adapter excels in accuracy, stability, generalization, and producing fewer hallucinations. Additionally, we found that fine-tuning the connector layers of MLLMs simultaneously does not always yield better results. Finally, comprehensive ablation studies were performed to understand the contributions of the location of PEFT modules, learning rate settings, and the size of training data on PEFT performance.
Limitations
All our experiments were conducted within the defined framework, which involves connector layers serving as the bridge between the visual encoder and LLM, and no additional modules were inserted on the LLM. Due to the limitation of computational resources, we have currently employed only a subset of datasets to conduct our analysis. Additionally, our choice of MLLMs on the analysis experiments is limited to LLaVA-1.5-7B or QwenVL-Chat. In the future, we plan to conduct an analysis on more datasets and MLLMs.

heading: A Datasets Setup
text: A.1 Detailed Description of Datasets Since OCRVQA and VQAv2 are very large, we randomly extract 20k samples from their training sets as the new training sets and another 5k samples from the test sets to create the new test sets. We utilized a variety of multimodal datasets for finetuning and evaluating. Detailed information for each dataset is provided in Table 8 below.
A.2 Seen Datasets for All MLLMs In our experiments, we divided the datasets into unseen datasets and seen datasets. In Table 9, we present the training datasets used by each MLLM in our experiments. It is worth noting that, due to the data used in both pre-training and fine-tuning stages being filtered or sampled from the datasets, to ensure a fair comparison, it is imperative that each seen dataset is fully seen by the model. We composed a mixed dataset consisting of three seen datasets: OKVQA, VQAv2, and OCRVQA. Subsequently, we kept the visual encoder frozen and conducted a Full Fine-Tuning on each model using this mixed dataset. During this process, we set the learning rate to 2e-5 and the global batch size to 128. We maintained all other settings the same as those used in the original paper for Full Fine-Tuning (Liu et al., 2023a; Bai et al., 2023b).
A.3 Instruction-following Data Template Multimodal Instruction-Following Tuning is a crucial component for the success of MLLMs. The MLLMs used in this experiment follow the same approach in data processing as the original models. Therefore, there are slight differences in the processing of LLaVA-1.5, ShareGPT4v, and QwenVL-Chat. They employ different image annotations, but the data instruction format remains consistent. Table 10 shows the template of all dataset types used in the experiments.

heading: B Models and Hyperparameters
text: B.1 Models LLaVA-1.5 consists of CLIP-ViT-L/14 (Ilharco et al., 2021), Vicuna-v1.5 (Chiang et al., 2023), and an MLP serving as the connector layer. During fine-tuning with multimodal instruction-following data, LLaVA-1.5 updates only the parameters of the MLP connector and the LLM, while keeping the parameters of the visual encoder frozen. ShareGPTv4 is obtained by fine-tuning LLaVA-1.5 using
the ShareGPT4v dataset (Chen et al., 2023), which comprises 100K high-quality captions from various images generated by GPT4-vision. Qwen-VL-Chat comprises ViT-G/16 (Ilharco et al., 2021), Qwen7B (Bai et al., 2023a), and a cross-attention module serving as the connector. During its vision instruction tuning phase, Qwen-VL-Chat updates only the parameters of the connector and the LLM.
B.2 HyperParameters Following Hu et al. (2023b), we conducted the following parameter selection experiments. Due to computational constraints, we concentrate on the SQA (all) dataset, which has a test set that contains both multimodal and text-only data. So, our goal is to enhance the model’s multimodal performance while maximizing its performance on text-only datasets.
We choose LLaVA-1.5-7B as the base model set the random seed to 42, freeze the visual encoder, and fine-tune the connector layers and LLM. Note that we do not consider IA3 in this experiment as it cannot change the number of trainable parameters. We look at different parameter settings: LoRA rank of {16, 32, 64, 128}, Adapter bottleneck size of {32, 64, 128, 256}, and virtual tokens of {10, 20, 30, 40} in Prefix-Tuning. Figure 5 shows the results with various parameter settings on SQA (all). We observe that for LoRA, a rank of 128 yielded an accuracy of 89.3%. Setting the adapter’s bottleneck size to 256 resulted in an accuracy of 87.4%. In the case of Prefix-Tuning, setting the virtual tokens to 20 achieved the best accuracy of 68.0%.
Based on our findings, for all remaining experiments, we use the following PEFT parameters: LoRA Rank=128, Adapter Bottleneck Size=256, and Prefix Virtual Token=20. More detailed hyperparameter settings are presented in Table 11. We utilized two NVIDIA A100 80GB GPUs and DeepSpeed for distributed training.

heading: C Training of Visual Encoder Analysis
text: Qwen-vl-chat unfroze the visual encoder during pre-training, which improved the model’s performance. However, most current MLLMs maintain the visual encoder frozen during task-specific finetuning. We fine-tuned the visual encoder on SQA and VizWiz for unseen tasks, and on OKVQA and OCRVQA for seen tasks. The results are shown in the Table 12. We found that although unfreezing the visual encoder does not significantly increase training resource consumption, the improvement in
13
model performance is limited and, in most cases, can even lead to performance degradation. Therefore, in our experiments, we adhered to the mainstream setting and kept the visual encoder frozen.

heading: D Location Analysis
text: We also conducted the Module Location experiment on Qwen-VL-Chat. Table 13 shows the results. We observe that for LoRA and IA3, Both settings achieved the best results. As for Adapter, inserting it only into the MLP layer yielded the best performance. It reveals that the results on QwenVL-Chat are consistent with those on LLaVA-1.57B.

heading: E Data Scale Analysis
text: Figure 6 shows the improvement in the average performance of the four PEFT methods as resources transition from low to high. When datasets transition from low to medium resources, all four PEFT methods achieve performance improvements of over 10%, higher than from medium to high resources. Thus, when computational resources are limited, fine-tuning on medium-resource datasets
is more efficient.

heading: F Stability Analysis
text: Figure 7 presents the training loss, showing that the stability of PEFT varies across different datasets. We observe that Prefix-Tuning and Adapter exhibit larger fluctuations in training loss at each step when trained with different seeds, followed by LoRA, while IA3 shows relatively smaller fluctuations.
We also investigate whether the learning rate correlates with stability. We conducted experiments with learning rates of {2e-4, 5e-5, 1e-5, 5e-6}. The results are shown in Table 14. It can be observed that IA3 and Prefix-Tuning demonstrate more stable performance at smaller learning rates, PrefixTuning tends to stabilize gradually as the learning rate decreases.

heading: G Overftitting and Generalization Analysis
text: Overfitting and Generalization experiments were conducted on three unseen VQA datasets. The train-loss curves for IconQA-txt and VizWiz are depicted in Figure 8. For IconQA-txt, all four PEFT
14
methods exhibit strong robustness, with LoRA performing the best. On Vizwiz, Prefix-Tuning shows the strongest robustness compared to the other three PEFT methods.
H Efficiency
We investigate the number of trainable parameters, the training and inference Flops for various MLLMs when fine-tuned with different PEFT methods in this section. Table 15 shows the results. We derive the training and inference FLOPs in accordance with the methodology outlined in Kaplan et al. (2020). Our analysis shows that models perform better when the connector is not frozen, which suggests that having more trainable parameters improves the performance, even though the overall parameter efficiency might decrease. Within the 7B model, the Adapter method without freezing connector is remarkably efficient, utilizing only 3.060% of trainable parameters and yet securing a high performance rate of 76.3%, showcasing an optimal balance between parameter efficiency and model efficacy.
Additionally, using the IA3 method without freezing the connector can reduce the computing effort needed for training. With more trainable parameters, the model becomes more efficient, pro-
ducing shorter and more accurate texts and thus requiring less computing power, even as the number of trainable parameters grows. In the case of the 13B model, the increase in the total number of parameters is not necessarily reflected in an increase in the percentage of trainable parameters to reach a good performance. According to Table 15, the IA3 method without a frozen connector yields significantly fewer trainable parameters compared to the Adapter and LoRA methods, but achieves the highest performance.
The details of the flops calculation are :
Training Flops Since the computational cost of the backward pass is approximately twice as the forward pass, we modify the formula as:
Train Flops = (2Pf + 4Pt)×Nt (5) where Pf and Pt represent the number of frozen and trainable parameters respectively, Nt is the number of input and model-generated tokens.
Inference Flops We calculate the inference flops based on the following equation:
Inference Flops = 2×(Pg+NlayerdmodelNt) (6)
where Pg indicates non-embedding parameters,
15
Nlayer is the number of model’s layers, and dmodel represents the dimension of the residual stream.

heading: I Case Study
text: I.1 Hallucination Analysis In this section, we explain how we tested the model’s hallucination. As the first step, we em-
ployed LLaVA-1.5-7B to generate captions for all images in the Flickr30k test set in a zero-shot manner. Subsequently, we randomly sampled 1k captions and manually curated 100 correct captions without hallucinations. Then, we utilized the LLaVA-1.5-7B model fine-tuned with four PEFT methods on IconQA-txt to generate captions for these 100 samples. Thereafter, we manually annotate the fine-tuned model-generated outputs and count the number of hallucination samples.
One sample is selected, as illustrated in Figure 9. In this example, the original LLaVA model delivers a hallucination-free description, accurately identifying the color and actions depicted in the image. On the other hand, the Adapter model incorrectly identifies the girl’s face as red. The IA3 model inaccurately attributes a mustache to the girl and misidentifies her hair color as red. The LoRA model also fails to recognize the girl’s hand and refers to a red substance, which is not present. As for the Prefix model, it attempts to provide a detailed description of the picture, including the mention of a ponytail and attributing emotions such as a "funny or playful gesture" to the subject. How-
16
ever, these emotions cannot be confirmed simply by viewing the picture. Additionally, the Prefix model presents more severe hallucinations compared to the others, as it "imagines" another hand that is not visible in the image. More examples at several epochs are illustrated in this section.
I.2 Qualitative Illustrations In this section, We randomly sampled several examples from each dataset and provided the original labels along with the outputs of various PEFT models. See Figures 14 to 23.
17
18
19
20

heading: A man jumps in the air in a park.
text: 21
22
23
24
25
26
27
28

heading: Future Work
text: In the future, we plan to conduct an analysis on more datasets and MLLMs