Multimodal Instruction Tuning with Conditional Mixture of LoRA (2024)

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in diverse tasks across different domains, with an increasing focus on improving their zeroshot generalization capabilities for unseen multimodal tasks. Multimodal instruction tuning has emerged as a successful strategy for achieving zero-shot generalization by fine-tuning pretrained models on diverse multimodal tasks through instructions. As MLLMs grow in complexity and size, the need for parameterefficient fine-tuning methods like Low-Rank Adaption (LoRA), which fine-tunes with a minimal set of parameters, becomes essential. However, applying LoRA in multimodal instruction tuning presents the challenge of task interference, which leads to performance degradation, especially when dealing with a broad array of multimodal tasks. To address this, this paper introduces a novel approach that integrates multimodal instruction tuning with Conditional Mixture-of-LoRA (MixLoRA). It innovates upon LoRA by dynamically constructing low-rank adaptation matrices tailored to the unique demands of each input instance, aiming to mitigate task interference. Experimental results on various multimodal evaluation datasets indicate that MixLoRA not only outperforms the conventional LoRA with the same or even higher ranks, demonstrating its efficacy and adaptability in diverse multimodal tasks1.

heading: 1 Introduction
text: The advent of Multimodal Large Language Models (MLLMs) (Li et al., 2023a; Liu et al., 2023; Driess et al., 2023; Dai et al., 2023) have revolutionized the field of artificial intelligence, demonstrating remarkable capabilities in processing and integrating information from various modalities, notably text and image. A key focus in advancing MLLMs is
1Our code is publicly available at https://github.com/ VT-NLP/MixLoRA
to enhance zero-shot generalization to novel multimodal tasks. In this pursuit, multimodal instruction tuning, which fine-tunes pre-trained models with diverse, instruction-based multimodal tasks, has demonstrated its efficacy in facilitating zero-shot generalization to unseen multimodal problems (Xu et al., 2023b; Liu et al., 2023; Ye et al., 2023).
Concurrently, the growing complexity and scale of MLLMs have spurred the development of various parameter-efficient fine-tuning (PEFT) techniques (Lee et al., 2019; Hu et al., 2021; Li and Liang, 2021; Karimi Mahabadi et al., 2021; Guo et al., 2021; Zaken et al., 2022). Among these, Low-Rank Adaption (LoRA) (Hu et al., 2021) has emerged as a powerful PEFT method that finetunes large pre-trained models by updating a small amount of injected adaption parameters. However, in multimodal instruction tuning, the effectiveness of conventional PEFT methods like LoRA diminishes due to their reliance on adjusting a limited portion of shared parameters to simultaneously accommodate diverse tasks, leading to task interference – a problem well-studied in multi-task learning (Yu et al., 2020; Liu et al., 2021; Navon et al., 2022), but insufficiently investigated in the context of parameter-efficient multimodal instruction tuning. The diverse nature of multimodal tasks significantly increases the risk of task interference. For instance, using the same limited set of adaptation parameters for distinct tasks like OCR and domain-specific classification can cause conflicting updates, potentially leading to suboptimal performance. Our research seeks to explore and address task interference in parameter-efficient multimodal instruction tuning. Specifically, we aim to answer two critical research questions: (1) Does task interference exist in parameter-efficient multimodal instruction tuning? (2) How can we effectively mitigate this issue for robust and versatile performance across various multimodal tasks?
To answer the first question, we investigate the
637
task-interference issue in parameter-efficient multimodal instruction tuning from the perspective of gradient direction (Liu et al., 2021) in Section 3.2. Our observations highlight notable task interference in this context, underscoring the necessity for more effective adaptation strategies to ensure robust and versatile performance across diverse multimodal tasks. In response to our second question, this paper proposes a novel multimodal instruction tuning framework – Conditional Mixture-ofLoRA (MixLoRA), designed to mitigate the task interference issue. As shown in Figure 1, unlike conventional LoRA which uses shared low-rank adaptation matrices A and B across all tasks and instances, MixLoRA dynamically constructs lowrank adaptation matrices A and B tailored to each input instance, by selecting their decomposition factors from two collections. MixLoRA introduces a dynamic factor selection mechanism, incorporating two Independent Factor Selection (IFS) routers and a Conditional Factor Selection (CFS) router. The two IFS routers independently select appropriate factors to dynamically construct LoRA A and B matrices tailored to each input. The CFS router further refines the selection for LoRA B based on the factors chosen for LoRA A, ensuring that the factors selections for LoRA A and B are not only tailed to input but also cohesively aligned.
To validate the effectiveness of MixLoRA, we conduct extensive experiments on MME (Fu et al., 2023), a comprehensive multimodal evaluation benchmark, and seven additional multimodal evaluation datasets that focus on various capabilities. Experimental results demonstrate that MixLoRA,
with its dynamic factor selection approach, consistently outperforms LoRA across various multimodal tasks when using the same number of ranks and remains competitive or superior even against LoRA with a higher rank number. This effectiveness is attributed to the dynamic factor selection mechanism and its ability to generalize to unseen tasks through adaptive factor activation, underscoring the potential of MixLoRA to generalize and perform effectively on unseen multimodal tasks.
Our contributions are summarized as follows: (1) We empirically investigate and demonstrate the existence of task interference in parameter-efficient multimodal instruction tuning. (2) We propose the Conditional Mixture-of-LoRA (MixLoRA) framework, aimed at alleviating task interference by dynamically constructing low-rank adaptation matrices for various inputs. (3) Comprehensive experiments demonstrate the effectiveness of MixLoRA, outperforming LoRA across various unseen multimodal tasks at equal or even higher ranks.

heading: 2 Related Work
text: Multimodal Instruction tuning Instruction tuning (Wei et al., 2021) significantly improves the generalization of large language models to unseen tasks based on natural language instructions (Ouyang et al., 2022; Taori et al., 2023). With the advent of multimodal large language models, the scope of instruction tuning has expanded to encompass multimodal and vision tasks, facilitated by the development of diverse multimodal instruction datasets, including both machine-generated (Liu et al., 2023; Zhao et al., 2023; Zhu et al., 2023;
Yin et al., 2023; Li et al., 2023b; Ye et al., 2023) and human-annotated (Xu et al., 2023b). Recently, Vision-Flan (Xu et al., 2023a) stands out as a comprehensive human-annotated visual instruction tuning dataset, covering a wide range of 187 tasks, making it ideal for our training.
Parameter-efficient fine-tuning (PEFT) Parameter-efficient fine-tuning (PEFT) (Lee et al., 2019; Hu et al., 2021; Li and Liang, 2021; Karimi Mahabadi et al., 2021; Guo et al., 2021; Zaken et al., 2022) strategies have become key in efficiently adapting large pre-trained models to various downstream tasks with minimal parameter adjustments. These methods typically involve freezing most of the pre-trained model while fine-tuning a small fraction of the parameters to facilitate the adaptation process. Among these, LoRA (Hu et al., 2021) demonstrates competitive trade-offs between performance and parameter efficiency, making it widely adopted. PEFT methods typically utilize shared adaptation parameters across diverse tasks or train task-specific adapters. However, when applied to multimodal instruction tuning, which requires simultaneous adaptation to diverse instruction tasks, PEFT can encounter task interference, highlighting the need for more adaptable and versatile PEFT methods to adeptly handle the complexities of multimodal instruction tuning.
Task Interference Task interference (Crawshaw, 2020) is a notable challenge in multi-task learning, where simultaneous training on multiple tasks can lead to performance decline due to conflicting gradients among tasks (Yu et al., 2020; Liu et al., 2021; Navon et al., 2022). To mitigate task interference in multi-task learning, researchers have explored various strategies, including dynamic adjustment of task loss contributions (Chen et al., 2018; Sener and Koltun, 2018; Liu et al., 2019) and parameter partitioning (Maninis et al., 2019; Bragman et al., 2019; Strezoski et al., 2019; Zhang et al., 2020). Despite the established understanding of task interference in multi-task learning, its presence and implications in instruction tuning, particularly in multimodal contexts, remain under-explored. Given the intrinsic complexity and diversity of multimodal instruction-based tasks, substantial task interference is likely to exist in multimodal instructiontuning scenarios. Our research delves into this area, specifically investigating task interference within parameter-efficient multimodal instruction tuning.

heading: 3 Task Interference in Multimodal Instruction Tuning with LoRA

heading: 3.1 Background: Low-Rank Adaptation
text: Low-Rank Adaptation (LoRA) (Hu et al., 2021) is a parameter-efficient fine-tuning method that finetunes only the trainable rank decomposition matrices injected in each layer of the Transformer (Vaswani et al., 2017). As illustrated in Figure 1 (a), consider a linear layer, represented by h̃ = Wh, where W ∈ Rdout×din denotes the pre-trained weight, with din and dout being the input and output dimensions, respectively. LoRA modifies the model parameters by injecting low-rank decomposition matrices as the weight adjustment matrices, which can be expressed as:
h̃ = Wh+∆Wh = Wh+ α ·BAh, (1) where ∆W = BA represents the trainable weight adjustment matrices formed by low-rank matrices A ∈ Rr×din and B ∈ Rdout×r, with the rank r ≪ min(din, dout). The scalar α ≥ 1 controls the influence of the weight adjustment matrices. During fine-tuning, only these low-rank decomposition matrices, referred to as LoRA A and LoRA B throughout this paper, are updated, allowing for rapid, task-specific adaptation by training distinct LoRA A and B for each downstream task.

heading: 3.2 Investigating Task Interference in
text: Multimodal Instruction Tuning
Our study delves into task interference in parameter-efficient multimodal instruction tuning by analyzing gradient direction conflicts between task pairs. For each task pair i and j, we first estimate the change in loss Li of task i, when optimizing the shared parameters θ according to the loss Lj of task j, following (Zhu et al., 2022): ∆jLi(xi) = Exj ( Li(xi; θ)− Li(xi; θ − λ
∇θLj(xj) ∥∇θLj(xj)∥ )
)
≈ λExj
( ∇θLj(xj)
∥∇θLj(xj)∥ T
∇θLi(xi) ) ,
(2) where xi and xj are sampled training batches for tasks i and j, and λ is the learning rate.
The interference of task j on task i is then quantified as follows:
Ii,j = Exi ( ∆jLi(xi)
∆iLi(xi)
) . (3)
Here, a positive Ii,j suggests aligned gradient directions between tasks i and j, while a negative
value implies divergent gradient directions, indicating that task j adversely impacts task i.
We conduct experiments on the fine-tuned LLaVa (Liu et al., 2023) model using LoRA with a rank of 4, computing the task interference among six diverse tasks from Vision-Flan (Xu et al., 2023a), including “ScienceQA” (Lu et al., 2022) (for “Complex Reasoning”),“COCO” (Lin et al., 2014) (for “Coarse-grained Perception”), “FairFace” (Karkkainen and Joo, 2021) (for “Finegrained Perception”), “iNaturalist” (Van Horn et al., 2018) (for “Knowledge Intensive”), “STVQA” (Biten et al., 2019) (for “OCR”), and “PACS” (Li et al., 2017) (for “Domain specific”). We compute the average task interference matrix I based on the gradients concerning LoRA A and B, across various layers. Figure 2 shows the task interference score for LoRA A and B at the 5-th and 25-th Transformer Layer for both MLP (Figure 2a) and Self-Attention (Figure 2b).
Our results reveal notable task interference at both shallow and deep Transformer layers for LoRA A and B. For instance, as shown in Figure 2b, at the 5-th layer for LoRA A, the domainspecific classification task “PACS” negatively impacts “COCO”, a coarse-grained perception task, with a negative interference score of −7.3. Meanwhile, positive influences are also observed. For example, Figure 2a shows that at the 5-th layer for LoRA B, “PACS” positively affects the OCR task "ST-VQA". The presence of both positive and negative interference suggests complex dynamics
among instruction tasks: positive scores (in red), suggest that the learning of one task can enhance the performance of another, while negative scores (in blue), imply that one task’s learning can hinder another. These findings highlight notable task interference in parameter-efficient multimodal instruction tuning and reinforce the need for effective adaption methods to ensure robust and versatile performance across diverse multimodal tasks.

heading: 4 Conditional Mixture-of-LoRA
text: Inspired by the concept of Mixture-of-Experts (Shazeer et al., 2016), we propose Conditional Mixture-of-LoRA (MixLoRA) which leverages low-rank decomposition factors as dynamically chosen experts to construct tailored decomposition matrices A and B for specific input instances. MixLoRA facilitates dynamic processing pathways for varying input instances, thereby enhancing the efficacy in handling diverse and complex multimodal instruction tasks.
The core of Conditional Mixture-of-LoRA lies in representing the weight adjustment matrices ∆W from Equation 1 via tensor decomposition:
∆W = BA = r∑
i=1
bi ⊗ ai, (4)
where {ai, bi}ri=1, ai ∈ Rdin×1, bi ∈ Rdout×1 are the rank r decomposition factors of ∆W ∈ Rdout×din .
Leveraging the concept that ∆W can be expressed as the sum of outer products of low-rank
decomposition factors ai and bi, MixLoRA introduces a Dynamic Factor Selection module. This module dynamically constructs unique ∆W for specific inputs by selecting r appropriate factors from an expanded pool of decomposition factors {ae}Ee=1, {be}Ee=1, E > r, as shown in Fig. 1 (b). Here, the number of factors E is much larger than the rank r. Following LoRA (Hu et al., 2021), we use a random Gaussian initialization for {ae}Ee=1 and zero for {be}Ee=1.

heading: 4.1 Dynamic Factor Selection
text: The Dynamic Factor Selection module uses two main components to dynamically constructs LoRA A and B. First, two Independent Factor Selection (IFS) routers (Section 4.1.1), independently select r relevant factors to form adaptation matrices LoRA A and B, ensuring precise, instance-specific adaptations. Second, a Conditional Factor Selection (CFS) router (Section 4.1.2) further refines the selection for LoRA B by conditioning the selection for B also on the factors chosen for LoRA A, promoting a coherent adaptation process.

heading: 4.1.1 Independent Factor Selection
text: MixLoRA employs two Independent Factor Selection (IFS) routers, RAIFS(·) and RBIFS(·), to select r relevant factors for LoRA A and B, respectively, as shown in Figure 3. IFS routers employ an instance-based routing method, which is more memory-efficient than conventional input-tokenbased routing, for selecting r decomposition factors. The routing strategy can be expressed as:
RAIFS(h) = Avg(h), (5)
where Avg(·) averages across the sequence dimension of the hidden states h ∈ Rseq×din from the preceding layer, resulting RAIFS(h) ∈ Rdin .
Factor Selection Process The factor selection process involves calculating vectors gA ∈ RE and gB ∈ RE to selectively identify specific subsets of decomposition factors from the set {ae}Ee=1 and {be}Ee=1, respectively. To compute gA, input RAIFS(h) ∈ Rdin is processed through a dense layer with weights WA ∈ RE×din , followed by a softmax normalization and top-k selection:
gA = topr(softmax(WA ·RAIFS(h))). (6)
This procedure ensures the selection of r factors for LoRA A, with gA[i] = 1 indicating the selection of factor i. The same process is applied to determine gB for LoRA B.

heading: 4.1.2 Conditional Factor Selection
text: While the factors for LoRA A and B have been independently selected so far, we hypothesize that an interdependence exists between the selections for LoRA A and B, which can be harnessed to improve the model’s overall adaptability and performance. To leverage this relationship, we propose a Conditional Factor Selection (CFS) strategy, wherein the selection of factors for the projection-up weight of LoRA B is also influenced by the factors chosen for the projection-down weight of LoRA A.
With the IFS router, LoRA A is assembled from chosen decomposition factors, denoted as A = [a1, · · · , ar]T , where A ∈ Rr×din . Following this, the CFS router employs a weight tensor WAB ∈ Rr×din×E to map each factor A[i] ∈ R1×din in A to an expert dimension E. The mapping process for each factor A[i], normalized via softmax and aggregated across r factors, is given by:
RBCFS(A) =
r∑
i=1
softmax(A[i] ·WAB[i]), (7)
where WAB[i] ∈ Rdin×E is the mapping matrix associated with A[i].
The factors selection for LoRA B integrates outputs from both the IFS RBIFS(·) and CFS RBCFS(·) routers via a late fusion strategy, forming the selection vector gB as follows:
pBIFS = softmax(W B IFS ·RBIFS(h)) (8)
pBCFS = softmax(R B CFS(A)) (9)
gB = topr(p B IFS + p B CFS), (10)
where, RBIFS(h) ∈ Rdout for LoRA B is computed similarly to RAIFS(h).
The probability distribution pBIFS ∈ RE and pBCFS ∈ RE reflect the selections from the independent and conditional routers, respectively. The final selection vector gB ∈ RE is determined by combining these distributions and identifying the top r factors. This CFS strategy enables the selection for LoRA B to be informed by factors selected for LoRA A, fostering a more cohesive selection process.

heading: 4.1.3 Reconstruction of Dynamic Adaptation Matrices
text: Finally, MixLoRA constructs dynamic adaptation matrices by leveraging the factor selection vectors gA and gB , gathering the chosen factors {ak, bk}k∈K , |K| = r, to assemble the final matrices for LoRA A and B. Consequently, in each forward pass, the weight adjustment matrix ∆W ∈ Rdout×din is dynamically calculated based on these selected factors, formulated as:
∆W = BA = [b1, · · · , br][a1, · · · , ar]T . (11)

heading: 5 Experimental Methodology

heading: 5.1 Datasets
text: Training Datasets To validate the effectiveness of MixLoRA, we perform instruction tuning on Vision-Flan (Xu et al., 2023a), a human-annotated multimodal instruction tuning dataset with 187 diverse tasks. Its diversity in visual instruction tasks makes it ideal for investigating task interference. To minimize computational cost, we utilize a scaled-down version with up to 1,000 instances per task, totaling 182,167 instances.
Evaluation Datasets We evaluate our method on MME (Fu et al., 2023), a comprehensive multimodal evaluation benchmark measuring both perception and cognition abilities across 14 subtasks, to assess the overall capabilities of MixLoRA. Alongside MME, we further probe the model’s various capabilities using 7 multimodal datasets. For Optical Character Recognition, we utilize TextVQA (Singh et al., 2019), and for reasoning, we employ the Visual Spatial Reasoning (VSR) dataset (Liu et al., 2022). Perception capability is tested on CIFAR-10/100 (Krizhevsky et al., 2009) and MNIST (LeCun, 1998), following the guidance of (Zhai et al., 2023). The SNLI-VE dataset (Xie et al., 2019) evaluates the Visual Entailment capabilities, and POPE (Li et al., 2023c) examines the tendency to objects hallucination.

heading: 5.2 Evaluation Metrics
text: For MME scores, we employ the official evaluation tool2, aggregating the Perception and Cognition metrics. For other multimodal datasets, we leverage Vicuna 1.5 13B (Chiang et al., 2023), the stateof-the-art open-source LLM to assess the accuracy of each prediction compared with ground-truth target output. More details are in Appendix C.

heading: 6 Results and Discussion
text: Comparison with LoRA We first present a detailed comparison between MixLoRA and the conventional LoRA, focusing on their performance in MME and 7 other multimodal tasks, as detailed in Table 1. We observe that MixLoRA consistently surpasses LoRA when both models operate at the same ranks on both MME and the additional multimodal tasks, and even demonstrate superior performance when compared to LoRA with a higher rank number. For instance, MixLoRA (with rank r=2 and factors E=16) outperforms LoRA (rank r=32) by 1.7% in MME and 1.6% on average across other multimodal evaluations.
Increase the Number of Rank We investigate the impact of increasing the rank number while keeping the number of factors constant. As shown in Table 1, MixLoRA exhibited a notable performance enhancement as the rank number increased from 2 to 4, when the factor number was fixed. Specifically, increasing the rank r from 2 to 4 leads to a performance uplift of 1.8% in MME and 3.1% in MMAvg with E = 16 factors, and a 3.5% improvement in MME and a 0.6% increase in MMAvg with E = 32 factors. However, further increasing the rank to 8 shows diminishing returns in performance gains. We hypothesize this decline might potentially be due to the expanded combination pool for constructing the adaptation matrices.
Increasing the Number of Factors In scenarios where the rank number is held constant, our findings reveal a general trend of performance improvement for MixLoRA, as shown in Table 1. This improvement can be attributed to the model’s increased capacity for providing a richer set of factors to tailor the model to specific multimodal tasks.
The Effect of Routing Strageties In this experiment, we examine different routing strategies for
2https://github.com/BradyFU/Awesome-MultimodalLarge-Language-Models/tree/Evaluation
the IFS router. In particular, we implement the Task-Specific Routing paradigm which leverages the definition of each multimodal instruction task to inform the selection of decomposition factors (details can be found in Appendix A). Table 2 shows that Instance-based Routing significantly outperforms Task-specific routing, achieving a higher MME score and average performance across the additional multimodal tasks. The superior performance of Instance-based Routing likely stems from its inherent flexibility. Unlike Task-specific Routing, which has the same selection of factors at different layers for inputs from the same task, Instance-based Routing adapts its selection based on the varying hidden states from previous layers, leading to a more flexible routing mechanism.
Furthermore, we investigate whether the superior performance is due to the introduction of extra expert parameters and not the routing mechanism. Table 2 reports the comparison with a random routing baseline, which randomly selects r factors. Our observations reveal that both Instance-based Routing and Task-specific routing surpass the random baseline, suggesting that the routing mechanism, rather than the inclusion of additional expert parameters,
is responsible for the performance enhancements.
Impact of Conditional Factor Selection We assess the impact of Conditional Factor Selection (CFS) through an ablation analysis, comparing MixLoRA’s averaged performance with and without the CFS across seven multimodal datasets. The comparative results, as shown in Figure 4 demonstrate that incorporating the CFS router in general consistently improves the performance across different factor and rank settings. This enhancement is hypothesized to stem from the CFS’s role in strengthening the interdependency between the factor selections of LoRA A and B.
Factor Selection Pattern on Unseen Tasks Our analysis delves into the factor selection patterns of LoRA A for unseen multimodal tasks. We randomly sample 300 instances from each of seven unseen multimodal tasks and visualize the factor selection within the MLP layer using t-SNE (Van der Maaten and Hinton, 2008), as shown in Figure
5. We observe that instances from identical tasks tend to cluster, indicating the effectiveness of an instance-based routing strategy in assigning diverse factor sets across tasks.
Furthermore, we visualize the factor selection patterns for similar seen and unseen tasks. We pair five distinct unseen tasks, each probing a different capability, with five similar seen tasks from the training set: SNLI-VE (unseen) with ImageText (seen) for assessing visual entailment, TextVQA (unseen) with InfoGraphicVQA (seen) for OCR capabilities, VSR (unseen) with GQA (seen) for reasoning, Pope (unseen) with VQA-ObjectPresence (seen) for hallucination detection, and CIFAR-10 (unseen) with ExDark (seen) for perception capabilities. The t-SNE visualization shown in Figure 6 depicts the distribution of factor selection across MLP layers, with the first row in the legend indicating the seen tasks, and the second row denoting the corresponding unseen tasks. Similar color schemes are used for each pair of similar seen and unseen tasks for clarity. Our observations reveal that MixLoRA effectively activates factors analogous to those employed in similar training tasks. This finding suggests that the model can adapt its factor selection strategies to new, unseen tasks based on its training on similar seen tasks.
Analysis of Task Interference To assess MixLoRA’s efficacy in mitigating task interference, we test it on the same six training tasks: “ScienceQA”, “COCO”, “FairFace”, “iNaturalist”, “STVQA”, and “PACS”, discussed in Section 3.2. For each task, we randomly sample 300 instances not included in the instruction-tuning phase for evaluation. We compare MixLoRA against both the conventional LoRA and task-specialized LoRA models (LoRASpecialist) that are fine-tuned with taskspecific adaptation parameters for each task. Table 3 shows that conventional LoRA models exhibit
varying degrees of performance degradation across tasks when compared to LoRASpecialist. In contrast, MixLoRA suffers less from performance degradation and demonstrates more consistent and robust performance across different tasks, suggesting its effectiveness in reducing task interference.
Moreover, we visualize the task interference scores using Equation 2 and 3. Given that MixLoRA dynamically selects a subset of factors (r out of E) for different instances, we record gradients concerning all E factors and compare the task interference scores between standard LoRA models (with r=16) and MixLoRA (with E = 16 and r = 4). Figure 7 visualizes the interference scores for both LoRA A and LoRA B aggregated across all adaptation layers, including MLP and self-attention layers. The analysis reveals that MixLoRA (E=16, r=4) exhibits lower negative interference scores compared to the standard LoRA (r=16), underscoring
MixLoRA’s efficacy in reducing task interference.

heading: 7 Conclusion
text: We introduce Conditional Mixture-of-LoRA, an innovative strategy that dynamically constructs low-rank adaptation matrices specific to individual inputs, to mitigate task interference during parameter-efficient multimodal instruction tuning. Comprehensive experiments across a variety of multimodal datasets have demonstrated the efficacy of MixLoRA, showcasing an enhanced performance on unseen multimodal tasks compared to conventional LoRA and demonstrating its effectiveness in mitigating task interference.

heading: 8 Limitations
text: Our study focuses on task interference within parameter-efficient multimodal instruction tuning, specifically for image and text modalities, leaving the integration of other modalities like sound and 3D point clouds as an avenue for future work. Moreover, due to the cost of training large models, our experimentation was conducted on a scaled-down version of Vision-Flan. Future studies could benefit from evaluating the effectiveness of MixLoRA when applied to more extensive multimodal instruction-tuning datasets. Additionally, our method introduces extra training overhead compared to standard LoRA of the same rank.

heading: Acknowledgments
text: This research is based upon work partially supported by the U.S. DARPA ECOLE Program #HR001122S0052, FoundSci Program #HR00112490370, and the Amazon - Virginia Tech Initiative for Efficient and Robust Machine Learning. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA or
the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Task-Specific Routing
text: The Task-Specific Routing paradigm leverages the distinct characteristics of each multimodal instruction task to inform the selection of decomposition factors. This strategy utilizes the detailed task definition, which includes a comprehensive description of the task’s requirements and the specific skills or modalities needed to successfully perform the task. For instance, consider the task “OK-VQA” (Marino et al., 2019), the task definition is: “Answer the question in natural language based on the content of the image. The questions require external knowledge to answer.” The task-specific routing strategy is formulated as:
RAIFS(z) = Avg(fϕ(z)), (12)
where fϕ(·) denotes a pre-trained Large Language Model (LLM) parameterized by ϕ, responsible for encoding the task definition z.
B Implementation Details
We leverage the stage-one LLaVA-v1 3 (before the visual instruction tuning stage) as our pre-trained large multimodal models, specifically employing LLaVA with Vicunna-7B v1.3. For all model variants, we fine-tune this stage-one LLaVa on the scale-down version of Vision-Flan for three epochs, using a total batch size of 128 and a learning rate of 4e− 5. The fine-tuning process for MixLoRA (E=16, r=4) takes approximately 20 hours on 4 A100 GPUs, with an effective batch size of 8 per GPU and a gradient accumulation step of 4. For LoRA, we set the hyper-paramter α in Equation 1 to be 2 × rank r and for MixLoRA, we define α as 2 × factors |E|. For the other configuration, we adopt LLaVA’s default setting for LoRA fine-tuning, as provided in its codebase. For the task-specific routing, we adopt the Vicunna (Chiang et al., 2023) as our pre-trained large language model fϕ(·) for encoding task definition. Notably, Vicuna also serves as the language backbone of the LLaVA model. Following a similar approach to LoRA, for the LLaVA model with 32 Transformer layers, we insert MixLoRA into all linear layers within the Transformer layers. During training, all parameters in the MixLoRA module are updated, while the rest of LLaVA’s parameters remain frozen.
3https://github.com/haotian-liu/LLaVA/

heading: C Evaluation Metrics
text: To evaluate the model performance on unseen multimodal datasets, we leverage Vicuna 1.5 13B (Chiang et al., 2023), the state-of-the-art open-source LLM to perform the evaluation. Specifically, we craft a prompt template that directs Vicuna to assess the accuracy of each prediction, considering the given task instructions and ground-truth target output. The prompt template used is as follows: “ A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user’s questions. USER: Decide if the prediction is correct given the question and the answer. Questions: {Question} Answer: {Ground-truth Answer} Prediction: {Prediction} Your response should only be Yes or No. ASSISTANT:” In this template, placeholders such as “{Question}”, “{Ground-truth Answer}”, and “{Prediction}” will be substituted with the specific details of each test instance. If Vicuna determines the prediction is correct, it outputs “Yes”, and “No” otherwise. As all tasks are classification tasks, we compute accuracy based on Vicuna’s judgments.

heading: Future Work
text: Our study focuses on task interference within parameter-efficient multimodal instruction tuning, specifically for image and text modalities, leaving the integration of other modalities like sound and 3D point clouds as an avenue for future work. Moreover, due to the cost of training large models, our experimentation was conducted on a scaled-down version of Vision-Flan. Future studies could benefit from evaluating the effectiveness of MixLoRA when applied to more extensive multimodal instruction-tuning datasets. Additionally, our method introduces extra training overhead compared to standard LoRA of the same rank