MokA: Multimodal Low-Rank Adaptation for MLLMs
Authors: Yake Wei
Yu Miao
Dongzhan Zhou
Di Hu
Venue: NeurIPS 2025 oral ()
Keywords: Multimodal
MLLMs
PEFT
fune-tuning
Primary area: deep_learning

Abstract: In this paper, we reveal that most current efficient multimodal fine-tuning methods are hindered by a key limitation: they are directly borrowed from LLMs, often neglecting the intrinsic differences of multimodal scenarios and even affecting the full utilization of all modalities. Inspired by our empirical observation, we argue that unimodal adaptation and cross-modal adaptation are two essential parts for the effective fine-tuning of MLLMs. From this perspective, we propose Multimodal Low-rank Adaptation (MokA), a multimodal-aware efficient fine-tuning strategy that takes multimodal characteristics into consideration. It compresses unimodal information by modality-specific parameters while explicitly enhancing cross-modal interaction, ensuring both unimodal and cross-modal adaptation. Extensive experiments cover three representative multimodal scenarios (audio-visual-text, visual-text, and speech-text), and multiple LLM backbones (LLaMA2, Qwen2, Qwen2.5-VL, etc). Consistent improvements indicate the efficacy and versatility of the proposed method. Ablation studies and efficiency evaluation are also conducted to fully asses our method. Overall, we think MokA provides a more targeted solution for efficient adaptation of MLLMs, paving the way for further exploration.

MokA: Multimodal Low-Rank Adaptation for MLLMs
Yake Wei Yu M2a Dongzhan Zhou', Di Hu 1,233
Gaoling School of Artificial Intelligence, Renmin University of China
Beijing Key Laboratory of Research on Large Models and Intelligent Governance Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE
*Shanghai Artificial Intelligence Laboratory yakewei@ruc. edu. cn, ymiaoOruc edu. cn zhoudongzhan@pjjatt org. en, dihu@ruc.
Abstract
In this paper, we reveal that most current efficient multimodal fine-tuning methods are hindered by key limitation: they are directly borrowed from LLMs, often neglecting the intrinsic differences of multimodal scenarios and even affecting the full utilization of all modalities. Inspired by our empirical observation, we argue that unimodal adaptation and cross modal adaptation two essential parts for the effective fine-tuning of MLLMs. From this perspective. we propose Multimodal low-rank Adaptation (MokA), multimodal-aware efficient fine-tuning strategy that takes multimodal characteristics into consideration. It compresses unimodal information by modality-specific parameters while explicitly enhancing cross- modal interaction, ensuring both unimodal and cross-modal adaptation. Extensive experiments cover three representative multimodal scenarios (audio-visual-text visual-text, and speech-text), and multiple LLM backbones (LLaMA2/3, Qwen2, Qwen2.5-VL, etc). Consistent improvements indicate the efficacy and versatility of the proposed method. Ablation studies and efficiency evaluation are also conducted fully asses our method. Overall, we think MokA provides more targeted solution for efficient adaptation of MLLMs, paving the way for further exploration. The project page at atthttts:///eeee lab. github. io/MokA
Introduction
Large language models (LLMs) have gained remarkable popularity due to their impressive ability to understand and generate content. extend their capabilities to more general multimodal scenarios, recent advancements of Multimodal Large Language Models (MLLMs) [38] 40) 20 have focused on aligning other modalities, such as images, with text tokens, thereby equipping LLMs with the ability to interpret and process content of other modalities. However, due to the massive parameter scale of LLMs, fully fine-tuning such models on downstream tasks is computationally prohibitive and inefficient in most cases.
promising direction has emerged in the field LLM fine-tuning before, which involves selectively updating subset of parameters rather than the full model. These Crtumeter-Efficicn Fine-Tuning Tuning (PEFT) strategies have seen widespread adoption and have been successfully extended to the fine- tuning of MILMs. In particular, LoRA [12] and its variants, which assume that over wner-parametrrieed models fact reside on low intrinsic dimension, have been broadly applied d 39), demonstrating strong adaptability and efficiency. However, the development of efficient multimodal LLM fine- tuning is at present obscured by "dark cloud": most current methods are directly borrowed from LLMs, often overlooking the fundamental differences of multimodal scenarios. Indeed, prior studies
CCorreponniigg Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LLM (LoRAO)
Text loken
Audio taken
Visual taken
Pretrained
Pretrained
Weights
Weights
LLM Tokenize
Projector
Quarloc Aadid//onng
Fnender
Encoder
tokio
taken
taken
(a) MLLM framework.
(b) Classic LoRA module.
(c) Partial modality inference.
MUSIC-AVOA
POPE
Banch -1e
34-
70
80
28
62-
Teeiorryy Pull modality Audio-ariy V Vause-orty
Teeienly Pull modaltty Vasueoory
Tost only Full modalltySpwchhany
(d) Audio-visual visual-text case.
(c) Visual-text case.
(1) Speech- text case.
Figure 1: (a): Common MLLM fine-tuning framework. (b): Sketch of classic LoRA module for MLLM fine-tuning, (c): Partial modality inference setting. Take tokens of visual modality as an example. (d-f): Partial modality inference performance of LoRA. Full modality: Regular case where all multimodal tokens are processed by the LoRA module. Text-/Audio- Sisiiio---ssua-Speeehhhoo vextaaudiovisuul///eeech tokens are passed through the LoRA pathway at the prefilling stage during inference. Results are based on LLaMA2.
Only
in multimodal learning have demonstrated that the inherent heterogeneity of different modalities necessitates modality-specifie utilization strategies, rather than fully unified way [2 34).
To this end, we are motivated observe the fine-tuning efficacy of the widely used LoRA strategy. A common MLLM fine-tuning framework is shown in Figure Ial encoded representation of non-text modality (c.g., audio or visual) is first aligned with the text embedding space via projector (usually Q-former MLP), after which the resulting multimodal tokens integrated and processed jointly by the LLM. In the efficient fine-tuning case, the LLM backbone frozen, and parameters of additional LoRA modules optimized. FFiuure 1b provides the sketch of classic LoRA module. A and B matrices are shared across different modalities.
To further observe how well tokens of different modalities are utilized, we conduct partial modality inference experiments. The training stage retains the original setting, wherein all multimodal tokens processed by the LoRA module. Specifically, tokens from a selected modality are passed through the LoRA adaptation pathway at the prefilling stage during inference. As illustrated in Figure Ic, visual token inference is used And should be noted that the pre-trained weights still receive full tokens of all modalities. Results shown in Figure TH- demonstrate surprising phenomenon across three representative multimodal scenarios, valfiovisuul-eee -exutlt-extttt case, visual-text case, and speech-text case. Text token inference can achieve quite comparable performance the regular full modalities case. However, Non-text token inference (e.g., audio or visual) leads to a noticeable drop in performance.
evaluate the model's performance when only
example.
The above results suggest that the optimization of all-modality-shared LoRA parameters is overly influenced by text tokens, resulting in non-text tokens being less effectively utilized during fine- tuning. Although these all-modality-sharl this phenomenon reveals the need to consider individual modality during fine-tuning. This fact inspires that unimodal and cross-modal adaptation equally critical in the fine-tuning of MLLMs, which is mostly ignored as mentioned above.
parameters implicitly improve cross- moosl interaction,
To this end, we propose the Multimodal low-rank Adaptation (MokA), fine-tuning strategy designed to achieve unimodal adaptation while explicitly enhancing cross-modal interaction. While MokA retains the widely adopted low-rank decomposition matrices, it redefines the roles of matrices A and to better accommodate multimodal characteristics. Specifically, matrix designed to be modality allowing each modality to compress information independently and thus avoid

---

interference from others. After that, a cross-attention mechanism is introduced to strengthen the interaction between text tokens and non-text tokens, emphasizing task-relevant features. Finally, shared multimodal matrix B projects the unimodal low-rank representations into a unified space, facilitating effective alignment across modalities. These three parts jointly ensure both unimodal and cross-modal adaptation. In experiments, noticeable improvement in multiple multimodal scenarios demonstrates the effectiveness of our method. We think MokA represents a first-step attempt at multimodal-aware adaptation, and further possibilities exist under our basis that simultancously accounts for both unimodal and cross-modal adaptation.
2 Method
2.1 Rethinking of low-rank adaptation the multimodal scenario
LoRA 12 is based the assumption that the weight updates during fine-tuning lie in subspace of low "intrinsic rank." Rather than updating the entire pre-trained weight matrix directly, LoRA introduces a low-rank decomposition approach, where the update AW Ral to pre-trained matrix Wo E R Ra kkkk, is parameterized as the product two much smaller matrices: B E Rdxr and A E IR" rx with < min(d, k). The resulting fine-tuned weight matrix W' is given by Wo + AW Wo BA. Therefore, for h = Wox, the modified update forward pass yields:
h Wox AWx W6x + BAx.
(1)
Here, Wo remains fixed during training, while only the matrices and are learned. To ensure stable training, A is initialized using uniform Kaiming distribution [II], and B is initialized to zero, leading to an initial update AW = BA = 0 at the beginning of fine-tuning, LoRA [12] and its variants have been extensively employed in the parameter-efficient fine-tuning of MILMs 6 39]. These methods typically employ shared parameters to uniformly process tokens from all modalities, implicitly facilitating cross- modal interactions during adaptation. However, empirical results reveal that such shared tuning leads to limited utilization of all modalities. This highlights the need to consider individual modality during fine-tuning.
To better support multimodal adaptation, we argue that both unimodal and cross-modal updates should be considered during fine-tuning. In other words, the model should be able learn from each modality independently while also ensuring the cross-mode interaction. Therefore, the design of the update mechanism should ensure that both types of information are properly captured during the forward pass:
h Wox + AWX Wox + [ x
m2
Wox + AW mi AW2x" AW, m.
+ AW cross m
m.
uninodal edaptation
crose-modal adaptation
where is the number of modalities. = is the token sequence of modality i. AW, is the unimodal update parameters of modality i, and AW is cross-modal update parameters.
2.2 Multimodal low-rank Adaptation (MokA)
Based on the above perspective, we propose Multimodal low-rank Adaptation (MokA) strategy, parameter- efficient fine-tuning method tailored for the multimodal nature MILMs. Considering the efficiency advantage of LoRA, A retains the core idea of low-rank adaptation, but redefines the roles of the projection matrices A and B to better reflect the characteristics of multimodal scenarios. By unimodal compression and explicitly reinforcing cross- modal interaction, MokA enables both unimodal and cross- modal adaptation, leading to more effective fine-tuning of MLLMs. The overall structure of MokA is depicted in Figure 2
Multimodil
Pretrained Weights
Cross Attention
Text A
Audio
Visual
Concretely, MokA has three core parts: unimodal matrix A, Figure 2: Illustration of MokA strategy.
task-centric cross-attention, and shared multimodal matrix
B. Here we take the andio-visual-ter case example, and other cases can be well extended.
3

---

2.2.1 Unimodal matrix A
For an arbitrary pretrained weight Wo in the LLMs, we suppose its input sequence is x [2:: -. N, =0;; ; = x2: x. Here (a..e) the token length of modality i. of is the first token of modality a, and so on. For simplicity, we use x' to denote the token sequence of modality i. Then the whole input sequence can be rewritten as = [xx;x* x"; x*)
To ensure the well compression of unique unimodal information and avoid the interruption from others, matrix A is designed to be individual for each modality, allowing tokens from different modalities to be processed independently through their respective parameter. The compressed sequence after matrix A is:
Ax = [A"x"; A"x"; A'x"];
where { (E.)e() is the parameter of modality i. After processing by unimodal matrix A, embed- dings of each modality are individually mapped into a low-rank space, without the potential influence of other modalities.
2.2.2 Task-centric cross-attention
In the fine-tuning process of MILMs, text and non-text tokens typically serve distinct roles. Specifi- cally, under supervised instruction tuning, text tokens often function task descriptions prompts, whereas non-text tokens (e.g., audio or visual inputs) primarily convey contextual information upon which the task is based. The following example illustrates typical instruction format:
<audio> <visual> Please answer the question: which clarinet makes the sound first?
In this case, <audio> and <visual> provide the event information. "Please answer the question: which clarinet makes the sound first?" describes the concrete task for LLMs. Successfully answering such questions relies effectively capturing the semantic association between the task description conveyed by text tokens and the event cues provided by non-text tokens. Therefore, it becomes intuitive and necessary explicitly emphasize the most relevant cross-modal information to support accurate rea- soning, Since unimodal information has been extracted individually after the processing of unimodal matrices A, this stage is well-suited for introducing cross-modal interaction. Additionally, the token embeddings are projected into a low-rank space, the computational burden of performing cross-modal interaction is significantly reduced. Hence, we place the cross- cros-aattnttion part after the low-rank compression ensure both effectiveness and efficiency. The concrete attention mechanism is illustrated in |Fiuure 3
Text taken Key
Tasi
loken
token
laken
codmnr
Query
Cross-attention part of MokA. Take the visual token as an example.
Figure 3:
and is conducted as follows:
(A"x* (A'x²)
Att (A"x". A xxxx A'x*) sof tmax
A'x',
(5)
VT (A'x")
Att (A'x", A'x" A'x',A''') = sof tmax
A'x',
(6)
VF
where is the rank. Then, the enhanced audio and visual tokens are: x + AgAtt (A"x", A'x', A'x'), x"x" " + A,Att (A"x", A'x', A'x',A'' A'x'), where Xa and Au are the hyperparameters that control the strength of explicit cross-modal interaction. Finally, the sequence after cross-attention is:
(7)
(8)
Ax = xA + A'Atto.,;iA'''
A'x
Here we use Att,,e.t to simply denote the cross- attentioen between modality and text. It should be noted that while we adopt a cross-attentin and non-text tokens, alternative designs that serve a similar purpose can also be considered. Further discussion provided in /Sectioo 4.4 In addition, in MokA, linear projections (W,, WA, and W.) are not included in the cross-attention module, since low-rank matrices A of each modality actually can be considered as the linear projection attention in this case. We also provide more discussion and comparison in Appendix B
module to explicitly enhance the interaction between text

---

2.2.3 Shared multimodal matrix B
After unimodal compression and explicit cross-modal interaction enhancement, it becomes crucial to project the resulting unimodal representations into shared space facilitate cross-modal alignment. To this end, shared multimodal matrix B is employed to perform this projection. The final output of the MokA pathway is thus given by:
BAx [B(Axx t,Atto,tadi B(A"x" A,Att BA'x''I
(10)
2.2.4 Overview
In conclusion, in MokA, for pretrained weight matrix Wo E RRdx its update AW E Bdxk is parameterized as the product much smaller matrices: B E Rdxr and {A' E R2 (E with min(dd k). For input sequence x, the forward pass yields:
h Wax AWx Wox + AW [x" xx;"",,
(11)
Wox
[B(A A.ttta,t,)): B(A"x" x" + A.t.t,,, BA'x'"
(12)
Wox xBA BA" BA
DAg BAtt,,,, X, BAt ON.
(13)
uninodal sdaptation
adaptation
where ON, denotes the zero vector of dimension N,, since text-token remains unchanged after cross- attention. During fine-tuning, Wo remains unchanged, with A' and B being subject to optimization. Also, A is initialized using the uniform Kaiming distribution [.] while B is initialized to zero. leads to an initial update A AW- at the beginning of fine- finettning, provide smooth starting point.
Based on Equation 13, MokA ensures both unimodal and cross moosllmodal adaptation, offering a more tailored solution for fine-tuning MILMs.
3 Training and evaluation details
3.1 Implement details
Our framework follows the common MLLM framework as illustrated Figure Ia but with MokA strategy. Text imput is processed by the corresponding LLM tokenizer, and non-text input is first encoded by its encoder, and then aligned with the text embedding space via a projector. Here use Q-former followed by two-layer MLP as the projector. Finally, all tokens are fed into LLM.
For the visual branch of audio-visual-ter and visual-text scenarios, the visual encoder to extract the last layer patch level embedding of each frame or image. For the audio branch of the andio-visual-texy scenario, we use the BEAT' (5] encoder to extract features. For the speech branch of the speech-text scenario, OpenAl's Whisper model [26 is used, The number of query tokens in Q-Former of all branches 32.
(25 as
3.2 Training procedure and benchmarks
Our experiment of MLLM follows the widely used two-stage training paradigm: pre-training stage that aims to cross cross-modal modal alignment and supervised instruction-tuning for downstream tasks.
Pre-training: LLM backbone frozen. Projectors are trainable for cross-modal alignment. For the visual branch of audio-visual-text and visual-text scenarios, trainable modules are trained on video-LLaVA [9] dataset, including the video captioning and the image captioning tasks. For the audio branch of the dataset on the audio captioning task. For the speech branch of the speech- scenario, trainable modules are trained GigaSpeech-M 4 dataset on the speech recognition task. During pre-training, using the Adam W optimizer with a cosine learning rate schedule. The initial learning rate is with warmup ratio of 0.03.
scenario, trainable modules are trained AudioCaps [14]
Instruction-tuning At this stage, we train the model on downstream tasks different scenarios. Trainable parameters include all projectors and our MokA module. For the audio-visual-te case, the model is fine-tuned on the train set of MUSIC-AVQA [17], and AVE (30, respectively. For the visual-text case, the model is fine-tuned on the LLaVA- -n-trutt 150K 20 and 12k subset of
5

---

Gale
Shared
Maxtmoddt B
Shared
LLM Backbone
LLM Backbone
TextA
Audio
Veew
MuximodatA
Mallmmat
Text tolion
Audio tolknn
Visual tokkn
Teat Jokan
Audio tolion
Visual takion
Uni LoRA MM LoRA.
(b) Uni LoRA + MM LoRA Gate.
Figure 4: Framework of baselines that also follow our multimodal-awan basis, yet relying more parameters and offering limited cross- moosll interaction.
A-OKVQA. For the speech-text case, the model is fine-tuned the LibriSpeech 24
using the
annotations provided by (28). Rank of low-rank matrices The remaining settings are the same as the first stage.
Inference: To well assess the effectiveness of our fine-tuning strategy, we evaluate our trained models
on in- in-domain test sets or public benchmarks. Details are provided in the supplementary materials.
Audio-visual-te visual-textt in-domain test set of MUSIC-AVQA and AVE dataset.
Visual-text public benchmarks: MMEpereer percep (9) MMBench 22 POPE [18), SEED BeDD-Bench [16].
Speech-text: public benchmarks: MMAU mini spoech
27 as well as the foundation subset
AIR-Bencee TRR-Benchspecck-.
[37
Large Language Model. For all three cases, ILaMA-2-70-Chat [3]]
(TO),
and Qwen2- 7B-Instruct (36] are used as the LLM base model, respectively. For the audio-visual-ter
case, Bwwe2.5--VL-7B-nnste
[2] is also used the LLM base model. Throughout the training process, weights of LLM are kept frozen. More experiments of Qwen3 are provided in Appendix A
4 Experiments
4.1 Audio-visual-tes scenario
To validate the effectiveness of our MokA fine-tuning strategy, compare it with LoRA 122 and its variants, including multiple LoRA, LoRAMoE [8, DoRA [21], HydraLoRA (2)] Uni-modal LoRA [1] In addition, also compare with two additional baselines, whose frameworks provided in Figure 4 Concretely, the Uni LoRA + MM LoRA strategy employs unimodal low-rank matrices A to extract unimodal information independently, while incorporating an additional fully shared multimodal LoRA module to implicitly promote cross- moddll interaction. The Uni LoRÅ + MM LoRA Gate variant further introduces gating mechanism dynamically integrate the outputs of the Uni LoRA and MM LoRA branches for improved fusion. These two baselines incorporate our multimodal-awe basis that ensures both unimodal and cross-modal adaptation, but involve more parameters and offer limited eross-modal interaction. Based on the results in Table we can have
the following observations:
Our proposed MokA method achieves the superior overall performance across multiple audio-visual- text datasets, consistently outperforming other baselines and compared methods. While MokA introduces a slight increase in parameter scale compared to standard LoRA, this does not account for the observed performance improvements. Based the TTalle multiple LoRA, baseline that uses 3 A matrices and B A matrices, underperforms both standard LoRA and MokA. Simply increasing the number of low-rank matrices does not necessarily lead to better fine-tuning performance. This suggests that MokA's advantage stems not from parameter quantity, but from its insurance for both unimodal and multimodal adaptation.
The mentioned two baselines, Uni LoRA + MM LoRA and Uni LoRA + MM LoRA + Gate, achieve competitive results. These results further support the validity of our multimodal-awen basis that unimodal and cross-modal adaptation are both essential for the tne-tuning of MLLMs. However, de- spite their effectiveness, MokA achieves superior results with fewer parameters and further enhanced cross-modal interactions.
In addition, Qwen2.5VL. with LoRA outperforms both LLaMA2 and Qwen2 under the LoRA fine- tuning setting. But when using MokA, the performance of Qwen2.5VL. is slightly lower than that of LLaMA2 and Qwen2. A possible reason is that the official visual connector in Qwen2.5VL.

---

Table 1: Evaluation results of LoRA, its variants, and our MokA on MUSIC-AVQA and AVE. #A and #B are the number of low-rank matrices. Here N = refers to the number modalities.
datasets,
LLM
Method
MUSIC-AVQA
AVE
#A
#B
LoRA [2]
73.41
69.84
Multiple LoRA
72.66
71.77
N
N
LoRAMoE [8]
73.57
72.81
N
N
DoRA 21
73.97
72.18
LLaMA2
HydraLoRA [29]
74.12
72.27
1
N
Uni-modal LoRA
74.37
71.44
N
N
Uni LoRA + MM LoRA
74.43
72.36
N +1
2
Uni LoRA MM LoRA Gate
74.94
73.56
N
2
MokA
75.71
74.68
N
LoRA
72.83
72.13
1
Multiple LoRA
72.71
72.11
N
N
LoRAMoE [
73.48
72.83
N
N
DoRA 2
73.29
72.91
1
Qwen2
HydraLoRA [29
73.14
72.59
N
Uni-modal LoRA
73.62
73.14
N
N
Uni LoRA MM LoRA
74.09
73.35
N
+
2
Uni LoRA MM LoRA Gate
74.71
73.96
N
+
1
2
MokA
75.26
74.48
N
LoRA
73.00
71.38
Multiple LoRA
73.13
71.27
N
N
LoRAMoE [8]
73.28
71.91
N
N
DoRA [2]]
73.37
71.06
Qwen2.5-VL
HydraLoRA 22
73.04
71.26
N
Uni-modal LoRA
73.46
72.11
N
N
Uni LoRA MM LoRA
73.75
72.27
N
2
Uni LoRA MM LoRA Gate
73.81
72.68
N
2
MokA
74.87
73.14
N
LoRA
78.31
76.91
LLaMA3
Multiple LoRA
78.63
77.02
N
N
MokA
79.15
77.81
N
which serves similar role to the projector used in our other LLM variants, remains frozen during fine-tuning. As a result, only the newly introduced audio branch trainable, which may have limited the full potential of our method. But MokA still introduces noticeable improvement in this case,
compared to other methods. In summary, our method achieves considerable improvement across various LLM backbones, demonstrating its broad versatility.
4.2 Visual-text and speech-text scenarios
To further validate our method across broader range of multimodal scenarios, we conducted case. Specifically, our method is further experiments beyond the challenging scenarios: visual-text and speech-text. For these tasks, we verified on two representative multimodal adopted three different LLM backbones, LLaMA2, LLaMA3, and Qwen2. The corresponding results are presented Table 2 and Table 3 The experimental results demonstrate that our method achieves stable and consistent performance gains across multiple benchmark datasets, further confirming its effectiveness. This indicates the versatility of MokA handling different multimodal combinations and LLM architectures.

---

Table Evaluation results of LoRA, its variants, and our MokA on visual-text benchmarks. #A and #B are the number of low-rank matrices. Here N = 2 refers to the number of modalities.
LLM
Method
MME percep
MMBench POPE BEED-Bence
#A
#B
LoRA
908.52
50.64
70.28
39.71
1
1
Multiple LoRA
882.87
49.83
68.20
38.44
N
N
LoRAMoE [
938.52
51.98
71.15
39.13
N I I N N + N + 2
N
DoRA 121
786.47
51.31
71.07
38.96
LLaMA2
HydraLoRA (29
774.47
47.33
70.87
38.81
N
Uni-modal LoRA
992.31
51.98
72.24
39.27
N
Uni LoRA + MM LoRA
972.87
50.96
73.39
39.74
2
Uni LoRA MM LoRA Gate
988.37
52.01
73.48
39.91
MokA
1025.86
52.74
74.23
40.45
N
LoRA
1062.34
57.89
81.17
55.25
Multiple LoRA
1103.28
57.01
80.96
55.13
N N
LoRAMoE [8]
1157.39
57.29
81.29
56.39
N
DoRA 21
1024.42
56.19
80.75
55.03
I
Qwen2
HydraLoRRA 29
1098.25
56.42
81.34
54.67
N
Uni-modal LoRA
1189.47
57.39
81.12
56.21
N + N +
Uni LoRA + MM LoRA
1191.81
57.17
81.46
56.84
N
Uni LoRA + MM LoRA + Gate
1201.49
57.91
81.72
57.18
MokA
1292.37
59.06
82.33
58.10
N
LoRA
1030.64
68.45
77.47
56.34
1
LLaMA3
Multiple LoRA
1032.74
68.79
78.73
56.06
N N
MokA
1072.67
69.90
79.27
56.60
N
POPE
ARR-BenDheemn
70
72
65
08
55
Tuaorrty Pull modllty Audic-orly Vaua
Ied-orty modaliy Vauak-ony
modellty Spaecherty
(a) Audio-visual-ten case.
(b) Visual-text case.
(c) Specch-text case.
Figure 5: Partial modality inference performance MokA w/o cross-attention. Full modality: Regular case where all multimodal tokens are processed by the MokA module w/o cross-attention. Text-/Audio-/Visual-AS -Ssuull-Sppecc-onny Only eext/andio/visual/spees tokens are passed through the LoRA pathway at the first generation step during inference. Results are based on LLaMA2.
4.3 Partial modality inference of MokA
To further examine how effectively MokA leverages tokens from different modalities, also conduct partial modality inference experiments where only tokens from a selected modality are passed through the LoRA adaptation pathway at the first generation during inference. It should be noted that the evaluated model MokA w/o cross-attention, as cross-attention computation requires the presence of both text and non-text tokens. The results, presented in FFiuure 5] show that MokA w/o cross- attention significantly enhances the utilization of individual modalities compared to LoRA (as shown in Figure 1f) These findings highlight that the multimodal-awan design of MokA facilitates more effective use of all available modalities.
4.4 Cross-modal interaction variants
In the original MokA framework, cross-attention is employed to explicitly strengthen the interaction between text and non-text tokens, thereby facilitating improved cross-modal adaptation. As previously discussed, alternative modules that similarly enhance this interaction also be considered. In this

---

Table 3: Evaluation results of LoRA, its variants, and our MokA on speech-text benchmarks,
MMAU mini- spini-speech and AIR LRR-Benchspeeee -en- #A and #B are the number of low-rank matrices. Here N refers the number of modalities.
LLM
Method
MMAU
AIR-Bench
#A
#B
LoRA
30.33
31.75
1
Multiple LoRA
29.73
31.91
N
N
LoRAMoF [8]
27.63
33,97
N
N
DoRA 21
26.73
34.36
I
LLaMA2
HydraLoRA [29
29.13
31.66
I
N
Uni-modal LoRA
38.14
35.14
N
N
Uni LoRA + MM LoRA
37.54
32.56
N +1
Uni LoRA MM LoRA Gate
32.13
33.04
N+1
W MokA
38.44
39.64
N
LoRA
50.15
44.55
I N
Multiple LoRA
50.45
41.13
N
LoRAMoE [8]
50.75
42.99
N
N
DoRA 21
52.55
44.11
I I
Qwen2
HydraLoRA 129
53.45
43.94
N
Uni-modal LoRA
54.05
47.01
N
N
Uni LoRA MM LoRA
54.16
43.55
N +1
2
Uni LoRA MM LoRA Gate
54.35
46.15
N+1
2
MokA
55.26
49.17
N
LoRA
46.25
43.04
1
1
LLaMA3
Multiple LoRA
44.74
43.87
N
N
MokA
51.05
44.39
N
Table 4: Evaluation results of MokA and variants on audio-visual-text and visual-text cases. Results
based on LLaMA2.
Method
Music-AVQA
AVE
MME percep
MMBench
POPE
BEED-Bench
LoRA
73.41
69.84
908.52
50.64
70.28
39.71
Multiple LoRA
72.66
71.77
882.87
49.83
68.20
38.44
Cross-attention
74.94
72.59
955.18
51.25
72.94
39.91
Naive interaction
75.04
73.18
996.73
51.49
73.52
40.17
MokA 75.71
74.68
1025.86
52.74
74.23
40.45
section, we explore several variants of the cross-modal interaction module, as summarized in Table 4 The cross-attention" * variant also adopts cross-attentin mechanism; however, it uses text tokens as queries. Consequently, the updated text tokens integrate information from the relevant non-text tokens-reverte the direction of interaction compared the original MokA. The naive interaction variant performs simple, uniform mapping from text tokens to non-text tokens without employing any attention mechanism.
Experimental results show that all proposed variants outperform the LoRA baseline, demonstrating the general effectiveness of explicilly enhancing cross-modal interactions. However, the cross-attention" * variant performs slightly worse than the others. One possible explanation is that, unlike in other variants where text tokens remain unchanged, this variant alters text tokens by integrating non-text features. Although cross-modal interaction is enhanced, the modification of text representations may adversely affect language modeling capabilities. In addition, while naive interaction yields competitive results, MokA achieves further improvement through its dynamic attention mechanism. These findings suggest that the core idea of explicitly reinforcing cross-modal interactions is beneficial, and the effectiveness is not restricted to specific module design.

---

4.5 Ablation study
Table 5:
Ablation study of MokA. CA denotes Cross-
To thoroughly validate the efficacy of our method, we conduct ablation stud- ies across three multimodal sce- narios. Results are shown in Table 5 Based on the results, even without the cross-attention module, MokA w/o CA outperforms the LoRA baseline, demonstrating the effectiveness of en- hancing unimodal adaptation. Furthermore, the introduction of the cross-attention module leads to additional performance improvements, indicating the benefit of explicitly enhancing cross-modal adaptation. These results indicate the necessity of each part in MokA.
Attention. Results based on LLaMA2.
Method
MUSIC-AVQA POPE
AIR-Bench
LORA 112
73.41
70.28
31.75
Multiple LoRA
72.66
68.20
31.97
MokA w/o CA
74.85
73.57
33.25
MokA
75.71
74.23
39.64
4.6 Efficiency evaluation
To enable comprehensive comparison, we further evaluate the proposed MokA and LoRA baselines on the proportion of trainable parameters in the full model, and inference latency. As reported in Tabll 6 although MokA introduces additional pa- rameters due to the inclusion of more low-rank matrices, the in- crease quite modest compared to the full LLM. Also, despite MokA incurs slight increase in inference latency compared to standard LoRA, it achieves notable performance gain of 3.95% on the POPE benchmark. These results suggest that the addi- tional computational cost introduced by MokA acceptable, and the performance improvement considerable. More detailed efficiency evaluations are provided in Appendix D)
Table 6: Efficiency evaluation and performance comparison. Here, trainable parameters include low-rank matrices and all projectors. Results are based on LLaMA2.
Trainable / Total
Avg. forward
POPE
Method
Parameters
time per sample
Ace
LoRA 12]
1.27%
1.000 x
70.28
Multiple LoRA
1.43%
1.006 x
68.20
MokA
1.33%
1.069 x
74.23
5 Related works
MLLMs built upon powerful LLM backbones are increasingly demonstrating impressive capabilities across diverse downstream tasks [3). However, fine-tuning these models remains computationally expensive, prompting growing interest in parameter-efficient fine-tuning (PEFT) techniques that reduce memory and storage overhead during adaptation. Among them, LoRA has emerged widely adopted, and researchers have proposed several variants to further improve its efficiency and flexibility [8 21 29 M. For instance, LoRAMoE [8] introduces multiple LoRA heads combined via gating mechanism, while DoRA (1 focuses solely on optimizing the gradient direction, enabling more efficient updates. Despite these advancements, most PEFT strategies for MILMs are direct extensions of L.M techniques and fail to account for the inherent characteristics of multimodal learning. To address this gap, we propose MokA, a fine-tuning strategy specifically designed for MLLMs. It explicitly ensures both unimodal and cross-modal adaptation to better preserve unimodal representations and enhance cross-modal interaction, offering a targeted solution for efficient and effective multimodal adaptation.
6 Discussion
In this paper, we argue that both unimodal adaptation and cross-modal adaptation are essential parts for the effective fine-tuning of MLLMs, yet have largely been neglected before. To this end, we propose Multimodal low-rank Adaptation (MokA) for efficient multimodal fine-tuning. MokA redefines the roles of low-rank matrices A and B, ensuring unimodal information is preserved while enhancing cross-modal interaction by cross-attention. We think MokA is a preliminary step toward multimodal-avv adaptation, highlighting the potential for future extensions that jointly consider both unimodal and cross-man modal adaptation.
10

---

Acknowledgement
The project was supported by the fund for building world-class universities (disciplines) of Renmin University of China, sponsored by CCF-Zhipu.AT Large Model Innovation Fund, and National Natural Science Foundation of China (NO.62 (012622 The project was also supported by the Shanghai Municipal Science and Technology Major Project, and Shanghai Artificial Intelligence Laboratory.
References
[1] AhdeIrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mfxture-of-loras, arXiv preprint 2rX1v:2503.00743, 2025.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang. Peng Wang, Shijie Wang. Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502.13925 2025.
[3] Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang. Maosong Sun, et al. Model composition for multimodal large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 11246-11262, 2024.
[4] Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspecch: An evolving, multi-domain ast corpus with 10,000 hours of transcribed audio. arXiv preprini arXiv:2 2021.
[5] Sanyuan Chen, Yu Wu, Chengyi Wang. Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei, Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058. 2022.
[6] Shaoxiang Chen, Zequn Jie, confficts in instruction finetuning milms. arXiv preprint 1.Xi:22401116660 2024.
Ma. Llava-mole: Sparse mixture lora experts for mitigating data
[7] Zhehuai Chen, He Huang, Andrei Andrusenko, Oleksii Hrinchuk, Krishna Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Ginsburg. Salm: Specch-augmented language model with in-context lcarning for speech recognition and translation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 13521-1352 IEEE, 2024.
[8] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang. Xiaoran Fan, et al. Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin. arXiv preprint 2xxx:23112.09977 2023.
[9] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models, 2024.
[10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
[11] Kaiming He, Xiangyu Zhang, Shanging Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[12] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
[13] Yue Jiang, Yueming Lyu, Bo Peng. Wei Wang, and Jing Dong. Cmsl: Cross-modal style learning for few-shot image generation. Machine Intelligence Research, pages 1-17, 2025.
[14] Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume I (Long and Short Papers), pages 119 132, 2019.
[15] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint 08iv:2001.0445// 2020.

---

[16] Bohao Li. Rui Wang, Guangzhi Wang. Yuying Ge. Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension, 2023.
[17] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9918-191111 2022.
[18] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, arXiv preprint 3xiv:2305.103555 2023.
[19] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint 1rX1v22311.0012,, 2023.
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. arXiv preprint (rXiv:2334.08488. 2023.
[21] Shih-Yang Yang Liu, Chien-Y Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Hing Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[22] Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang. Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2024.
[23] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 JEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[24] Xiaokang Peng. Yake Wei, Andong Deng, Dong Wang. and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8238-8247, 2022.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[26] Alce Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-seale weak supervision. In International conference on machine learning, PMLR, 2023.
pages
[27] S Sakshi, Utkarsh Tyagi. Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint 200iv:2400.1911666 2024.
[28] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilitics for large language models. arXiv preprint 12X2222300.12289 2023.
[29] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37:9565-9584, 2024.
[30] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained vidcos. In Proceedings of the European conference on computer vision (ECCV), pages 247-263. 2018
[31] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099: 092888, 2023.
[32] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, arviv preprint arXiv:2006.04700 2020.
[33] Xiao Wang. Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao Yong Wei, Yaowei Wang. Yonghong multi-modal
Tian, and Wen Gao. Large-scale multi modal pre-trained models: A comprehensive survey. Machine Intelligence Research. 20(4):44--45 2023.
12

---

[34] Yake Wei, Di Hu, Henghui Du, and Ji-Rong Wen. On-the-fly modulation for balanced multimodal learning. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 2024.
[35] An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang. Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Kegin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang. Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Tianyi Tang, Wenbiao Yin, Xingzhang Ren. Xinyu Wang. Xinyu Zhang, Xuancheng Ren. Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025.
[36]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jianxin Yang. Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li. Tianyu Liu, Wenbin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei. Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao. Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, 7eyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024.
[37] Qian Yang. Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. Air-bench: Benchmarking large audio-language models via generative comprehension. arXiv preprint arXiv:2402.07723 2024.
[38] Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[39] Jeong Hun Yeo, Seunghee Han, Minsu Kim, and Yong Man Ro. Where visual speech meets language: Vsp- IIm framework for efficient and context-aware visual speech processing. arXiv preprint 151151 2024.
[40] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, propose MokA, efficient fine- fine-tuning strategy for MLLMs. Experiments covering multiple multimodal scenarios verify the effectiveness of MokA.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
13

---

Justification: In Section 4.6 we discuss the efficiency and efficacy of MokA and baselines. MokA is with a slight increase in parameters and inference latency, but also brings noticeable improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide experimental information in Section 3
Guidelines:
14

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide source code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
stteeeeuubmssiioooooooooc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//iips.cc/publi///guu
CodeeSumiiii wwttubbmmssiiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
15

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the information of used dataset in Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The resource-intesd -inturceinttessvee nature of LLM training prevents from conducting multiple runs or including variance estimates.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
16

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp://porrrss.c//bbb..//eeeeet
Answer: [Yes]
Justification: Yes, have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide this discussion in the supplementary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [No]
Justification: Our model is used for multimodal understanding, which has low risk for misuse.
Guidelines:
17

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In Section 3 we have cited all used models and datasets.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release our source code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
18

---

Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: In this paper, we provide a new fine-tuning strategy for multimodal LLM. And in /eccioon 3] we provide the training process.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy o /ttp///rrppccc//ooooeeeeeee///// for what should or should be described.
19

---

A Extension to Qwen3
To further validate our MokA method, we equip with the latest Qwen3 [35] model. Qwen3-8B is used as the LLM base model. Throughout the training process, weights of LLM are kept frozen.
We conduct experiments in the audio-visual-text case. Our method can still deliver improvements compared to the LoRA baseline, with the latest Qwen3 model. These findings provide additional evidence of the effectiveness and scalability of MokA.
Results shown in Table 7
Table 7: Evaluation results of LoRA, its variants, and our MokA on audio-visual-ter datasets, MUSIC-AVQA and AVE. #A and #B are the number of low-rank matrices. Here N = 3 refers to the number modalities.
Method
MUSIC-AVQA
AVE
#A
#B
LoRA
78.57
74.17
1
Multiple LoRA
78.24
74.01
N
N
MokA
79.54
75.38
N
B MokA with linear projection
In MokA, linear projections (W,. Ws. and W;,) are not included in the cross-attention module:
(A'x")
Att (A"x", A'x", A'x") = sof tmax
A'x'
(14)
VT
(A'x")
Att (A"x", A'x', A'x') sof tmax
A'x'.
(15)
Vr
The reason is that low-rank matrices A each modality actually can considered as the linear projection in attention in this case. Therefore, we do not introduce other linear projections in the cross-attention module. In addition, projections of key and value are shared in this case (A*). In fact, this kind of projection sharing strategy has been widely used to increase the efficiency of For example, the Linformer 332 it also utilizes the sharing key-value projection strategy reduce computation cost. What's more, here non-text tokens used queries to merge textual information into non-text modalities. In this way, cross-modal interaction explicitly enhanced, while tokens are kept unchanged avoid potential disruption the model's original strong text understanding capability.
attention 15 32
To further validate the idea of cross-attention, we conduct experiments that include linear projections in cross-attention. The concrete attention mechanism with linear projection is conducted as follows, and the notation is consistent with the main manuscript:
(W; A'x" (W (W7,'x
Att (A"x*, A'x!, A'x!)
sof tmax
W'
(16)
(W
Att (A"x", A'x', A'x') softmax
A'
(17)
Here, We is the linear projection of the audio query, and the others similar.
In Table II we provide the results in the audio-visual visual-text and visual-text cases. Based on the results, MoKA with linear projection, yields improvements compared with LoRA baseline. However, it does not consistently outperform the original MoKA and introduces additional trainable parameters along with increased computational overhead.
C More than three modality case
In this section, we extend our experiments to scenarios with more than three modalities. Specifically, consider a four-modalig setting involving audio, visual, point cloud, and language data, and
20

---

evaluated it on the MCUB-3 benchmark [3. In this 3+ modality case, LoRA has an accuracy of 37.41%. Our MokA has an accuracy of 45.58% indicating its scalability and effectiveness under 3+ modality cases.
D Efficiency evaluation
Compared to LoRA, MokA introduces additional A matrices and cross-attention module. However, should be noted that the additional computational cost MokA only comes from the cross-attention module, which is flexible and can be replaced by a more efficient strategy if needed. In this section, we provide a more detailed efficiency analysis comparing FLOPs, GPU memory usage, and average forward time per sample (proportional to training time). For clarity, we report these metrics for the two-modality (VL), three-modality (AVL), and four-modality (AVPL) settings. As shown in the tables, MokA's extra computational and memory cost remains limited and acceptable for typical multimodal scenarios.
Table 8: Efficiency evaluation between LoRA and MokA on various datasets. Results are based on LLaMA2.
VL (MME percep)
FLOPs
Memory Usage
Avg. forward time/sample
LoRA
1,000x
1.000x 1.001x
1.000x
MokA
1.009x
1.069x
AVL (MUSIC-AVQA)
FLOPs
Memory Usage
Avg. forward time/sample
LoRA
1.000x
1.000x
1.000x
MokA
1.021x
1.001x
1. 1.134x
AVPL (MCUB-3)
FLOPs
Memory Usage
Avg. forward time/sample
LoRA
1.000x
1.000x
1.000x
MokA
1.013x
1.002x
1.213x
E
Audio-visual interaction
In the original MokA, only the attention between text and non-text tokens considered. It is motivated by the fact that text modality typically conveys the question or task description, while the audio and visual modalities provide environmental information, in the instruction. Therefore. cross-attention is applied to explicitly enhance the interaction between the task (text token) and environment (non-text token). It is also worth exploring whether further interactions between the scene modalities themselves-ic. audio and visual-cne can be beneficial. To this end, we conduct additional ablation studies. Experiments are conducted based on LLaMA2. The table reports the results for audio-visual attention with both audio as query and video as query. The results show that introducing additional audio-visual attention can bring gains, but it is not very noticeable. The potential benefit from further enhancing explicit audio-visual interactions is relatively limited.
Table 9: Evaluation results of LoRA and MokA variants audio-visual datasets. Results based on LLaMA2.
Method
MUSIC-AVQA
AVE
LoRA
73.41
69.84
MokA
75.71
74.68
MokA w/ audio query att.
75.78
74.53
MokA w/video query att.
75.76
74.81
21

---

Ablation study of rank
In this section, conduct experiments of MokA with different ranks. It consistently outperforms the LoRA baseline across different rank settings.
Table 10: Evaluation results of LoRA and MokA on MUSIC-AVQA and AVE with different ranks. Results are based on LLaMA2.
Method
MUSIC-AVQA
AVE
Rank
LoRA
73.41
69.84
r=4
LoRA
73.56
70.01
r=8
LoRA
73.73
70.07
r=12
V MokA
75.71
74.68
r=4
MokA
74.68
74.71
r=8
MokA
74.89
74.36
1=12
G Case study of cross-attention in MokA
In this section, we conduct a case study on the cross-attention module in MokA. This module explicitly integrates task description information from text tokens with non-text tokens, thereby facilitating cross-modal interaction. Here we provide two samples from an audio-visual-text scenario, as shown in Figure 6 The visualization shows the cross-attention weights of the Oproj the 10-th layer. The LLM model is LLaMA2. The results indicate that during the process of explicit cross-modal integration (e.g., cross-attention), text tokens that are more related to given modality can receive higher attention weights. For example, the token "sound" receives greater attention in relation to the audio modality. This cross-man moosll integration can better facilitate the alignment and interaction between text tokens and non-text tokens.
Table 11: Experiments of MokA with linear projection under the audio-visual-text and visual-text cases. The LLM backbone LLaMA2.
Method
Music-AVQA
MME peroep
MMBench
POPE
SEED-Bench
LoRA
73.41
908.52
50.64
70.28
39.71
Multiple LoRA
72.66
882.87
49.83
68.20
38.44
MokA w/ linear projection
73.83
926.77
53.97
72.43
41.01
MokA
75.71
1025.86
52.74
74.23
40.45
H Broader impacts
In this paper, we aim to contribute to the efficient fine-tuning of MLLM, particularly how they well process and integrate information from different modalities. Improvements in this area may support downstream applications in fields like autonomous driving and education. At the same time, this line of research carries certain risks. For example, there is a possibility that MLLM could reflect or amplify biases present in the training data, or be misused in sensitive contexts. We do not directly address these issues in this work, but acknowledge them as important areas for future research. All datasets used are publicly available, and we follow standard filtering procedures to reduce exposure harmful content.
I Datasets
Information about the datasets used in our experiments is provided in this section.
Video-LLaVi VA 19 used mixed dataset of images and videos for video captioning and image captioning tasks. The dataset includes 665k image-text instruction and 100k video-text instruction. This dataset is used for pre-training the visual branch.
22

---

now
many
types
musical
nnttucments
sound
the
Video
Wiight of suddoooed ateriioe
types
musical
instrucments
soundd
the
Video
(a) Sample 1.
sound
Wiiight visca Haust stauniion
Instrument
Sample 2.
Figure 6: Cross-attention weight visualization, where deeper colors indicate higher attention weights.
AudioCaps 14 dataset used for the audio captioning task. It consists of 46K pairs of audio clips and text descriptions. This dataset is used for pre-training the audio branch.
GigaSpeech-M 4 is 1000h dataset for speech recognition task. This dataset is used for pre-training the speech branch.
MUSIC-AVQA 117 is an audio-visual-ten dataset, which is introduced support spatio-temporal understanding of musical content. offers 45K QA pairs across 33 question templates that span multiple modalities and question types.
AVE 30 an audio-visse audio-visual-terr visual--ext dataset. focuses on the audio-visual event localization task. This dataset covers 28 event classes and consists of 4, 4,143 samples.
LLaVA-Instruct-1500 20 is of GPT-generated multimodal instruction-follloinng data. It is used for instruction fine-tuning for the visual-text case.
LibriSpeech 23 is 960-hour dataset. We use the instruction from (28 for instruction fine-tuning of the speech-text case.
MMEpereep
9 is the perception subset of the MME benchmark, covering total of 10 subtasks for the evaluation of the visual-text perception ability.
MMBench (22 is a collection of benchmarks to evaluate the visual-text understanding capability. has 3,000 multiple-choice questions covering object detection, text recognition, action recognition, image captioning, relation reasoning, and so on.
POPE [8 is benchmark that is used for evaluating the visual-text understanding ability of MLLM.
The used image the test set of MSCOCO dataset.
SEED-Bench [6 consists of 19K multiple-choice questions with accurate human annotations for evaluating the visual-text understanding ability of MILMM.
MMAU JAMAUmin-ssa mprAUs for evaluating the speech-text understanding ability of MIL.M.
27 is the speech subset of MMAU-mini benchmark. This benchmark used
the English speech subset of the foundation part AIR-Bench. This benchmark used for evaluating the speech-text understanding ability of MLLM.
A TTR--enchspeeck--a
23


Reviews:
[{"review_id": "l51nidg5Cy", "reviewer": "Reviewer_RCVF", "summary": "This paper introduces MokA, a novel fine-tuning strategy for Multimodal Large Language Models (MLLMs) that addresses the limitations of directly applying LoRA—originally designed for LLMs—to MLLMs. The key insight is that shared LoRA parameters across modalities overly favor text, undermining the adaptation of non-text (e.g., visual, audio, speech) modalities. MokA uses modality-specific low-rank matrices (A) to capture unimodal features and introduces cross-modal attention between text and non-text tokens for explicit interaction, and projects all adapted features using a shared matrix (B) to unify representations.", "questions": "1. The paper should provide a comprehensive efficiency analysis of MokA, including computational overhead, training time, and memory requirements compared to baseline methods. Please elaborate on the trade-offs between performance and cost, and explain whether the performance gain comes from additional training, especially in resource-constrained environments.\n2. The current formulation of \"unimodal adaptation\" in Equation 13 uses a shared B matrix across modalities, which contradicts the true definition of modality-specific adaptation. Additionally, the cross-modal interaction design omits Audio-Visual interactions. The authors should explain this design choice and provide ablation studies on whether incorporating Audio-Visual interactions would further enhance performance.\n3. The authors should address how MokA would perform in scenarios with many modalities (3+) and propose potential solutions to mitigate the growing computational/inference burden in such cases.\n4. The authors should explore and compare alternative attention mechanisms (such as bidirectional attention, dynamic attention weights, or gated attention) and explain why the current design was selected over potentially more powerful alternatives.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**：\n\n1. The paper is well-written with clear motivation addressing the issue of text-dominance in LoRA-based MLLM adaptation. Additionally, the authors present the proposed algorithm MokA with clear illustrations and rigorous mathematical formulations, making it easy for readers to understand the methodology.\n2. MokA is a logical extension of LoRA with minimal overhead, specifically adding modality-specific A matrices and a lightweight cross-attention module to effectively improve the performance of LoRA fine-tuning for MLLMs. While simple, the approach proves to be quite effective.\n3. The extensive experimental results demonstrate that MokA outperforms multiple strong baselines across diverse benchmarks (MUSIC-AVQA, MMBench, POPE, AIR-Bench, etc.). Importantly, these performance gains are consistent across different LLM backbones.\n\n\n\n**Weakness**：\n\n1. In the experimental section, the paper primarily focuses on performance metrics while overlooking crucial efficiency indicators. The introduction of additional A matrices and the cross-attention module incurs extra computational and training time overhead during fine-tuning. Although MokA demonstrates performance gains, these improvements might simply result from increased training resources. The tradeoff between performance and cost is not deeply analyzed (e.g., FLOPs, wall-clock training time, memory usage).\n2. Regarding Equation 13, the unimodal adaptation component is not entirely equivalent to \"unimodal adaptation\" since matrix B is shared across modalities rather than being modality-specific. Additionally, for the \"cross-modal adaptation\" component, we observe only Text-Visual and Text-Audio interactions, while Audio-Visual interactions are missing. The paper does not discuss whether introducing Audio-Visual interactions would further enhance model performance.\n3. With each modality requiring its corresponding A matrix, the scalability of MokA becomes questionable when dealing with numerous modalities. This concern is particularly evident in Table 5, where even with just a few modalities, there's already a 1.13× increase in inference overhead. This overhead would grow substantially with additional modalities.\n4. The design of the cross-attention module is fixed and simple. More flexible designs (e.g., bi-directional or dynamic attention) are not explored or compared.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns in the rebuttal, and their clarifications have improved my understanding of the work. Therefore, I would like to raise my score to 4.", "rebuttal": "We sincerely thank the reviewer for these valuable suggestions, which have greatly strengthened our work. We will add these additional experiments and related discussion to the revised manuscript.\n\n### **1. Efficiency evaluation & 3+ modality extension**\n\nWe thank you for the valuable suggestions about efficiency evaluation. As suggested, we analyze and compare LoRA and MokA from computational cost, GPU memory usage, and extension under 3+ modality cases. Experiments are conducted based on LLaMA2.\n\n1. **On Computational Cost:**\n    \n    Compared to LoRA, MokA introduces additional A matrices and a cross-attention module. **However, it should be noted that the additional computational cost of MokA only comes from the cross-attention module, which is flexible and can be replaced by a more efficient strategy if needed.** \n    \n    Concretely, in LoRA, each token is processed through *one* A matrix and *one* B matrix. Similarly, in MokA w/o cross-attention, each token passes through *one* modality-specific A matrix and *one* shared B matrix. This design maintains the same computational cost as LoRA while enabling modality awareness. In addition, as discussed in Sec. 4.4 and shown in the following table, **MokA w/o cross-attention already brings notable performance improvements over LoRA without additional computational burden.**\n    \n    ---\n    \n    |  | **MUSIC-AVQA** | **POPE** | **AIR-Bench** |\n    | --- | --- | --- | --- |\n    | LoRA | 73.41 | 70.28 | 31.75 |\n    | MokA w/o cross-attention | 74.85 | 73.57 | 33.25 |\n    | MokA | 75.71 | 74.23 | 39.64 |\n    \n    ---\n    \n    As per the cross-attention module, it is employed to explicitly strengthen the interaction between text and non-text tokens, thereby facilitating cross-modal adaptation. **Alternative strategies, like more efficient cross-modal interaction methods, can also be considered** as mentioned in the manuscript. Here we provide one variant, MokA w/ naive interaction, which performs a simple, uniform mapping from text tokens to non-text tokens without employing any attention mechanism. And this more efficient variant also achieves improvement. \n    \n    ---\n    \n    |  | **MUSIC-AVQA** | **AVE** | **MME_${percep}$** | **MMBench** | **POPE** | **SEED-Bench** |\n    | --- | --- | --- | --- | --- | --- | --- |\n    | LoRA | 73.41 | 69.84 | 908.52 | 50.64 | 70.28 | 39.71 |\n    | MokA | 75.71 | 74.68 | 1025.86 | 52.74 | 74.23 | 40.45 |\n    | MokA w/ naive interaction | 75.04 | 73.18 | 996.73 | 51.49 | 73.52 | 40.17 |\n    \n    ---\n    \n2. **On GPU Memory Usage:**\n    \n    MokA’s additional GPU memory usage mainly comes from having modality-specific A matrices. In practice, the A matrices are lightweight by design (following LoRA’s low-rank parameterization), so the increase of memory usage is small and remains within acceptable limits. We will provide more information in the following parts.\n    \n3. **3+ Modality Extension:**\n    \n    Based on the suggestion, we extend our experiments to scenarios with more than three modalities. Specifically, we consider a **four-modality setting** involving audio, visual, point cloud, and language data, and evaluate it on the MCUB-3 benchmark. In this 3+ modality case, LoRA has an accuracy of 37.41%. Our MokA has an accuracy of 45.58%, indicating its scalability and effectiveness under 3+ modality cases. We also provide the efficiency comparison in the following parts.\n    \n4. **On Comprehensive Efficiency Analysis:**\n    \n    Following the reviewer’s valuable suggestion, we have added a more detailed efficiency analysis comparing **FLOPs, GPU memory usage**, and **average forward time per sample** (proportional to training time). For clarity, we report these metrics for the two-modality (VL), three-modality (AVL), and newly added four-modality (AVPL) settings. As shown in the tables, MokA’s extra computational and memory cost remains limited and acceptable for typical multimodal scenarios, which aligns with the above analysis. **Overall, we obtain significant performance gains with only modest extra resource consumption.**\n    \n    ---\n    \n    | **VL (MME_${percep}$)** | **FLOPs** | **Memory Usage** | **Avg. forward time/sample** |\n    | --- | --- | --- | --- |\n    | LoRA | 1526.25 GFLOPs (1.000x) | 27315.75 MB  (1.000x) | 1.75 s (1.000x) |\n    | MokA | 1539.55 GFLOPs (1.009x) | 27329.78 MB (1.001x) | 1.87 s (1.069x) |\n    \n    ---\n    \n    | **AVL (MUSIC-AVQA)** | **FLOPs** | **Memory Usage** | **Avg. forward time/sample** |\n    | --- | --- | --- | --- |\n    | LoRA | 12116.72 GFLOPs  (1.000x) | 27961.74 MB (1.000x) | 5.53 s (1.000x) |\n    | MokA | 12375.90 GFLOPs (1.021x) | 27996.49 MB (1.001x) | 6.27 s  (1.134x) |\n    \n    ---\n    \n    | **AVPL (MCUB-3)** | **FLOPs** | **Memory Usage** | **Avg. forward time/sample** |\n    | --- | --- | --- | --- |\n    | LoRA | 12620.51 GFLOPs  (1.000x) | 28329.56 MB (1.000x) | 1.64 s (1.000x) |\n    | MokA | 12778.88 GFLOPs (1.013x) | 28377.37 MB (1.002x) | 1.99 s  (1.213x) |\n    \n    ---\n    \n\n---\n\n### **2. Uni-modal adaptation in Eq. (13)**\n\nSpecifically, in Eq. (13), the unimodal adaptation component is formulated as $\\[BA^a\\mathbf{x}^a; BA^v\\mathbf{x}^v; BA^t\\mathbf{x}^t\\]$, where $A^a$, $A^v$, $A^t$ are modality-specific matrices and $B$ is a shared projection matrix. Input of modality $i$ first undergoes a unique modality-specific transformation through its respective $A^i$ matrix. And then, the subsequent shared $B$ matrix applies an **identical mapping** to all modalities. Therefore, **the modality-specific characteristics introduced by $A^i$ and $\\mathbf{x}^i$ are preserved, since $B$ is identical to all modality-specific terms and it does not explicitly mix information across modalities.**\n\nTherefore, we consider this component an **approximation of unimodal adaptation**, since the distinct modality-specific paths remain independent. In the revised manuscript, we will refine the related analysis.\n\n---\n\n### **3. Audio-visual interaction**\n\nIn the original MokA, only the attention between text and non-text tokens is considered. It is motivated by the fact that text modality typically conveys the question or task description, while the audio and visual modalities provide environment information, in the instruction. Therefore, cross-attention is applied to explicitly enhance the interaction between the task (text token) and environment (non-text token). \n\nAs suggested, it is also worth exploring whether further interactions between the scene modalities themselves—i.e., audio and visual—can be beneficial. To this end, we conduct additional ablation studies. Experiments are conducted based on LLaMA2. The table reports the results for audio-visual attention with both audio as query and video as query. The results show that introducing additional audio-visual attention can bring gains but is not very noticeable.  The reason could be that the original cross-attention in MokA has effectively captured task-relevant information within the audio and visual inputs. As a result, the potential benefit from further enhancing explicit audio-visual interactions is relatively limited. We will include this new ablation and the corresponding discussion in the revised manuscript.\n\n---\n\n|  | **MUSIC-AVQA** | **AVE** |\n| --- | --- | --- |\n| LoRA | 73.41 | 69.84 |\n| MokA | 75.71 | 74.68 |\n| MokA w/ audio query att. | 75.78 | 74.53 |\n| MokA w/video query att. | 75.76 | 74.81 |\n\n---\n\n### **4. More attention strategies**\n\nWe thank the reviewer for this valuable suggestion regarding the flexibility of the cross-attention module. **As mentioned above, our cross-attention design primarily aims to explicitly strengthen cross-modal interactions, and it is flexible and can be replaced by different strategies if needed.** Based on the suggestion, we further explore bi-directional attention and dynamic attention as alternative designs. Experiments are conducted based on LLaMA2.\n\nThe results show that these alternative designs also achieve improvements over the LoRA baseline, further demonstrating the flexibility and effectiveness of the cross-modal interaction module. We appreciate the reviewer’s suggestion, which has strengthened the paper by motivating us to include these informative comparisons.\n\n---\n\n|  | **MUSIC-AVQA** | **AVE** | **MME_${percep}$** | **MMBench** | **POPE** | **SEED-Bench** |\n| --- | --- | --- | --- | --- | --- | --- |\n| LoRA | 73.41 | 69.84 | 908.52 | 50.64 | 70.28 | 39.71 |\n| MokA | 75.71 | 74.68 | 1025.86 | 52.74 | 74.23 | 40.45 |\n| bi-att. | 75.13 | 74.92 | 1015.31 | 52.21 | 74.07 | 40.56 |\n| dynamical att. | 74.97 | 74.48 | 1001.14 | 52.42 | 74.67 | 40.28 |\n\n---"}, {"review_id": "NQjGIGqRMU", "reviewer": "Reviewer_x9zT", "summary": "This paper advances along the LoRA direction, proposing a targeted variant, MokA, for Multimodal-LLM. The variant is motivated by the observation that the update of vanilla LoRA is dominated by text tokens. Accordingly, the author allocates modality-specific LoRA A parameters to decouple their update. Additionally, cross-modal communication is injected within the MokA module.", "questions": "Questions:\n\n1. Considering the LoRA baseline, your ablations have already shown that 'Multiple LoRA' does not enhance performance, but I wonder if increasing the rank of LoRA baseline could lead to better results? I mean, if in practice I find my LoRA model not performing well, my first reaction would be to try increasing the hidden rank instead of trying a much more complicated variant.\n2. \"Specifically, we evaluate the model’s performance when only tokens from a selected modality are passed through the LoRA adaptation pathway at the first generation step during inference. \"  Could the author clarify what the highlighted \"at the first generation step during inference\" means?  I guess this means something similar to \"during prompt prefilling\" or \"during building the KV cache\", but I am not sure.\n3. The comparison result between \"Multiple-LoRA\" and \"MokA w/o CA\" is counterintuitive to me. The result means that a shared B matrix outperforms multiple modality-specific B matrices. Could the author provide some analysis/explanation?\n3. Arrows in Fig.4 are messy, and the lines cross arbitrarily. I strongly suggest refining this figure\n5. While the \"Uni LoRA + MM LoRA\" and \"Uni LoRA + MM LoRA + Gate\" baselines are interesting, their relationship with the proposed MokA is not sufficiently explained in the paper. What do the authors intend to verify by introducing these baselines?\n\nOverall, I would be happy to raise my score if some of the above questions can be satisfactorily answered.", "limitations": "See questions", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The paper has a clear logical flow, which makes it easy to follow.\n- Experiment designs are reasonable and well support the paper's claims\n\nWeaknesses:\n- The proposed method breaks the simplicity of LoRA: the side branch can no longer be merged into the main LLM parameters, and the introduced cross-attention module adds additional complexity conceptually, engineerly, and computationally.", "was_revised": true, "final_justification": "After the discussion, this paper is complete and sound to me. I am therefore happy to raise my score.\n\nMeanwhile, since I have not been closely tracing the recent literature on this topic, for novelty and comparative experiments, I would suggest the AC to give more weight to the opinions of other reviews.", "rebuttal": "We greatly appreciate the reviewer for the constructive and detailed comments, which help us improve the quality, clarity and presentation of the manuscript. We will add these experiments and analysis to the revised manuscript.\n\n### **1. Ablation studies about rank**\n\nBased on the suggestions, we explore the effect of varying the rank. Experiments are conducted based on LLaMA2. As shown in the following table, **simply increasing the rank does not necessarily lead to consistent performance gains.** This observation aligns with the finding reported in the original LoRA paper [1]: “*We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient.*”\n\nMore importantly, our observation in Figure 1 of the manuscript shows that the optimization of all-modality-shared LoRA parameters is overly influenced by text tokens, resulting in non-text tokens being less effectively utilized. Therefore, **even with an increased rank, the parameters remain fully shared across modalities, which still structurally prevents independent utilization of unimodal information.** The issue is not just about capacity, but more about whether the module structure can effectively capture modality-specific characteristics and cross-modal interactions.\n\nBased on this, **MokA is designed to explicitly introduce modality-specific parameters and cross-modal interaction, addressing this structural limitation that increasing rank alone cannot resolve.** We also conduct experiments of MokA with different ranks. It consistently outperforms the LoRA baseline across different rank settings.\n\n---\n\n|  |  | **MUSIC-AVQA** | **AVE** | **MME_${percep}$** | **MMBench** | **POPE** | **SEED-Bench** |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| LoRA | r=4 | 73.41 | 69.84 | 908.52 | 50.64 | 70.28 | 39.71 |\n| LoRA | r=8 | 73.56 | 70.01 | 947.24 | 51.23 | 70.63 | 39.57 |\n| LoRA | r=12 | 73.73 | 70.07 | 916.76 | 51.37 | 70.17 | 39.46 |\n| MokA | r=4 | 75.71 | 74.68 | 1025.86 | 52.74 | 74.23 | 40.45 |\n| MokA | r=8 | 74.68 | 74.71 | 1027.84 | 52.83 | 74.43 | 40.38 |\n| MokA | r=12 | 74.89 | 74.36 | 1007.33 | 52.81 | 74.18 | 40.97 |\n\n---\n\n[1] LoRA: Low-Rank Adaptation of Large Language Models, ICLR, 2022.\n\n---\n\n### **2. First token generation process**\n\nHere, “at the first generation step during inference” refers specifically to the prefilling stage, where the model processes the entire prompt sequence and generates the first output token.\n\nThank you for pointing this out, and we will revise the related parts of the manuscript to make it more readable.\n\n---\n\n### **3. Discussion about multiple-LoRA & MokA w/o CA**\n\nTo clarify, the following explains the differences between LoRA, Multiple-LoRA, and MokA w/o CA. Suppose the number of modalities is $N$:\n\n- **LoRA**: uses one matrix A and one matrix B, both shared by all modalities.\n- **Multiple-LoRA**: uses $N$ matrices A and $N$ matrices B. But all matrices (both A and B) are still fully shared across modalities. *There are no multiple modality-specific matrices.* The multiple heads simply increase capacity without distinguishing modality-specific characteristics.\n- **MokA w/o CA**: uses $N$ modality-specific matrices A and one shared matrix B. Here, the matrices A adapt each modality independently, while the shared matrix B maps the separated unimodal subspaces back into a common latent space.\n\n**Multiple-LoRA** can be viewed as simply expanding the capacity of standard LoRA by adding multiple heads. However, *it still treats all modalities identically, i.e. all parameters are shared across all modalities*. There are no modality-specific matrices in this method. In this fully shared case, as discussed above, optimization of parameters is overly influenced by text tokens, which results in non-text tokens being less effectively utilized. Therefore, simply increasing the number of LoRA heads does not provide a significant benefit in addressing the multimodal challenge.\n\nBy contrast, **MokA w/o CA** explicitly captures unimodal information by introducing modality-specific matrices *A*. This design better captures unique modality characteristics before merging them back with the shared B matrix.\n\nTherefore, the better performance of MokA w/o CA comes from its more effective use of modality-specific information.\n\n---\n\n### **4. Discussion about \"Uni LoRA + MM LoRA\" and \"Uni LoRA + MM LoRA + Gate\"**\n\nAs mentioned in Sec 2.1, one of the core motivations for our work is that existing LoRA variants for multimodal adaptation often overlook the need to jointly address both unimodal adaptation and cross-modal adaptation. Our MokA method is proposed to address this limitation.\n\nSince most LoRA variants do not explicitly consider multimodal characteristics, **we further introduce two more relevant and competitive baselines to provide a more comprehensive comparison.**\n\nConcretely, these two baselines, Uni LoRA + M LoRA and Uni LoRA + MM LoRA + Gate, are straightforward designs that aim to address both unimodal and multimodal adaptation:\n\n- “Uni LoRA + MM LoRA” directly uses unimodal LoRA to fit unimodal adaptation, and multimodal LoRA to fit multimodal adaptation.\n- “Uni LoRA + MM LoRA + Gate” further adds a gating mechanism to adaptively integrate the unimodal and multimodal branches.\n\n**These baselines demonstrate that introducing a multimodal-aware design—i.e., accounting for both unimodal and cross-modal adaptation—can indeed lead to improved performance, confirming that jointly considering both aspects is necessary.** \n\nHowever, these variants rely on simply stacking separate LoRA modules and thus require substantially more parameters. \n\nIn contrast, **MokA** achieves both unimodal and multimodal adaptation more neatly and efficiently through its modality-specific A matrices and flexible cross-attention module, **achieving better performance with a simpler and more cohesive structure**.\n\n---\n\n### **5. Discussion about the simplicity of LoRA**\n\nThank you for pointing this out. Indeed, MokA makes the side branch separate from the main LLM parameters and thus it can no longer be merged back. However, **multimodal scenarios are inherently more complex, and it is often inevitable to introduce additional cost to better capture multimodal characteristics.**\n\nWe have aimed to keep the structure of MokA as neat and efficient as possible. Based on our efficiency evaluation in terms of FLOPs, GPU memory usage, and average forward time per sample, MokA introduces only modest additional resource consumption while achieving significant performance gains. Therefore, we believe MokA is practical with noticeably better performance.\n\n---\n\n| **VL (MME_${percep}$)** | **FLOPs** | **Memory Usage** | **Avg. forward time/sample** |\n| --- | --- | --- | --- |\n| LoRA | 1526.25 GFLOPs (1.000x) | 27315.75 MB  (1.000x) | 1.75 s (1.000x) |\n| MokA | 1539.55 GFLOPs (1.009x) | 27329.78 MB (1.001x) | 1.87 s (1.069x) |\n\n---\n\n| **AVL (MUSIC-AVQA)** | **FLOPs** | **Memory Usage** | **Avg. forward time/sample** |\n| --- | --- | --- | --- |\n| LoRA | 12116.72 GFLOPs  (1.000x) | 27961.74 MB (1.000x) | 5.53 s (1.000x) |\n| MokA | 12375.90 GFLOPs (1.021x) | 27996.49 MB (1.001x) | 6.27 s  (1.134x) |\n\n---\n\n### **6. Enhancing Fig. 4**\n\nThank you for this suggestion. We will revise Fig.4 to make the arrows/lines clearer and more organized, improving its clarity."}, {"review_id": "l873ROgAxd", "reviewer": "Reviewer_DUve", "summary": "This paper proposes Multimodal low-rank Adaptation (MokA), consisting of unimodal matrix A, task-centric cross-attention, and shared multimodal matrix B. MokA provides a more targeted solution for efficient adaptation of MLLMs instead of borrowing from LLM. Experiments demonstrate that MokA achieves better performance across multiple benchmarks.", "questions": "1. Are all the methods listed in the table trained under the same settings?\n2. Since tokens from different modalities are processed by separate matrices A, does MokA mitigate catastrophic forgetting of the LLM’s pretrained knowledge? To verify, for example, this is particularly relevant for the MME reasoning subset, which relies heavily on the model’s original language capabilities. Could you report the performance on the MME reasoning subset specifically?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. **Clear motivation.** This paper is motivated by designing an efficient fine-tuning strategy tailored for the multimodal nature of MLLMs. A preliminary study clearly demonstrates the limitations of the existing unimodal PEFT strategies.\n2. **Technical contributions.** This paper innovatively decouples the projection matrices A and B and designs a cross-modal interaction mechanism between them.\n3. **Comprehensive experiments.** This paper conducts experiments across audio-visual-text, visual-text, and speech-text benchmarks, and makes comparisons with a variety of PEFT methods.\n\n### Weaknesses\n\n1. **Lack of discussion regarding hyperparameters.** The rank of low-rank matrices is 4, which is not consistent with the optimal rank 8 reported in the LoRA paper. How was this parameter selected, and why not experiment with the optimal configuration?\n2. **Limited related works discussion.** There are several existing works, like Omni-SMoLA[1] already propose PEFT methods suitable for multimodal learning by assigning tokens from different modalities to different LoRAs. However, this paper only discusses methods that are not designed for multimodal learning.\n\n---\n\n[1] Wu, JIalin, et. al. “Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts”", "was_revised": true, "final_justification": "The author supplements hyperparameter experiments and related work comparisons in the rebuttal, which address all of my concerns. Given the clear motivation and technical contribution, I would recommend accepting this paper. However, since this paper does not bring breakthrough progress or significant performance improvement, I am unable to continue increasing my score.", "rebuttal": "We are deeply grateful to the reviewer’s positive feedback and helpful comments. Based on suggestions, we have added new ablation studies, comparisons, and evaluations, which will be included in the revised manuscript. \n\n### **1. Ablation studies about rank**\n\nThank you for the valuable suggestion. In our experiments, we did not specifically tune the rank hyperparameter, as our primary focus is to provide a multimodal-aware PEFT method that is rank-agnostic. Based on this suggestion, we conducted ablation studies with different ranks, and the results are shown in the following table. Experiments are conducted based on LLaMA2. These results indicate that simply increasing the rank does not always lead to performance gains, which aligns with the observation in the original LoRA paper [1]: “*We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient.*”\n\nMore importantly, MokA consistently outperforms the LoRA baseline across different rank settings, demonstrating its scalability. We will include this discussion and the new results in the revised manuscript. Thank you again for highlighting this point.\n\n---\n\n|  |  | **MUSIC-AVQA** | **AVE** | **MME_${percep}$** | **MMBench** | **POPE** | **SEED-Bench** |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| LoRA | r=4 | 73.41 | 69.84 | 908.52 | 50.64 | 70.28 | 39.71 |\n| LoRA | r=8 | 73.56 | 70.01 | 947.24 | 51.23 | 70.63 | 39.57 |\n| LoRA | r=12 | 73.73 | 70.07 | 916.76 | 51.37 | 70.17 | 39.46 |\n| MokA | r=4 | 75.71 | 74.68 | 1025.86 | 52.74 | 74.23 | 40.45 |\n| MokA | r=8 | 74.68 | 74.71 | 1027.84 | 52.83 | 74.43 | 40.38 |\n| MokA | r=12 | 74.89 | 74.36 | 1007.33 | 52.81 | 74.18 | 40.97 |\n\n---\n\n[1] LoRA: Low-Rank Adaptation of Large Language Models, ICLR, 2022.\n\n---\n\n### **2. Comparison to related methods**\n\nWe thank the reviewer for pointing out the related method, Omni-SMoLA.\n\n**Compared to MokA, Omni-SMoLA adopts a more complex design with noticeably more parameters**. For example, in an AVL (Audio-Visual-Language) scenario, Omni-SMoLA employs separate SMoLA modules for each modality (visual, audio, and text), in addition to an extra multimodal SMoLA. Each SMoLA module internally integrates multiple LoRA branches.\n\nIn contrast, MokA only uses a few low-rank A matrices (one per modality) and a single shared low-rank B matrix, without stacking multiple independent LoRA blocks. Therefore, Omni-SMoLA can have several times more parameters than MokA. \n\nConcretely, in experiments, we set each SMoLA to have two LoRA branches. Therefore, in the AVL case, Omni-SMoLA has four SMoLA modules (audio, visual, text and multimodal), and each SMoLA has two LoRA branches. Then the total number of low-rank matrices (both A and B) is 16, which is much more than MokA. The VL case is similar.\n\nThe comparison results in the following table show that **MokA achieves comparable or even better performance than Omni-SMoLA, while maintaining a simpler and more lightweight design**. Experiments are conducted based on LLaMA2.\n\nWe will add this comparison and discussion to the revised manuscript.\n\n---\n\n| **AVL case** | **MUSIC-AVQA** | **AVE** | **#A** | **#B** |\n| --- | --- | --- | --- | --- |\n| LoRA | 73.41 | 69.84 | 1 | 1 |\n| MokA | 75.71 | 74.68 | 3 | 1 |\n| Omni-SMoLA | **75.73** | **74.70** | 8 | 8 |\n\n---\n\n| **VL case** | **MME_${percep}$** | **MMBench** | **POPE** | **SEED-Bench** | **#A** | **#B** |\n| --- | --- | --- | --- | --- | --- | --- |\n| LoRA | 908.52 | 50.64 | 70.28 | 39.71 | 1 | 1 |\n| MokA | **1025.86** | 52.74 | **74.23** | 40.45 | 2 | 1 |\n| Omni-SMoLA | 972.64 | **53.01** | 73.43 | **41.68** | 6 | 6 |\n\n---\n\n### **3. Evaluation on MME reasoning benchmark**\n\nWe thank the reviewer for this helpful suggestion to evaluate performance on the MME reasoning benchmark to assess potential catastrophic forgetting of the LLM’s pretrained knowledge. Following this advice, we conduct an evaluation on the MME reasoning benchmark. Experiments are conducted based on LLaMA2. \n\nOn the MME reasoning benchmark, the LoRA baseline achieves an accuracy of 16.78%, while MokA reaches 18.01%, indicating that MokA has stronger capacity in this aspect.\n\nIt should be noted that both LoRA and MokA do not obtain high scores on this benchmark, as our work focuses only on the supervised fine-tuning (SFT) stage and does not include additional designs specifically aimed at enhancing reasoning ability. Nevertheless, under the same setting, MokA achieves consistently better results, demonstrating its effectiveness.\n\nWe will include the new MME reasoning results and discussion in the revised manuscript.\n\n---\n\n### **4. Training setting**\n\nYes, all the compared methods under the same scenarios are conducted under the same settings to ensure fairness."}, {"review_id": "b30rDJrtL7", "reviewer": "Reviewer_3Vnj", "summary": "This paper introduces a new mechanism for multimodal low-rank adaptation for MLLMs. In a nutshell, conventional LoRA uses matrix A to project down to a low-rank space, then uses matrix B to project to original dimension. And for multimodal LLMs, A and B are shared across 3 modalities: text, audio, and vision.\n\nThis paper proposes to: (1) use 3 different A matrices, one for each modality. (2) use a cross-attention among the 3 modalities in the projected space, and update the embeddings for audio and vision. (3) use a shared B matrix.", "questions": "Line 219: \"a baseline that uses 3 A matrices and B A matrices\" - what does this mean? If it's a typo, please fix.\n\nTable 4: Why is \"Multiple LoRA\" worse? And why is \"MokA w/o CA\" better? What are the major differences? Is sharing or not sharing B matrix the only difference? \n\nSection 2.2.2: So this cross attention mechanism does not introduce any new parameters, and only introduce new computes? If my understanding is correct, this is really worth highlighting (both for the audience to understand, and for emphasizing as a strength of the method).\n\nSection 2.2.2: So Query is from audio/vision, Key is from text, and Value is from text? Does this use any positional encoding?\n\nWhich part of the LLM do we apply LoRA or MokA in the experiments? In another word, what is the \"W\" matrix that you are adapting in the experiments? Did you adapt all the weight matrices?", "limitations": "The cross attention vs causal decoding seems an important limitation here, especially for MLLMs generating multimodal tokens. The paper did not cover or discuss them.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nPaper is well written. Very easy to follow. Very readable.\n\nFigure 1 is a great example showcasing the problem of conventional LoRA on multimodal tasks: \"LoRA parameters is overly influenced by text tokens\". I am very convinced by how this was presented.\n\nThe math part is also very clear. All variables are clearly defined. Equations are easy to follow and all technically correct. \n\nTable 1 did a great job explaining the differences between alternative approaches. Using #A and #B makes it very easy to understand.\n\nThe proposed method outperforms alternative methods almost on all multimodal setups.\n\nWeaknesses:\n\nThe efficiency evaluation in Section 4.5 looks a bit problematic to me, especially the Inference Latency part. How is latency measured? What is the unit? Why Multiple LoRA has more latency than LoRA? To me it seems they should be same, if the implementations are not too different. MokA should have the highest latency because the computation of the cross attention is expensive. If the implementations are not apple-to-apple comparable, I would suggest using FLOPs instead of latency.\n\nThe paper did not cover an important technical detail: how does decoding work with MokA, especially with the cross attention? By introducing cross attention, the model is no longer causal. Every time when the model decodes a new text token, do we recompute the projected embeddings for all audio/vision tokens? Or the cross attention only applies to the prompt part, not the decoded / response part? What about MLLMs that are capable of generating audio/vision tokens?", "was_revised": true, "final_justification": "The authors have addressed all of my raised concerns. We stick with my original rating.", "rebuttal": "We sincerely thank the reviewer for the positive feedback on our work. We also appreciate the helpful suggestions regarding the efficiency evaluation, the discussion of the cross-attention module, and other valuable advice. In response, we will include related experiments and discussions in the revised manuscript to make the paper clearer. \n\n### **1. Efficiency evaluation**\n\nIn the previous implementation of Multiple-LoRA, we did not merge the multiple LoRA heads during inference, to keep the implementation modular easily extensible to other LoRA variants. As a result, each token passes through multiple A and B matrices independently, bringing additional cost. This could be the reason that Multiple-LoRA has a longer latency in the previous experiments. \n\nThank you for pointing this out. We improve the implementation of Multiple-LoRA, merging multiple LoRA heads during inference. In addition, we also add additional efficiency metrics, including FLOPs and GPU memory usage, to better evaluate the efficiency. The following table shows results under Vision-Language case as an example.\n\nBased on the results, Multiple-LoRA-merged during inference has a similar computational cost. MokA obtains noticeable performance improvement with only modest extra resource consumption. We will add these experiments and discussion to the revised manuscript.\n\n---\n\n| **VL (MME_${percep}$)** | **FLOPs** | **Memory Usage** | **Avg. forward time/sample** |\n| --- | --- | --- | --- |\n| LoRA | 1526.25 GFLOPs (1.000x) | 27315.75 MB  (1.000x) | 1.75 s (1.000x) |\n| Multiple LoRA-merged | 1526.67 GFLOPs (1.000x) | 27349.53 MB (1.001x) | 1.76 s (1.006x) |\n| MokA | 1539.55 GFLOPs (1.009x) | 27329.78 MB (1.001x) | 1.87 s (1.069x) |\n\n---\n\n### **2. Discussion about cross-attention module**\n\nHere we will provide a more detailed discussion about our cross-attention module. \n\n1. **Cross-attention only applies to prompt tokens**\n    \n    In the original MokA, cross-attention is motivated by the fact that text modality typically conveys the question or task description, while the audio and visual modalities provide environment information, in the prompt. Cross-attention is applied to explicitly enhance the interaction between the **task (text token in the prompt) and environment (non-text token in the prompt).**\n    \n    **Therefore, the cross-attention is only applied to the prompt parts.** Concretely, during training, only the prompt tokens (excluding response tokens) are considered in the cross-attention module. During inference, cross-attention only operates on the prefilling stage, where the model processes the entire prompt sequence and generates the first output token. The decoding stage is not applied.\n    \n    Accordingly, **MokA does not greatly break the causal nature when generating response. It only enhances the comprehension of prompt information.**\n    \n2. **Query and positional encoding**\n    \n    Yes, Query is from audio/vision tokens. Key and Value are from text tokens. Regarding positional encoding, we retain the original positional encoding of the LLMs, and do not introduce any new encoding specifically for the cross-attention for efficiency. \n    \n3. **No additional parameters**\n    \n    Yes, this module does not introduce any new parameters, and only introduces new computes. Before passing into the cross-attention module, tokens of each modality are processed by their own low-rank matrices $A$. **The low-rank matrices $A$ of each modality can be approximately regarded as the linear projection in the standard attention in this case.** In addition, projections of key and value are shared in this case (since key and value are both text tokens that are processed by $A^t$). And this kind of projection sharing strategy has been shown to be effective for improving attention efficiency [1]. **Therefore, we do not introduce other linear projections in the cross-attention module to be more efficient.** Thank you for this helpful advice, and we will add the related discussion to the revised manuscript.\n    \n\n[1] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.\n\n---\n\n### **3. MLLMs can generate audio/vision tokens**\n\nAt present, our work focuses on scenarios where the MLLMs generate text tokens while conditioning on audio/vision/text inputs. MLLMs that can generate audio/vision tokens are indeed an important and exciting direction for future research. We sincerely thank the reviewer for inspiring us to consider this promising direction.\n\n---\n\n### **4. Discussion about Multiple-LoRA & MokA**\n\nHere we first provide the differences between LoRA, Multiple-LoRA, and MokA w/o CA. Suppose the number of modalities is $N$:\n\n- **LoRA**: uses one matrix A and one matrix B, both shared by all modalities.\n- **Multiple-LoRA**: uses $N$ matrices A and $N$ matrices B. But all matrices (both A and B) are still fully shared across modalities.\n- **MokA w/o CA**: uses $N$ modality-specific matrices A but one shared matrix B. Here, the matrices A adapt each modality independently, while the shared matrix B maps the separated unimodal subspaces back into a common latent space.\n\n**Multiple-LoRA** simply expands standard LoRA’s capacity by adding multiple heads but still treats all modalities the same—ignoring their unique characteristics. However, just increasing capacity while fully sharing parameters across all modalities does not solve the specific challenges of multimodal scenarios. It still does not consider both unimodal adaptation and multimodal adaptation. \n\nIn contrast, **design of MokA w/o CA is multimodal-aware. It explicitly ensures better unimodal adaptation. MokA w/o CA explicitly captures unimodal information by introducing modality-specific matrices A**. It ensures that each modality’s unique information is individually captured before merging. This simple but crucial design helps better utilize unimodal adaptation. This explains why MokA w/o CA outperforms Multiple-LoRA, even with fewer total parameters.\n\n---\n\n### **5. Applied weights of LoRA/MokA**\n\nIn experiments, we apply LoRA or MokA modules to important weights within each transformer block. Specifically, we insert them into:\n\n- q_proj, k_proj, v_proj, o_proj (the self-attention projection weights),\n- gate_proj, up_proj, and down_proj (the MLP feed-forward layers).\n\n---\n\n### **6. Typos**\n\nWe thank the reviewer for pointing out the typos. We have carefully revised the manuscript to correct these errors. Thank you for helping us improve the clarity and presentation of the paper."}]

Author rebuttal:

Meta-review: Lora fine-tuning, when applied directly to MLLMs, overly favors text, and undermines the adaptation of non text modalities. To address this, the authors present Moka, which uses modality specific low rank matrices for unimodal features, and a cross-modal attention to model the interaction explicitly, and finally, projects all features to a shared space for unification. This outperforms multiple strong baselines across diverse benchmarks (MUSIC-AVQA, MMBench, POPE, AIR-Bench, etc.). Importantly, these performance gains are consistent across different LLM backbones.

Strengths 

- All reviewers find the paper very well motivated, clearly illustrated with a clear logical flow and excellent figures, and mathematically rigorous, making the paper easy to read and understand. 
- The approach is simple i.e. only adds a modality specific module and a cross-attention module, but yet achieves very strong results. 
- All reviewers appreciate the strong experimental design and the strong results on multiple benchmarks.

Weaknesses

1. The primary weaknesses/questions by reviewers have all been promptly and very reasonably addressed by the authors. These include:
    1. Full absolute FLOPs, memory, and latency numbers on both Lora and Moka.
    2. Adding audio-visual interaction in addition to text-audio and text-visual interactions.
    3. Stronger versions of the cross-attention module
    4. Ablations of higher/lower ranks for Lora - doesn’t help Lora
    5. Scaling to 3+ modalities - Moka scales well
    6. Comparison with Omni-SMoLA - Moka matches it with a much simpler and more lightweight design. 
    7. Catastrophic forgetting - On the MME reasoning benchmark, the LoRA baseline achieves an accuracy of 16.78%, while MokA reaches 18.01%, indicating that MokA has stronger capacity in this aspect.

The paper has all four accepts, including a strong accept. It is a solidly executed paper with a clear motivation and a strong/easily readable manuscript. Based on results, this paper can serve as the de-facto PEFT method for MLLMs going forward. I recommend this be accepted.
Decision: Accept (oral)