Title: MokA: Multimodal Low-Rank Adaptation for MLLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal

MLLMs

PEFT

fune-tuning
Abstract: In this paper, we reveal that most current efficient multimodal fine-tuning methods are hindered by a key limitation: they are directly borrowed from LLMs, often neglecting the intrinsic differences of multimodal scenarios and even affecting the full utilization of all modalities. Inspired by our empirical observation, we argue that unimodal adaptation and cross-modal adaptation are two essential parts for the effective fine-tuning of MLLMs. From this perspective, we propose Multimodal Low-rank Adaptation (MokA), a multimodal-aware efficient fine-tuning strategy that takes multimodal characteristics into consideration. It compresses unimodal information by modality-specific parameters while explicitly enhancing cross-modal interaction, ensuring both unimodal and cross-modal adaptation. Extensive experiments cover three representative multimodal scenarios (audio-visual-text, visual-text, and speech-text), and multiple LLM backbones (LLaMA2, Qwen2, Qwen2.5-VL, etc). Consistent improvements indicate the efficacy and versatility of the proposed method. Ablation studies and efficiency evaluation are also conducted to fully asses our method. Overall, we think MokA provides a more targeted solution for efficient adaptation of MLLMs, paving the way for further exploration.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new mechanism for multimodal low-rank adaptation for MLLMs. In a nutshell, conventional LoRA uses matrix A to project down to a low-rank space, then uses matrix B to project to original dimension. And for multimodal LLMs, A and B are shared across 3 modalities: text, audio, and vision.\n\nThis paper proposes to: (1) use 3 different A matrices, one for each modality. (2) use a cross-attention among the 3 modalities in the projected space, and update the embeddings for audio and vision. (3) use a shared B matrix.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nPaper is well written. Very easy to follow. Very readable.\n\nFigure 1 is a great example showcasing the problem of conventional LoRA on multimodal tasks: \"LoRA parameters is overly influenced by text tokens\". I am very convinced by how this was presented.\n\nThe math part is also very clear. All variables are clearly defined. Equations are easy to follow and all technically correct. \n\nTable 1 did a great job explaining the differences between alternative approaches. Using #A and #B makes it very easy to understand.\n\nThe proposed method outperforms alternative methods almost on all multimodal setups.\n\nWeaknesses:\n\nThe efficiency evaluation in Section 4.5 looks a bit problematic to me, especially the Inference Latency part. How is latency measured? What is the unit? Why Multiple LoRA has more latency than LoRA? To me it seems they should be same, if the implementations are not too different. MokA should have the highest latency because the computation of the cross attention is expensive. If the implementations are not apple-to-apple comparable, I would suggest using FLOPs instead of latency.\n\nThe paper did not cover an important technical detail: how does decoding work with MokA, especially with the cross attention? By introducing cross attention, the model is no longer causal. Every time when the model decodes a new text token, do we recompute the projected embeddings for all audio/vision tokens? Or the cross attention only applies to the prompt part, not the decoded / response part? What about MLLMs that are capable of generating audio/vision tokens?\n\nQUESTIONS:\nLine 219: \"a baseline that uses 3 A matrices and B A matrices\" - what does this mean? If it's a typo, please fix.\n\nTable 4: Why is \"Multiple LoRA\" worse? And why is \"MokA w/o CA\" better? What are the major differences? Is sharing or not sharing B matrix the only difference? \n\nSection 2.2.2: So this cross attention mechanism does not introduce any new parameters, and only introduce new computes? If my understanding is correct, this is really worth highlighting (both for the audience to understand, and for emphasizing as a strength of the method).\n\nSection 2.2.2: So Query is from audio/vision, Key is from text, and Value is from text? Does this use any positional encoding?\n\nWhich part of the LLM do we apply LoRA or MokA in the experiments? In another word, what is the \"W\" matrix that you are adapting in the experiments? Did you adapt all the weight matrices?\n\nLIMITATIONS:\nThe cross attention vs causal decoding seems an important limitation here, especially for MLLMs generating multimodal tokens. The paper did not cover or discuss them.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Multimodal low-rank Adaptation (MokA), consisting of unimodal matrix A, task-centric cross-attention, and shared multimodal matrix B. MokA provides a more targeted solution for efficient adaptation of MLLMs instead of borrowing from LLM. Experiments demonstrate that MokA achieves better performance across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Clear motivation.** This paper is motivated by designing an efficient fine-tuning strategy tailored for the multimodal nature of MLLMs. A preliminary study clearly demonstrates the limitations of the existing unimodal PEFT strategies.\n2. **Technical contributions.** This paper innovatively decouples the projection matrices A and B and designs a cross-modal interaction mechanism between them.\n3. **Comprehensive experiments.** This paper conducts experiments across audio-visual-text, visual-text, and speech-text benchmarks, and makes comparisons with a variety of PEFT methods.\n\n### Weaknesses\n\n1. **Lack of discussion regarding hyperparameters.** The rank of low-rank matrices is 4, which is not consistent with the optimal rank 8 reported in the LoRA paper. How was this parameter selected, and why not experiment with the optimal configuration?\n2. **Limited related works discussion.** There are several existing works, like Omni-SMoLA[1] already propose PEFT methods suitable for multimodal learning by assigning tokens from different modalities to different LoRAs. However, this paper only discusses methods that are not designed for multimodal learning.\n\n---\n\n[1] Wu, JIalin, et. al. “Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts”\n\nQUESTIONS:\n1. Are all the methods listed in the table trained under the same settings?\n2. Since tokens from different modalities are processed by separate matrices A, does MokA mitigate catastrophic forgetting of the LLM’s pretrained knowledge? To verify, for example, this is particularly relevant for the MME reasoning subset, which relies heavily on the model’s original language capabilities. Could you report the performance on the MME reasoning subset specifically?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper advances along the LoRA direction, proposing a targeted variant, MokA, for Multimodal-LLM. The variant is motivated by the observation that the update of vanilla LoRA is dominated by text tokens. Accordingly, the author allocates modality-specific LoRA A parameters to decouple their update. Additionally, cross-modal communication is injected within the MokA module.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper has a clear logical flow, which makes it easy to follow.\n- Experiment designs are reasonable and well support the paper's claims\n\nWeaknesses:\n- The proposed method breaks the simplicity of LoRA: the side branch can no longer be merged into the main LLM parameters, and the introduced cross-attention module adds additional complexity conceptually, engineerly, and computationally.\n\nQUESTIONS:\nQuestions:\n\n1. Considering the LoRA baseline, your ablations have already shown that 'Multiple LoRA' does not enhance performance, but I wonder if increasing the rank of LoRA baseline could lead to better results? I mean, if in practice I find my LoRA model not performing well, my first reaction would be to try increasing the hidden rank instead of trying a much more complicated variant.\n2. \"Specifically, we evaluate the model’s performance when only tokens from a selected modality are passed through the LoRA adaptation pathway at the first generation step during inference. \"  Could the author clarify what the highlighted \"at the first generation step during inference\" means?  I guess this means something similar to \"during prompt prefilling\" or \"during building the KV cache\", but I am not sure.\n3. The comparison result between \"Multiple-LoRA\" and \"MokA w/o CA\" is counterintuitive to me. The result means that a shared B matrix outperforms multiple modality-specific B matrices. Could the author provide some analysis/explanation?\n3. Arrows in Fig.4 are messy, and the lines cross arbitrarily. I strongly suggest refining this figure\n5. While the \"Uni LoRA + MM LoRA\" and \"Uni LoRA + MM LoRA + Gate\" baselines are interesting, their relationship with the proposed MokA is not sufficiently explained in the paper. What do the authors intend to verify by introducing these baselines?\n\nOverall, I would be happy to raise my score if some of the above questions can be satisfactorily answered.\n\nLIMITATIONS:\nSee questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MokA, a novel fine-tuning strategy for Multimodal Large Language Models (MLLMs) that addresses the limitations of directly applying LoRA—originally designed for LLMs—to MLLMs. The key insight is that shared LoRA parameters across modalities overly favor text, undermining the adaptation of non-text (e.g., visual, audio, speech) modalities. MokA uses modality-specific low-rank matrices (A) to capture unimodal features and introduces cross-modal attention between text and non-text tokens for explicit interaction, and projects all adapted features using a shared matrix (B) to unify representations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**：\n\n1. The paper is well-written with clear motivation addressing the issue of text-dominance in LoRA-based MLLM adaptation. Additionally, the authors present the proposed algorithm MokA with clear illustrations and rigorous mathematical formulations, making it easy for readers to understand the methodology.\n2. MokA is a logical extension of LoRA with minimal overhead, specifically adding modality-specific A matrices and a lightweight cross-attention module to effectively improve the performance of LoRA fine-tuning for MLLMs. While simple, the approach proves to be quite effective.\n3. The extensive experimental results demonstrate that MokA outperforms multiple strong baselines across diverse benchmarks (MUSIC-AVQA, MMBench, POPE, AIR-Bench, etc.). Importantly, these performance gains are consistent across different LLM backbones.\n\n\n\n**Weakness**：\n\n1. In the experimental section, the paper primarily focuses on performance metrics while overlooking crucial efficiency indicators. The introduction of additional A matrices and the cross-attention module incurs extra computational and training time overhead during fine-tuning. Although MokA demonstrates performance gains, these improvements might simply result from increased training resources. The tradeoff between performance and cost is not deeply analyzed (e.g., FLOPs, wall-clock training time, memory usage).\n2. Regarding Equation 13, the unimodal adaptation component is not entirely equivalent to \"unimodal adaptation\" since matrix B is shared across modalities rather than being modality-specific. Additionally, for the \"cross-modal adaptation\" component, we observe only Text-Visual and Text-Audio interactions, while Audio-Visual interactions are missing. The paper does not discuss whether introducing Audio-Visual interactions would further enhance model performance.\n3. With each modality requiring its corresponding A matrix, the scalability of MokA becomes questionable when dealing with numerous modalities. This concern is particularly evident in Table 5, where even with just a few modalities, there's already a 1.13× increase in inference overhead. This overhead would grow substantially with additional modalities.\n4. The design of the cross-attention module is fixed and simple. More flexible designs (e.g., bi-directional or dynamic attention) are not explored or compared.\n\nQUESTIONS:\n1. The paper should provide a comprehensive efficiency analysis of MokA, including computational overhead, training time, and memory requirements compared to baseline methods. Please elaborate on the trade-offs between performance and cost, and explain whether the performance gain comes from additional training, especially in resource-constrained environments.\n2. The current formulation of \"unimodal adaptation\" in Equation 13 uses a shared B matrix across modalities, which contradicts the true definition of modality-specific adaptation. Additionally, the cross-modal interaction design omits Audio-Visual interactions. The authors should explain this design choice and provide ablation studies on whether incorporating Audio-Visual interactions would further enhance performance.\n3. The authors should address how MokA would perform in scenarios with many modalities (3+) and propose potential solutions to mitigate the growing computational/inference burden in such cases.\n4. The authors should explore and compare alternative attention mechanisms (such as bidirectional attention, dynamic attention weights, or gated attention) and explain why the current design was selected over potentially more powerful alternatives.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}