MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization

Authors: JiangyongYu, Sifan Zhou, Dawei Yang, Shuoyu Li, Shuo Wang, Xing Hu, XUCHEN, Zukang Xu, Changyong Shu, Zhihang Yuan
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodal Large Language Models, Quantization

Abstract:
Recently, multimodal large language models (MLLMs) have garnered widespread attention due to their ability to perceive and understand multimodal signals. However, their large parameter sizes and substantial computational demands severely hinder their practical deployment and application. While quantization is an effective way to reduce model size and inference latency, its application to MLLMs remains underexplored. In this paper, we conduct an in-depth analysis of MLLMs quantization and identify several challenges: slow inference speed of the visual tokens, distributional differences across modalities, and visual outlier clipping degrades performance.
To address these challenges, we propose **MQuant**, a quantization framework tailored for MLLMs. Specifically, 1) we design Modality-specific Quantization (MSQ) and Attention-Invariant Flexible Switching (AIFS) to support per-tensor static quantization and facilitate efficient inference. 2) we introduce a unified LayerNorm-to-RMSNorm transformation, achieving seamless integration of the MLLM vision encoder with Hadamard rotation. 3) we propose Rotation Magnitude Suppression (RMS) to mitigate outliers introduced by Hadamard rotation. Experiments conducted on five mainstream MLLMs demonstrate the superior performance and broad applicability of MQuant. For example, it maintains around 98\% of the floating-point accuracy under the W4A8 setting. To the best of our knowledge, **MQuant** is the first quantization solution for MLLMs, paving the way for future advancements in their application.

Decision: reject
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730284551895

thread_type:
official_review

<thread_object>
creator: reviewer_gy4o

<content>
summary:
This paper proposes MQuant, an accurate and efficient post-training quantization solution for multimodal large language models (MLLMs). MQuant reduces the time to first token (TTFT) with per-tensor static quantization and introduces modalityspecific quantization (MSQ) to handle distribution discrepancies between visual and textual tokens. Experiments on five mainstream MLLMs demonstrate that MQuant attains state-of-the-art PTQ performance.

strengths:
Strength：

1. Extensive experiments demonstrate the approach's effectiveness in the PTQ of MLLMs.
2. The motivation is clear and quantization for MLLM is an important topic.
3. This paper is well-organized and clearly-written.

weaknesses:
Weakness:

1. My only concern is that i'm not familiar with quantization. So i will adjust my rating depending on the other reviewers' opinions.

questions:
Please see the comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704991399

thread_type:
official_review

<thread_object>
creator: reviewer_ftkm

<content>
summary:
This paper proposes a quantization method which is specifically tailored towards MLLM. Because of the distributional differences between visual tokens and text tokens, the authors intuitively calculate separate quantization scales for two modalities and calibrate the attention mask accordingly. Further, they adapt some techniques from the LLM quantization literature to visual encoders in MLLM. By combining these two, MQuant maintains lower performance degradation under challenging quantization settings on multiple state-of-the-art retrained MLLM models.

strengths:
1. The paper follows an intuitive approach to study MLLM quantization. The authors identify the issues based on some observations in the experiments and resolve the problem in a step-by-step manner.
2. The efficacy of the method is supported by extensive experiments.  The paper shows the quantization performance of 5 mainstream MLLM models on various multi-modal tasks. The ablation studies demonstrate the usefulness of different components in maintaining the performance near the float-point baseline.

weaknesses:
1. The delivery of the paper needs significant improvement. The text is highly redundant. 
- Introduction: The content of the second last paragraph mostly overlap the main contribution part. It could be beneficial if these two parts are reorganized or condensed.
- Methodology: In 4.1, there are abundant words to explain the reason why we need MSQ and AIFS and the benefits brought by these two. To me, these are intuitive and simple operations which only need concise words for explanation. For 4.2 and 4.3, which are the techniques adapted from LLM quantization, it would be better if the authors could emphasize their novel improvements or adaptations rather than putting too many words to explain other people's contributions. 
- Although using separate figures for different components are informative, it could be easier for the readers to follow without reading the algorithm 1 in Appendix first if the authors could add a figure to show the overall quantization pipeline with the novel parts highlighted. 
- For some abbreviations used in the paper, like GEDD and W4A8, it would be friendly to readers not in the area if adding the explanations in the first place.

2. The paper does not demonstrate enough novelty.  First, both LayerNorm-to-RMSNorm transformation and Hadamard rotation are borrowed from LLM quantization literature (Ashkboos et al., 2024a, b). Second, although adopting a simple Divide-and-Conquer strategy like paper does to cope with the distribution outliers or differences may be sufficient, it is worth thinking about other systematic alternatives after getting more insights from the observations in the experiments. For now, the paper is more like a technical report. The paper should be concise and highlight the actual novel contributions.

3. Experiments: It would be better to see the latency comparisons among the proposed quantization methods could be added in Table 5. 

4. Minor Errors:

- The font size of the legend in Figure 1 (left side) is too small to read.
- Line 85-87: the meaning of the sentence Is not clear. Two "slightly" exist.
- For Table 3/4. the arrow directions showing the relative difference are counter-intuitive. Showing the decrease of latency with down arrows and adding "lower is better" could be an alternative.
- In Table 5, should that be "MSQ" rather than "MDQ"?

questions:
1. In Eq (6), should the denominator of equation $s$ be $2^b-1$? since for b-bit, the value range would be (0,  $2^b-1$).
2. In line 321, "easier to quantize". What does easy mean in this context?
3. In line 287, what do the "outliers" mean? Extremely low or high values?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710523630

thread_type:
official_review

<thread_object>
creator: reviewer_xqur

<content>
summary:
This paper introduces several techniques to enhance the accuracy and reduce the inference latency of Multimodal Large Language Models (MLLMs), which are affected by the additional vision encoder/adaptor. Empirical results demonstrate that the quantized model obtained using the proposed method outperforms other quantization methods in terms of accuracy and inference speed under certain settings.

strengths:
1. The paper is well-written and easy to follow.
2. The modality-specific quantization and Layernorm-to-RMSNorm transformation are well-motivated by the distributional differences of various modality modules and architectural designs.
3. Comprehensive experimental results are provided on various MLLMs, with comparisons to several popular recent LLM quantization methods.

weaknesses:
1. Attention-Invariant Flexible Switching (AIFS) Scheme: The authors claim that the proposed AIFS scheme is computationally equivalent to the original attention computation. However, it is unclear whether the corresponding positional embeddings are adjusted accordingly. If not, the equivalence may not be ensured.

2. Experiment Settings: There are concerns regarding the experimental settings. In Section 5.1, the authors conducted experiments under the "text-image-text" setting with 15 textual tokens. However, inference settings can be more complex:
- In a batch, the number of textual tokens varies, resulting in different attention masks after AIFS.
- There can be interleaved image-text inference with more image-text turns.
- There can also be multi-image inference with single or multiple turns.
More clarifications under these cases are required to further show the efficacy of the proposed method.

questions:
1. For the proposed AIFS scheme, are the positional embeddings adjusted accordingly as the attention mask changes?
2. What batch sizes were used when evaluating the inference latency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721168465

thread_type:
official_review

<thread_object>
creator: reviewer_zdjf

<content>
summary:
This paper studies the quantization problem in Multi-modal LLMs. Specifically, the authors investigate three aspects that lead to performance degradation when applying the straightforward per-tensor static quantization for prefilling multimodal tokens. To address these challenges, this paper presents MQuant with Modality-specific Quantization (MSQ), Attention-Invariant Flexible Switching (AIFS), LayerNorm-to-RMSNorm transformation and Rotation Magnitude Suppression (RMS).

strengths:
1. This paper focuses on a valuable question, i.e. quantization in MLLMs.
2. Well presented with figures and tables.
3. Overall performance is superior to some LLM quantization baselines.

weaknesses:
1. MSQ and AIFS are simply trivial adaptions of per-token dynamic quantization to MLLMs. It's better that this serves as a baseline model.
2. MSQ and MSQ + AIFS exhibit marginal improvement over the per-tensor static baseline in Table 4.
3. Please discuss the overhead of MSQ, otherwise why don't we use token-specific quantization?
4. Although MSQ + AIFS is proposed to address the token increase brought by larger resolution of images, the speedup fails to exhibit great advantages over per-token dynamic baseline with resolution scaling.
5. SliceGPT [1] has already proposed converting LayerNorm to RMSNorm and provides a solution, which you do not mention in the related work. Please discuss the difference between your method in Section 4.2 and the one in SliceGPT.
6. Lack of sufficient technical contribution. Most of the techniques used are from previous work and adapt to MLLM with trivial modifications.
7. Typos. e.g. whthin in line 304 and grammatic errors, e.g. 305 (should be "to show how to transform xxx")

[1] Ashkboos, Saleh, et al. "Slicegpt: Compress large language models by deleting rows and columns." arXiv preprint arXiv:2401.15024 (2024).

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732193358242

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response1: Background of related quantization techniques

comment:
**In per-tensor static quantization, the quantization parameters (i.e., scale and zero-point) are precomputed for an entire tensor (e.g., weights or activations) and remain fixed throughout inference**. While efficient, this approach often leads to large and unacceptable accuracy loss in MLLMs due to their diverse activation distributions across varying inputs.

**In contrast, per-token dynamic quantization computes quantization parameters on-the-fly for each input token during inference**. This approach incurs significantly higher computational overhead, as the quantization parameters must be recalculated for every input token, along with multiple additional memory traversals.  Such requirements make per-token dynamic quantization unfriendly or impractical for edge devices and some AI accelerators, which struggle with fine-grained dynamic operations [1].  This issue is especially severe in MLLMs, where the token count increases significantly with higher image resolution or more video frames.

**Our MQuant is a novel per-modality quantization approach specifically designed to address the unique challenges of MLLMs quantization**.  MQuant achieves the same efficiency as per-tensor static quantization while maintaining near-lossless accuracy comparable to the original FP32 model.

[1]. MobileQuant: Mobile-friendly Quantization for On-device Language Models, EMNLP 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1732203513133

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response and revision update notice

comment:
Dear Reviewers,

We thank you for your  precious reviews. The paper has been revised according to your suggestions. We've carefully examined all the questions and provided answers. Please feel free to discuss with us if any new questions arise. All changes in manuscript are marked with blue.

* Revised the second-to-last paragraph and summarized the main contributions to enhance clarity (suggested by Reviewer FtkM) (Section 1).
* Revised description of Equation (6), and GEMM and W4A8 to eliminate ambiguity (suggested by Reviewer FtkM) (Section 3).
* Added a detailed explanation of the positional embeddings transformations in AIFS. (suggested by Reviewer xqur) (Section 4.1).
* Added differentiation from SliceGPT and made our contribution clearer (suggested by Reviewer zDjf) (Section 4.2).
* Revised the experimental description (suggested by Reviewer xqur) (Section 5.1).
* Added latency measurements in Table 5 for clarity (suggested by Reviewer FtkM) (Section 5.2).
* Added detailed description of position embedding in AIFS and clarified Figure 1 in the Supplementary Materials (suggested by Reviewer xqur).

Sincerely,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732711026362

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Sincerely looking forward to your more feedbacks

comment:
Dear Area Chairs and Reviewers,

We sincerely appreciate everyone’s efforts put into the reviewing process, which has significantly contributed to the refinement of our manuscript. We've carefully examined all the constructive questions and have accordingly revised our work. To make the best use of the discussion period and to improve our work, we are eager to know whether our answers well address your concerns, as it is crucial for us to have a candid and thorough discussion to continuously strengthen our method. Please share your thoughts on viewing our reply. We hope to resolve your doubts with our best efforts.

We are ready to respond to any further issues raised. Please let us informed.

Sincerely,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732852385420

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response and further revision update notice

comment:
Dear Reviewers,

We thank you again for your valuable reviews. The paper has been **further revised according to your suggestions**. We've carefully examined all the questions and provided answers. Please feel free to discuss with us if any new questions arise. All changes in the manuscript are marked in blue.

- Revised the discussion about SliceGPT in **Related Work**, specifically discussing the methodological differences (suggested by Reviewer zDjf) (Section 2.2).
- Revised the advantages of MSQ+AIFS and moved them to the corresponding experimental sections to enhance solidity and clarity (suggested by Reviewer FtkM) (Sections 4.1, 5.1, 5.2).
- Revised and streamlined the description of Post-LN+Rotate to highlight our core contribution (suggested by Reviewer FtkM) (Section 4.2).
- Revised the description of RMS to reduce redundant text while adding theoretical support and analysis (suggested by Reviewer FtkM) (Section 4.3).
- Updated Table 4 and added accuracy and speed comparisons for clarity (suggested by Reviewer zDjf) (Section 5.2).
- Added speedup of the prefill and decode stages for MSQ+AIFS in the Appendix to highlight the substantial acceleration effects (suggested by Reviewer zDjf) (Appendix A.2).
- Added algorithm and generalization experiments of RMS on LLMs in the Appendix to demonstrate their effectiveness (suggested by Reviewer FtkM) (Appendices A.7, A.8).
- Added a detailed schematic of the Post-LN+Rotate scheme in the Appendix to highlight the differences with SliceGPT (suggested by Reviewers zDjf and FtkM) (Appendix A.14).

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary of Scientific Claims and Findings**  
The paper introduces MQuant, a post-training quantization (PTQ) framework tailored for multimodal large language models (MLLMs). The proposed method addresses challenges such as distributional discrepancies between visual and textual modalities, inference latency due to visual tokens, and performance degradation from visual outlier clipping. The authors propose techniques such as Modality-Specific Quantization (MSQ), Attention-Invariant Flexible Switching (AIFS), LayerNorm-to-RMSNorm transformation, and Rotation Magnitude Suppression (RMS), claiming improvements in both accuracy and speed.

**Strengths**  
- The paper tackles an important and underexplored problem of MLLM quantization.  
- The methodology is supported by extensive experiments across multiple mainstream MLLMs.  
- The authors provided detailed rebuttals and additional experiments to clarify questions raised during the review process.  

**Weaknesses**  
1. **Limited Novelty**:  
   - Several proposed methods, such as the LayerNorm-to-RMSNorm transformation and the use of Hadamard rotation, are adaptations of existing techniques from LLM quantization literature (e.g., SliceGPT and other prior works). The novelty of the contributions is incremental rather than groundbreaking.  
   - The core contribution of MSQ and AIFS, while specific to MLLMs, is primarily a direct application of existing concepts like per-tensor static quantization and data reordering, raising concerns about the lack of fundamental innovation.

2. **Lack of Generalization**:  
   - While the authors added experiments during rebuttal to address multi-turn and multi-batch inference, these setups are still limited in scope. Broader and more diverse use cases, such as varying batch sizes or more complex sequences in real-world scenarios, are not fully addressed.

3. **Writing and Organization**:  
   - Initial drafts of the paper were verbose and poorly structured, devoting excessive focus to prior methods rather than highlighting the novelty of the authors' contributions. Despite revisions, the paper still lacks conciseness and clarity in some sections.  

4. **Marginal Improvements**:  
   - Performance gains over baselines are relatively minor in key metrics, particularly in Table 4, where speed improvements of MSQ+AIFS are comparable to per-tensor static quantization but fail to showcase clear advantages in practical scenarios.  

**Decision Rationale**  
While the problem addressed is important and the work demonstrates technical competence, the paper’s contributions are incremental and build heavily on prior work without providing sufficient innovation. The practical significance of the improvements is also limited, with minor gains in performance that do not strongly justify the proposed methods. Moreover, concerns about generalizability and the lack of a significant leap in methodology weigh against acceptance.