Title: QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision language model

singular value decomposition

quantization
Abstract: Vision-Language Models (VLMs) are integral to tasks such as image captioning and visual question answering, but their high computational cost, driven by large memory footprints and processing time, limits their scalability and real-time applicability. In this work, we propose leveraging Singular-Value Decomposition (SVD) over the joint query (Q), key (K), and value (V) weight matrices to reduce KV cache size and computational overhead. We in addition introduce an efficient rank allocation strategy that dynamically adjusts the SVD rank based on its impact on VLM accuracy, achieving a significant reduction in both memory usage and computational cost. Finally, we extend this approach by applying quantization to both VLM weights and activations, resulting in a highly efficient VLM. Our method outperforms previous approaches that rely solely on quantization or SVD by achieving more than $10$% accuracy improvement while consuming less hardware cost, making it better for real-time deployment on resource-constrained devices. We open source our code at https://github.com/SAI-Lab-NYU/QSVD.


Peer reviews:
comments: SUMMARY:
This paper proposes an efficient method for compressing Vision-Language Models (VLMs) by applying Singular Value Decomposition (SVD) to the QKV weight matrices, reducing KV cache size and computation. A dynamic rank allocation strategy is introduced to balance compression and accuracy. The approach is further enhanced with quantization of both weights and activations. Experiments show that the method achieves better accuracy–efficiency trade-offs than prior work, making it suitable for real-time, resource-constrained settings.

STRENGTHS AND WEAKNESSES:
Strengths:

1.	The paper is well-structured and easy to follow.
2.	The experiments and ablations are thorough and clearly demonstrate the effectiveness of the proposed method.

Weaknesses:

1.	My main concern is that the method appears to lack VLM-specific design—it primarily targets the QKV weights, which is a general approach applicable to LLMs as well. This raises the question: why is the method not directly evaluated on standard LLM benchmarks and compared against existing LLM compression techniques, if it's not tailored to vision-language settings?
2.	The paper lacks comparisons with state-of-the-art post-training quantization methods in terms of inference time, memory usage, and accuracy.
3.	The figures need to be more refined. For example, Figure 2 alone is hard to interpret—the connections and meanings of the different subplots are unclear.

QUESTIONS:
See Weakness

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
This paper proposes QSVD, a framework that leverages joint Singular Value Decomposition (SVD) of QKV weight matrices and quantization to reduce computational overhead and KV cache size in Vision-Language Models (VLMs). Key innovations include:  
1. Joint QKV-SVD: Concatenating Q/K/V weight matrices for unified low-rank decomposition, reducing parameters and computation

2. Dynamic Rank Allocation: Singular value truncation based on impact on VLM accuracy (Eq. 10).  



3. Low-Rank Activation Quantization: Orthogonal transformations to suppress activation outliers (Eq. 11-12).  

Experiments show QSVD outperforms baselines (ASVD, SVD-LLM) on LLaVA and SmolVLM, reducing KV cache to 9.38% in W8A4 quantization with <1% accuracy drop on ScienceQA.

STRENGTHS AND WEAKNESSES:
**Strengths:**  
- Efficiency: Joint SVD+quantization reduces KV cache (η<sub>qsvd</sub> = *rL* vs. η<sub>fp</sub> = *2LE*) and computation (FLOPs ↓ when *r* < 0.75*E*).
- Novel Ranking Strategy: Global singular value importance scoring (Eq. 10) outperforms uniform/Fisher-based allocation (Table 3).  
- Strong Validation: Tested on 5 VLMs and 3 benchmarks with W4A4 robustness (Table 2).  

**Weaknesses:**  
- Attention-Layer Limitation: No optimization for FFN layers (Sec. 5).

QUESTIONS:
1. Does the O(E²) cost of Eq. 10 affect real-time performance?  

2. How does r selection adapt to long sequences (e.g., high-res images)?  

3. Could visual-textual feature misalignment hinder joint SVD?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
This paper introduces a QSVD method that combines quantization and SVD. It cocnats QKV weight matrix together and performs a low-rank SVD on it. Then it introduces a cross‑layer, importance‑based singular‑value ranking scheme for adaptive rank allocation. This method can reduce the GPU memory usage and computation cost, and experiments on five VLMs across ScienceQA, VizWiz, and SEED‑Bench demonstrate that QSVD outperforms SVD‑only and quantization‑only baselines.

STRENGTHS AND WEAKNESSES:
1. The joint QKV SVD formulation is clearly derived, with precise comparisons against single matrix SVD and analysis of parameter/FLOPs/cache reductions. The results support the method proposed.
2. The figures are intuitive. A small drawback is that the order of the sub-graphs is inconsistent with the order in the text. 2. The parameter beta is only used in Quantization, so it’s confusing in Section 3.1 without explanation.
3. QSVD works on different models and datasets, but why do we need to concatenate the three matrices together? Is it just to reduce cache overhead? Or is there a deeper reason for this design? A deeper analysis is encouraged. 
4. While prior work apply SVD to Q, K, V separately, this paper’s joint factorization over the concatenated matrix is novel (despite the lack of a deeper motivation)   
5. The knockout approach to measuring the importance of singular values is novel.

QUESTIONS:
1. Analyze why we can get better results from SVD on concated QKV. Either theory or experiment is encouraged.
2. What is the beta obtained on the calibration dataset? Can authors study the beta values on different datasets? This ablation is meaningful for practical applications.
3. Experiment on GPU memory usage between different methods is encouraged.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper proposes QSVD, a unified framework for compressing Vision-Language Models by applying joint Singular Value Decomposition to the combined query, key, and value weight matrices, alongside post-training quantization of weights and activations. The goal is to reduce memory, computational cost, and KV cache size without degrading model accuracy.

STRENGTHS AND WEAKNESSES:
Strengths:
- This paper proposes a novel compression paradigm that effectively combines SVD and PTQ.
- The buffered mechanism and auto-regressive decoding work well together.
- The figures well demonstrate the method design.
- Extensive experiments on popular VLMs show superior performance over existing works.

Weaknesses:
- The method's effectiveness is quite sensitive to $\beta$.  And $\beta$ is determined by a very small calibration set. I'm not sure if the generalization ability of such pattern is enough. 
- The method can only be used on QKV matrices and can not be used on other layers like FC layers, restricting its compression capacity.

QUESTIONS:
- Can we replace post-training quantization by quantization-aware training and further improve the performance?
- What about the training time? Does QSVD add heavy additional cost compared with normal training?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03