QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models
Authors: Yutong Wang
Haiyu Wang
Sai Qian Zhang
Venue: NeurIPS 2025 spotlight ()
Keywords: Vision language model
singular value decomposition
quantization
Primary area: deep_learning

Abstract: Vision-Language Models (VLMs) are integral to tasks such as image captioning and visual question answering, but their high computational cost, driven by large memory footprints and processing time, limits their scalability and real-time applicability. In this work, we propose leveraging Singular-Value Decomposition (SVD) over the joint query (Q), key (K), and value (V) weight matrices to reduce KV cache size and computational overhead. We in addition introduce an efficient rank allocation strategy that dynamically adjusts the SVD rank based on its impact on VLM accuracy, achieving a significant reduction in both memory usage and computational cost. Finally, we extend this approach by applying quantization to both VLM weights and activations, resulting in a highly efficient VLM. Our method outperforms previous approaches that rely solely on quantization or SVD by achieving more than $10$% accuracy improvement while consuming less hardware cost, making it better for real-time deployment on resource-constrained devices. We open source our code at https://github.com/SAI-Lab-NYU/QSVD.

QSVD: Efficient Low-rank Approximation for Unified Query-Key-Vale Weight Compression in Low-Precision Vision-Language Models
Yutong Wang Haiyu Wang Sai Qian Zhang 1.2 "Tandon School Engineering, New York University CCurantt Institute of Mathematical Sciences, New York University Cyw6594, hw3689, sai. znanan
Abstract
Vision-Language Models (VLMs) integral to tasks such as image captioning and visual question answering, but their high computational cost, driven by large memory footprints and processing time, limits their scalability and real-time ap- plicability. In this work, we propose leveraging Singular- Value Decomposition (SVD) over the joint query (Q), key (K), and value (V) weight matrices to reduce KV cache size and computational overhead. We in addition introduce an efficient rank allocation strategy that dynamically adjusts the SVD rank based on its impact on VLM accuracy, achieving a significant reduction in both memory usage and computational cost. Finally, extend this approach by applying quantization to both VLM weights and activations, resulting in a highly efficient VLM. Our method outperforms previous approaches that rely solely on quantization or SVD by achieving more than 10% accuracy improvement while consuming less hardware cost, making it better for real-time deployment on constrie devices. We open source our code at https:: //g thub. com -Aa- LLUU-LLD--UUUQVV
Vinuular-Value
Introduction
Vision-Language Models (VLMs) crucial for advancing artificial intelligence by bridging the gap
between visual perception and natural language understanding. By enabling machines to interpret and generate both visual and textual information, VLMMs open up wide range of applications, such as
image captioning [63] I7 8 [I] visual question answering [7] 3 47), and content- search [22]
43 These models are vital for tasks where visual context is needed to fully understand textual queries
or vice versa, such as healthcare [34] 3 18), education [61]. and interactive entertainment [44] 26).
Despite their strong performance, Vision-Language Models (VLMs) incur substantial computational costs due to the intensive processing required to integrate high- high-dimensional visual and textual data. Additionally, their autoregressive token generation places significant pressure memory bandwidth, becoming major bottleneck for inference speed. To enable practical deployment in latency-sensitive and resource-constrained environments, it is essential to reduce both the computational overhead and the size of the Kalue (KV) cache, without compromising model accuracy.
To address this issue, particularly the high memory usage introduced by Multi-Head Attention (MHA), several variants have been proposed, such as Grouped-Quee Attention ① and Multi-Query Attention 42 DI which aim to reduce the number of KV projections while maintaining performance. recent proposal, Multi-Head Latent Attention (MLA) in the DeepSeek-v3 model [3], offers a novel approach to improving VLM efficiency. It significantly reduces the KV cache size by compressing the KV cache into a latent vector, thereby enhancing inference efficiency.
Authors contributed equally; the order of authorship was assigned randomly.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

*An innge of two go Lden retrievers"
Callbration dataset
Stop f: Joint SVD over QKV Weight
QSVD
Q: What the cuslomee reciety
Step Post-trainin Quantization
Language
umage
Q: What brand the
Projecton
Step Adaptive alocation for SVD
Outller smoothing
Embedding
Visual
Encoder
Embedding
Visual encoder
Text
'Describe the image'
Image
Projection
Quantiationn
Language model
Loss
Token Concatenation
(a)
(b)
Figure 1: (a) An example vision-language model. (b) An overview of QSVD.
Building on the insights from MLA, this work proposes the application of Singular- Value Decompo- sition (SVD), which has proven effective in reducing both KV cache size and computational cost in prior research [51 60 52
5 49], to the joint weight matrices of the query, key, and value. This approach significantly reduces the KV cache size by storing only the latent vectors instead of separate key and value vectors. Additionally, we introduce a novel rank allocation scheme, which investigates the importance of each singular value in relation to VLM accuracy. This results in a minimal SVD rank with the minimal impact on the model accuracy. Finally, we extend this approach by applying quantization to both VLM weights and activations. The proposed framework, termed QSVD, results in extremely efficient VLM that outperforms previous methods that relied solely on quantization or SVD. Specifically, our contribution can be summarized as follows:
28
25
QSVD proposes applying singular value decomposition to the combined weight matrices of the query, key, and value projections. This technique substantially reduces the size of the KV cache, computational overhead, and weight storage, resulting in significant improvements in hardware efficiency.
To improve the accuracy of SVD-based compression in VLMs, QSVD proposes novel importance scoring method that quantifies each singular value's contribution to overall model performance, allowing for rank-based truncation that minimizes accuracy degradation.
Quantization is applied alongside SVD decomposition to both the weights and activations of the VLM. We propose an efficient method eliminate outliers under the SVD framework, enabling low- oowpprccision operation that reduces the memory footprint of both the KV cache and model weights, while incurring minimal impact on accuracy
Background and Related Work
2.1 Vision Language Model
extend the capabilities of Large Language Models (LLMss) by incorporating visual inputs alongside text, enabling tasks such as visual question answering (VQA) and image captioning. Models such as BLIPP and InstructBIIIP 24 23 employ data filtering and visual instruction tuning to better align model outputs with human preferences in zero-shot settings. A commonly used architecture, illustrated in Figure (a), processes input images through a visual encoder to produce visual tokens, which are then concatenated with tokens and passed to a language model for response generation. This concatenation- based design is adopted by widely used models including the LLaVA series [3]. SmoIVLM [39], PaLI-Gemma 4 and Qwen- VL [48). Although VLMs demonstrate impressive capabilities, their large size presents challenges in terms of computational efficiency and deployment, particularly on resource-constrained devices. This has led to the development of lightweight alternatives. TinyGPT-V [59] and L.nyLLaVA (2) explore efficient designs at smaller scales, while SmoIVLM 39 introduces a family of compact models (500M and 2B parameters) that maintain strong performance with much reduced hardware cost.
Vision-Language Models (VLMs) 24
23 33
4
13
48
All experiments and data processing conducted New York University.

---

2.2 Singular Value Decomposition for Large Models
Singular Value Decomposition (SVD) 19 is a widely used matrix factorization technique that decomposes matrix W E R into three components: W = UEVE, where U and V are orthogonal matrices containing the left and right singular vectors of W. and Σ is diagonal matrix of non-negative singular values arranged in descending order. By retaining only the top singular values and corresponding vectors, we obtain a rank-r approximation:
W W R U.EVV
with U, E R Σ, E R and V, E R Equivalently, the approximation can be expressed as W x AB by defining A= U, B = EV and B = Σ V. Such low-rank factorizations pre- serve the most salient structure of W while reducing its dimensionality, enabling matrix compression and accelerating downstream computations.
SVD has been extensively studied as compression method for LLMs 51 60
52
28
49) Early
efforts 40 directly applies standard SVD to weight matrices but encountered significant compression
errors. To address this, FWSVD [16] prioritizes parameters based on Fisher information (37. while ASVD [60] incorporates activation outliers into the factorization. SVD-LLM [52) further reduces compression loss by explicitly minimizing the contribution of each truncated singular value. Most of these methods focus on compressing model weights. In contrast, Palu [6] and [58] have shifted attention compressing the KV-Cache. leveraging SVD and low-rank projections to reduce memory footprint. Recent advances include AdaS (28), which adaptively compensates for truncation errors and dynamically allocates compression rates according to layer importance, and SVD-LLM V2 [5]] which further optimizes singular value truncation via theoretical loss estimation.
Recently, DeepSeek introduces Multi-Heat Latent Attention [31, a novel mechanism that integrates low-rank projections directly into the attention computation. Instead of computing attention over the full key and value matrices, this approach projects them into lower-dimensional latent space using learned projection matrices, effectively reducing the computational and memory costs multi-head attention without significantly impacting model performance. This latent factorization can be viewed as an implicit low-rank approximation applied dynamically during inference, offering complementary benefits static weight compression methods such SVD.
2.3 Quantization for Large Models
Post-training quantization (PTQ) has become one of the most used approaches for enabling efficient inference of large models [38 [55 2 29 [2] 41 30 57 45 53 20, 9. For example, AffineQuant [38]
replaces the traditional scaling factor with learned affine transformation to better align weight with the quantization grid.
Another line of work focuses on smoothing outliers in activation distributions, which have shown that activations in LLMs contain severe outliers at the per-channel level [2] 29 35 30 (55) resulting in
substantial quantization errors during activation quantization. To address this issue, SmoothQuant (55)
reduces activation outliers by shifting part of the activation outliers into the weights, promoting more balanced quantization. Building upon these ideas, techniques like QuaRot (2). DuQuant 229] and SpinQuant (35] incorporate orthogonal transformations to further enhance quantization performance. These transformations maintain computational invariance by preserving the model's output while effectively suppressing outliers. Specifically, let W and X represent the weight and activation matrices, respectively, where X exhibits channelwise outliers, and let Y = XW denote the output. To eliminate outliers in X, an orthogonal matrix H is introduced, satisfying H H HH = T. This transformation yields an equivalent formulation Y = XW = X'W', where W" = H W and X' XH. To minimize runtime overhead, W H W can be precomputed offline, and X' = XH can be efficiently integrated into the weight matrices of the previous layer, incurring no additional computational cost. The resulting transformed activation X' exhibits smoother distribution with significantly fewer outliers, thus lowering the quantization errors. In parallel, methods such GPTQ [12), OmniQuant [1], and AWQ [30] focus on optimizing scaling factors and channel wise equalization during the calibration process.
In the realm of VIMMs, quantization presents unique challenges due to the integration of visual and textual modalities. QSLAW (56) introduces quantization-aware scale learning method with multimodal warmup strategy that progressively incorporates linguistic and multimodal samples
3

---

SE
Wn
We
W
We
Wi.
We
Wennoet
E
SE
We
Cx
PW
Wk
Wx
W2
WW
W
X
Σ
W"
X
X
X
X
X
WS
3E
E W
EW
We
W.
=
W:
Kr
W!
E
Wijes
Xr
[W
We
wg
(a)
(b)
(c)
(d)
(e)
(f)
Figure 2: Efficiency analysis of different SVD schemes. (a)(b) are original Q/K/V matrix applied
SVD. (c) are proposed concatenated QKV SVD and their corresponding computing process.
stabilize training. It also emphasizes group-wise scaling to better handle activation outliers. Q- VLM [6) addresses cross-layer dependency in quantization by leveraging activation entropy as proxy to guide block partitioning. It formulates quantization strategy that balances discretization error and search cost, and further optimizes the visual encoder to disentangle cross-layer interactions, enabling more efficient calibration. MBQ (27) proposes a modality- bbllnced quantization approach that accounts for the distinct gradient distributions of visual and textual tokens during calibration. applies a modality-aware loss to improve the accuracy of scaling factor estimation. However, to the best of our knowledge, prior work has combined quantization with SVD for efficient VLM processing in the manner proposed by QSVD.
Methodology
An overview of QSVD is shown in Figure T [bb, comprising three key components; joint SVD over the combined QKV V weights (Section 3.1) adaptive singular value truncation (Section 3.2). and PTQ
over low-rank VLMs (Section) 3.3
3.1 Singular- Singular-Value Value Decomposition over Joint QKV Weights
We introduce efficient SVD-based approach reduce computation within the multi-head self- attention block, as illustrated in Figure 2, where each subfigure denotes: (a) Original QKV matrix in vision-language model (VLM) without SVD. (b) Applying SVD separately the weight matrices of Q, K, and V, where each of W,, WA. and W. factorized into a down- and up-projection pair, (c) Our proposed approach: concatenating QKV weights before applying SVD. (d) Standard KV computation during prefilling: the input X multiplied by W, and W.. and the resulting C2 and C₂ are stored in memory. (e) Computation with per-matrix SVD: during prefilling, X must be read from memory and multiplied with the down-projection matrices W and W to generate low-rank representation of K and V. (f) Storage and computation in QSVD: since W Wi, and W, share common down-projection matrix Waku X is multiplied by gko once to produce the intermediate Cyku: which is stored and later used reconstruct the KV vectors. Let o. 1). and denote the weight parameter size, KV cache size, and the computational cost in FLOPs of QKV multiplication, respectively. In the original design, assuming single-head attention module for simplicity, the combined weight matrices W,, W1. and W, collectively contain a total of a fr 3E" parameters, where E represents the embedding dimension (Figure (a)). The corresponding KV cache requires
a memory footprint of 75p 2LE, where L is the input sequence length (Figure 2 (d)). The total
computational cost in FLOOPs for generating the key, query, and value vectors is 7 3LE where three matrix multiplications are required, each with size of (L x E) and (E E).
QSVD adopts a more efficient strategy that reduces the number of weight parameters, KV cache size, and overall computational cost. Specifically, the weight matrices Wq. WK, and Wy, each of size E x E, are concatenated into a single matrix W Waomaal& E Ex3E A low-rank SVD is then applied to this combined matrix to achieve compression.
[W We. W.] = W Weoneat& W:d x E, x W.
(2)
Wdd = W Wak [W WH. W = A
(3)
[W,, We, W.] Wid skv x [W"
" WT. W
(4)
where Wd gku E Exr WH, WE, W E R and 3 satisfies 0 S 3 < 1. After decomposition,
the QKV components share common down-projection matrix W'd gkes while each maintains its

---

own distinct up- op-projection matrix. This results in total weight size of ageed = Er (Figure (c)). The computational cost for generating the query, key, and value vectors is Vasond 4Lr which arises from two steps: first, multiplying the input X with W to generate Cykus and then performing a second multiplication with the concatenated matrices [W",, W W During inference, the intermediate products Cykeo between the input and the down-projection matrix W gku are buffered to compute the KV vectors, yielding a total buffer size of ngaod = r (Figure (f)), and the KV vectors can be easily recomputed
K = C.keoWk V= Cqku
(5)
In comparison, our method achieves reduced weight size and computational cost when 4rE < 3F and 4Tr < 3LF², which holds when r < 0.... This condition is easily satisfied with negligible accuracy loss, demonstrated by the evaluation results in Section Furthermore, our method consistently reduces the buffered size for the intermediate data, since r is always smaller than 25 given that r < E. During decoding, the cached intermediate representation Coke is used to reconstruct the key and value matrices via W and W which are then combined with the current query (sequence length 1=1) to compute the attention outputs.
Previous methods 60 52) apply SVD individually to the weight matrices, as illustrated in Figure[2]b resulting in total of Cind 6r E parameters. During inference, the intermediate products C and Cz, which is computed from the input X and the down-projection matrices W and W must be buffered, leading to total buffer size of Aind 2r (Figure (()).. The computational cost for generating the query, key, and value vectors is "iind = 6Tr and the buffer size for C& and C, is consistently larger than that required to store the unified Cokeu in our method. Finally, our method always achieves lower weight size, computational cost and intermediate storage.
3.2 Cross-layer Rank Allocation for Low-rank SVD
Performing SVD on the joint QKV weights can lead to hardware efficiency gains in both computation and storage, provided that the rank r of W is sufficiently reduced without compromising the final accuracy performance. A key challenge, therefore, is determining how to truncate the singular values across all self-attention blocks in the VLM. While prior work has used Fisher information (37 to assess the importance of individual weight matrix or a group of singular values (6] [6]. QSVD proposes more effective method that evaluates the importance of each singular value in a way that minimizes degradation in model accuracy.
Given the SVD of a weight matrix W = UUVT, it can also be expressed as a summation: W Ei u.o,"," where o, is the i-th singular value, and u, U; are the corresponding left and right singular vectors. Truncating singular value by setting a; 0 effectively removes its associated single-rank component from the matrix, resulting in a modified representation W', of W, we have:
AW. = W - W = n,,eo
(6)
The truncation of a will affect the final output of the VLM and lead an increase in the training loss L. The corresponding change in training loss can be estimated through first-order expansion:
aL.
L. (.W,,) = (W - AW..) R L.WW)
> AwW,, J,k
U.A]
DW J. K
where (.W,,)) denotes the training loss after the weight matrix is modified to W W., [j, k) represents the (j. k--tth element of the matrix Wo,, Let Gw represent the gradient of the loss with respect to the original weight matrix W, the changes on the training loss can be expressed as follows:
AL. L.W) - L.W.) ~ Σ Jaw,, Li k] G. = A.W,,Gwi)
(8)
J.k
where (F denotes the Frobenius inner product over matrix elements. This formulation enables estimation of each singular value's contribution (e.g., o;) to the change in training loss, providing a principled basis for rank selection by measuring the sensitivity of the loss function to each truncated component across all layers, which can be used evaluate the importance for each singular value. Specifically, by evaluating ALg. across multiple calibration samples and computing its squared
expectation, we derive the Importance Score lo; for each singular value o,, which serves as an
5

---

empirical approximation of the diagonal Fisher information:
f., EEED ( 27
Σ
Σ 3.k
Lww,LiA G.
(9)
N
where D denotes the calibration dataset, indexes individual samples, and N is the total number of samples in D. However, computing the importance score as defined Equation [ poses a significant memory burden, primarily due to the need to construct and store AW., [5. k] for all singular values. Since each A AW, is a full x E matrix and there are E such singular values from the joint SVD, the total memory cost scales as OEE") per layer, making naive computation impractical for large models. To address this, the importance score Lo, can alternatively be computed follows:
1
Σ of U
Glv) v]
(10)
(4.8)
where U and V are the left and right singular vectors from the SVD, or is the i-th singular value. The notation (i, refers to the i-th diagonal element of the transformed gradient matrix U G V. The proof is given in Appendix A..I] This form eliminates the need to compute and store AWa, for each singular value, requiring only (EE²) memory instead of (EE3) per layer.
The overall SVD procedure is as follows. Starting with the original model, we first concatenate the QKV weight matrices and apply joint SVD, following the method outlined in Section 3.1 We adopt the activation-awes awwre SVD technique from ASVD 600 to extract the singular values across all layers. For each singular value, we then compute its corresponding importance score based on the calibration dataset, as defined in Equation TO After computing the importance scores, we perform cross-layer ranking by globally sorting all singular values based on their scores. We retain only the top& & singular values with the highest importance scores, where termed runk budget. All remaining singular values are truncated. This global ranking strategy ensures that the most critical components are preserved regardless of the layer they originate from, allowing for an optimal allocation of rank capacity throughout the VLM. For QSVD, we apply the rank selection mechanism to the self-attention layers throughout the entire VLM.
3.3 Bost-Training Quantization Scheme for Low-Rank VLMs
Building on the efficient low-rank SVD ap- proach described in Section 3.J and Sec- tion 3.2 this section presents an efficient quantization scheme applied to the result- ing low-rank VLMs for further hardware efficiency enhancement. To analyze the outlier distribution in the VLM, we profile the input activation distribution of LLaVA- v1.5 13B B33. Specifically, we examine the input activations X across the self- mod-
Talon bngthi
Star 100 Tokon larth
Charmal
Charaal
Enfandeel
(b)
())
Figure 3: Input activation distribution within VLM. Only partial channel are shown.
attention modules and ules within the language model of the VLM, as illustrated in Figure 3 (a) and (b), respectively. Our analysis reveals prominent chan- nelwise outliers in X across all three components, which poses the great challenge when quantizing these VLM activations for (ow-precision operations.
To address this issue, we adopt the rotational method introduced in [2) and outlined in Section 2.3 to smooth channelwise outliers. However, since the self-attention architecture of the VLM has been modified by the application of SVD, we develop an efficient quantization approach that accounts for this change. Let X denote the input to the weight matrices, and the output be expressed as Y XW W gku Cyk. W gkv. where Wd
W and gke Σ/ x!-"W" as notated in Equation 3 The distribution of Cykv. which is buffered for KV vector recomputation, is shown in Figure (c). We observe that Cyke exhibits channelwise outliers, rendering it unsuitable for low-precision quantization. To climinate the channelwise outliers in both X and the compressed representation Cake, we introduce orthogonal matrices H1 and H,, The self-attention computation together with its quantized counterpart are then reformulated as follows:
oko
Y = (XH, (H1 W H2 ((H2 ( Y' Q(Catt ((H22
(11)

---

Table I: Accuracy evaluation of different methods. For ASVD and SVDLLM, their R1, R2 shared. Detailed results can be found in Appendix A.2
Muthel
Adc
20dd
Arc
The one
Anc
Fhe c01
Are
Adc
Flk. and
Arc
ASVD
R1:
11005
Ra 1.
R1 170.055
6.085C
Ra: 11005
SmoVEM
MA3756
3.00%
0.1335
28
USVD-aaQ
83.76%
Rg 37..5%
1100%
K1.70%
Rg
35.78%
148
7A.57%
50.%
-9.0%
77.64%
R2
26.25%
70.0%
40.67%
R2
A6 -100%
337.5%
40,67%
Rg 38.0%
LLAW-NNo Noa.
ASVID
56.735
16..3%
47-1555
10,9655
R1 56.77%
38 2166
:50.395
68.33%
19 1948198
R1: 56.776
SVDLLM
05.04%
-22 555
66.14%
64.9065
Rg 1 17.55
62.8776
R 15.0%%
45.01%
222 555
47.749%
R2 17.555
75
QUVD
15 00.0%
68.2216
10 (53.3%
67.00%
10 46.7%
65.35%
35 .0.0%%
54.38%
45 100 8%
$1.42%
Ri 46.
FP16
69.55 516
Aucerwcy
40/9
ASVD
64.70%%
R. -66 35%
56.92%
Rs 66.05%
46.5016
R. 56-77%
42.79%
R. -38.3%
44.8415
40.011%
Ru. 56-755
SVOLLM
72.048
22
71.4655
30.0%
71.2900
70. 50.5
115.0%
0390
222
48 20.119%
R, 17.35%
138
QSVD-ooQ
71.70%
Rg -2.55%
71.74%
Rg
2000%
71.74% 9
17.85%
70.80%
36g 15.00%
56.15%
+22.5% F2
58.79%
11.55%
Accernor
where Cyku can be approximately computed using the quantized input and weight as:
Cykre Q(XH) (Q(H, (o.
(12)
The computations presented in Equation I1 and Equation 12 support low-precision execution while reducing the size of both the weight parameters and the intermediate results Cgkur thereby lowering the overall memory footprint and reduce the processing latency.
Although the introduction of H1 and H2 helps mitigate outliers in X and Cake, respectively, we observe that these transformations do not fully eliminate the severe outliers, particularly those present in Cgku- To analyze this issue, we examine the distribution of W gku: which directly influences the distribution of Cękv- We observe that W gkv
W is strongly affected by the parameter B. Since E, is diagonal matrix whose entries are singular values that can vary significantly in magnitude, raising them to the power 3 can amplify the disparity. This, in turn, exacerbates the presence of outliers in Cakes as shown by the following derivation:
Cake XW gkn XW X diadgag
of) = (XW:),, of (XW:))) (13)
where (XW:)); denotes the i-th column of XW which can significantly influence the channelwise outlier distribution in Cake- To address this, we propose learning an optimal value for /3 by optimizing it over the calibration dataset D, namely:
min Σ 118a -
(14)
JED
where Y. and Y denote the self-attentin block outputs with and without quantization for the d-th sample the calibration dataset D. respectively. The parameter 8 optimized individually for each layer within the VLM. Finally, we apply the quantization operations to both the visual encoder and all layers of the language model, resulting an end-to-end efficient VLM computation.
Evaluation Results
We evaluate QSVD on five VLMs: LLa 7B 333, LLa 13B. LLaVA-Next 7B, LLaVA- Next 13B, and
39 To determine the optimal rank allocation and 3 parameters, 256 samples from the ScienceQA training dataset (36). following the procedures outlined in Section 3.2 and Section 33. For evaluation, we adopt three widely used benchmark datasets, ScienceQA [36.. VizWiz [15] and CGEDD-Bench-MG (2I]. line with prior work such as LLaVA, We compare QSVD against baseline methods by implementing them on the aforementioned VLM models, the baselines include the SVD approaches (ASVD 60), SVD-LLM (52) and quantization approach (QuaRot 2 DuQuant (29. QVLM (46). Specifically, for ASVD and SVD-LLM, we follow their official implementations by applying SVD separately to the Key and Value matrices, while avoiding decomposition of the Query matrices to prevent performance degradation. Additionally, SVD is not applied to other linear layers within the VLM. All methods are evaluated using the same calibration samples and random seeds to ensure faimess, and we report their best performance. For QuaRot and DuQuant, we apply them to the various VLMS by strictly following the detailed procedures provided in their respective code repositories.
7

---

Table 2: Quantization evaluation across different models and datasets. R2 is omitted since they are similar for different methods. Detailed results can be found in Appendix A.2
Model Bir
Dhoguant
QVLM
QANVD
Ohr
Rg
SUIUIT
VIZW
SEED7
R2
SOAJKT
VIEWIE
SEEDD
Rg
SEEDT
A2
SUIUNT
VI2W12T
SEEDD
Smaol IL Next ULa Next (LAVk-11 (LaV5-115
1916
10075
600 01155
511/1/15
06: 60:18/5
1007
05.0115
50111115
061 181
18375
0511115
5010155
1001
0911155
50.195
60: 1W5
W848
50%
66,533
49 86%
58.6298
50%
64. 6555
50.6490
51,82%
505
52,955 9595
48.511%
53.92%
18.75%
07.57%
54.06%
60.22%
W8A4
25%
57.350
54.11%
25%
48.35%
11.99216
41.26%
52.18%
58.89%
77%
12.6115
10.48%
55.16%
100%
53,6395
100%
71.78%
53:6315
10096
53.6316
71.7856
158
WRAB
50%
09.6646
50.7316
62.70%
50%
70.65%
50.32%
62.36%
50%
2516
54.533%
61 01.345
15.75%
72.12%
55.42%
62.91%
WKA
25R
67.222%
53.07%
61.43%
25%
66.46%
49.05%
22%
25%
65. 36.95
52.611
59.30%
9.58%
70.12%
53.20%
62.95%
W444
25%
65.80P%
49,37%
$9.28
59.2899
25%
64. 64.36
48,57%
41.07%
25%
20.3554
$7.5%5
20.9698
9,38%
65.82%
56.82%
61.79%
100R
68.6015
54,409
09 09.00%
54.400%
69.02%
10015
0005
51, 06/5
09:025
54,00
09.03
$2.05%L
47.55%
69.09%
W844
259
66.34%
2652
636448
25%-
48.55%
15%
43.37%
48.6 68.65
635
9.8%%
53.72%
65.63%
W444
25R2
58.57%
52.00%
62.95%
25%6
55.30%
48.58%
45.24%
25%
19.17%
3.30%
13.68%
9.38%
59.67%
52.00%
62.08%
F716
10072
75:23154
57.7255
71.30%
1007
73:339%
57 57.7355
71.90%
10095
05:2215
57.925
71 71.309
1007
75.2144
57.728
71. 1.
WRA8
50%
61.115%
54.388%
7000798
50%
69.3669
49.8995
69. 69.225%
50%
71.522%
55.135
67.887%
15.75%
72.38%
58.33%
71.23%
WRA4
25%
70.220%
52.43%
25%
15.2
48.9885
64.8516
53.135
66.54%
70.43%
69.21%
53.766
52.23%
12.85%
14.64%
54,27%
65.34%
F716
100%
$4.6899
37,0775
100%
84.68%
37,075%
10095
$4.6855
37.00%
68.18%
$4.6855
37.077%
WRA3
50R:
57.80%
35.52%
62.65%
50%
55.30%
33.145
55.739%
50%
11.94%
0.00%
9.13%
18.75%
76.20%
37.82%
66.89%
WKAA
25R
55.92%
34.09%
45.82%
25%
53.52%
30.43%
42.24%
254
3.92%
0,00%
1.23%
9,38%
62.35%
36.91
55.35%
Given that QSVD performs joint SVD with quantization, we first evaluate its SVD-only performance in by comparing it with ASVD and SVD-LLM under equivalent hardware costs, including inter- mediate data storage for KV recomputation, weight size, and VI.M computational cost in FLOPS. Specifically, we express it terms of the ratio between ours and the FP16 without compression. We denote the SVD-onll approach depicted in Section 4.] as QSVD-noQ. We then apply the quantization techniques introduced in Section 3 to the SVD-compressed VLM and compare the results with advanced quantization methods such as DuQuant [29] and QVLM [66, as well as quantized version of ASVD (QASVD). For QASVD, we apply QuaRot 2 to the SVD-truncated VLMs obtained from ASVD. The corresponding results are presented in Section 4.2
We evaluate all performance results on NVIDIA RTX A6000 GPUs using VLMEvalKit 100, and we report results under three weight-activation quantization configurations: W8A8 (8-bit weights and 8-bit activations), W8A4, and W4A4. For activation quantization, we apply per-token symmetric quantization. For weight quantization, we use round-to-nearest (RTN) with per-channel symmetric quantization and learnable clipping ratio, determined via linear search to minimize squared error, following [2]. We present ablation studies in Section 4.3 and evaluate the latency improvements of QSVD on GPU in Section 4.4 Additional results are included in Appendix A.2
4.1 Accuracy Evaluation on QSVD-noQ
We begin by evaluating the QSVD-noQ performance in FP16 under four different rank budgets k. To ensure a fair comparison, we adjust the rank configurations of all methods such that our approach consistently maintains the lowest hardware cost in terms of intermediate data storage (1)), weight size (a), and computational overhead (7), as mentioned in Section 3.1 Importantly, as noted in Section 3.1] the relative ratios among the weight sizes afp- aqsvd. and Crind identical to the ratios among the computational costs "Ip- Yywdd and Tind This equivalence allows us to report a single normalized metric to represent both weight parameter reduction and computational efficiency. Therefore we have R₁ and Rg, defined as:
76
7 Vfrr
R1
R2
(15)
fxf
T/P
where i can either be "qsvd" or "ind". The evaluation results are presented in Table I Our method outperforms ASVD and SVD-LLM in accuracy while incurring minimal or comparable hardware cost. On LLaVA-v1.55 5 13B, QSVD-noQ results in less than 1% drop in ScienceQA-IMG accuracy compared to the original FP16 baseline, and notably even surpasses the FP16 performance on VizWiz. For instance, with R1 = 46.7% and R2 = 17.5 17.5%. QSVD-noQ achieves an accuracy of 55.79%, exceeding the FP16 counterpart by more than 2%. This may be due to the low-rank approximation effectively mitigating hallucinations [32 in the VLM; however, further investigation is needed to confirm this hypothesis. Moreover, our approach consistently achieves higher accuracy than ASVD and SVD-LLM under reduced parameter and cache ratios (R₁ and R2), with the performance gap
8

---

widening as these ratios decrease. For instance. in the SmolVLM setting, our method attains over 70% accuracy on ScienceQA-IMG while both ASVD and SVD-LLM fail to operate effectively.
4.2
Accuracy Evaluation of QSVD
The low-rank SVD components are subsequently quantized using the techniques described in Sec- We compare QSVD with DuQuant [29 and QVLM [46]. as well QASVD, which integrate QuaRot's quantization approach with the low-rank SVD outputs from ASVD, respectively. Quantization is applied consistently across the entire VLM, including both feed-forward and self- attention layers in the language model and visual encoder. Evaluations are conducted on three benchmark datasets: ScienceQA, VizWiz, and SEEDBench. All methods maintain similar R& of approximately 50%, while R2, which has greater impact on inference latency and cache size, varies across approaches. Notably, QSVD consistently achieves lower R2 compared to all other baselines.
tion 3.3
As shown in Table 2 under the W8A8 setting. QSVD consistently outperforms other baselines in most scenarios. On large-scale models like LLaVA-1.5 13B, it reaches accuracy comparable to the FP16 baseline while reducing QKV weights and compute by 50%, and cutting intermediate data size to just 18. 85%. Under the more aggressive W8A4 setting, QSVD surpasses all baselines and approaches FP16-level performance using as little as 9.38% of the original KV cache. Finally, Table shows the quantization results under the W4A4 setting. Under this configuration, QASVD fail to operate properly (yielding zero accuracy). Despite the challenging conditions, QSVD consistently delivers the highest performance among all models while maintaining the lowest hardware cost in terms of Rí and R2.
4.3 Ablation Study
Effectiveness of Cross-layer Rank Allocation Scheme To evaluate the effectiveness of our cross-layer rank al- location strategy (Section 3.2) . we compare with two baseline methods. The first, referred to as the Uniform- rank (UR) scheme, applies SVD to the joint QKV weights using the same rank across all VLM blocks. The second, denoted as the Fisher Information Based (FIB) scheme, also applies SVD the joint QKV weights but distributes ranks across layers based on Fisher information. This approach has been adopted in prior work for SVD-based compression in LLMs (5). All methods operate under the same hardware budget, defined by R2 and R2. As shown in Table 3
Table 3: Accuracy performance under
varying rank allocation strategies.
Melied
Scieme-QA-IMG
Ri
60.0%
$3.95
46.7%
40.0%
22.5%
20.03
17.5%
15.0%
71.84%
70.40%
70.405
67.72%
FIR
70.608
70.499
70.155 158
69.9698
QSVD-noQ
71.79%
71.74%
71.74%
70,80%
under aggressive compression, QSVD-noQ consistently outperforms both baselines and maintains accuracy close to the FP16 model.
Impact of Learning 3 As described in Equation 14 we train 3 to suppress outliers in the intermediate result Cyke- Table 4 presents presents the impact of the learnable 3 on VLM accuracy under W4A4 setting over LLaVA 7Bs on Science QA. We compare QSVD with bascline methods using fixed (3 across the entire VLM. QSVD consistently achieves the highest accuracy, outperforming all fixed-3 baselines, highlighting the importance of learning 3 for effective low-bit quantization.
Table 4: Impact of B learning.
Model
0.0
QSVD
0.4
0.8
v1.5-7h Next-bb
54.53%
55.16%
54.83%
6.09%
58.80%
59.67%
56.56%
15.12%
Long Sequence Scenarios To further evaluate the adaptability of our QSVD method under long sequence conditions, we conduct experiments on the HRBench-
Table Evaluation results on HRBench-4K.
Methed
HRRBencb-KK
Ase
Hu cost
Avc.
Ho cost
Acc.
Ho cost
4K dataset [50]. which consists of 4K-
44.35%
44.12%
43.12%
resolution images. We follow the same evaluation setup as mentioned above and use VLMEvalKit [10 to report the "Aver- age All" accuracy metric. Both LLaVA- Next 13B and VAav1-11.5 13B are evalu-
QSVDsneQ)
64.88%
R2 32.5%
R: 60.01%
44.12%
f2 20.0%
16 -3.3%
45.80%
R2 .7.5%
Ri .6.7%
LL-WW-11
ASVI
1258
Rs -22.5%
Ri 63.3%
$8.691%
Ro (20,00)
R. .50.0%
16.62%
R. -1.5%
R, 56 76.7%
QSVD>asQ
39.58%
.22.5%
38.75%
.1..5%
1716
Ancimatyy 14: 1776

---

Table Accuracy evaluation results (1) on HallusionBench under different compressed parameter size ratios (R,) ) FP16 indicates uncompressed original models.
LLa VA.. 138
LAVVA-Nett 13B
R,
aAce
fAcc
qAce
Overall
aAcc
fAcc
qAce
Overall
90%
49.63%
21.10%
17.58%
29.44%
57.73%
26.01%
26.59%
36.78%
80%
48.90%
20.52%
16.92%
28.78%
58.25%
26.01%
26.81%
37.03%
70%
50.26 %
22.83%
17.80%
30.30%
58.46% %
26.88%
27.25%
37,53%
FP16
44.69%
19.36%
16.04%
26.70%
56.78%
26.01%
25.27%
36.02%
ated under our QSVD-noQ configuration and compared against ASVD and FP16 baselines. The results are summarized in Table
As shown in T QSVD-noQ consistently outperforms ASVD in all evaluation settings. Moreover, the relative performance trends on HRBench-4K closely mirror those observed on ScienceQA- and VizWiz, indicating that our rank allocation strategy generalizes effectively to long sequence scenarios resulting from high-resolution visual inputs.
Impact of QSVD on Hallucination We further evaluate the impact of QSVD on VLM hallucination using HalcusionBencch [14], following the same evaluation setup mentioned above. Metrics include aAcc, fAcc, and Ace from Hallusion and their overall average score. As shown in Table both LLa 13B and LLaVA-Next 13B exhibit noticeable improvements in groundedness metrics after QSVD-noQ. For LLaVA-Next 13B, the overall score increases from 36.02 to a peak of 37.53 R₁ 70%. Similarly, LLaVA-v 13B improves from 26.70 76.70 to 30.30 R 70%, marking clear reduction in hallucination. These findings confirm that QSVD-noQ not only reduces model and cache size but also acts as an effective regularizer against hallucinations. This effect explains why, on certain benchmark datasets such as VizWiz, the DSVD-compresee models occasionally outperform their original FP16 counterparts in terms of end-task accuracy.
4.4 Latency Evaluation on VLM
QSVD leverages both SVD and quantization to jointly compress model weights and KV cache, making it well-suited for deployment on memory-constrained hardware. We evaluate inference latency of the layer-wise LLaVA-v1.55 7B on NVIDIA RTX 4070 GPU with
FP16
noQ-40%
noQ-30%
INT8
70%
50%
15
12.7x 1x.1x
Speedup
10
9.7x
12GB memory. The batch size is set to I and the token length to 4K. As shown in Figure 4] under FP16 precision, due to limited GPU memory, both the FP16 baseline and QSVD-noQ require partial offloading to CPU memory. However, QSVD-noQ with 40% and 30% (denoted as noQ-40% and noQ-30%) rank retention benefits from reduced data movement enabled by effective SVD compression, achieving up to a 2.1x speedup over the baseline. Furthermore, QSVD with WSA8 quantization, under 70% and 50% rank retention, completely avoids offloading and achieves significant speedup of up to 13.1x.
1x
1.7x 2.1x
Figure 4: Normalized speedup of QSVD-noQ and QSVD W8A8 on low-end GPU.
5 Conclusion and Limitation
In this work, we proposed QSVD, unified framework that applies joint singular value decomposition and quantization to compress VLMs efficiently. By decomposing the combined QKV weight matrices and introducing an adaptive cross-layer rank allocation strategy, QSVD significantly reduces computational cost, KV cache size, and model storage with minimal impact on accuracy. Although quantization is applied to all layers of the VLM, compression is mainly focused on the QKV weights in self-attention layers. Future work will explore joint optimization across all model blocks. Additionally, improving VLM efficiency may also make powerful models more accessible, which raises concerns about potential misuse in areas such as surveillance, misinformation, and privacy violations. Further investigation is needed to address these risks.
10

---

References
[1] Joshua Ainslic, James Thorph Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-hcad checkpoints. arXiv preprint arXiv:2305.13245 2023.
[2] Salch Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37: 370100333-000240, 2024.
[3] Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Laila Bashmal, and Mansour Zuair. Vision language model for visual question answering in medical imagery. Rioengineering, 10(3)380, 2023.
[4] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, al. Paligemma:
A versatile vim for transfer. CoRR, 2024.
[5] Chi-Chih Chang, Wei-Cheng Lin, Chien- Yu Lin, Chong- Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Kv-cache compression with low-rank projection. The Thirteenth International Conference on Learning Representations.
[6] Chi-Chih Chang, Wei-Cheng Lin, Chien- Lin, Chong- Yan Chen. Yu-Fang Hu, Pei-Shuo Wang. Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21 18, 2024.
[7] Christel Chappuis. Valérie Zermatten, Sylvain Lobry, Bertrand Le Saux, and Devis Tuia. Prompt-rsvqa: Prompting visual context to language model for remote sensing visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1372-1381, 2022.
[8] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny, Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 18030-180444 2022.
[9] Tim Detimers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 Q:: 8-bit matrix multipli- cation for transformers at scale. Advances in neural information processing systems, 35:30318 803332, 2022.
[10] Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11198 11201, 2024.
[11] Maksim Dzabraev, Alexander Kunitsyn, and Andrei Ivaniuta. Vlm:: Vision-langunge models act reward models for image captioning. arXiv preprint arXiv:2404.0199 2024.
[12] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post- cost-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 2022.
[13] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint arXiv.2407.21788. 2024.
[14] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language ballucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375-14385, 2024.
[15] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jicbo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. CVPR, 2018
[16] Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint ar ar::::2077 007.00012, 12, 2022.
[17] Xiaowei Hu, Zhe Gan, Jianfeng Wang. Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision- languge pre- pre-training for image captioning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 17980-17989, 2022.
[18] Jia Ji. Yongshuai Hou, Xinyu Chen, Youcheng Pan, and Yang Xiang. Vision Vision-language model for generating textual descriptions from clinical images: Model development and validation study. JMIR Formative Research, 8:e32690, 2024.
11

---

[19] Ian T Jolliffe and Jorge Cadima. Principal component analysis: review and recent developments. Philoxophical transactions of the royal society A: Mathematical, Physical and Engineering Sciences, 374(2065):20 2016.
[20] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong. Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.076.22 2023.
[21] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang. Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299-13308 2024.
[22] Chuanhao Li, Zhen Li, Chenchen Jing, Shuo Liu, Wenqi Shao, Yuwei Wu, Ping Luo, Yu Qiao, and Kaipeng Zhang. Scarchlvims: A plug-and- playy framework for augmenting large vision-language models by searching up-to-date internet knowledge. arXiv preprint arXiv:2405. 14554, 2024.
[23] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image image pre-training with frozen image encoders and large language models. In International conference on machine learning. pages 19730-19742. PMLR, 2023.
[24] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping ergguage-image pre- ve-training for unified vision-language understanding and generation. In International conference on machine learning. pages 12888-12900. PMLR, 2022.
[25] Muyang Li, Yujun Lin, Zhekai Zhang. Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng. Jun-Yan Zhu, and Song Han. Svdqunat: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint 24.0007 2024.
[26] Muyao Li, Zihao Wang, Kaichen Hc, Xiaojian Ma, and Yitao Liang. Jarvis-vla: Post-training large-scale vision language models to play visual games with keyboards and mouse. arXiv preprint 2rXiv:2503.1636 2025.
[27] Shiyao Li, Yingehun Hu, Xuefci Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, et al. Mbq: foolality-balanced quantization for large vision-language models. arXiv preprint 1rXiv:2412.19509, 2024.
[28] Zhiteng Li, Mingyuan Xia, Jingyuan Zhang, Zheng Hui, Linghe Kong, Yulun Zhang, and Xiaokang Yang. Adasvd: Adaptive singular value decomposition for large language models. arXiv preprint arXiv:2502.01400. 2025.
[29] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized Ilms. Advances in Neural Information Processing Systems, 37:87766 87800, 2024.
[30] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation- aware weight quantization for on-device lIm compression and accelcration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[31] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19. 19437. 2024.
[32] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint 242.2002.,00233, 2024.
[33] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
[34] Junling Liu, Ziming Wang. Qichen Ye, Dading Chong, Peilin Zhou, and Yining Hua. Qilin-med-vl. Towards chinese large vision-language model for general healthcare. arXiv preprint a7X1922100.19556 2023.
[35] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations. arXiv preprint urXiv:2405.1 16406, 2024.
[36] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chi Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
12

---

[37] Alexander Ly. Maarten Marsman. Josine Verhagen, Raoul PPP Grasman, and Eric-Jan Wagenmakers. A tutorial fisher information. Journal of Mathematical Psychology, 80:40-55, 2017.
[38] Yuexiao Ma. Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403. 12544 2024.
[39] Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, al. Smolvim: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299 2025.
[40] Matan Ben Noach and Yoav Goldberg. Compressing pre- praited language models by matrix decomposition. In Proceedings of the Ist Conference of the Asia Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 884-889, 2020.
[41] Wengi Shao, Mengzhao Chen, Zhaoyang Zhang. Peng Xu, Lirui Zhao, Zhiqjan Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
[42] Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv: 09111911111100 150, 2019.
[43] Zelong Sun, Dong Jing, Guoxing Yang, Nanyi Fei, and Zhiwu Lu. Leveraging large vision-language model as user intent-aware encoder for composed image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7149-7157, 2025.
[44] Mohammad Reza Taesiri and Cor-Paul Bezemer. Videogamebunny; Towards vision assistants for video games. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1413. IEEE, 2025.
[45] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codchooks. arXiv preprint arXiv:2402.043366 2024.
[46] Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang. Jie Zhou, and Jiwen Lu. Q-vim: Post-training quantization for large vision-language models. arXiv preprint arXiv:2410.081 19, 2024.
[47] Guankun Wang, Long Bai, Wan Jun Nah, Jie Wang, Zhaoxi Zhang, Zhen Chen, Jinlin Wu, Mobarakol Islam, Hongbin Liu, and Hongliang Ren. Surgical-IvIm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery. arXiv preprint 2xXi:24405.10444 2024.
[48] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan. Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- ssiinnllagguae model's 's perception the world at any resolution. arXiv preprint arXiv:2409. 12191, /2191, 2024.
[49] Qinsi Wang, Jinghan Ke, Masayoshi Tomizuka, Yiran Chen, Kurt Keutzer, and Chenfeng Xu. Dobi-svd: Differentiable svd for Ilm compression and some new perspectives. arXiv preprint arXiv:2502.02723, 2025.
[50] Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free free framework for high-resolution image perception in multimodal large language models. In Proceedings of the AAAI Conference Artificial Intelligence. volume 39, pages 1907-7915. 2025.
[51] Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-Iim v2: Optimizing singular value truncation for large language model compression. arXiv preprint arXiv:2503. 12344 2025.
[52] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Syd-Ilm: Truncation-aware singular value decompo- sition for large language model compression. arXiv preprint arXiv:2403.07377. 2024.
[53] Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit integers, 2023.
[54] Jingyang Xiang and Sai Qian Zhang. Dfrot: Achieving outlier-feee and massive activation-fret free for rotated Ilms with refined rotation. arXiv preprint arXiv:2412.00644 2024.
[55] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pages 38087-38099 PMLR, 2023.
13

---

[56] Jingjing Xie, Yuxin Zhang, Mingbao Lin. Liujuan Cao, and Rongrong Ji. Advancing multimodal large language models with quantization-wware scale learning for efficient adaptation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 10582-10591, 2024.
[57] Ke Yi, Zengke Liu. jianwei zhang, Chengyuan Li, Tong Zhang, Junyang Lin, and Jingren Zhou. Rotated runtime smooth: Training-fres activation smoother accurate INT4 inference. In The Thirteenth International Conference on Learning Representations. 2025.
[58] Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. Effectively compress kv heads Ilm, arXiv preprint X1vi:22406.077055 2024.
[59] Zhengqing Yuan, Zhaoxu Li, Weiran Huang, Yanfang Ye, Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint urXiv:2312.1663 2023.
[60] Zhihang Yuan, Yuzhang Shang. Song. Qiang Yan Yan, and Guangyu Sun. Asvd: Activation aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.0882, 2023.
[61] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[62] Baichuan Zhou, Ying Hu, Xi Weng. Junlong Jia, Jie Luo, Xien Liu, Ji Wu. and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402. 14289, 2024.
[63] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision- language pre-training for image captioning and vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13041 10041-13060 13049, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, our abstract and introduction accurately reflect the main contributions and scope of the paper. We clearly state that the work proposes novel application of Singular Value Decomposition (SVD) to the joint QKV matrices in Vision-Language Models (VL.Ms), with the goal of reducing KV cache size and computational overhead. This aligns with the technical content, which introduces a dynamic SVD rank allocation strategy balance memory and accuracy. Furthermore, the abstract discusses the extension of this approach through quantization of both weights and activations, which is supported by experimental results in the paper showing better comparable performance against previous methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitation in the last section of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We will inlcude full derivation of our importance score in Appendix A.1
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Yes, the paper discloses sufficient information to reproduce its main results. clearly describes the methods for joint QKV SVD, rank allocation, and quantization, with supporting equations and implementation details. The evaluation setup, including models, datasets, and calibration procedures, is well-documented.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We submit the code as the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https wwwedeeeeeoeebbmmeeeeeee ssionollccy for more details.
//nips.cc/put wwtpcccccppccccceee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer:
[Yes]
Justification:
We include necessary experiments specifications in supplementary materials
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, report the average results over 5 random seeds in all our quanization part. And for calculation of our reported accuracy we follow the opensource VLM evaluation framework.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All the details are provided the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neerrrppi httpprisssccpppppc/eeeeeee cttceeeeeeecccccceeeeeeeee
Answer: [Yes]
Justification:
We have read and acknowledge the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We mentioned it in the last section of the paper.
18

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not involve the release of data or models that carry a high risk of misuse, and therefore no specific safeguards are necessary in this context.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All code, data, models utilized in this paper are cited and credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Our paper does not release new models and assets.
Guidelines
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper do not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects
Guidelines: The paper does not involve crowdsourcing nor research with human subjects. The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
This work does not use large language models (LLMs) as an important, original, non-standard component of the core methodology. LLMs were only used minimally for editing or polishing writing and did not influence the research's 's scientific content, methodology, or conclusions.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not non Please refer to our LLM policy (https: /ttpriip/neurps... http://urrpsscccoaffeeeeeeeeeeee cc//oooeeeeccee//22555555 for what should or should not be described.
involve L.MMs as any important, original, non- non-standard components.
21

---

A
Technical Appendices and Supplementary Material
Outline of Appendices
A.1
Detailed derivation for Importance Score in Sec. 3.2
A.2
Additional experimental results.
A.3
Case study for QSVD
A.1
Importance Score derivation
Proof.
Recall that the singular value decomposition (SVD) of W is given by:
W UEVT
where u, and oy the i-th left and right singular vectors, and a; is the i-th singular value. When truncating oi the change in W is:
AW. = o,u,y
Then, the inner product between AW, and the gradient G( is:
Σ
AW., (ik)) Glv Ci (i,k) (AW G(P))
j.k
where (,, .) denotes the Frobenius inner product.
Substitute AW,, oqu.o?
(AW.,, G(W)))))>))))>>)))))) (o,uo, G(W)r au(uue cW
Using the property of the Frobenius inner product:
(A, B) = I)
we have:
auuu = (( o, trv,,uu G
By cyclic property trace:
o, trru (() o, ur(u, G(u v,)
Since up G(W vs is scalar:
os tr( G(r) 0.) = (u( G(T) vz)
Note that:
UT Rex
and the (i, i)-th diagonal entry is:
22

---

VFW = uz G
Therefore:
Σ (..k GW o. [ Vl(.
Σ
AW (i Gw(j,)) (AW., G o2 Gw
J.K
Finally, the importance score fo can be computed follows:
f..
1 N
Σ
of
U
G(
(6.1)
If whitening transformation such ASVD or SVD-LLM is applied prior SVD, that is, UEVT = SVD(WS) 5) where S denotes the whitening matrix, the corresponding importance score can be reformulated as:
i.
1 of 1UT [
S S-
(1.2)
where the term S converts the gradient from the original weight space to the whitened space in which the SVD is performed.
A.2 Full table for experiments
Here we include more detailed experiments table for QSVD method. We evaluate performance reuslts on NVIDIA RTX A6000 GPUs, we report results under QSVD-noQ and under three weight- activation quantization configurations: W8A8 (8-bit weights and 8-bit activations), W8A4, and W4A4. For activation quantization, we apply per-token symmetric quantization. For weight quantization, we use round-to-nearest (RTN) with per-channel symmetric quantization and learnable clipping ratio, determined via linear search to minimize squared error, following (2). For QSVD baseline, we add QuaRot without SVD as an baseline, also for ours method, we use activation clip ratio of 0.85 for vit model and 0.9 for language model, under this setting, we have updated some QSVD accuracy results higher than main paper report.
QSVD-noQ results. Table presents detailed results of QSVD-noQ on SmolVLM 39), LLaVA- v1.5 33 series, and LLaVA- VA-Next series, along with the corresponding preserved ratios. QSVD-noQ consistently outperforms ASVD and SVD-LLM in accuracy under reduced parameter and cache ratios (R1 and R2), with the performance gap widening as the compression becomes more aggressive, For instance, in the SmoIVLM setting, our method maintains over 70% accuracy on ScienceQA-IMG (36], while both ASVD 60 and SVD-LLM (1) fail to function effectively.
QSVD Results. We evaluate our proposed QSVD quantization strategy on LLaVA v1.5 and Next series: 7B, 13B, across three benchmarks: ScienceQA, VizWiz, and SEEDBench. Table summarizes performance under two low-bit settings, W8A8 and W8A4.
Under the W8A8 setting, QSVD matches or exceeds prior methods such Duquant 29 QVLM 46). and QASVD, while reducing the KV cache and intermediate data sizes by up to 50%. Compared to QSVDLLM, our approach avoids the need for manually re-optimizing decomposed matrices while still achieving superior performance.
In the more challenging W8A4 configuration, QSVD continues to deliver robust outputs, reaching levels comparable to the FP16 baseline using just 9.38% of the original KV cache. This demonstrates the scalability of our guantization design under aggressive memory constraints.
23

---

Method
ScienceQA-IMGi
VizWiz 1
FP16
84.53%
37.07%
R1
100%
90.0%
80.0%
70.0%
100%
90.0%
80.0%
70.0%
SmoIVLM-2B
Ra
50.0%
42.5%
35.0%
27.5%
50.0%
42.5%
35.0%
27.5%
ASVD
53.84%
7.88%
0.69%
0.10%
6.68%
0.00%
0.00%
0.00%
SVDLLM
65.89%
34.61%
9.07%
3.02%
14.86%
1.62%
0.13%
0.00%
R1
100%
90.0%
80.0%
70.0%
100%
90.0 %
80,0 %
70,0 %
Ra
37.5%
33,75% %
30.0%
26.25% %
37.5%
33.75%
30,0 %
26.25 %
QSVD-noQ
83.78% %
81.70% %
79.57 %
77.64 %
40.67
39.88 %
40.67%
43.84 %
FP16
68.0 %
50.03%
LLaVA-v1.5 7B
R1 63.3%
60.0%
56.7%
53.3%
63.3%
60.0%
56.7%
53.3%
Ra
22.5%
20.0%
17.5%
15.0%
22.5%
20.0%
17.5%
15.0%
ASVD
22.36%
19.09%
15.22%%
10.81%
47.70%
39.02%
10. 10%
8.87%
SVDLLM
55.23%
55.03%
49.23%
51.17%
50.10%
50.71%
50.47%
49.99%
R₁
60.0% %
53.3% %
46.7 %
40.0 %
60.0 %
53.3%
46,7 %
40,0 %
Ra
22.5%
20.0% %
17.5%
15.0 %
22.5 %
20.0 %
17.5
15.0 %
QSVD-noQ
66.12%
65.64% %
64.06%
61.68 %
53.84% %
54.19 %
53.88%
52.40%
FP16
71.78%
53. 63%
LLaVA-vI 5-133 5-13B
R1 63.3%
60.0%
56.7%
53.3%
63.3%
60.0%
56.7%
53.3%
Ra
22.5%
20.0%
17.5%
15.0%
22.5%
20.0%
17.5%
15.0%
ASVD
64.70%
56.92%
46.50%
42.79%
44.48%
40.63%
40.01%
37.87%
SVDLLM
71.44%
71.44%
71.29%
70.50%
51.03%
51.15%
49.37%
46.49%
R1 60.0%
53.3%
46.7 %
40,0 %
60.0%
53,3%
46.7%
40.0
Ra 22.5%
20.0% %
17.5% %
15.0 %
22.5 %
20.0 %
17.5%
15.0 %
QSVD-noQ
71.79%
71.74%
71.74%
70.80
56.15%
56.05
55.79
54.04%
FP16
69.51% %
54.46%
LLaVA-Next 7B
R; 63.3%
60.0%
56.7%
53.3%
63.3%
60.0%
56.7%
53.3%
Ra
22.5%
20.0%
17.5%
15.0%
22.5%
20.0%
17.5%
15.0%
ASVD
50.72%
47.15%
40.26%
25.73%
47.78%
47.3%
39.41%
6. 6.69%
SVDLLM
65.94%
66.14%
64.90%
62.87%
48.01 %
48.41%
47.74%
47.73%
R; 60.0% %
53.3%
46.7 %
40.0
60.0
53.3%
46.7
40.0
Ra 22.5%
20.0%
15.0
22.5
20.0
17.5%
15.0%
QSVD-noQ
68.22%
67.03%
52.31
51.42%
49.86
FP16
73.23%
57.72%
LLaVA-Next-1 13
R1 63.3%
60.0%
56.7%
53.3%
63.3%
60.0%
56.7%
53.3%
22.5%
20.0%
17.5%
15.0%
22.5%
20.0%
17.5%
15.0%
ASVD
69.71%
68.86%
67.43%
64.01%
55.42%
54.97%
54.50%
52.95%
SVDLLM
70.30%
69.71%
69.56%
68. 53%%
53 33.08%
52.54%
52.52%
51.77%
R1 60.0%
53.3%
46.7%
40.0%
53.3%
46.7%
40.0%
22.5%
20.0%
17.5%
15.0%
22.5%
20.0%
17.5%
15.0%
QSVD-noQ
72.63%
72.29
72.34%
71.64%
55.48
55.14%
54.99%
55.77%
Table Accuracy ScienceQA-IMG and Viz datasets. The R1, R2 denotes the proportion of preserved QKV parameters and the corresponding cache ratio.
For completeness, Appendix Tables and greport full results across all model variants and bitwidth configurations. Notably, our method consistently ranks highest near-highest across settings, while maintaining favorable compression ratios of 50%/18.75% W8A8 and 50%/9.35 in W8A4. These results highlight the ability of QSVD balance compression and output quality across diverse range of architectures and tasks.
24

---

Method
W8A8
RiRRa
W8A4
Ri/Ra
SeiQAt
VizWizt
SEEDI
SeiQAt
VizWizt
SKBDT
FP16
68.01%
50.03%
60.18%
100%/100%
68.01%
50.03%
60.18%
100%/1000%
LLaVA-1,5-70
QuaRot
67.90%
49. 49.95%
60.11%
50%
63.19%
49.82%
58.18%
50%
Duquant
66.53%
49. 86%
58.622 62%
50.52%/50%
57.36%
50.07%
54.11%
50.52%
QVLM
64.65%
50.64%
51.82%
50%/550%
55.24%
48.33%
50.13%
50%//25%
QASVD
52.95%
48.31
53.92%
50%6/50%
41.92%
47.85%
41.26%
50%k/25%
QSVDLLM
66.14%
51.93%
56.47%
50%6/5018
30.38%
45.00%
37.00%
5036/25%
QSVD
67.57%
54,06 %
60.20%
50 50%18.75%%
65.61
52.18%
58.49
50 96/9.9998 %
FP16
71.00%
53.63%
62.544 62.54%
100%
71.80%
53.63%
62.54%
100%
LLaYA-1.5-1B3
QuaRot
71.64%
53.64%
62.57%
50%/550
68.02%
54.57 %
58.53%
50%
Duquant
69.66%
50.73%
62.70%
51.67%
67.22%
53.07%
61.43%
31.67%
QVLM
70.65%
50.32%
62.36%
50%//50%
66.46%
49.03%
59.22%
50%%
QASVD
70.25%
54.93%
61.84%
50%/55%%
65. 35.344%
52.61%
59.30%
50/6/25%
QSVDLLM
70.65%
56.32%
62.35%
50%6/50%
60.20%
50.52%
55.03%
50%6/25%
QSVD
72.12%
55.42%
62.91%
50 %0...%55%%
70.12%
53.20%
62.95%
50%/9.38 %
LLaVA M---Nex--77 -B
69.60%
54.46%
69, 02%
100% 000%
69.60%
54.46%
69.02%
100%s/100%;
QuaRot
69.19%
52. 86%
65.60%
50%
64.53%
51.27%
65.08%
50%\
Duquant
66.34%
52.05%
67.91%
50.52%
66.34%
50.26%
63.64
QVLM
64.70%
47.55%
66, 66.82%
50%/50%
60.60%
48.55%
50.38%
50%%
QASVD
64.94%
47.30%
66.87%
50%//50%
43.37%
48.65%
49.63%
50%
QSVDLLM
64.70%
47.55%
66. 83%%
50%/550%
33.83%
46.05%
39. 39.08%
/25%
QSVD
69.09%
53.42%
68.92%
50 %
66.10%
53.72%
65.63%
50 99.%/9938 %
M-A-Nex--11l 13B
FP16
73.23%
57.72%
71.30%
100%%/100P
73.23%
57.72%
71.30%
100%//1006
LLaVA-Next-11
QuaRot
72.04%
58.03%
67.29%
50%/o50%
66.98%
55.56%
70.15%
50%
Duquant
61.13%
54.38%
70.07%%
51 5666%
70.20%
52.43%
66.15%
51.67%%/55%
QVLM
69.86%
49, 89%
69.28%
50%/55%%
65.28%.
48.98%
65.39%
QASVD
71.52%
55.13%
67.87%
50%/55%%
64.85%
53.13%
66.54%
50%%
QSVDLLM
69.85%
49.89%
69.27%
5018/5018
61.25%
45.05%
65.03%
5036/25%
QSVD
72.38%
58.33
71.23%
50%/18755%
70.43%
58.52%
69.21%
50 9...%/9989
Table Quantization results on W8A8 and W8A4.
V.. Series
LLaVA-Next Series
Bit
Method
ScienceQA
SEED
VizWiz
ScienceQA
SEED 1
VizWiz
R. RR
FP16
68.01%
60.18%
50.03%
99.600%
69.02%
54.46%
100%
W4A4
QuaRot
49.08%
50.54%
49.96%
55.57%
59.81%
55.25
25% 25%
W4A4
Duquant
52.56%
49.51%
48.77%
58.36%
62.95%
52.00%
27.08% 25%
QVLM
25%. 25%
W4A4
QASVD
12.61%
10.48%
1.23%
19.17%
13.68%
3.30%
25% 25%
W4A4
QSVDLLM
6.18%
5,53%
0.00%
10. 13%
8.64%
2,55%
25% 25%
QSVD
55.16%
52.70%
62.97%
99.38%
FP16
71.80%
62.54%
53.63%
73.23%
71.30%
57.72%
100% 100%
W4A4
QuaRot
62.74%
60.14%
55.62%
57.47%
62.95%
50.13%
25% 25%.
W4A4
Duquant
65.80%
$9, 28%
49.37%
58. 16%
63.15%
53.26%
26.67% 25%
W4A4
QVLM
64.86%
41.07%
48.57%
57.33%
60.55%
52.23%
25% 25%
W4A4
QASVD
20.35%
37.5%
20.96%
12.85%
14.64%
4,44%
25% 25%
W4A4
QSVDLLM
10.53%
7.65%
1.01%
15.57%
10.89%
2.22%
25% 25%
W4A4
QSVD
65.82%
61.79%
56.82%
63.61%
65.34%
$4.27%
25% 99.8%
Table 9; Quantization results on W4A4.
25

---

A.3 Case study for QSVD
For case study, we randomly selected examples from the ScienceQA [3 set, which demonstrate our method's 's superior performance over QVLM and QASVD baselines.
As shown in Case 2 and where the FP16 model fails to produce the correct answers, QSVD exhibits more consistent responses between W8A8 and W4A4 settings, and does not contradict the FP16 outputs.
QSVD also demonstrates surprisingly better performance under low-bit settings in Case 7 where the FP16 model answers incorrectly but QSVD at W4A4 produces the correct response. This may be attributed to the quantization process reducing model noise, as discussed in Dobi-SVD [49], and may partially explain why QSVD sometimes surpasses FP16 accuracy, as observed Appendix A.2
In Case and 8 QSVD successfully answers questions that FP16 answers correctly but where other baselines fail under W4A4, highlighting its robustness under extreme quantization.
Furthermore, in Case under 4-bit settings, while QSVD maintains coherent and relevant responses.
and 7 other baselines output random tokens repeat content
26

---

Case 1: LLLVA--1.5-22
The diagrams below show two pure samples of gas in identical closed, rigid containers. Each colored ball represents one gas particle. Both samples have the same number of Compare the average kinetic energies of the particles in each sample. Which sample has the higher temperature?
Question:
particles.
A. sample B
B. neither; the samples have the same temperature
C. sample A
Sample
Sarrpte i
Average
speed:
FP16 Response: 1. Compare the average kinetic energies of the particles in each sample. Sample A: The average kinetic energy of the particles in Sample A is 18000000000000000000000000000000 [truncated]
The image shows two jars filled with green balls, which represent the particles in the gas. The jars filled with the same number of particles, and the balls are of the same color. The image also provides information about the average kinetic energy of the particles in each jar.
QVLM W8A8 Response:
To determine which jar has the higher temperature, we need to compare the average kinetic energy of the particles in each jar. If the particles jar A have higher average kinetic energy. then jar A has higher temperature. If the particles in jar B have higher average kinetic energy, then jar B has a higher temperature.
However, the image does not provide enough information to determine the average kinetic energy of the particles each jar. Therefore, we cannot confidently determine which jar has the higher temperature.
QASVD W8A8 Response: The diagrams below show two pure samples gas in identical closed, rigid containers. Each colored ball represents one gas particle. Both samples have the same number of particles. 2. Compare the average kinetic energies of the particles in each sample. Which sample has the higher temperature? A. sample B B. neither; the samples have the same temperature C. sample
QSVD W8A8 Response: 1. The diagrams below show two pure samples of gas in identical closed, rigid containers. Each colored ball represents one gas particle. Both samples have the same number of particles. 2. Compare the average kinetic energies of the particles in each sample. Which sample has the higher temperature?
A. sample B B. neither; the samples have the same temperature C. sample A
QSVD W4A4 Response: A. sample
Note that QASVD and QVLM W4A4 output nothing or random number.
27

---

Case 2: LLaVA-v1 .5-
The diagrams below show two pure samples of gas in identical closed, rigid containers. Each colored ball represents one gas particle. Both samples have the same number of Compare the average kinetic energies of the particles in each sample. Which sample has the higher temperature?
Question:
particles.
A. sample B
B. neither; the samples have the same temperature
C. sample A
Sample
Sarrpt 8
saridle FP16 Response: The average kinetic energy of the particles in sample A higher than that of sample B. This is because the particles in sample A are moving faster than those in sample B. The higher average kinetic energy in sample indicates that the temperature of the gas in sample A is higher than that of the gas in sample B.
QVLM W8A8 Response: The average kinetic energy the particles in each sample can be determined by calculating the average kinetic energy of each colored ball. The higher the average kinetic energy, the higher the temperature of the sample.
In the image, there are 28 colored balls in each sample. The average kinetic energy of the
particles in sample calculated follows:
(1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28)
(1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28)
(1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28)
(1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28) (1/28)
(1/28) (1/28) (1/
QASVD W8A8 Response: The average kinetic energy of the particles sample A is higher
than that of sample B. This means that sample A has higher temperature than sample B.
QSVD W8A8 Response: The diagrams show two samples of gas, each in closed, rigid container. The containers have the same number of colored balls, which represent the gas particles. The samples have different average kinetic energies, which can be determined by comparing the average kinetic energy of the particles in each sample.
Sample A has higher average kinetic energy than Sample B. This is because the balls in Sample are moving faster than the balls in Sample B. The average kinetic energy of the particles in Sample A is higher than that of Sample B, which means that Sample A has higher temperature. QSVD W4A4 Response: The average kinetic energy of the particles in sample A is higher than that of sample B.
Note that QASVD and QVLM W4A4 output nothing or random number.
28

---

Case 3: LLaVA-v1.5-1
Question: What is the capital of Louisiana?
A. Atlanta
B. Frankfort
C. New Orleans
D. Baton Rouge
FP16 Response: D. Baton Rouge
QVLM W8A8 Response: C. New Orleans
QVLM W4A4 Response:
C. New Orleans
QASVD W8A8 Response: New Orleans
QSVD W8A8 Response: D. Baton Rouge
QSVD W4A4 Response: The capital of Louisiana is D. Baton Rouge.
Note that QASVD W4A4 output random number.
Case 4: LLaVA -1.VAAvv..5 v1.5-13b
Question: Figure: Kermadec Are. The Kermadec Arc is deep sea ecosystem in the southern Pacific Ocean. Which better describes the Kermadee Arc ecosystem?
A. It has bright sunlight. It also has organisms that crawl or stick to the ground. B. It has water at the bottom of the ocean. It also has sunlight.
FP16 Response: B. It has water at the bottom of the ocean. also has no sunlight.
QVLM W8A8 Response: B. has water at the bottom the ocean. It also has no sunlight..
QVLM W4A4 Response: B. has water the bottom of the ocean. It also has sunlight..
QASVD W8A8 Response: B. It has water at the bottom of the ocean. It also has no sunlight.. The Kermadec Are deep sea ecosystem in the southern Pacific Ocean. Which better describes the Kermadec Arc ecosystem? B. It has water at the bottom of the ocean. also has no sunlight.
QASVD W4A4 Response:
QSVD W8A8 Response: B. has water the bottom the ocean. also has no sunlight. QSVD W4A4 Response: B. It has water at the bottom of the ocean. It also has no sunlight..
29

---

Case 5: LLLVA-v1..--1
The model below represents silver. Silver is precious metal that is used to make many items, such as jewelry and coins. Complete the statement. Silver is ().
Question:
A. an elementary substance
B. a compound
FP16 Response: A, an elementary substance
A. an elementary substance QVLM W4A4 Response: Silver is precious metal that is used to make many items, such as jewelry and coins. QASVD W8A8
QVLM W8A8 Response:
Response: A. elementary substance
QSVD W8A8 Response:
A. an elementary substance
QSVD W4A4 Response: A. an elementary substance
Note that QASVD W4A4 output a random number.
30

---

Case 6: LLaVA-v1 1W1.-vv1.--13-00
The images below show two pairs of magnets. The magnets in different pairs do affect each other. All the magnets shown are made of the same material. Think about the magnetic force between the magnets in each pair. Which of the following statements is true?
Question:
A. The magnetic force is stronger Pair 1.
B. The strength of the magnetic force is the same in both pairs.
C. The magnetic force is stronger Pair 2. Par
Fuir
FP16 Response: 1. The magnetic force is stronger in Pair 1.
In the image, there are two pairs of magnets, each consisting of two magnets. The magnets in different pairs do not affect each other, and all the magnets are made of the same material. The statement "The magnetic force stronger in Pair suggests that the magnetic force between the magnets in Pair is stronger than the magnetic force between the magnets in Pair 2. This could be due to various factors such as the orientation, size, or shape the magnets each pair. However, without more information clear visual comparison, is not possible to definitively determine which pair has stronger magnetic force.
QVLM W8A8 Response: The magnetic force is stronger in Pair 2. QVLM W4A4 Response: 1. The magnetic force is stronger in Pair 1. QASVD W8A8 Response: The magnetic force is stronger in Pair 2.
QSVD W8A8 Response: 1. The magnetic force is stronger in Pair 1. In the image, there are two pairs of magnets, each consisting of three magnets. The magnets in each pair are arranged in a straight line, and they are made of the same material. The magnetic force between the magnets each pair is determined by their strength, size, and arrangement. Since the magnets in Pair I are larger and more closely spaced, the magnetic force between them likely to be stronger than the force between the magnets Pair 2. QSVD W4A4 Response: The magnetic force is stronger in Pair 1.
Note that QASVD W4A4 output random number.
Case 7: LLaVA vv. 1..... 5-b
Question: Which ocean highlighted?
A. the Indian Ocean
B. the Atlantic Ocean
C. the Pacific Ocean
D. the Southern Ocean
FP16 Response: D. the Southern Ocean
QVLM W8A8 Response: D. the Southern Ocean
QVLM W4A4 Response:
the Southern Ocean
QASVD W8A8 Response: C. the Pacific Ocean
QASVD W4A4 Response: 2. C. the Pacific Ocean
QSVD W8A8 Response: D. the Southern Ocean
QSVD W4A4 Response: B. the Atlantic Ocean
31

---

Case 8: -LaVA-v1.5--1
This organism is Fromia monilis. It is member of the animal kingdom. Fromia monilis is commonly called tile sea star. Did you notice that this tile sea star missing one of its arms? If a tile sea star loses an arm, it can regrow the arm over time. This ability is called regeneration. Can Fromia monilis cells make their own food?
Question:
A. yes B. no
FP16 Response: 1. Fromia monilis is type of star, also known tile star. 2. The star is missing one of its arms, which can regrow over time. Fromia monilis cells cannot make their own food, as they not capable of photosynthesis.
QVLM W8A8 Response: 1. Fromia monilis type of star, also known tile star. The sea star is missing one of its arms, which can regrow over time. 3. Fromia monilis cells cannot make their own food, as they are not capable of photosynthesis.
QVLM W4A4 Response: 1. Fromia monilis is type of sea star that belongs to the animal kingdom. Fromia monilis is commonly known as a tile sea star. The tile sea
missing one of its arms. 4. Fromia monilis can regrow its arms over time, process called regeneration. Fromia monilis cells can make their own food.
QASVD W8A8 Response: Fromia monilis is a type of sea star, which is an invertebrate. Invertebrates are animals that do have backbone. They are typically found aquatic environments, such oceans and seas.
Sea stars, like Fromia monilis, are known for their ability to regrow lost body parts, such as arms. This process is called regeneration.
Sea stars are not capable of making their own food, as they are primarily herbivores. They feed on various types of algac and other plant materials found in their environment.
QASVD W4A4 Response: A. Yes
QSVD W8A8 Response: Fromia monilis is a type of sea star, also known as a tile sea star. The star is missing of its arms, which regrow over time. 3. Fromia monilis cells cannot make their own food, as they are not capable photosynthesis.
QSVD W4A4 Response: Fromia monilis is type of sea that commonly found in the ocean. It is a member of the animal kingdom, and commonly known as a tile sea star. Fromia monilis has the ability regenerate lost body parts, such regeneration.ie In terms of whether Fromia monilis cells can make their own food, the answer is no. Fromia monilis carnivorous organism, which means relies other organisms for food. It feeds on small marine animals such as crabs, clams, and other invertebrates.
This ability is called
32


Reviews:
[{"review_id": "T1Bc6COj2Z", "reviewer": "Reviewer_Ue5o", "summary": "This paper proposes QSVD, a unified framework for compressing Vision-Language Models by applying joint Singular Value Decomposition to the combined query, key, and value weight matrices, alongside post-training quantization of weights and activations. The goal is to reduce memory, computational cost, and KV cache size without degrading model accuracy.", "questions": "- Can we replace post-training quantization by quantization-aware training and further improve the performance?\n- What about the training time? Does QSVD add heavy additional cost compared with normal training?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper proposes a novel compression paradigm that effectively combines SVD and PTQ.\n- The buffered mechanism and auto-regressive decoding work well together.\n- The figures well demonstrate the method design.\n- Extensive experiments on popular VLMs show superior performance over existing works.\n\nWeaknesses:\n- The method's effectiveness is quite sensitive to $\\beta$.  And $\\beta$ is determined by a very small calibration set. I'm not sure if the generalization ability of such pattern is enough. \n- The method can only be used on QKV matrices and can not be used on other layers like FC layers, restricting its compression capacity.", "was_revised": true, "final_justification": "See comments", "rebuttal": "Thank you for your insightful comments. Below, we summarize the weaknesses and questions raised in your review and provide detailed responses to each.\n\n**Q1: The method's effectiveness is quite sensitive to $\\beta$ . And $\\beta$ is determined by a very small calibration set. I'm not sure if the generalization ability of such pattern is enough.**\n\n**Ans:** We explored the beta learning experiments on different sample size of 64, 128, 256 and 512. Furthermore, to show the effectiveness of beta learning, we also show the QSVD performances when fixing beta to 0.2, 0.4, 0.8 and 1. The results for W4A4 and W8A4 are shown in table below:\n| LLaVA v1.5 7B |  |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  |$\\beta$ | QSVD| $\\beta$=0.2 | $\\beta$=0.4 | $\\beta$=0.8 | $\\beta$=1 |\n| n=64 | W4A4 | **52.31** | 48.44 | 42.24 | 39.32 | 30.05 |\n|  | W8A4 | **64.46** | 63.36 | 61.38 | 58.40 | 49.43 |\n| n=128 | W4A4 | **46.85** | 42.54 | 35.30 | 36.76 | 30.05 |\n|  | W8A4 | **65.74** | 62.02 | 63.96 | 61.38 | 47.79 |\n| n=256 (QSVD default setting) | W4A4 | **55.16** | 54.79 | 42.37 | 49.22 | 40.45 |\n|  | W8A4 | **65.61** | 64.06 | 64.35 | 60.27 | 58.58 |\n| n=512 | W4A4 | **41.65** | 33.12 | 26.56 | 17.09 | 10.55 |\n|  | W8A4 | **63.91** | 62.17 | 63.71 | 59.44 | 13.93 |\n\nThe table shows that the selection of $\\beta$ matches in different sample size settings, and this learning $\\beta$ method generalizes well over different calibration sizes.\n\nMoreover, it is standard in post-training quantization and low-rank compression literature to use small calibration sets to estimate sensitive hyperparameters. For example: AWQ [1] Section 5.3 Figure 8(a) use 8-256 samples to determine the scaling factor exponent. ASVD [2] Section 4.1 use only 64 samples of calibration dataset in all their experiments and hyperparameter search.\n\n[1] Lin, Ji, et al. \"Awq: Activation-aware weight quantization for on-device llm compression and acceleration.\" Proceedings of machine learning and systems 6 (2024): 87-100.\n\n[2] Yuan, Zhihang, et al. \"Asvd: Activation-aware singular value decomposition for compressing large language models.\" arXiv preprint arXiv:2312.05821 (2023).\n\n**Q2: The method can only be used on QKV matrices and can not be used on other layers like FC layers, restricting its compression capacity.**\n\n**Ans:** As noted in earlier works [1-4], self attention layers, particularly the KV vectors, contribute significantly to the overall latency, energy consumption, and memory usage of large models, and can severely limit inference speed. Therefore, optimizing the QKV matrices using singular value decomposition is critically important and can lead to substantial reductions in latency for large language models.\n\nIn addition, as mentioned in Line 315 of our paper, quantization is applied to the FC layers within the feedforward network (FFN), as well as the output FC layers within the self attention mechanism. While SVD can also be applied to these FC layers, we find them to be highly sensitive to such decomposition. Applying quantization alone is sufficient to compress these layers effectively. This design choice aligns with prior work [5, 6]. Future research may explore more advanced strategies to further optimize the FFN components.\n\n[1] Hooper, Coleman, et al. \"Kvquant: Towards 10 million context length llm inference with kv cache quantization.\" *Advances in Neural Information Processing Systems* 37 (2024): 1270-1303.\n\n[2] Kim, Minsu, et al. \"Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization.\" *Proceedings of the 52nd Annual International Symposium on Computer Architecture*. 2025.\n\n[3] Kwon, Woosuk, et al. \"Efficient memory management for large language model serving with pagedattention.\" *Proceedings of the 29th symposium on operating systems principles*. 2023.\n\n[4] Zhang, Zhenyu, et al. \"H2o: Heavy-hitter oracle for efficient generative inference of large language models.\" *Advances in Neural Information Processing Systems* 36 (2023): 34661-34710.\n\n[5] Yuan, Zhihang, et al. \"Asvd: Activation-aware singular value decomposition for compressing large language models.\" *arXiv preprint arXiv:2312.05821* (2023).\n\n[6] Chang, Chi-Chih, et al. \"xkv: Cross-layer svd for kv-cache compression.\" *arXiv preprint arXiv:2503.18893* (2025).\n\n[7] Chang, Chi-Chih, et al. \"Palu: Compressing kv-cache with low-rank projection.\" *arXiv preprint arXiv:2407.21118* (2024).\n\n**Q3: Can we replace post-training quantization by quantization-aware training and further improve the performance?**\n\n**Ans:** Yes, incorporating quantization-aware training (QAT) has the potential to further improve quantization performance by allowing model weights to adapt to low-precision constraints during training. However, this approach incurs significant training cost, as backpropagation must be performed across the entire vision-language model, resulting in substantial computational overhead and increased training latency. In contrast, our primary goal in this work is to develop a highly effective **post-training quantization (PTQ)** method that is both efficient and practical.\n\nSpinQuant [1] demonstrates that end-to-end QAT can yield superior results compared to QuaRot under aggressive settings (e.g., W4A4), achieving up to **28.6-percent improvements** (see SpinQuant Table 5, Section 4.3.4). While impressive, this comes at the cost of additional training and online adaptation. Our method, like QuaRot, is **fully PTQ and no E2E training**, and we believe that with modest QAT integration (e.g., low-rank-aware finetuning or selective stage-wise updates), **QSVD has the potential have an even better performance**, which we leave as promising future work.\n\n[1] Liu, Zechun, et al. \"Spinquant: Llm quantization with learned rotations.\" arXiv preprint arXiv:2405.16406 (2024).\n\n**Q4: What about the training time? Does QSVD add heavy additional cost compared with normal training?**\n\n**Ans:** Thank you for the question. QSVD is designed as a post-training compression method and does not require full model retraining. The additional computation involved comes from three components: (1) low-rank SVD decomposition of the QKV matrices, (2) importance score–based rank allocation, and (3) calibration-based tuning of the quantization parameter $\\beta$ and the quantization process. The latency overhead introduced by QSVD is negligible compared to standard training procedures. Furthermore, it introduces no additional cost during inference.\n\nTo study the detailed training cost, QSVD introduces only a lightweight calibration and tuning stage, rather than full fine-tuning. For a fair comparison, we benchmarked the complete QSVD pipeline for LLaVA-v1.5-13B on a single A100 80G GPU, using the same 256-sample calibration set as described in the paper. The total time is approximately 96 minutes. The detailed breakdown is:\n\n| Step | Time |\n| --- | --- |\n| Input calibration | 1 min |\n| SVD (on all layers) | 2 min 15 s |\n| Gradient collection & rank allocation | 10 min 12 s |\n| SVD results Fusion | 30 s |\n| $\\beta$ tuning & quantization (fake quant) | 82 min |\n| Total Calibration Time | **96 min** |\n\nIn comparison, the typical training cost of a vision-language model is approximately **204 GPU hours** [1] on A100, making the computational overhead of QSVD negligible relative to full model training.\n\n[1] Liu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024."}, {"review_id": "oC7iwJTuLY", "reviewer": "Reviewer_bUxq", "summary": "This paper introduces a QSVD method that combines quantization and SVD. It cocnats QKV weight matrix together and performs a low-rank SVD on it. Then it introduces a cross‑layer, importance‑based singular‑value ranking scheme for adaptive rank allocation. This method can reduce the GPU memory usage and computation cost, and experiments on five VLMs across ScienceQA, VizWiz, and SEED‑Bench demonstrate that QSVD outperforms SVD‑only and quantization‑only baselines.", "questions": "1. Analyze why we can get better results from SVD on concated QKV. Either theory or experiment is encouraged.\n2. What is the beta obtained on the calibration dataset? Can authors study the beta values on different datasets? This ablation is meaningful for practical applications.\n3. Experiment on GPU memory usage between different methods is encouraged.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "1. The joint QKV SVD formulation is clearly derived, with precise comparisons against single matrix SVD and analysis of parameter/FLOPs/cache reductions. The results support the method proposed.\n2. The figures are intuitive. A small drawback is that the order of the sub-graphs is inconsistent with the order in the text. 2. The parameter beta is only used in Quantization, so it’s confusing in Section 3.1 without explanation.\n3. QSVD works on different models and datasets, but why do we need to concatenate the three matrices together? Is it just to reduce cache overhead? Or is there a deeper reason for this design? A deeper analysis is encouraged. \n4. While prior work apply SVD to Q, K, V separately, this paper’s joint factorization over the concatenated matrix is novel (despite the lack of a deeper motivation)   \n5. The knockout approach to measuring the importance of singular values is novel.", "was_revised": true, "final_justification": "Post rebuttal, my concerns about method explanations, ablation experiments and GPU consumption have been addressed. Thus I am leaning to accept the paper.", "rebuttal": "Thank you for your insightful comments. We have summarized weakness and questions raised in your review below and provided detailed responses to each.\n\n**Q1: The figures are intuitive. A small drawback is that the order of the sub-graphs is inconsistent with the order in the text.**\n\n**Ans:** We will clarify the figure in the final version of the paper.  And please see **Reviewer** **HD9e Q3** \n\n\n**Q2: The parameter beta is only used in Quantization, so it’s confusing in Section 3.1 without explanation.**\n\n**Ans:** The parameter $\\beta$, although introduced earlier in Section 3.1, does not affect the outcome of the SVD itself. Specifically, when SVD is applied to a weight matrix $W$, the process can be expressed as:\n\n$W = U \\Sigma V^T = (U \\Sigma^\\beta)(\\Sigma^{1-\\beta} V^T) = W_{d}W_{u}$\n\nThe choice of $\\beta$ does not alter the reconstruction result of the SVD. We note this here to support the discussion of $\\beta$ in Section 3.3.\n\nHowever, once quantization is introduced, the value of $\\beta$ becomes impactful on the final accuracy result, as it influences the magnitudes of $W_d$, $W_u$, and the intermediate product $C_{qkv} = XW_d$, and their corresponding quantized representations. Therefore, learning an appropriate value for $\\beta$ is crucial for optimizing quantization performance.\n\n**Q3: QSVD works on different models and datasets, but why do we need to concatenate the three matrices together? Is it just to reduce cache overhead? Or is there a deeper reason for this design? Analyze why we can get better results from SVD on concated QKV. Either theory or experiment is encouraged.**\n\n**Ans:** In addition to reducing the KV cache size, another potential advantage of applying SVD to the concatenated QKV weights is the improved flexibility in assigning singular values compared to applying SVD to each of the Q, K, and V weights separately. For example, given a total rank budget of 9, performing SVD on Q, K, and V individually would require a fixed allocation (such as 3, 3, and 3), which may result in truncating many important singular values and lead to suboptimal performance. In contrast, applying SVD to the concatenated QKV weights allows for a global comparison of singular values across all three matrices. This enables more adaptive and effective rank allocation under the same total budget, potentially yielding better overall performance.\n\nTo illustrate the advantages of QKV concatenation over separate QKV decomposition for SVD, we provide a quantitative comparison below, summarized from Table 1 and Table 3 of our paper. The comparison uses LLaVA-v1.5-13B under different preserved parameters across both settings, the definitions of $R_{1}$ and $R_{2}$ follow equation 15 in the paper.\n\n| Source | SVD Type | Acc. | Hw cost | Acc. | Hw cost | Acc. | Hw cost | Acc. | Hw cost |\n|--------|----------|------|-----------------------------|------|-----------------------------|------|-----------------------------|------|-----------------------------|\n| From Table 1, LLaVA-v1.5 13B, ASVD | Separately SVD | 64.70% | R1: 63.30%; R2: 22.50% | 56.92% | R1: 60.00%; R2: 20.00% | 46.50% | R1: 56.70%; R2: 17.50% | 42.79% | R1: 53.30%; R2: 15.00% |\n| From Table 3, LLaVA-v1.5 13B, UR | Joint SVD       | 71.84% | R1: 60.00%; R2: 22.50%  | 70.40% | R1: 53.30%; R2: 20.00% | 70.40% | R1: 46.70%; R2: 17.50% | 67.72% | R1: 40.00%; R2: 15.00%  |\n\nAs the table shows, joint SVD improves accuracy by more than **7.14%** under the same compression budget. This trend holds consistently across various preserved ratios, R1 and R2 are defined in Equation 15.\n\nFinally, concatenating the Q, K, and V weights allows for a shared down projection, which significantly reduces the KV cache size during inference. As shown in Figure 2 and discussed in Section 3.1, our design stores a single intermediate matrix $C_{{qkv}}$ instead of computing and caching separate key and value projections for each $W_{{k}}$ and $W_{{v}}$. This results in approximately a twofold reduction in KV cache size compared to traditional approaches.\n\n**Q4: What is the beta obtained on the calibration dataset? Can authors study the beta values on different datasets? This ablation is meaningful for practical applications.**\n\n**Ans:**  In our paper. We follow the convention of QVLM [1] to use samples from ScienceQA training set as our calibration dataset, and the resulting beta is around 0.12. In contrast, MBQ [2] uses the COCO [3] dataset for its calibration process. To align with this, we extended our beta learning experiments using COCO data and observed a similar optimal beta value of approximately 0.10. This consistency indicates that beta remains stable across different calibration datasets, which significantly enhances the practicality of QSVD, since a beta value of 0.1 can be reliably applied across various datasets.\n\n[1] Wang, Changyuan, et al. \"Q-vlm: Post-training quantization for large vision-language models.\" NeurIPS, 2024.\n\n[2] Li, Shiyao, et al. \"Mbq: Modality-balanced quantization for large vision-language models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[3] Lin, Tsung-Yi, et al. \"Microsoft coco: Common objects in context.\" European conference on computer vision. Cham: Springer International Publishing, 2014.\n\n**Q5: Experiment on GPU memory usage between different methods is encouraged.** \n\n**Ans:** To provide a fair and concrete comparison with standard PTQ methods in memory cost, we conducted an experiment on one transformer layer of LLaVA-v1.5-13B, focusing on the decoding stage. Specifically, we compare the following approaches:\n\n- **W8A8 PTQ:** Linear layers, KV cache and attention use standard INT8 quantization.\n\n- **QSVD-8bit:** Low-rank decomposition with int8 quantization, using our fused attention kernel optimized for compressed latent cache structure.\n\nFor the W8A8 PTQ baseline, we adopt the per-tensor RTN (round to nearest) quantization scheme, one of the simplest and most memory-efficient PTQ strategies, with no additional quantization parameters. This sets a conservative lower bound on memory usage. Most other recent PTQ methods introduce additional per-channel scales, zero-points, or outlier handling, which increase memory usage beyond RTN.\n\nWe measured the peak memory usage in MB during decoding under varying sequence lengths (1024 and 2048) and batch sizes (16, 32, 160). The results are summarized below (Peak Memory Usage in MB):\n\n| Seq Len: 1024 | Memory  | Usage | (MB) |\n| --- | --- | --- | --- |\n| Batch Size | 16 | 32 | 160 |\n| W8A8 | 702.76 | 944.18 | 2875.52 |\n| QSVD-8bit | **545.97** | **633.14** | **1335.66**|\n\n| Seq Len: 2048 | Memory  | Usage | (MB) |\n| --- | --- | --- | --- |\n| Batch Size | 16 | 32 | 160 |\n| W8A8 | 943.51 | 1424.93 | 5276.27 |\n| QSVD-8bit | **626.97** | **824.12** | **2275.41** |\n\nAs shown, under the same 8-bit precision, QSVD achieves significantly lower peak memory usage than standard W8A8 PTQ, especially under longer sequence lengths and larger batch sizes."}, {"review_id": "xozz1myALj", "reviewer": "Reviewer_3Q9H", "summary": "This paper proposes QSVD, a framework that leverages joint Singular Value Decomposition (SVD) of QKV weight matrices and quantization to reduce computational overhead and KV cache size in Vision-Language Models (VLMs). Key innovations include:  \n1. Joint QKV-SVD: Concatenating Q/K/V weight matrices for unified low-rank decomposition, reducing parameters and computation\n\n2. Dynamic Rank Allocation: Singular value truncation based on impact on VLM accuracy (Eq. 10).  \n\n\n\n3. Low-Rank Activation Quantization: Orthogonal transformations to suppress activation outliers (Eq. 11-12).  \n\nExperiments show QSVD outperforms baselines (ASVD, SVD-LLM) on LLaVA and SmolVLM, reducing KV cache to 9.38% in W8A4 quantization with <1% accuracy drop on ScienceQA.", "questions": "1. Does the O(E²) cost of Eq. 10 affect real-time performance?  \n\n2. How does r selection adapt to long sequences (e.g., high-res images)?  \n\n3. Could visual-textual feature misalignment hinder joint SVD?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**  \n- Efficiency: Joint SVD+quantization reduces KV cache (η<sub>qsvd</sub> = *rL* vs. η<sub>fp</sub> = *2LE*) and computation (FLOPs ↓ when *r* < 0.75*E*).\n- Novel Ranking Strategy: Global singular value importance scoring (Eq. 10) outperforms uniform/Fisher-based allocation (Table 3).  \n- Strong Validation: Tested on 5 VLMs and 3 benchmarks with W4A4 robustness (Table 2).  \n\n**Weaknesses:**  \n- Attention-Layer Limitation: No optimization for FFN layers (Sec. 5).", "was_revised": true, "final_justification": "5", "rebuttal": "Thank you for your insightful comments. Below, we summarize the key concerns and questions raised in your review and provide detailed responses to each point.\n\n**Q1: Attention-Layer Limitation: No optimization for FFN layers (Sec. 5).**\n\n**Ans:** As noted in earlier works [1-4], self attention layers, especially the key and value (KV) vectors, contribute significantly to the overall latency, energy consumption, and memory usage of large models and can severely limit inference speed. Therefore, optimizing the QKV matrices using SVD is critically important and can lead to substantial reductions in latency for large language models.\n\nIn addition, as noted in Line 315 of the paper, quantization is applied to the FFN layers and already yields significant computational savings. When applying SVD to the FFN layers, we observed that FFN are highly sensitive to SVD, so we chose to use quantization alone, consistent with the observation of prior works [5–7]. Future research can further explore methods to better optimize the FFN layers.\n\n\n[1] Hooper, Coleman, et al. \"Kvquant: Towards 10 million context length llm inference with kv cache quantization.\" *Advances in Neural Information Processing Systems* 37 (2024): 1270-1303.\n\n[2] Kim, Minsu, et al. \"Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization.\" *Proceedings of the 52nd Annual International Symposium on Computer Architecture*. 2025.\n\n[3] Kwon, Woosuk, et al. \"Efficient memory management for large language model serving with pagedattention.\" *Proceedings of the 29th symposium on operating systems principles*. 2023.\n\n[4] Zhang, Zhenyu, et al. \"H2o: Heavy-hitter oracle for efficient generative inference of large language models.\" *Advances in Neural Information Processing Systems* 36 (2023): 34661-34710.\n\n[5] Yuan, Zhihang, et al. \"Asvd: Activation-aware singular value decomposition for compressing large language models.\" *arXiv preprint arXiv:2312.05821* (2023).\n\n[6] Chang, Chi-Chih, et al. \"xkv: Cross-layer svd for kv-cache compression.\" *arXiv preprint arXiv:2503.18893* (2025).\n\n[7] Chang, Chi-Chih, et al. \"Palu: Compressing kv-cache with low-rank projection.\" *arXiv preprint arXiv:2407.21118* (2024).\n\n**Q2: Does the $\\mathcal{O}(E^2)$ cost of Eq. 10 affect real-time performance?**\n\n**Ans:** Thank you for the question. The $\\mathcal{O}(E^2)$ cost in Equation 10 only applies during the **offline calibration phase**, where we compute importance scores to allocate ranks using a small calibration dataset. We benchmarked the complete QSVD pipeline for LLaVA-v1.5-13B on a single A100 80G GPU, using the same 256-sample calibration set as described in the paper. The total time is approximately 96 minutes. This is much shorter than the normal training time (204 GPU hours for LLaVA-v1.5-13B [1]). QSVD generates low-rank and quantized weights, which are stored and efficiently loaded during inference, in line with approaches such as AWQ [2] and GPTQ [3]. There is no need to recompute any part of the calibration during inference, making QSVD a one-time lightweight process with minimal runtime overhead.\n\n[1] Liu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[2] Lin, Ji, et al. \"Awq: Activation-aware weight quantization for on-device llm compression and acceleration.\" Proceedings of machine learning and systems 6 (2024): 87-100.\n\n[3] Frantar, Elias, et al. \"Gptq: Accurate post-training quantization for generative pre-trained transformers.\" arXiv preprint arXiv:2210.17323 (2022).\n\n**Q3: How does r selection adapt to long sequences (e.g., high-res images)?**\n\n**Ans:** Thank you for the thoughtful comment. To evaluate the adaptability of our rank selection method to long sequences, we conduct experiments on the HRBench4K dataset [1], which consists of 4K resolution images, and use VLMEvalKit [2] for evaluation. We follow the same setup as the original paper and report the “Average All” accuracy metric in HRBench4K. Both LLaVA-v1.6-13B and LLaVA-v1.5-13B are evaluated under the QSVD-noQ setting and compared with ASVD and FP16 baselines. The results are shown below, where R1 and R2 follows the same definition as Equation 15 in the paper:\n\n**LLaVA-v1.6-13B**\n\n| Method     | Acc.   | Hw cost                    |   | Acc.   | Hw cost                   |    | Acc.   | Hw cost                     |  | Acc.   | Hw cost                       |\n|------------|--------|-----------------------------|--|--------|-----------------------------|--|--------|------------------------------|-|--------|-------------------------------|\n| ASVD       | 44.38% | R1: 63.30%; R2: 22.50%      |  | 44.12% | R1: 60.00%; R2: 20.00%       | | 43.12% | R1: 56.70%; R2: 17.50%     |   | 42.62% | R1: 53.30%; R2: 15.00%        ||\n| QSVD-noQ   | **44.88%** | R1: 60.00%; R2: 22.50%      |  | **44.12%** | R1: 53.30%; R2: 20.00%      |  | **43.88%** | R1: 46.70%; R2: 17.50%       | | **44.12%** | R1: 40.00%; R2: 15.00%        |\n| FP16       | 45.63% |                              |      |                             |       |                              |       |                              |\n\n**LLaVA-v1.5-13B**\n\n| Method     | Acc.   | Hw cost                      | | Acc.   | Hw cost                   |    | Acc.   | Hw cost                   |    | Acc.   | Hw cost                       |\n|------------|--------|-----------------------------|--|--------|------------------------------|-|--------|-------------------------------|----|----|-------------------------------|\n| ASVD       | 39.12% | R1: 63.30%; R2: 22.50%      |  | 38.62% | R1: 60.00%; R2: 20.00%      |  | 36.62% | R1: 56.70%; R2: 17.50%       | | 37.25% | R1: 53.30%; R2: 15.00%     |   |\n| QSVD-noQ   | **39.88%** | R1: 60.00%; R2: 22.50%    |   | **38.75%** | R1: 53.30%; R2: 20.00%      |  | **39.00%** | R1: 46.70%; R2: 17.50%  |  | **39.00%** | R1: 40.00%; R2: 15.00%      |  |\n| FP16       | 39.12% |                              |       |                              |       |                              |       |                              |\n\nAs shown above, QSVD-noQ outperforms ASVD, and the overall trends observed on HRBench4K are consistent with those on ScienceQA-IMG and VizWiz (see Table 1 in our paper). This indicates that our rank selection method generalizes well to long sequences resulting from high-resolution images.\n\n[1] Wang, Wenbin, et al. \"Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 8. 2025.\n\n[2] Duan, Haodong, et al. \"Vlmevalkit: An open-source toolkit for evaluating large multi-modality models.\" Proceedings of the 32nd ACM international conference on multimedia. 2024.\n\n**Q4: Could visual-textual feature misalignment hinder joint SVD?**\n\n**Ans:** Thank you for this insightful question. Visual-textual feature misalignment is a well-known challenge in VLMs and can significantly impact overall model accuracy. However, it does not directly hinder the effectiveness of our proposed joint SVD approach in QSVD, as QSVD is designed to maintain the original VLM's performance even under feature misalignment, while simultaneously enhancing computational efficiency. Furthermore, by compressing Q, K, and V jointly, QSVD inherently smooths parameter variations across modalities. This may even help mitigate alignment-induced instability by regularizing the QKV projection layer and encouraging a more cohesive representation space. Additionally, the evaluation in Section 4 shows consistent performance across diverse VLMs (e.g., SmolVLM, LLaVA) and datasets (e.g., ScienceQA, VizWiz), despite differences in visual-textual alignment quality. This suggests that joint SVD is robust to potential modality misalignments at the embedding level. \n\nFinally, we simulate the effect of feature misalignment by injecting Gaussian noise into the visual features generated by the visual encoder. Specifically, we perturb the output of the projection layer with Gaussian noise at varying levels, using a standard deviation of 0.1. The corrupted visual features are then concatenated with the textual embeddings and passed through the subsequent layers for processing. Following the experimental setup in the main paper (Table 4), we use the LLaVA-v1.5-7B model and evaluate performance on the ScienceQA benchmark.\n\nThe results, shown in the table below, indicate that QSVD exhibits a certain degree of robustness to feature misalignment, performing noticeably better than ASVD under the same conditions. This observation suggests the potential resilience of joint SVD to modality noise, which warrants further investigation in future work.\n\n\n| Setting            | FP16-ASVD | FP16-QSVD-noQ | W4A4-QSVD |\n|--------------------|-----------|---------------|-----------|\n| W/o noise in feature | **66.34** | **68.46**      | **55.16** |\n| W/ noise in feature  | 46.24     | 50.02          | 48.29     |"}, {"review_id": "Uw1u2SUytf", "reviewer": "Reviewer_HD9e", "summary": "This paper proposes an efficient method for compressing Vision-Language Models (VLMs) by applying Singular Value Decomposition (SVD) to the QKV weight matrices, reducing KV cache size and computation. A dynamic rank allocation strategy is introduced to balance compression and accuracy. The approach is further enhanced with quantization of both weights and activations. Experiments show that the method achieves better accuracy–efficiency trade-offs than prior work, making it suitable for real-time, resource-constrained settings.", "questions": "See Weakness", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper is well-structured and easy to follow.\n2.\tThe experiments and ablations are thorough and clearly demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n\n1.\tMy main concern is that the method appears to lack VLM-specific design—it primarily targets the QKV weights, which is a general approach applicable to LLMs as well. This raises the question: why is the method not directly evaluated on standard LLM benchmarks and compared against existing LLM compression techniques, if it's not tailored to vision-language settings?\n2.\tThe paper lacks comparisons with state-of-the-art post-training quantization methods in terms of inference time, memory usage, and accuracy.\n3.\tThe figures need to be more refined. For example, Figure 2 alone is hard to interpret—the connections and meanings of the different subplots are unclear.", "was_revised": true, "final_justification": "Thank you for the authors’ response. I feel that most of my concerns have been addressed. However, the explanation of how the proposed method specifically tackles the unique challenges of VLMs remains somewhat unclear. Therefore, I have changed my rating from borderline reject to borderline accept.", "rebuttal": "Thank you for your insightful comments. Below, we summarize the key concerns and questions raised in your review and provide detailed responses to each point.\n\n**Q1: QSVD primarily targets the QKV weights, which is a general approach applicable to LLMs as well. This raises the question: why is the method not directly evaluated on standard LLM benchmarks and compared against existing LLM compression techniques, if it's not tailored to vision-language settings?**\n\n**Ans:** Efficiency enhancement techniques specifically targeting VLMs have not been studied as extensively as those for language models. Given the growing computational demands of VLM deployment, we believe QSVD is both timely and essential. Our joint QKV compression approach directly addresses the unique efficiency challenges of multimodal inference, setting it apart from conventional unimodal compression methods used in language models.\n\nIn addition, whether the proposed techniques can be effectively applied to LLMs requires further investigation. This is because the quantization scheme in QSVD is specifically tailored to the input activation distributions observed in VLMs. As shown in our analysis (Section 3.3, Figure 3), VLM activations display distinct channel-wise outlier patterns. The quantization methods we introduce are designed based on this distribution. Consequently, applying the same techniques to LLMs may not yield comparable effectiveness, as their activation distributions can differ significantly from those of VLMs due to the absence of multimodal integration. We plan to explore this direction in the future.\n\n**Q2: The paper lacks comparisons with state-of-the-art post-training quantization methods in terms of inference time, memory usage, and accuracy.**\n\n**Ans:** In our original manuscript (Table 2), we presented **accuracy comparisons** between QSVD and several leading post-training quantization methods, including Duquant [1] and QVLM [2]. Additionally, we extended our evaluation by comparing QSVD with **more recent PTQ approaches**, including MBQ [3] (targeting VLMs) and AffineQuant (designed for LLMs) [4]. The table below shows the evaluation results in terms of accuracy:\n\n| **Accuracy**     |         | **SQA**                     |                      |                      | **VizWiz**                  |                      |                      |\n|------------------|---------|-----------------------------|----------------------|----------------------|-----------------------------|----------------------|----------------------|\n|                  |         | MBQ                         | AffineQuant          | Ours QSVD            | MBQ                         | AffineQuant          | Ours QSVD            |\n| llava v1.5 7B     | W4A4    | 39.27                       | 46.23                | **55.16**            | 35.61                       | 42.63                | **52.05**            |\n|                  | W8A4    | 54.62                       | 59.45                | **65.61**            | 43.85                       | 46.22                | **52.18**            |\n| llava v1.5 13B    | W4A4    | 50.59                       | 58.55                | **65.82**            | 40.00                       | 45.56                | **56.82**            |\n|                  | W8A4    | 55.51                       | 67.83                | **70.12**            | 48.72                       | 50.25                | **53.20**            |\n| llava v1.6 7B     | W4A4    | 50.69                       | 53.72                | **59.67**            | 40.13                       | 49.90                | **52.00**            |\n|                  | W8A4    | 63.62                       | 65.87                | **66.10**            | 45.25                       | 52.75                | **53.72**            |\n| llava v1.6 13B    | W4A4    | 60.00                       | 61.24                | **63.61**            | 45.00                       | 49.11                | **54.27**            |\n|                  | W8A4    | 63.12                       | 68.97                | **70.43**            | 50.01                       | 56.41                | **58.52**            |\n\nWe observe that QSVD consistently achieves comparable or superior **accuracy**, particularly under aggressive quantization settings such as W8A4 and W4A4.\n\nTo provide a fair and concrete comparison with standard PTQ methods in **memory usage**, we conducted an experiment on one Transformer layer of LLaVA-v1.5-13B, focusing on the decoding stage. Specifically, we compare the following configurations:\n\n- **W8A8 PTQ:** Linear layers, KV cache and attention use standard INT8 quantization.\n\n- **QSVD-8bit:** Low-rank decomposition with int8 quantization, using our fused attention kernel optimized for compressed latent cache structure.\n\nFor the W8A8 PTQ baseline, we adopt the per-tensor RTN (round-to-nearest)  quantization scheme, one of the simplest and most memory-efficient PTQ strategies, with no additional quantization parameters. This sets a conservative lower bound on memory usage. Most other SOTA PTQ methods introduce additional per-channel scales, zero-points, or outlier handling, which increase memory usage and inference time beyond RTN.\n\nWe measured the peak memory usage in MB during decoding under varying sequence lengths (1024 and 2048) and batch sizes (16, 32, 160). The results are summarized below (Peak Memory Usage in MB):\n\n| Seq Len: 1024 | Memory |  Usage | (MB) |\n| --- | --- | --- | --- |\n| Batch Size | 16 | 32 | 160 |\n| W8A8 | 702.76 | 944.18 | 2875.52 |\n| QSVD-8bit | **545.97** | **633.14** | **1335.66** |\n\n| Seq Len: 2048 | Memory  | Usage | (MB) |\n| --- | --- | --- | --- |\n| Batch Size | 16 | 32 | 160 |\n| W8A8 | 943.51 | 1424.93 | 5276.27 |\n| QSVD-8bit | **626.97** | **824.12**| **2275.41** |\n\nAs shown, under the same 8-bit precision, QSVD achieves significantly lower **peak memory usage** than standard W8A8 PTQ, especially under longer sequence lengths and larger batch sizes.\n\nFigure 4 in the paper shows the **inference latency** comparison between QSVD-8bit and W8A8. The same advantage is observed, QSVD-8bit achieves up to $1.4\\times$ speedup over standard W8A8 PTQ during decoding. This improvement is due to both reduced memory traffic, as QSVD compresses the attention cache and lowers key/value memory reads, and fused kernel execution, which eliminates intermediate memory writes and enables more efficient computation by directly operating on the compressed representation.\n\n[1] Lin, Haokun, et al. \"Duquant: Distributing outliers via dual transformation makes stronger quantized llms.\" Advances in Neural Information Processing Systems 37 (2024): 87766-87800.\n\n[2] Wang, Changyuan, et al. \"Q-vlm: Post-training quantization for large vision-language models.\" Advances in Neural Information Processing Systems 37 (2024): 114553-114573.\n\n[3] Li, Shiyao, et al. \"Mbq: Modality-balanced quantization for large vision-language models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[4] Ma, Yuexiao, et al. \"Affinequant: Affine transformation quantization for large language models.\" arXiv preprint arXiv:2403.12544 (2024).\n\n**Q3: The figures need to be more refined. For example, Figure 2 alone is hard to interpret—the connections and meanings of the different subplots are unclear.**\n\n**Ans:** We thank the reviewer for the helpful feedback. To improve clarity and readability, we have clarified the connections and meanings as follows. We will also enhance the quality of Figure 2 in the final version of the paper. Specifically\n\nSubplot **(a)** presents the original QKV architecture in VLM without applying SVD.\n\nSubplot **(b)** illustrates the application of SVD separately to the weight matrices of $Q$ , $K$ and $V$ , where each of $W_{q}$ , $W_{k}$ , and $W_{v}$ is decomposed into a pair of down and up projection matrices.\n\nSubplot **(c)** shows our proposed QKV weight concatenation followed by the SVD process.\n\nSubplot **(d)** depicts the standard KV vectors computation process. At each decoding step, the input $X$ is multiplied with $W_{k}$ and $W_{v}$, and the resulting $C_{k}$ and $C_{v}$ are stored in memory for future use.\n\nSubplot **(e)** illustrates the recomputation process when SVD is applied separately to $W_{k}$ and $W_{v}$. During decoding, the intermediate input $X$ must be read from memory, and the $K$ and $V$ vectors are recomputed by multiplying $X$ with the down-projection matrices $W^{d}_k$  and  $W^{d}_v$, respectively.\n\nSubplot **(f)** demonstrates the storage cost in QSVD. During decoding, since $W_{q}$, $W_{k}$, and $W_{v}$ share a common down-projection matrix $W_{qkv}^{d}$ , the input $X$ is multiplied by $W_{qkv}^{d}$ to produce the intermediate data $C_{qkv}$, which is then stored in memory for later recomputation of the KV vectors."}]

Author rebuttal:

Meta-review: Driven by the high computation cost and memory footprint of vision language models,  the authors propose a method to reduce the KV cash size. The authors suggest to concatenate the Key, Query and Value matrices and perform a Singular Value Decomposition with an efficient rank allocation.

Strengths

All reviewers found this interesting and novel.

The paper demonstrated strong performance.

Through experiments across various models.

Weakness

The main limitation pointed by the reviewers is the applicability to the attention layers only, however the authors showed that the work can be combined with FC quantization.
Decision: Accept (spotlight)