WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models
Authors: Haiyu Wang
Yutong Wang
Jack Jiang
Sai Qian Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Vision language model
singular value decomposition
quantization
Primary area: foundation or frontier models, including LLMs

Abstract: Singular Value Decomposition (SVD) has become an important technique for reducing the computational burden of Vision Language Models (VLMs), which play a central role in tasks such as image captioning and visual question answering. Although multiple prior works have proposed efficient SVD variants to enable low-rank operations, we find that in practice it remains difficult to achieve substantial latency reduction during model execution. To address this limitation, we introduce a new computational pattern and apply SVD at a finer granularity, enabling real and measurable improvements in execution latency. Furthermore, recognizing that weight elements differ in their relative importance, we adaptively allocate relative importance to each element during SVD process to better preserve accuracy, then extend this framework with quantization applied to both weights and activations, resulting in a highly efficient VLM. Collectively, we introduce Weighted SVD (WSVD), which outperforms other approaches by achieving over $1.8\times$ decoding speedup while preserving accuracy. We open source our code at: https://github.com/SAI-Lab-NYU/WSVD.

Published as a conference paper at ICLR 2026
WSVD: WEIGHTED LOW-RANK APPROXIMATION FOR FAST AND EFFICIENT EXECUTION OF LOW-PRECISION MODELS
Haiyu Wangb
Yutong Wang' Jack Jiang Sai Qian Zhang
Tandon School of Engineering, New York University Courant Institute of Mathematical Sciences, New York University (8w3689, yw6594, 94, i jj2513, sai. chang))nyu.es.
ABSTRACT
Singular Value Decomposition (SVD) has become an important technique for reducing the computational burden of Vision Language Models (VLMs), which play a central role in tasks such as image captioning and visual question answer- ing. Although multiple prior works have proposed efficient SVD variants en- able low-rank operations, we find that in practice it remains difficult to achieve substantial latency reduction during model execution. To address this limitation, we introduce a new computational pattern and apply SVD at finer granularity, enabling real and measurable improvements in execution latency, Furthermore, recognizing that weight elements differ in their relative importance, we adap- tively allocate relative importance each element during SVD process to better preserve accuracy, then extend this framework with quantization applied to both weights and activations, resulting highly efficient VLM. Collectively, we in- troduce Weighted SVD (WSVD), which outperforms other approaches by achiev- ing over 1.8x decoding speedup while preserving accuracy. We open source our code at:
com/ SAI Lab NYU/WSVD
INTRODUCTION
Vision-language models (VLMs) represent key frontier in artificial intelligence, they connect visual recognition with natural language comprehension. By jointly processing imagery and text, these models enable diverse applications, including automatic image description (Zhou et al. 2020 Hu et al. 2022 Chen et al. 2022; Dzabraev et al. 2024), visual question answering (Chappuis et al, 2022 Bazi et al. 2023 Wang et al. 2024b)), and semantic search over multimodal data (Li al,
2024b Sun et al. 2025) However, the impressive capabilities of VLMs come at the expense of significant resource demands. The joint encoding of large-scale visual and linguistic inputs requires heavy computation, and the autoregressive generation of tokens further stresses memory bandwidth, introducing major inference bottlenecks.
To reduce the computational cost of large models, low-rank decomposition has recently attracted in-
creasing attention (Wang et al. 2025c Yuan al. 2023b Wang et 2024d Li et al. 2025 2024c]
Chang et al., 2024 Wang et al. 2025a). By factorizing the query (Q), key (K), and value (V) matrices within attenntto blocks into low-rank components, prior work has shown significant reductions in computational complexity and weight storage, thereby improving efficiency. However, based on our practical seste implementation, we observe that applying SVD-based decomposition to the QKV matrices does not consistently yield latency improvements; in fact, it can sometimes incur even higher computational costs for some VLMs.
To investigate this, first evaluate the latency of VLMs and find that the root cause lies in the re- computation of the KV vectors introduced by low-rank factorization, which requires multiple rounds of memory access to the latent data and ultimately increases memory traffic. To overcome this lim- itation, we propose a new computational pattern that applies SVD at a finer granularity, thereby achieving tangible and measurable improvements in execution latency.

---

Published as a conference paper at ICLR 2026
Calibration dataset
Step 1: Fine-grained SVD operation
WSVD Framework
How many dogs in it?
Text
Traegg Oiagg
Step
Quantization- finetuning over low-rank weights
Q: What animal this is?
Embedding
Visual encoder
Step 2:
Weighted finetuning with the importance score n
Quantization
Projection
Language model
min
W
W
Outtier smoothing
(a) Figure 1: (a) Architecture of vision-language model. (b) Overview of WSVD framework.
(b)
Furthermore, building on prior work (Yu et al. 2024b) demonstrating that certain weight elements play critical role VLM accuracy, we note that standard SVD operations treat all weights equally when truncating them for low-rank approximation. To address this, we adaptively allocate relative importance for each weight element during SVD better preserve performance. To further enhance computational efficiency, we apply low-precision quantization to the low-rank VLM and finetune to mitigate accuracy loss. Collectively, these steps yield a low-precision, low-rank VLM with exceptionally low execution latency. Our contributions are summarized as follows:
The WSVD scheme applies SVD separately each attention head, fundamentally reduc- ing memory access and computational cost during the decoding stage, and resulting in significantly lower VLM execution latency compared to prior solutions.
To mitigate the accuracy drop introduced by the per-head SVD scheme, WSVD incor- porates local weighted finetuning, where importance score is assigned to each weight element during the SVD stage. This weighted decomposition produces low-rank weight matrices with minimal impact on VLM accuracy.
WSVD applies quantization alongside SVD decomposition both the weights and activa- tions of the VLM. To further enhance efficiency, it incorporates an outlier elimination strat- egy within the SVD framework and locally finetunes the decomposed matrices, achieving improved accuracy while substantially reducing memory and computational cost.
RELATED WORK
2.1 VISION LANGUAGE MODEL.
2023 Liu et al. 2023 Dai al. 2023 Beyer Vision Language Models (VLMs) (Li al. 2022,
et al. 2024 Grattafiori et al. 2024 Wang et al. 2024c) build on the foundation of Large Lan- guage Models (LLMs) by incorporating visual signals in addition to textual input, thereby enabling multimodal tasks such as image captioning and visual question answering (VQA). Representative systems like BLIP and InstructBLIP (Li
2022 2023) leverage large-scale data curation and visual instruction tuning to better align their responses with human intent, particularly under zero- shot evaluation. A widely adopted framework, shown in Figure ! (a), encodes images into visual tokens via a vision backbone, concatenates them with text tokens, and feeds the combined sequence into a language model for output generation. This simple yet effective concatenation strategy un- derpins popular VLMs such as the LL family (Liu et al.) 2023). SmolVLM (Marafioti et al.) 2025), PaLI-Gemma (Beyer et al. 2024), and Qwen-VL (Wang et al. 2024c). Despite their strong performance, these models are often computationally heavy and difficult to deploy on devices with limited resources. To address this, compact designs have been introduced. TinyGPT-V (Yuan et al, 2023a) and TinyLLa VA (Zhou et al. 2024) pursue scaled-down yet efficient alternatives, while SmolVLM (Marafioti et al. 2025) presents family of lightweight models with one to three billion parameters that preserve competitive accuracy while significantly lowering hardware requirements.
2

---

Published as a conference paper at ICLR 2026
2.2 SINGULAR VALUE DECOMPOSITION FOR LARGE MODELS
Singular Value Decomposition (SVD) Jooliitee & Cadima 2016) is a fundamental tool in matrix factorization that represents a matrix W I= as W UVV", where and V are orthogonal matrices containing the left and right singular vectors, and Σ is diagonal matrix with non -eg-teete singular values sorted descending order. By retaining only the leading singular values and their associated vectors, one obtains compact rank-r approximation:
W = UU
with U, E Rmxr, E, Rrxr, and V, E Raxr, This form can equivalently be written as W R AB, where A = U, and B = $./2VT Such low-rank approximations capture the dominant structure of W, allowing dimensionality reduction, compression, and faster computation.
SVD has been extensively studied as compression strategy for LLMs (Wang et al. 2025c Yuan
al. 2023b; Wang al. [2024d] Li al., 2025 2024c; Chang al. 2024 Wang al.) 2025a). Early work (Noach & Goldberg 2020) applied vanilla SVD directly to weight matrices, but the method suffered from considerable approximation errors. Subsequent techniques refined this ap-
proach: FWSVD (Hsu et al. 2022) incorporates Fisher information (Lyy et al. (2077) 7) to rank param- eter importance, ASVD Yuan al. 2023b) accounts for activation outliers, and SVD-LLM (Wang ct al. 2024d) explicitly minimizes the loss introduced by discarded singular values.
While most efforts have focused on compressing model weights, it can also be used for KV cache compression (Chang et al.] 2024 Yu et al. 2024a). In particular, the key and value projection matrices can be factorized as WK = Ak and Wy = By. For a given input X, this allows the KV cache to store only the low-dimensional latent vectors Cx XAx and Cv XAy, thereby reducing cache size. During decoding, the original KV representations are reconstructed via K = CR Bx and V Cv By. More recent innovations include AdaSVD (Li et al., (025), which dynamically adjusts compression rates based on per-layer sensitivity, and SVD-LLM2 (Wang et al., 2025ch, which optimizes truncation strategies using theoretical error analysis.
2.3 FISHER-BASED IMPORTANCE AND WEIGHTED MATRIX FACTORIZATION
Fisher information has been widely used measure of parameter importance in continual learn- ing (Kirkpatrick et al. 2017) and in pruning and compression (Liu et al., 2021 Singh & Alistarh, 2020). Weighted low-rank approximation has been explored in matrix completion and recommen- dation, where each entry carries confidence weight (Srebro & Jaakkola, 2003 More recently, FWSVD (Hsu et al., 2022) incorporates Fisher information into low-rank factorization by assign- ing single Fisher-based weight to each row and applying SVD to pre-scaled matrix, yielding coarse row-wise weighting. On the interpretability side, gradient- based attribution and layer- wise relevance propagation methods (Ancona et al. 2017 Bach et al. 2015) also importance weights, but primarily for explanation rather than compression. In contrast, WSVD uses element-wise Fisher weights directly guide both local fine-tuning and quantization-awan training.
2.4 FLASH DECODING
FlashAttention (Dao et al. 2022 is an IO-aware attention algorithm that leverages tiling and kernel fusion to reduce memory traffic and improve GPU utilization. By keeping query, key, and value tiles in on-chip memory and streaming them through fused kernel, FlashAttention avoids materializing large intermediate attention matrices, thereby lowering memory footprint and achieving substantial speedups in training and inference.
Building on this idea, Flash Decoding (Dao et al. 2023) extends FlashAttention to the autoregressive decoding setting. Instead of materializing and reloading the entire KV cache for each step. it streams K and V in sequence tiles and incrementally updates online softmax statistics. This bisek-wise processing exposes additional parallelism along the sequence length dimension, complementing the existing head- and batch-level parallelism in Altetint,, and thereby improves GPU utilization. As a result, Flash Decoding achieves both lower memory traffic and higher throughput, and has become the de facto baseline for efficient inference in large language and vision-language language models. Our WSVD system further builds on Flash Decoding by integrating low-rank reconstruction into the fused kernel pipeline, reducing memory overhead while preserving its efficiency (see Section 3.4).
3

---

Published as a conference paper at ICLR 2026
ontention IFFN
30
XB Kh
Cled
X B1
Kh1
Latency
20
10
Mi...
E
A
XR
Bh
E
Ceaa
x
Kn2
Bh
Flash
Ah
Cer x Bn
Khn
Decoding
(a)
(b)
Figure 2: (a) Latency evaluation of VLM including self-attention (SA) and feed-forward (FFN) modules. (b) Conventional SVD: the left side illustrates SVD of Wx. and the right side shows the reconstruction of K2, from the shared latent. (c) Per-head SVD: the left side illustrates per-head SVD of WK., and right side shows per-head reconstruction of K₁, from per-head latent.
3 METHOD
An overview of WSVD is presented in Figure T (b), which consists of three key components: (i) Per-head SVD operations for reduced latency (Section [.T]. (ii) WSVD with dynamic importance scoring (Section 3.2), and (iii) quantization-awaa finetuning for low-rank VLMs (Section 3.3
3.1 FINE-GRAINED PER-HEAD SVD OPERATION FOR REDUCED LATENCY
Prior studies have shown that VLM decoding is predominantly memory- boundd, as long image-token sequences enlarge the KV cache and each generated token requires accessing the large KV cache, with overall latency bottlenecked by memory access. As discussed in Section 2.2 conventional SVD-based approaches (Chang et al. 2024 Wang et al. 2025d 2024d) address this by decomposing projection matrices (e.g., Q. K. and V). thereby reducing parameter count and storing low-rank latent representations CR and Cy. This strategy not only decreases computation and runtime in the prefill stage but also reduces cache size, offering potential I/O savings during the decoding stage.
However, in practice, we find that reconstructing K and V from low-rank latents introduces sub- stantial overhead, leading to even higher decoding latency than the original uncompressed model. Specifically, we profile the single-layer decoding latency of LLaVe-Next 7B (Zhou et al. (024) on an RTX 4090, comparing standard flash decoding without SVD against an SVD baseline that caches low-rank latents. In this baseline, both the rank ratio and cache size are reduced to 50% as before. With batch size of 16 and KV cache length of 8192, the results (Figure 2 (a)) show that SVD scheme incurs substantially higher latency compared to flash decoding.
To pinpoint the cause of this latency growth, we observe that the overhead arises from decomposing the entire K and V matrices. Taking K as an example, after SVD we obtain Wx = Ag Bx. where AK E IR ExR and Bx E IR Rx with E denoting the embedding dimension and the truncated rank. For each head h, the key projection is computed as WKA Ax BKi. where Bky, E R f and H is the head dimension (Figure (b)) During inference, the latent representation CK XAx E RI is cached across sequence length L, and each head's key vector is reconstructed as K₁,- = CK BKA. This reconstruction introduces a computational cost of 7svd = LRH per head. Compared with directly storing the K matrix of size L.F,, although caching Ck reduces storage to LR, reconstructing WK for each head requires accessing the entire Ck, which has a size LR. As result, the effective memory footprint becomes 7/vd LR per head, thereby increasing decoding latency. Similar argument holds trues for the computation of value vector V.
To mitigate this overhead, our WSVD approach applies SVD directly to the submatrices of WK and Wy corresponding to each head, rather than decomposing the entire matrices, as illustrated in Figure 2 (c). Specifically, for head h. the submatrix WK Ex is decomposed as WKb AIa. B kh. where AKa E Exr and BK E RrxH The rank r is obtained by truncating the H singular values of WKK- Since H < F,, the per-head rank r typically much smaller than R. For each head h, the latent representation is computed as CK
XAgk E R Lxr and stored in the cache. During decoding, the corresponding key vector reconstructed as K₁, = BRA. Unlike the conventional SVD approach shown in Figure ( (b), this design eliminates the need to repeatedly load large shared latent representation CK. since each head can be reconstructed directly from its own latent CKE- With this design, the memory footprint is reduced to Awwvd Lr, since only the latent vector CKn needs to be stored, and the computational cost of reconstructing K₁, becomes Twsed La H. where r & R. A similar computation applies to the reconstruction of V.
4

---

Published as a conference paper at ICLR 2026
To evaluate the saving analytically, the per-head SVD scheme shown in Figure (c) reduces both memory traffic and computational cost, thereby enabling practical decoding accelcration, as demon- strated in Section 4.4 In particular,
Twavd
Twsvd
<< R
7svd
V/vvd
R2'
Thus, both computational cost and memory footprint for latent storage are reduced by factor of r/R. Compared to the original SVD-based scheme, WSVD further reduces the weight parameter count from Crorig = EH H per head to awwvdd = Er + rH, and lowers the KV-cache size from /oig = LH Tlwsvd Lr. These improvements are quantified by the parameter size ratio pl and the cache size ratio P2 for KV vector storage.
pt
Owwsvd
Garig
[E + xr (1
Fx
H r F H
P2
Thavvd
Jorig
π
(3)
However, the per-head SVD in the WSVD scheme also amplifies approximation errors, making accuracy degradation harder to control compared to conventional SVD applied to the full WR. Next, we describe a local weighted finetuning scheme to mitigate the accuracy drop.
3.2 SVD WITH LOCAL WEIGHTED FINETUNING
Conventional SVD converts full-rank input matrix into low-rank representation, but limita- tion is that it cannot control the relative contribution of different weights after decomposition. Prior work (Yu et al. 2024b) has shown that in large models, weights vary significantly their impor- tance to final accuracy. In particular, some "superweights" are highly sensitive, where even small changes in magnitude can cause a substantial drop accuracy. Therefore, it is crucial to incorporate this notion of importance when performing SVD, resulting in a weighted low-rank decomposition.
The first question is how to evaluate the importance of weight element. To formalize this, let DD denote the data distribution over calibration samples and let a(W;r) denote the training loss of sample x. The importance score each element in WK with respect to final accuracy can be estimated as:
= EzDD Vwx ((;
(4)
A weight entry with large gradient magnitude indicates that even small change in this element has a substantial effect on the expected model loss. Accordingly, GK can be interpreted as an importance score that links parameter updates their impact performance.
This estimation of training loss impact can be refined using the Fisher Information Matrix (FIM), which quantifies parameter importance the expected sensitivity of the ligeliied with respect to model parameters. A second-order Taylor expansion of the expected loss around the current parameter values yields:
AC = EC==EnD[(W + AW: -(W:))(W::)))
(5)
2 AWW [E.| VW ((W;))))) AW.
(6)
To make the computation of the Hessian tractable, it can be approximated by a diagonal matrix, where each diagonal entry corresponds to the Fisher importance score of the parameter. For example, the vector of Fisher information score Fx for Wx can be computed as:
Fx En-D[gx(()) (x) @ gı (x)], gK(x) Vwx ((W;
where O denotes elementwise multiplication. Motivated by these observations. we propose weighted local fine-tuning mechanism that performs SVD while incorporating the relative impor- tance of each weight element, quantified by its Fisher information score. Specifically, the objective function can be described as:
min 1 の (Wx AK Bx Kg
(8)
where Ak. BK are the low-rank decomposition described in Section 3.1 this optimization can be applied across the SVD for the weight matrices for each head h, and the objective function can be depicted as:
estimate WK. In the context of per-head SVD
Σ
1//2
(WKL. AxnBrul
Agn.Bgh
5

---

Published as a conference paper at ICLR 2026
VRAM
On-chip
2
Ckth
Buffer
Reconstructt centinsttuuttt Kh (MaiMMul & RoPE)
On-chip
Reconstruct (MatMuu & RoPE)
On-chip
Buffer
Buffer
Cin Tile
VRAM
VRAM
Tile
Tile
Bich
1980
Ceh
3
CKt
Cios Tile
Kn Tile
Kh
Kn
Ctn
Cvh Tile
Cvn Tile
(b)
Figure 3: (a) Naive reconstruction requires materializing and writing back full K₁, to VRAM (global GPU memory), leading to excessive memory usage and I/O. (b) Our fused kernel consumes CK and Cvs, tiles on-chip with flash decoding, reducing both peak memory footprint and I/O traffic. All the step numbers are shown in circle.
where Age and Kg denote the low-rank approximation of WK.- Since no analytical solution exists for this problem, it is solved by fine-tuning AKж and Bgg until convergence. The same loss formulation can be applied to other projection matrices in the model (e.g., We. Wv, or feed-forward layers), providing a general framework for graghtet-weighted fine-tumin fine-tuning after SVD truncation. The WSVD procedure summarized in Algorithm
3.3 LOCAL QUANTIZATION-ANIE TRAINING FOR LOW-PRECISION WSVD
To further reduce model size and cache footprint, we apply low-precision quantization the low- rank model parameters and the input and mitigate accuracy loss using local quantization-aware train- ing (QAT). To address chamnel-wise outliers in the input X and latent representations Ck. Cy, we follow previous work (Ashkboos et al. (2024, Xiang & Zhang 2024) and introduce two orthogonal matrices S and S2, and Si also Hadamard matrix with predefined binary elements. With these transformations, the quantized Q, K, V computation for each head h can be reformulated as:
XA, > Y, (XS An (s, = ((XSS)QQSSAA))
(s,
(10)
where S7 = S S2 = I, Q(-) denotes the quantization operator, and we omit the QKV subscripts for simplicity of presentation. We further finetune the rotational matrices Sy together with A,, Bn, to minimize the change on the low-rank weights caused by quantization, with the objective as follows:
min Sa.A,,
(1%) [S.W. Q.. Q(22..) ll2
(11)
where F & (e)g(()))(z)) @ (Sig(z))). F2 is the Fisher information matrix associated with the transformed weight S, Wh computed element-wise as the root of the expected squared gradient S1g(2) over the calibration dataset D. This acts ters with higher sensitivity and guiding the QAT objective to more effectively preserve accuracy under quantization. During QAT, we jointly update An, S2, and B,, while S₁ is fixed Hadamard matrix size E E, determined by the model embedding dimension E. This update design enables the factorized components flexibly adapt quantization noise while preserving the orthogonal transformation imposed by S,, thereby maintaining the low-rank structure and im- proving the approximation accuracy and stability of low-precision training. Since this procedure is QAT performed locally, it incurs much lower time and memory overhead than end-to-en finetuning.
importance weight, emphasizing parame-
exact
3.4 WSVD SYSTEM IMPLEMENTATION
naive PyTorch implementation of WSVD results in excessive memory consumption during the re- construction of K and V, as illustrated in Figure (a). Taking the key K₁ of head h as example, with K₁. = CRI BKG where CKk h Rlxr and Bgn E IR*xH the GPU operation proceeds as fol- lows. First, the latent representation Cкa is loaded from VRAM. Next, reconstruction Cra. BKn is performed, materializing the full K₁, E L in VRAM. The reconstructed K₁, is then written back to VRAM and later reloaded for attention. Since K2, and V,, cannot fit into limited on-chip buffers, they must be stored along with the latent CKE. Cvh. which largely increases I/O traffic and peak memory usage, in some cases exceeding that of the original model without low-rank decomposition.
To address this problem, we design a fused kernel in Triton Tiileet et al. 2019) that integrates low- rank reconstruction directly into the flash decoding pipeline, as shown in Figure (b). At tile

---

Published as a conference paper at ICLR 2026
granularity, the kernel streams tile t of Ckk. denoted CKA.1 Rixr, from VRAM (step 1), where 1 is the tile size along the sequence dimension L that fits into on-chip memory. The up- projection weight Bxn. is then loaded once into on-chip storage (step 2), and the temporary key tile Ka,t = CKA. BKa is formed in registers or shared memory (step 3). This process is executed within single fused kernel that proceeds directly into the flash decoding pipeline: the temporary KE, is immediately contracted with the query tile qn to compute qn K,,,,, update the online softmax statistics, and apply the normalized attention weights to the corresponding value tile CVn,, (step 4).
In this design, both Cкa and Cya are consumed in place, and all intermediate tensors remain on- chip without being written back to VRAM. The fused kernel integrates reconstruction, g ac- cumulation, softmax normalization, and the V multiplication into a single workflow, eliminating redundant kernel launches and memory transfers. Memory usage now scales only with the tile size r and Bg&), which significantly reduces peak footprint and I/O traffic while preserving the effi- ciency of flash decoding. The design exposes parallelism at two levels: across tiles, where multiple tiles are processed concurrently within each head, and across heads, where different heads execute in parallel, fully utiliziig GPU compute resources in accordance with flash decoding scheduling. Finally, the V-path up-projection Byb is fused into the output projection, which avoids explicit reconstruction of Vn. following Palu (Chang et al. 2024). Collectively, these optimizations elimi- nate redundant memory operations while maintaining high parallelism, enabling WSVD to achieve practical inference acceleration without any loss of accuracy.
Beyond kernel fusion, WSVD applies per-head SVD to the Query, Key, and Value projections to reduce parameters and improve efficiency. Decomposing WK and Wy decreases model size and accelerates both prefilling and decoding, while decomposing Wo further reduces parameters and speeds up pre- filling. During prefilling, the imput sequence is pro- jected into low-rank K, V latents, with K, V la- tents stored as cache.
Down- Projection
Ch
New Cache
Stored
Cache
Keree
Aa
Bah
9
Akn
Cknn
Latent
Ckh
Bkh
Cache
Avh
Cvn
Ckn,
Cvn
Cvil
@Cvh
Figure 4: WSVD decoding pipeline. Each token is down-projected to low-rank latents, and K and V latents are appended to the cache, while Q latent is up-projected and consumed together with cached CKE: Cvh in the fused kernel.
During the decoding stage, as shown in Figure 4 each new token is processed through per-head down- projections to generate low-rank latents for Q K, and V. The latents of K and V are stored in the cache, while the latent of Q is immediately up- projected to form q for the current attention step. The kernel then loads the cached latents CK and Cyv together with the current qh. performing highly parallelized computation that integrates low-rank reconstruction with flash decoding. This unified pipeline eliminates redundant material- ization of full K and V, preserves compact latent caches throughout decoding, and enables efficient attention computation with reduced memory footprint.
4 EVALUATION
We conduct experiments on five representative vision-language models: LLaVA-v1.5 7B (Liu et al., 2023 VLa 13B, LLaVA-Next 7B, LLaVA-Next 13B, and SmooVLLM-Istruet (Marafioti ct al. 2025). For local weighted fine-tuning and QAT, we use 256 samples randomly drawn from the ScienceQA training split (Lu et al.) 2022). following the procedures described in Section 3.2 and
Section 3.3 Evaluation is conducted on two widely used benchmarks, Science (Lu et al. 2022)
and JEED-Bench-IMG (Li et al. 2024a). consistent with prior studies on VLMs such as LLa VA, using VL.Eval (Duan et al. 2024) tool. For comparison, WSVD is benchmarked against sev- eral baselines, including SVD-based approaches (ASVD (Yuan et al. (0236), SVD-ILLM (Wang ce al. 2024d), QSVD WWang et al. (025dg) and quantization-base techniques (DuQuant (Lin al. 2024). ) QVLM (Wang et al. 2024a)). For ASVD, SVD-LLM and QSVD, we follow their official implementations and apply S VV independently the Q, K, V matrices to ensure fair comparison with WSVD, while leaving other linear layers unchanged. More results are shown in the Appendix.
To isolate the impact of SVD from quantization, we introduce WSVD-noQ (Section 3.2 which applics only the SVD techniques described in Sections 3.1 and 3.22 We compare it with ASVD, SVD-LLM, and QSVD-noQ (unquantized version of QSVD) We then apply QAT in Section 3.3 on
7

---

Published as a conference paper at ICLR 2026
Table Accuracy evaluation of different methods under FP16 (detailed results in Appendix A.3
Acc.
Medhod
ScienesQ4-IMG
SERID-Bncch
Arg.
Pi: :8095
Pr: 90%
p 7005
p): 0096
Ph: 1.00%
P:: 19000%
(1 :80%
(P: 7.0%
Pi: 0009
Pv: 1.00%
ASVD
49.93%
50 50%12%
47.10%
36.09%
19.19%
54.27%
53.53%
48.35%
37.17%
24.17%
42.05%
SVD LLM
65.44%
63.71%
61.92%
57.41%
55.53%
57.89%
57.50%
55.33%
54.64%
55.31%
58.47%
QSVD-unt)
67.72%
68.22%
67 0855
68.055
62.57%
59 84.84%
59.07%
59. 69.78%
59.0095
58.235
62.64%
WSVD-noQ
68.17%
67.72%
67.28%
65.89%
65.49%
60.10%
60.17%
59.89
60.18%
60.46%
63.54%
FP16
Accuracy: 95.01%
Accursry: 60) 1878
64.10%
M.aa Mr-NNN
SVD SVD LLM
ASVD
72.53%
71.24%
72.2 7
70066755
71.71%
44.44%
71.15%
71,385
69.81%
70.55%
70.76%
70.89%
70.63%
70.26%
70.01%
70.25%
69 96%
69.69%
69.01%
69.58%
70.45%
70,9498
1:30
QSVD-an))
71.94%
72. 14%
71.74%
72.
71.39%
71.23%
71.02%
71.06%
70.925
70.40%
71 71.44%
WSVD-aaQ
72.88% %
72.98%
73.57%
73.48%
73.28%
71.29%
71.17%
71.25
70,95%
70.81 %
72.17%
FP16
Accuracy: 73.23%
Accurccy: 71,30%
72,27%
990%
800%
70%
50055
80%
70%
SmalVT.M
ASVD
29.30%
3,975
0.20%
17.85%
L,50%
0,95%
8.96%
28
SVD-LLM LLM
40.06%
17.20%
3.82%
32.49%
15. 1..89%
4. 4.60%
19.01%
QSVD-ar0
77.00%
62.77%
42.59%
64. 80.00
50.469
36.24%
55,64%
60.93%
65.78%
63.29%
54.45%
65.42%
FP16
Accuracy: 84.53%
Accuncyy 68.47%
76.53%
top of WSVD-noQ, benchmarking against DuQuant, QVLM, and QSVD. We also evaluate QASVD, which applies QuaRot (Ashkboos et al.] 2024) to the SVD-truncated VLMs produced by ASVD. For fine-tuning and QAT, we adopt lightweight local optimization to minimize overhead. A₁ and Bg are updated with Adam (Kingma & Ba 2014) (learning rate 1 x 10 for fine-tuning and 1 x 10-6 for QAT), while S2 is updated during QAT using the Cayley optimizer (Wen & Yin 2013). Local fine-tuning is performed for 100 steps and QAT for 50 steps, ensuring effective adaptation while keeping the additional latency negligible. All experiments are conducted on NVIDIA H100 GPUs.
4.1 ACCURACY EVALUATION ON WSVD-NOQ
We first evaluate the FPI6 performance of WSVD-noQ under different rank budgets. To ensure fairness, we align the parameter ratio P1 across all methods. For WSVD, P1 is defined in Equation 3 while for other SVD-based baselines, P1 is defined as the proportion of parameters relative to the original model after SVD is applied.
The evaluation results are summarized in Table T (details in Appendix A.3 Under the same pa- rameter ratio fl, WSVD-noQ surpasses ASVD, SVD-LLM, and QSVD-noQ in accuracy in most cases. On large-scale models such as LLL 13B and LLaVA-Next 13B, WSVD-noQ incurs less than 1% accuracy drop on ScienceQA-IMG and SEED-Bench compared the FP16 base- line. Notably, for LLaVA-Next 13B, when P1 < 70%, WSVD-noQ even outperforms the FP16 model on ScienceQA-IMG. For example, at 01 70%. WSVD-noQ reaches 73.57% accuracy, exceeding the FP16 baseline by more than 0.3%. This suggests that low-rank approximation may implicitly mitigate hallucinations (Liu et al., 2024a). though further validation is required. Fur- thermore, WSVD-noQ delivers consistently higher average accuracy across datasets and parameter ratios. The advantage over other baselines becomes increasingly evident as Pi decreases. For ex- ample, SmolVLM, WSVD-noQ attains over 60% accuracy ScienceQA-IMG, while baselines fail to yield usable results under the same parameter ratio settings.
4.2 ACCURACY EVALUATION OF WSVD
We present results under two weight-activation quantization configurations: W8A8 for WSVD with rank settings P1 = 50% and P2 50%, and W8A4 for all other baselines. This design keeps cache size and parameter size comparable across methods, while WSVD's rank truncation further reduces its parameter budget, ensuring fairness in comparison.
For activation quantization, we adopt per-token symmetric quantization. For weight quantization, we employ round-to-neares, (RTN) with per-channel symmetric scaling and a learnable clipping ratio, where the clipping value is selected via linear search to minimize squared error, following QuaRot (Ashkboos et al.) 2024). This quantization scheme is applied to the per-head Q/K/V weight matrices and all remaining attention and feed-forward modules, ensuring that the dominant matrix multiplications in each transformer block are executed in low precision. As shown in Table 2 WSVD consistently outperforms the baselines in most cases, despite using smaller parameter budget and the same cache size. On average across models and datasets, WSVD incurs only a modest accuracy drop of just over 1 1% relative the FP16 baseline, while reducing cache size to
8

---

Published as a conference paper at ICLR 2026
Table 2: Accuracy evaluation of different methods under low-precision on LLaVA-v1.5 7B, LLaVA-
v1.5 13B, LLaVA-Next 7B and LLaVA-Nexx 13B.
ScienceQA-IMG1
SEED-Bench
Mothod
Avg. T
v1.5 7B
v1.5 13B
Next 7B
Next 13B
vB.5
v1.5 13B
Next 7B
Next 13B
DuQuant
57.36%
67.22%
66.34%
70.20%
54.11%
61.43%
63.64%
66. 1.5%
63.31%
QVLM
55.24%
66.46%
60.60%
65.28%
50.13%
59.22%
65.39%
59.09%
QASVD
41.92%
65.34%
49.37%
64.85%
41.26%
59.30%
49.63%
66.54%
54.78%
QSVD
65.61 %
70.12%
66. 10%
70.43%
58.49%
62.95% %
65.63%
69.21%
66.07%
WSVD
64.25%
72.14%
66.94%
73.08 %
60.23%
62.01%
67.49%
70.67%
67.10%
FP16
68.10%
71.83%
69.60%
73.23%
60.18%
62.54%
69.02%
71.30%
68.23%
Table 3: Results of weighted finetuning ablation under different p1 settings.
Acc.
Method
Pr m1=00%
px=70%
01=500 50%
Acc.
Method
900
p.=70%
550%
FP16
68.01 %
FP16
69.60%
v1'. 78
WSVD-1oFT
67.82%
66.82%
65.09% 65.49 %
Next
7B
WSVD-noFT
69.76%
68.61%
66.46%
WSVD-noQ
68.17%
67.28
WSVD-noQ
69.81 %
69.36%
67.87%
25% of the FP16 model. At the same time, WSVD achieves more than I % higher average accuracy than all baselines, demonstrating that the inlegration of per-head SVD and quantization with WSVD only lead to minimized accuracy loss.
4.3 ABLATION STUDY
Effectiveness of Weighted Local Finetuning We evaluate the impact of WSVD fine-tuning, as described in Section 3.2] on accuracy performance using SciG for WSVD-noQ. The comparison is made against the WSVD-nnQ baseline, which applies standard SVD without account- ing for the relative importance of weight elements, while keeping all other settings identical. As shown in Table 3 WSVD-noQ consistently outperforms the no- no-finetuning variant (WSVD-noFTT) demonstrating that incorporating relative weight importance during the SVD process leads to signif- icantly improved performance over standard SVD.
Effectiveness of QAT We further examine the impact of local QAT on the low-rank model, as described in Section [.3] Specifically, compare WSVD against a baseline that uses the same quantization settings but does not fine- tune Sz, A,, or B₁ mentioned in Section 3.3 while keeping all other settings identical. As shown in Table under W8A8, WSVD consistently surpasses the bascline across all models. These results demonstrate that local QAT effectively re- covers the performance lost due low- oowpreccsion quantization.
Table 4: Results of local QAT ablation.
Method
ScienceQAAIMG
Avg.
v1.57B
v1.5
7B
We QAT WSVD
63.91%
71.99%
66.59% 66.94%
72.68% 73.08 %
68. 79%
64.25%
72.14%
69.10%
4.4 SYSTEM EVALUATION ON VLM
Lawany (mn)
15
9.5x
9
We assess the system-level performance of WSVD-noQ, with focus decoding-stage acceleration. Specifically, we measure the layer-wise decoding latency of LLaVA-Next 7B across the attention and feed-forward mod- ules using our fused kernel implementation de- scribed in Section 3.4 on RTX 4090 and 5090
6
3.8x
4,3x
1.8%
1.3x
1x
Enger Attention
Palu
Decoding
Flash
WSVD.
naC
Attandion
Eager
Palu
Decoding
Flaoh
WEVD- noQ
(a)
(b)
Figure 5: Latency evaluation and normalized la-
tency on: (a) RTX 4090 and (b) RTX 5090.
GPUs. For comparison, we include Eager Attention without Flash Decoding, Palu Chang et al. 2024), and Flash Decoding (Dao et al. 2023) as the baseline algorithms. For Flash Decoding, we adopt scaled dot-product attention (SDPA), while Palu is evaluated using its official repository. Both Eager Attention and Flash Decoding operate the full KV cache, while Palu and WSVD-noQ re- strict the latent size to P2 = 50%, corresponding to P1 x 51.5% for WSVD. All measurements are conducted with a batch size of 16 and a sequence length of 8192. Since Palu supports only batch size 1, we use an equivalent sequence length of 16 x 8192 for fair comparison. In addition, we report latency results of full-matrix SVD and per-head SVD in Appendix A.7
9

---

Published as a conference paper at ICLR 2026
As shown in Figure 5 WSVD-noQ consistently outperforms all baselines both GPUs in lateney. Relative to Flash Decoding, achieves over 1.8x speedup, driven by reduced I/O overhead and negligible reconstruction cost enabled by our scheme. Compared with Palu, WSVD-noQ attains lower latency through two advantages: algorithmically. per-head SVD provides finer-grained com- pression than Palu's group-head SVD;; system-wiss decoding pipeline. These results demonstrate that WSVD, together with our fused kernel design, of- fers effective system-level solution that alleviates I/O bottlenecks and enables practical decoding acceleration in VLMs while maintaining accuracy performance as the original model.
fused kernel is fully integrated into the flash
Table 5: Latency (ms) on RTX 4090 (left) and RTX 5090 (right) for different sequence lengths.
Seq Len
1024
2048
4096
8192
16K
32K
Seq Len
1024
2048
4096
8192
16K
32K
Flash Decoding
0.92
1.77
2,92
5.14
9.64
Flash Decoding
0.65
0.86
2.14
3.81
7.18
WSVD-noQ
0.70
0.83
1.12
1.66
2.89
5.28
WSVD-noQ
0.58
0.66
0.83
1.15
1.79
3.06
Speedup
13x
16xx
1.8x
Speedup
11x
13x
19x
2.1x
2.3x
Table 6: Latency (ms) on RTX 4090 (left) and RTX 5090 (right) for different batch sizes.
Batch Size
4
8
16
32
64
Batch Sizee
8
16
32
Flash Decoding
1.13
1.71
2.92
5.11
9.67
Flash Decoding
0.75
1.07
2.14
3.38
5.97
WSVD-noQ
0.82
1.11
1,66
2.91
5.33
WSVD-noQ
0.66
0.84
1.15
1.79
3.11
Speedupp
1.4x
1.5%
188
1.8%
1.8%
Speedup 1.1x 1.3x
1.9%
1.x
1.9%
Impact of Sequence Length and Batch Size We further perform an ablation over various quence lengths (Table 5) and batch sizes (Table ( for LLaVA-Next 7B under the same setting, and report the layer-wise decoding latency of Flash Decoding and WSVD-noQ 5090 GPUs With batch size 16, as the sequence length grows from 1K to 32K tokens, WSVD-noQ improves over Flash Decoding by about 1.3x to 1.8x on RTX 4090 and up to 2.35x on RTX 5090. For fixed 8192 sequence context, increasing batch size from 4 to 64 yields roughly 1.4x to 1.9x speedups on both GPUs. This trend reflects that longer contexts make KV-cache I/O increasingly dominant, so our WSVD-based compression and decoding kernel delivers larger relative gains.
RTX 4090 and RTX
Impact of Rank Ratio Using the same setting as Section 4.4 we vary the rank ratio Ps E (90%. 70%, 50%} for WSVD-noQ and measure the latency on RTX 4090 and RTX 5090 GPUs. Table 7 summarizes the impact of rank ratio on decoding latency. Smaller #2 values (i.e., lower ranks) consistently yield lower latency, demonstrating that WSVD's fused kernel can effectively translate rank reduction into tangible decoding speedups over the Flash Decoding baseline.
Table 7: Latency (ms) under different P2-
Table 8: Latency (ms) on RTX 3060 (p2 50%).
GPU
Flash Dec.
0s.90%
Pa:70%
p2:50%
Seq Len
1024
2048
4096
8192
16K
4090
2.92
2,83
2.53
1.66
Flash Decoding
3.37
4.88
7,81
13.27
24.59
WSVIDnnOQ
2.18
2.68
3.62
5.54
9.49
5090
2.14
1.87
1.75
1.15
Specdup
1.5x
1.8x
2.2x
2.4x
2.6x
Speedup on Low-end GPU To evaluate our method on more modest hardware, we benchmark the latency of LLa VA-Next 7B with WSVD-noQ on an RTX 3060 (Table () under the same setting, and compare it with the Flash Decoding baseline. On RTX 3060, WSVD-noQ reduces latency from 3.37 ms to 2.18 ms at 1K tokens (1.55x) (15x)amm and from 24.59 ms to 9.49 ms at 16K tokens (2.59x). These speedups are larger than on 4090/5090-class GPUs because the lower memory bandwidth of RTX 3060 makes K V-cache I/O more dominant. By shrinking the KV cache and using fused decoding kernel, WSVD reduces memory traffic and achieves larger latency gains on low-end devices.
5 CONCLUSION
In this work, we present WSVD, weighted low-rank approximation framework that integrates her-head SVD, weighted models. By aligning algorithmic design with system-level optimization through our fused kernel, WSVD achieves over 1.8x decoding speedup while preserving accuracy under aggressive compression.
and quantization-an awwrr training to compress and accelerate
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work focuses on model compression and acceleration techniques for vision-language mod- els. All datasets used this study (ScienceQA-IMG and SEED-Bench) are publicly available and widely adopted in the community. Our research does not involve human subjects, private or sensitive data, or personally identifiable information. The proposed method aims to improve the efficiency of large models, which may contribute to reducing the computational and environmental costs of deployment. We aware of any direct ethical concerns specific to this work.
REPRODUCIBILITY STATEMENT
We take reproducibility seriously and provide the following details:
Code and models: We will release the full implementation of WSVD, including training and inference code, as well as evaluation scripts, upon publication.
Datasets: All datasets used in this work are publicly available. In particular, we evaluate ScienceQA-IMG and 3EED-Bench both of which be accessed without restriction. We will also provide preprocessing scripts to reproduce the exact input formats used in our experiments.
Experimental setup: Hyperparameters, optimization settings, and evaluation protocols are described detail Section 4
Randomness: All experiments are run with fixed random seeds in the scripts, ensure consistent results.
Our experiments are conducted on NVIDIA H100, RTX 4090 and RTX 5090 GPUs as described in Section 4
Compute resources:
Limitations: Some large-scale experiments (e.g., on 13B- -araaammet models) require ac- cess to high-end GPUs, which may limit reproducibility for groups without such resources.
We believe that with the released code, scripts, and dataset accessibility, other researchers will be able to reproduce our results and build upon our method.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06100 2017.
Salch Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li. Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213-100221 2024.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robed Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308.12900 2023.
Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Laila Bashmal, and Mansour Zuair. Vision-language model for visual question answering in medical imagery. Bioengineering. 10(3):380, 2023.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. CoRR, 2024.
Chi-Chih Chang, Wei-Cheng Lin, Chien- Yu Lin, Chong- Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiane Sai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprini arXiv:2407.21 2024.
Christel Chappuis, Valérie Zermatten, Sylvain Lobry, Bertrand Le Saux, and Devis Tuia. Prompt- rsvqa: Prompting visual context to a language model for remote sensing visual question answer- ing. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp- 1372-1381, 2022.
Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adap- tation of pretrained language models for image captioning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 18030-18040. 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: towards general-purpose vision-language models with instruction tuning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359 2022.
Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context inference. http:s:://pytrrr..rroooo/// hwwpb/ooog/laaa--cccoddddo decodideciingg... October 2023. Accessed: 2025-09-22.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art the-art laniuuelanguage models. In Proceedings of the Com- puter Vision and Pattern Recognition Conference, pp. 91-104, 2025.
Haodong Duan. Jumming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluat- ing large multi-modallty models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198-11201, 2024.
Maksim Dzabraev, Alexander Kunitsyn, and Andrei Ivaniuta. Vlr:: Vision- languae models act as reward models for image captioning. arXiv preprint arX1v:2404.01911, 2024.
12

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arX 2024.
2024.
Yen- Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112 12, 2022.
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up usion-language pre-training for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17980-17989, 2022.
Ian T Jolliffe and Jorge Cadima. Principal component analysis: review and recent developments. Philosophical transactions of the royal society A: Mathematical, Physical and Engineering Sci- ences, 374(2065):20150202 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barrkkikka al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3522 2017.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEÉF/CVF Conference Computer Vision and Pattern Recognition, pp.
2024a.
Chuanhao Li, Zhen Li, Chenchen Jing, Shuo Liu, Wenqi Shao, Yuwei Wu, Ping Luo, Yu Qiao, and Kaipeng Zhang. Searchlvims: plug-and-play framework for augmenting large vision-language models by searching up-to-date internet knowledge. arXiv preprint 2xxx:22405.14544 2024b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference machine learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Muyang Li, Yujun Lin, Zhekai Zhang. Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. Svdqunat: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05007 2024c.
Zhiteng Li, Mingyuan Xia, Jingyuan Zhang, Zheng Hui, Linghe Kong, Yulun Zhang, and Xiaokang Yang. Adasvd: Adaptive singular value decomposition for large language models. arXiv preprint arXiv:2502.01400. 2025.
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quan- tized Ilms. Advances Neural Information Processing Systems, 37:87766-87800, 2024.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00255 402500253, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296-26306, 2024b.
13

---

Published as a conference paper at ICLR 2026
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pp. 7021-7032. PMLR, 2021.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12)220010,, 2024c.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Kei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul PPP Grasman, and Eric-Jan Wagenmak- ers. A tutorial on fisher information. Journal of Mathematical Psychology, 80:40-55, 2017.
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Za- kka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint urXiv:2504.05299 2025.
Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix de- composition. In Proceedings of the 1st Conference of the Asia- Asia-Pacificc Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 889, 2020.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-orde approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098-11 109, 2020.
Nathan Srebro and Tommi Jaakkola. Weighted low-rank approximations. In Proceedings of the 20th international conference on machine learning (ICML-03), pp. 720-727, 2003.
Zelong Sun, Dong Jing, Guoxing Yang, Nanyi Fei, and Zhiwu Lu. Leveraging large rision-language model as user intent-aware encoder for composed image retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7149-7157, 2025.
Philippe Tillet, Hsiang- Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPIAN International Workshop on Machine Learning and Programming Languages, pp. 10-19, 2019.
Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Q-vim: Post- training quantization for large vision-language models. arXiv preprint arXiv:2410.088 19, 2024a.
Guankun Wang, Long Bai, Wan Jun Nah, Jie Wang. Zhaoxi Zhang, Zhen Chen, Jinlin Wu, Mo- barakol Islam, Hongbin Liu, and Hongliang Ren. Surgical-Ivim: Learning to adapt large vision- language model for grounded visual question answering robotic surgery. arXiv preprint 1005 2024b.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024c.
Qinsi Wang, Jinghan Ke, Masayoshi Tomizuka, Yiran Chen, Kurt Keutzer, and Chenfeng Xu. Dobi-svd: Differentiable svd for I1m compression and some new perspectives. arXiv preprint mrxiv:2502.02223. 2025a.
Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training- free framework for high-resolution image percep- tion in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7907-7915, 2025b.
Xin Wang. Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint a7X.v.2403.07378. 2024d.
14

---

Published as a conference paper at ICLR 2026
Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-Ilm v2: Optimizing sin- gular value truncation for large language model compression. arXiv preprint 123440, 2025c.
Yutong Wang, Haiyu Wang, and Sai Qian Zhang. Qsvd: Efficient low-rank approximation for unified query-key-value weight compression in low-precision vision-language models. arXiv preprint arXiv:2510.16292. 2025d.
Zaiwen Wen and Wotao Yin. A feasible method for optimization with orthogonality constraints. Mathematical Programming, 142(1):397-334, 2013.
Jingyang Xiang and Sai Qian Zhang. Dfrot: Achieving outlier-free and massive activation-free for rotated mims with refined rotation. arXiv preprint arXiv:2412.00648, 2024.
Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. Effectively compress kv heads for Ilm. arXiv preprint arXiv:2406. 400667056,6..77 2024a.
Mengxia Yu, De Wang, Qi Shan, Colorado J Reed, and Alvin Wan. The super weight large language models. arXiv preprint 1rXiv:2411.07191 2024b.
Zhengqing Yuan, Zhaoxu Li, Weiran Huang. Yanfang Ye, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint 12. 2023a
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05522 2023b.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint 2rXiv:2402.14288, 2024.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings the on artificial intelligence, volume 34, pp. 13041-1 13049, 2020.
the AAAI conference
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LLMS
Large language models (LLMs), such as ChatGPT, were used exclusively for language polishing and minor stylistic editing of the manuscript. All technical ideas, analyses, and experimental results were conceived, implemented, and verified by the authors. The authors carefully reviewed and validated all text to ensure accuracy.
A.2 WSVD ALGORITHM
Algorithm Weighted SVD Fine-tuning (WSVD) pseudo code
Require: Calibration dataset X, model parameters {W}, rank r
Ensure: Fine-tuned low-rank factors {A, B}
1: for each sample 24 E do
2: Compute forward pass and loss L(x,)
3: Backpropagate to obtain gradients (V w
4: Accumulate importance weights F Σ Σx 2
5: end for
6: for each weight matrix W (WQ, WK, wy.... do
7: Compute SVD: W, = B,, with AA E mmxrr BA E R
8: Define weighted loss:
LwsVD (An, Bb) || o (Wn Bx
where FA, Wh, As and B, is each head's importance weight, weight. decomposed matrices. 9: Locally fine-tune An, B₁ using LWSVD
10: end for
return Fine-tuned low-rank factors (Ap, B₂) for all matrices
A.3 DETAILED RESULTS OF WSVD-NOQ
We inlcude the detailed results of WSVD noQ and other baselines in Table D
Table 9: Accuracy evaluation of different methods under FP16.
Acc.
Metbod
ScienceQA JMG
SERD-Beechh
Avg.
900%
70%
500%%
PI: p. 80%
100%
49.93%
50.125
47.10%
36.69%
19.19%
54.27%
53.53%
48. 35%
57.17%
24.17%
42.05%
LLAAA-1
SVD-LLM
65.44%
57. 57.895
55.31%
28
QSVD-ant)
WSVD-aoQ)
67.28%
FP16
Accurac::
Accomay:
LLAV--115
SVD-LLM
71.05%
855
70.33
70.300
62.284
62. 02:255
62.08%
66.48%
178
OSVD-nnQ
71.54%
62.64%
62.63%
62.52%
WSVD-aeQ
71.99%
71.81%
72.53%
71.59%
71.44%
62.52%
62.68%
62.38%
62.57%
62.37%
67.17%
Accornyy: 21.8%
Accumcy: 60.51%
LLAVA NoN
ASVD
SVD LLM
68.27%
64.20%
67.92%%
63.36%
62 07%
66
60.19%
66.
55.28%
65.54%
67.38%
68.50%
68.31%
66.96%
66. 66.24%
67.65%
67.45%
65.13%
61.52%
66.28%
63.23%
67 07.29%%
7M
QSVD-anD
70.10%
69 16%
69 01%
68.27%
66.195 19%
68.86%
68.05%
68 41%
67.985
67.27%
68.42%%
WSVD-aaQ
69.81%
60.56%
69.36%
68.229
67.87%
69.18%
69.27%
69.15%
69.16%
68.599%
69.02%
FP16
Accuracy: 69 60%
Accurscy: 69.025%
69. 31%
L.a.% Neat
SVD SVD LLM
ASVD
72.53%
71.24%
72.20 72.24%
71 44%
71.74%
71.15%
71.38%
69.81
20.55%
70.767
70.88%
20.26%
70.63%
70.01%
70.25%
69 96%
69.6955
69.01%
69.58%
70.43%
70.94%
1338
QSVD-ant)
71.34%
72. 14%
71.74
14%
21.79%
71.23%
71.02%
71.06%
70.9256
70.40%
21,44%
WSVD-as0
72,88%
72.98%
73.57%
73.49%
73.28%
71.29%
71.17%
71.25% %
70.95%
70.81%
72.17%
FP16
Accuracy: 73.23%
Accumay: 71.30%
72.27%
900%
801%
70%
5005
80%
70%
SmalVL.M M
ASVD
29.30%
3,970%
0.20%
17.85%
1,50%
0.95%
8.96%
28
SVD-LLM
40.06%
17, 17.20%
3.82%
12.49%%
15 8.%9
A.60%%
19.01%
QSVD-au0
77.00%
62.77%
42.50%
64 3055
50.46%
36.249
55.64%
76.30%
71.74%
60.93%
65.78
63.29%
54.45%
65.42%
FP16
N4.53%
Accuncy: 68.47%
76.53%
16

---

Published as a conference paper at ICLR 2026
A.4 COMPARISON WITH DOBISVD
For completeness, we additionally evaluate DobiSVD on LLaVA-v1.5 7B using the ScienceQA- IMG benchmark. Following the official implementation, DobiSVD applied the Q, K, V ma- trices while leaving other linear layers unchanged. Calibration is performed using the same set of samples and initialized with the same random seed as the main experiments to ensure fairness. As shown in Table 10 WSVD-noQ achieves higher accuracy than DobiSVD under the same compres- sion ratio.
Table 10: Accuracy evaluation of different methods under FP16.
ScienceQA-IMC
Acc.
Method
Pr : 90%
80%
P1 70%
Pr 60%
50%
LLa VA.v.
DobiSVD
67.19%
60.94%
59.38%
56.64%
54.69%
ASVD
49.93%
50.12% 12%
47. 47%10%
36.69%
19.19%
SVD-LLM
65.44%
63.71%
61.92%
57.41%
55.53%
QSVID-noQ)
67.72%
68.22%
67.08%
65.05%
62.37%
WSVD-noQ
68.17% %
67.72%
67.28%
65.89 %
65.49 %
FP16
Accuracy: 68.01%
A.5 SUPPLEMENTARY RESULTS ON MORE VLMs
We additionally apply WSVD to Qwen-VI 7B (Bai al. 2023 and Molmo-7B-O (Deitke et al. 2025) to further examine the generality of WSVD beyond the LLaVA family and SmolVLM, We fol- low exactly the same experimental setting Section 4 and evaluate the FP16 and SVD compressed models on ScienceQA-IME and SEED-Bench As shown in Table WSVD-noQ consistently out- performs all SVD-based baselines (ASVD and SVDLLM) across all singular-value ratios, and it also matches or slightly improves over the FP16 model. For example, on ScienceQA, WSVD-noQ im- proves over SVDLLM and ASVD by up 3-5%, and on SEED-Bench yields the best/comparable accuracy among all compressed variants at every ratio. These results indicate that WSVD transfers well to VLMs with different vision languaelanguage fusion designs, supporting the general applicability of our method.
Table 11: Accuracy evaluation of different methods under FP16 on Qwen- VL 7B and Molmo-7B -B-O.
Method
ScienceQA-IMGG
SEED-Bench f
Avg.
90%
Pj 80%
Pi 70%
60%
Pi 900
: 80%
Pj 70%
Pi 60%
ASVID
63.86%
63.11%
60.54%
59.49%
61.67%
61.00%
59.94%
58.69%
61 61.04%
SVD-LLM
65.29%
65.29%
64.55%
65.29%
62.99%
62.80%
62.69%
62.51%
63.93%
Quenn
7B
QSVI-noQ
66. 66.78%
66.24%
66.68%
65.20%
63.25%
63. 30.00%
62.06%
G1. 61.26%
64.33 %
WSVD=noQ
68.77%
68.12%
67.23%
65.49%
63.32% %
63.53%
63.60%
63.68%
65.47 %
FP16
Accuracy:
68.32%
Accuracy:
63.52%
65. 92.
ASVD
94,99 44.99%
19%
10%
93.06%
74, 74.42%
74.21
73.73%
73.62%
84.17%
Malma
7B-0
QSVD-noQ
SVD-LLM
95.09%
95.54%
95.09%
94. 94.99%
94.59%
94.99%
95.09%
93.85%
74.68 %
74.47%
74.46%%
74.44%
74.23%
74.41%
74.37%
71%11%
84.58%
84.72%
WSVD-noQ
95.59%
95.49 %
95.34%
95.09%
74.61%
74.52%
74.48%
74.38%
84.94 %
FP16
Accuracy: 95.78%
Accuracy: 74.74%
85.26%
A.6 SUPPLEMENTARY RESULTS ON MORE DATASETS
To further assess generalization, we additionally evaluate LLaVA-Next 13B with WSVD-noQ on two additional benchmarks: HRBench-44 (4K high-resolution images) (Wang al. 2025b), and OCRBench (text-centric images) (Liu et al. 2024c). Following Section 4 we reuse the same 256- sample calibration set drawn from the ScienceQA training set and keep all other settings identical, while sweeping the parameter ratios Pt. As summarized in Table [2 WSVD-noQ consistently matches or outperforms all baselines across nearly all ratios on these datasets, despite being cali- brated only once on the ScienceQA training set. These results indicate that WSVD generalizes well across tasks and datasets. Moreover, WSVD's decoding speedup is independent of the evaluation dataset: once the model is calibrated and compressed, runtime is determined solely by the resulting model size and context length, so a fixed compressed model yields essentially the same speedup across benchmarks.
17

---

Published as a conference paper at ICLR 2026
Table 12: Accuracy evaluation of different methods under FPI6 on OCRBench and HRBench-4K.
Method
OCRBench
HRDDecc
Pr:
p
pi:
Pr.
p1
Pr: PI:
pr:
LLaV Noxt
30%
47.20%
SVD-LLM
10%
51. 90.9
009%
39.909%
48.20%
43.000%
44.25%
42.62%
43.75%
43.25%
47.0085
1.38
QSVD no
$2.80%
32.40%
52 40%5
51.40%
48.90%
44.25%
44.88%
43.889
43.37%
42.88%
47.7218
WSVD-noO
53.30%
53.30%
53.50%
52.20%
48.70%
46.13%
44.88%
44.50%
44.88%
44.50%
48.59%
FP16
Aceerccy: 53,50%
Accuracy; 45.6396
49.47%
A.7 LATENCY COMPARISON OF FULL AND PER-HEAD SVD
We further compare the decoding latency of applying SVD the full QKV matrices versus adopting WSVD's fine-grained per-head SVD. To enable this comparison, we minimally modify our kernel support reconstruction with larger matrix sizes under the full SVD setting (as discussed in Sec- tion 3.2). while still fusing the reconstruction with flash decoding. This variant is denoted as "W/o per-heade " Both approaches are evaluated under the same P2, ensuring equal overall cache size, with batch size, sequence length and other settings kept identical to the setup described above.
Table 13: Decoding latency on RTX 4090.
Table 14: Decoding latency on RTX 5090.
P2
W/o per-head
WSVD-noQ
Specdup
Pz
W/ per-head
WSVD-noQ
Speedup
90%
51.31
2.83
18.1x
90%
40.40
1.87
21.6x
80%
46.03
2.60
17.7x
80%
35.99
1.77
20.3x
70%
39.46
2.53
15.6x
70%
31.14
1.75
17.8x
60%
33.54
2.25
14.9x
60%
26.85
1.56
17.2x
50%
28.37
1.66
17.1x
50%
21.44
1.15
18.6x
As shown in Tables 13 and 14 WSVD-noQ consistently achieves more than an order-of-magnituud speedup over the full-matrix SVD variant ("W/o per-head")) across all compression ratios P2- On RTX 4090, the speedup ranges from 14.9x to 18.1x, while on RTX 5090 it further increases to 17.2x-21.66 These results confirm that per-head SVD substantially reduces reconstruction over- head and I/O traffic, enabling efficient decoding.
A.8 TRAINING CoST OF WSVD
WSVD first applies SVDLLM's whitening method (Wang et al. 2024d) to per-head weight matri- ces before performing SVD, then uses QSVD's importance--core-basee rank allocation (Wang al.) 2025d) to truncate the model, and subsequently performs lightweight, Fisher-information- based cal fine-tuning and local quantization-awae training the truncated low-rank weights to better preserve the most sensitive weight elements and to mitigate the degradation of per-head SVD and low-precision inference.
Table 15: Calibration time breakdown for QSVD and WSVD.
QSVD
WSVD
Step
Time
Step
Time
Input calibration
min
Input calibration
min
SVD on all layers
2 min 15
SVD on all layers
min
Gradient collection & rank allocation
10 min 12
Gradient collection & rank allocation
13 min 40
SVD results fusion
30
Local FT
min
3 tuning & quantization
82 min
Local QAT & quantization
min
Total calibration time
96 min
Total calibration time
38 min
We quantify the computational overhead of WSVD's local fine-tuning and QAT and compare it with QSVD on LLaVA-1.5 13B. We extract QSVD's reported training time from its OpenReview page and, for faimnsss, benchmark WSVD on the same GPU type (A100). As shown in Table 15 QSVD requires 96 minutes x 1.6 A100 GPU-hours, whereas WSVD takes only 38 minutes z 0.63 A 100 GPU-hours, about 2.5x less tuning time. The peak GPU memory usage of WSVD's
18

---

Published as a conference paper at ICLR 2026
local FT and QAT stages LLaVA-1.3 13B is only 15 GB, since only perform local updates low-rank weights rather than end-to-end fine-tuning and do not store full intermediate activations, so the whole procedure fits comfortably on single A100-80GB. For additional context, the official LLa report (Liu et al. 2024b) states that training the 13B model requires at least 204 A100 GPU-hours. Thus, WSVD's tuning is only small fraction of the original training cost, while still delivering practical decoding speedups, indicating that the efficiency gains comfortably justify the modest local fine-tuning and QAT overhead.
19


Reviews:
[{"review_id": "y2StCWEsFp", "reviewer": "Reviewer_ewCT", "summary": "This paper proposes WSVD. By leveraging SVD, local weighted fine-tuning, and quantization-aware training, the method addresses the issues of high latency and accuracy loss in low-precision quantization of VLMs. Its fine-grained SVD reduces memory access overhead, and weighted decomposition preserves the importance of critical weight. The authors claim that the method achieves a 1.8× speedup in VLM decoding.", "questions": "I mainly have some questions about the validation of the method.\n1. I am quite curious about the basis for selecting rank r in per-head SVD. Is there any quantitative analysis on the specific impacts of different rank settings on model performance and latency?\n2. When using the Fisher information score as the weight importance indicator in weighted fine-tuning, has it been compared with other indicators (such as gradient magnitude)? What is the proportion of its contribution to performance improvement?\n3. In quantization-aware training, what impact do the settings of orthogonal matrices S₁ and S₂ have on the quantization effect?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper adopts per-head SVD operations, reducing memory overhead and latency in KV cache reconstruction. The method has a clear motivation, targeting the high memory access latency of traditional SVD. The paper has clear writing logic and readability, with a rigorous structure.", "weaknesses": "The paper lacks sufficient analysis on the theoretical basis for per-head SVD rank selection, and the sensitivity validation of weighted scoring is absent. Baseline comparisons fail to include some of the latest low-rank quantization methods, and the performance stability in extreme low-rank scenarios has not been fully explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TyvMbVegoT", "reviewer": "Reviewer_9Jd6", "summary": "This paper introduces WSVD, a weighted low-rank approximation framework for efficient low-precision vision-language models (VLMs). It addresses latency issues of existing SVD methods by applying fine-grained per-head SVD, assigning weight importance via Fisher information to preserve accuracy, and integrating quantization-aware training. A fused kernel merges low-rank reconstruction with Flash Decoding to cut I/O overhead. Experiments on 5 VLMs show WSVD achieves up to 1.8× decoding speedup, maintains accuracy (≤1% drop even at 50% parameter compression), and reduces KV cache to 25% of FP16 models, outperforming baselines like ASVD and SVD-LLM", "questions": "1. Could you provide experimental results of WSVD on other representative VLMs (e.g., BLIP-2, Qwen-VL) beyond the LLaVA family and SmolVLM? This would clarify whether WSVD’s design adapts to VLMs with different vision-text fusion mechanisms .\n2. Can you supplement latency and accuracy data of WSVD under variable sequence lengths (not just 8192)? This helps assess its adaptability to real-world scenarios with short/long inputs .\n3. Could you quantify the computational overhead (e.g., training time, GPU memory) of WSVD’s local fine-tuning and QAT, and compare it with baselines like QSVD? This enables evaluating if its efficiency gains justify the tuning costs .", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In terms of originality, the paper innovatively combines fine-grained per-head SVD, Fisher information-based weight importance scoring, and quantization-aware training with a fused kernel design, effectively addressing the latency-accuracy trade-off in VLM optimization that prior SVD methods failed to resolve.\n2. Regarding quality, the work demonstrates rigorous experimental validation across 5 VLMs, 2 benchmarks, and multiple GPUs, with detailed ablation studies confirming the effectiveness of each component and fair comparisons against diverse baselines.\n3. For significance, the WSVD framework achieves up to 1.8× decoding speedup while preserving accuracy and reducing KV cache to 25% of FP16 models, providing a practical solution for efficient VLM deployment on resource-constrained devices.", "weaknesses": "1. The paper’s evaluation of WSVD’s generalization across VLM architectures is limited, as it only tests models from the LLaVA family and SmolVLM—excluding other representative VLMs like BLIP-2 or Qwen-VL. This gap makes it unclear if WSVD’s per-head SVD and fused kernel design work equally well for VLMs with different vision-text fusion mechanisms (e.g., Qwen-VL’s high-resolution perception), and expanding tests to these models would strengthen its generalizability.\n2. The analysis of WSVD’s performance under varying sequence lengths is insufficient; experiments only use a fixed sequence length of 8192, while real-world VLM use cases (e.g., long image captions or multi-image QA) involve variable lengths. Without testing how WSVD’s latency and accuracy scale with shorter/longer sequences, it is hard to assess its adaptability to diverse input scenarios, which is critical for practical deployment.\n3. The paper lacks a detailed comparison of WSVD’s computational overhead during the fine-tuning phase. While it mentions local fine-tuning (100 steps) and QAT (50 steps) are lightweight, it does not quantify this overhead (e.g., training time, GPU memory usage) against end-to-end finetuning or baselines like QSVD’s optimization process. Providing such data would help users evaluate if WSVD’s efficiency gains justify its fine-tuning costs, especially for resource-limited teams.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ttEYT7xTX", "reviewer": "Reviewer_9c8b", "summary": "WSVD proposes a practical way to make vision-language models decode faster—not just smaller—by combining per-head low-rank factorization with smart tuning, quantization, and a fused kernel. It starts from the observation that plain SVD shrinks parameters and KV cache but often slows decoding because full keys/values must be reconstructed every step. WSVD instead factorizes each attention head separately so each head caches tiny latents and reconstructs only what it needs; then it uses a Fisher-weighted objective to fine-tune the low-rank factors so the most important weights are preserved. Afterward, it applies rotation-aware low-precision quantization and a short round of QAT to recover accuracy. On the systems side, a Triton kernel folds reconstruction directly into Flash Decoding so K/V are never materialized in VRAM, cutting memory traffic and kernel launches. Across LLaVA-Next and SmolVLM families on 4090/5090-class GPUs, this yields up to ~1.8× lower latency at similar cache sizes (≈25% of FP16) with ~1% average accuracy drop, and ablations show both the weighted objective and QAT are key to those results. The approach shines in memory-bound decoding (long contexts, image-heavy prompts) if brief local QAT is acceptable, though gains can vary with model and hardware.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Overall, this paper excels by sharply diagnosing why plain SVD can slow decoding and giving a compact analysis showing that per-head SVD cuts reconstruction compute and memory by a factor of r/R; it then backs that theory with a well-engineered Triton fused kernel that integrates into Flash Decoding to avoid materializing full K/V in VRAM; and it demonstrates measured gains—up to 1.8× faster decoding—while maintaining a strong accuracy–efficiency trade-off (≈1% average drop with the KV cache shrunk to ~25% of FP16). Careful ablations isolate the contributions of Fisher-weighted local fine-tuning and local QAT, and the rotation-aware quantization design (Hadamard S1 + learnable S2) is practical and lightweight compared with full end-to-end training.", "weaknesses": "WSVD’s evidence is somewhat narrow and system-dependent: accuracy and tuning are demonstrated mainly on ScienceQA-IMG and SEED-Bench-IMG with just ~256 calibration samples and a small set of VLMs, so generalization is unclear; its headline latency gains are reported for decoding on LLaVA-Next-7B under specific, favorable settings (e.g., batch 16, 8K tokens) on 4090/5090-class GPUs, and the paper itself notes access to high-end hardware as a limitation; practical speedups hinge on a custom Triton fused kernel integrated with Flash Decoding, raising portability questions to other runtimes/accelerators; per-head SVD increases approximation error, so Fisher-weighted fine-tuning and local QAT are required to recover accuracy—adding calibration data needs and engineering complexity; some baseline comparisons (e.g., Palu) adapt settings due to Palu’s batch-1 constraint, which complicates fairness; the method focuses on attention (Q/K/V) rather than also optimizing FFN blocks that can dominate runtime; and, until code/models are publicly released, independent replication and broader portability studies remain gated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nIiZDxqkmC", "reviewer": "Reviewer_YkZ1", "summary": "This paper proposes a method that accelerates vision-language model inference by combining Weighted Singular Value Decomposition (WSVD) with low-precision quantization. Unlike standard SVD-based compression, WSVD assigns importance weights to parameters to better preserve critical information during low-rank approximation. The approach further integrates quantization, leading to significant latency reduction, reportedly achieving faster decoding with minimal accuracy loss. Overall, WSVD aims to offer a practical and effective way to deploy large VLMs more efficiently without sacrificing performance.", "questions": "- The computational cost of reconstruction for each head is $LrH$. Wouldn’t it be equivalent to simply reducing $R$ to $r$ in the original $LRH$ formulation? How is this approach fundamentally different?\n- Why is the square root used in the Fisher matrix in Equation (8)?\n- There exist several established methods for measuring parameter importance [1, 2]. Why were these approaches not explored or compared? Is the proposed method guaranteed to be optimal?\n- The paper states that FWSVD utilizes Fisher information. How exactly do FWSVD and WSVD differ in the way they incorporate Fisher information?\n\nReferences\\\n[1] Ancona, M., Ceolini, E., Öztireli, C., & Gross, M. (2017). Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104.\\\n[2] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a practical and well-motivated approach to enhancing the efficiency of vision-language models. In particular, it provides a comprehensive exploration of memory reduction strategies from multiple perspectives, including low-rank decomposition and quantization, and proposes a corresponding system implementation to support these ideas.\n- The paper is clearly written and well organized, making the contributions easy to follow.", "weaknesses": "I'm not very familiar with this specific research area, such as quantization and system implementation. Therefore, it is difficult for me to precisely assess the novelty of the proposed method relative to existing work.\n\n- From my perspective, the contribution over prior work appears incremental. Please refer to the question below for clarification.\n- The experimental validation is limited to only two datasets, and broader comparisons across more diverse datasets are missing.\n- The related work section omits several important studies on weighted matrix factorization, which weakens the paper’s novelty claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)