LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models
Authors: Pengcheng Zheng
Chaoning Zhang
Jiarong Mo
GuoHui Li
Jiaquan Zhang
Jiahao Zhang
Sihan Cao
Sheng Zheng
Caiyan Qin
Guoqing Wang
Yang Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Multimodal Models
Model Compression
Fourier Domain
Matrix Approximation
Primary area: foundation or frontier models, including LLMs

Abstract: Large multimodal models (LMMs) have achieved impressive performance on various vision-language tasks, but their substantial computational and memory costs hinder their practical deployment. Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors, especially in multimodal architectures with cross-modal redundancy. To address this issue, we propose LLaVA-FA, a novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain. By leveraging the de-correlation and conjugate symmetry properties of Fourier transform, LLaVA-FA achieves more compact and accurate weight representations. Furthermore, we introduce PolarQuant, a polar-coordinate quantization method tailored for complex matrices, and an optional diagonal calibration (ODC) scheme that eliminates the need for large-scale calibration data. Extensive experimental results demonstrate that our proposed LLaVA-FA outperforms existing efficient multimodal models across multiple benchmarks while maintaining minimal activated parameters and low computational costs, validating its effectiveness as a powerful solution for compressing LMMs.

Published as a conference paper at ICLR 2026
LLAVA-FA: LEARNING FOURIER APPROXIMATION FOR COMPRESSING LARGE MULTIMODAL MODELS
Pengcheng Zheng', Chaoning Jiarong Mo', GuoHui Li', Jiaquan Zhang', Jiahao Zhang", Sihan Cao', Sheng Zheng', Caiyan Qin', Guoqing Wang', Yang Yang!
'University of Electronic Science and Technology of China
2Chengdu University of Information Technology Harrbn Institute of Technology, Shenzhen xpc777@std std.d.. .est...eeet... edu.cn, cn, chaoningzhan 9908 gmai i com
ABSTRACT
Large multimodal models (LMMs) have achieved impressive performance on vari- vision-languse tasks, but their substantial computational and memory costs hinder their practical deployment. Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors, especially in multimodal architectures with cross-modal redundancy. To ad- dress this issue, we propose LLaVA-FA, novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain. By leveraging the de-correlation and conjugate symmetry properties of Fourier transform, LLaVA- FA achieves more compact and accurate weight representations. Furthermore, we introduce PolarQuant, polar-coordinate quantization method tailored for complex matrices, and optional diagonal calibration (ODC) scheme that eliminates the need for large-scale calibration data. Extensive experimental results demonstrate that our proposed LLaVA-FA outperforms existing efficient multimodal models across multiple benchmarks while maintaining minimal activated parameters and low computational costs, validating its effectiveness powerful solution for compressing LMMs.
INTRODUCTION
In the past few years, large multimodal models (LMMs) have shown its extraordinary performance on multiple down-stream tasks, such as zero-shot classification (Ge al., 2023 [dandarao et al. 2024). image-text retrival (Long et al. 2023 Liu et al. (2025) and multimodal dialogue generation (Liu et al. 2024a; Koh et al. 2023). Despite their substantial applications, LMMs are computationally expensive, which limits their broader usage. For instance, the training of LLaVA 70B models necessitates a total of over 800 GPU hours, as calculated based on NVIDIA A100 GPUs (Li et al., 2024a). The substantial computational costs of LMMs present ongoing challenges. Aside from training, the inference of LMMs requires significant electricity consumption, which raises concerns about environmentally friendly AI. In light of this, developing efficient LMMs to reduce the memory and computational requirements is becoming crucial to ensure their broad accessibility.
Considering the correlated nature of multimodal semantics during training, the weight matrices of LMMs often exhibit redundancy, which manifests as a low-rank structure (Saha et al. 2024). To this end. one promising framework for memory -ffoorrefffiiientt L.MMs through garameter-efficatt fine-tuning methods, e.g., low-rank adaptation (LoRA) (Hu et al.) 2022), where the pretrained models weight matrix W reparameterized as W + L, L2. Recent studies such as LoRD (Kaushal al.. 2023). ASVD (Yuan et al., 2023), FWSVD HHsu et al.) LASER (Sharma et al. 2023). LQER (Zhang et al. 2024), and ZeroQuant- V2 Yao et al. 2024b) have explored the efficacy low-rank structures in large language models (LLMs) weights, utilizing LoRA technique compress LLMs. However, these approaches treat low-rank factorization and quantization independently, which leaves the rank-selection stage blind to the impending quantization noise and inevitably compounds the final reconstruction error (lacovides et al.] 2024). Therefore, Saha et al. (Saha et al. 2024). approach
"Corresponding Author

---

Published as a conference paper at ICLR 2026
this by uniquely formulating joint optimization problem for generic matrices, providing additional flexibility for compression.
Due to its simplicity and astounding accuracy, the low-rank with quantization paradigm dominates in the field of efficient LLMs recently. However, we argue that this paradigm still faces storage challenges when handling extensive customization approximation for large vision-language models. For example, unlike pure-text LLMs, large vision-language models carry an extra tower of image encoders whose cross-modal adapter ranks balloon with every new visual domain, so the same low-rank with quantization recipe that tamed LLMs still leaves their multimodal stack uncomfortably obese. To this end, we naturally ask the question: How can we aggressively compress learnable parameters even further for compressing large vision- -aiionla models?
Previous works have demonstrated the powerful expressiveness of Fourier transform in data com- pression, where extremely sparse spectral information can be used to recover high fidelity 1D signal
vectors (Rudelson & Vershynin 2008 Duarte & Baraniuk 2013 Zwartjes & Gisolf 2007 Luo
et al., 2026) and 2D image matrices |Vlarddnnerbbrek & Boer 2013 [Song et al.) (021). More importantly, when dealing with more general matrices, e.g., weight matrices of neural networks that lack strong spatial semantics and are not frequency-spare, Fourier transform can still handle approximation effectively (Chen & Chi [013 Yang & Xie 2016). Due to the inherent de-correlation capabilities of frequency domain, observe that the weight matrices of LMMs in the frequency domain have a more compact spread of singular values as compare to spatial domain. Furthermore, provide both theoretical and analytical evidence demonstrating that the accumulated error for matrix factorization in the frequency domain, utilizing a low-rank approximation, is smaller than that in the spatial domain at the same rank. This effectively enhances generalization performance. Additionally, benefiting from the conjugate symmetry of Fourier transform, approximating weight matrix in the frequency domain can save nearly half learnable parameters compared to the spatial domain. Motivated by these analyses | we investigate the potential for compressing the weight matrix of LMMs with matrix approximation in the frequency domain. We therefore ask a natural question: Can we harness the de-correlation, conjugate symmetry, and mneryyccoppattion merits of Fourier space to perform LoRA-style low-rank plus quantization compression in a single shot?
To answer this question, we design novel efficient and high-quality LMM with Fourier approx- imation, coined as VA-FA, which approximately decomposes weight matrices into low-rank plus quantized weight in the frequency domain. We devise a novel PolarQuant codec that separately discretizes amplitude and phase in polar coordinates. The conjugate symmetry of Fourier transform allows to store only half of the coefficients, while the energy concentration guarantees smaller Frobenius error for the same rank compared with spatial- an optional diagonal calibration scheme that approximates the full Hessian with row/column means, eliminating the need for large-scale calibration data P. Extensive experiments demonstrate that LLaVA-FA surpasses existing works across various benchmarks while maintaining minimal activated parameters and low computational costs. In conclusion, the main contributions of this paper are summarized as follows.
truncation. Moreover, we derive
We introduce LLaVA-FA, a novel efficient LMM that decomposes the weight matrices into a low-rank plus quantized weight via Fourier approximation.
We design PolarQuant, amplitude-and-phant polar codec that tailored for quantizing complex matrix. It preserves complex structure and stabilizes the low-bit reconstruction.
We introduce an optional diagonal calibration (ODC) scheme, which can approximate the full Hessian matrix with row/column means, which enables more robust compression without large- scale calibration sets.
Extensive experimental results demonstrate that LLaVA-FA surpasses existing works across various tasks, including comprehension-oria and callucination-orientee benchmarks, while maintaining minimal activated parameters and low computational costs.
We give detailed proof of these analyses in Appendix B.1
"This approximation leverages the empirical observation that Hessian matrices in deep networks often exhibit diagonal dominance or low-rank structure, enabling accurate low-rank approximation via row/column means (LeCun al. 1989 Botev al. 2017.
2

---

Published as a conference paper at ICLR 2026
6044
033
2303
Training Samples (NQ
1382
302
MocChea
30
96.3
Figure I: Comparisons training cost and performance. LLaVA-FA achieves comparable performance with advanced LMMs using lower training costs while outperforming current efficient LMMs by large margin.
2
RELATED WORKS
2.1 LARGE MULTIMODAL MODELS
The evolution of LMMs compression begins with LLM-Pruner (Ma et al. 2023 which adapts task-agnostic pruning from LLMs to LMMs. Subsequent studies highlighted that, beyond weight parameters, the predominant efficiency bottleneck LMMs arises from the proliferation visual tokens. To address this, ALaVA-PruMerge (Shang et al. 2024) introduces input-level optimization by adaptively retaining key tokens via attention scores, reducing up to 14x tokens while preserving performance. DivPrune (Alvar et al. (025) advances this line by emphasizing token diversity rather than mere importance, mitigating redundancy and ensuring richer visual representation. Cross- GET (Shi et al.) further pushes token-centric acceleration by ensemble-guided token selection, achieving > 2x throughput on VQA benchmarks. Complementary to token pruning, UPop (Shi et al. 2023) proposes progressive magnitude-based pruning scheme that trims 40% of weights across vision-language layers with < 1% accuracy drop. Finally, CASP (Gholami et al. 2025) shifts focus inward, exploiting cross-modal attention sparsity to compress attention matrices W Wi with theoretical guarantees, thus marking the transition from generic LLM techniques to multimodal- specific compression paradigms. Unlike the token-centric CrossGET (Shi et al.) and weight-centric UPop (Shi et al., 2023), our method directly compresses both vision and language parts in one shot without pruning tokens altering the architecture. Instead, approximate the full weight matrix in the frequency domain via joint low-rank plus quantization, enabling calibration free deployment and offering a drop-in replacement for weight-pruning when zero-shot compression desired.
2.2 FOURIER TRANSFORM
LLaVA-FA
Y,
In deep learning, applications of the Fourier transform in the fre- quency domain have gradually evolved from straightforward detec- tion tasks to broader computational optimization. Jeong et al. (Jeong
Fourter LLM
et al. 2022) introduce BiHPF, amplifying frequeney-domain artifacts Via bilateral high-pass filters for improving cross-domain general- ization. Advanced with this, FreqNet [Tan al., 2024) enforces continuous high-frequene focus through frequency domain learn- ing and convolution layers on phase/amplitude spectra, significantly enhancing detection on unseen domains. Recently, Yang al. (Yang) ct al., 2025) extend frequency analysis to model acceleration with TS. which separates diffusion model token into high and low- frequency subsets, achieving markedly speedup without retraining by selectively reducing low-freque www-freuency features. To the best of our knowledge, we make the first attempt to perform a joint low-rank plus quantization approximation directly in the frequency domain.
H
Hp
Fourler
f
X Muttimeds
X,
Fourier- Vision Encoddr
Figure The architecture of proposed LLaVA-FF
3

---

Published as a conference paper at ICLR 2026
2.3 Low RANK ADAPTATION
The immense computational and storage costs of fully fine-tuning LLMs have spurred the develop- ment of Parameter-Efficiae Efficient Fine-Tuning (PEFT) techniques. Among these, Low-Rank Adaptation (LoRA) (Hu et al. 2022) has become a foundational method. Subsequent research has focused on enhancing LoRA's efficiency and efficacy. For efficiency, QLoRA (Dettmers al. 2023) makes a significant impact by propppaaggg gradients through frozen, 4-bit quantized base model into the LoRA adapters. To improve efficacy, DoRA (Liu al. 2024d) decomposes pre-trained weights into magnitude and direction, applying LoRA specifically y to the directional component, which outperforms standard LoRA. The application LoRA has extended multimodal and multi-task learning, LLaVA-MoLE (Chen et al., 2024b) leverages sparse mixture of LoRA experts to reduce data conflicts by routing tokens to task task-specialized modules. Our approach integrates LoRA with quantization techniques to construct more efficient LMM.
3 METHODOLOGY
We introduce LLa a novel framework for building an efficient LMM via Fourier approximation. As shown in Fig. 2 LLaVA-FA consists of four main modules: Fourier-vision encoder tasked with receiving and processing visual inputs X., a text tokenizer to map text prompts X, to the text tokens Hş. and a Fourier Visual-Language Projector (Fourier- which functions as a bridge to align the two modalities. Then, text tokens H, and the visual tokens H, are concatenated as the input of the Fourier-based LLM, which outputs the final response sequence Y, in an autoregressive manner:
HYa.HH Hg) = II p(u: He, Hg. Vei),
where L denotes the length of Y.. LLaVA-FA relies an elaborate factorization scheme, which decomposes each pretrained matrix into two low-rank complex matrices plus a quantized complex matrix via 2D-Fourier transform.
3.1 LOW-RANK PLUS QUANTIZED WEIGHT DECOMPOSITION
Due to the nature of language syntax and visual semantics learned during training, the weight matrices of LMMs often exhibit redundancy, which manifests low-rank structure. As shown in Fig. 3 the singular value profile of the weight matrices in LMMs follow a decaying profile. To this end, we can approximate the weight matrix of a neural network, W, as a low-rank plus quantized weight decomposition, i.e. W = Q + L2 The low-rank factors L1 and L2 capture the effect of the large singular component of W with full precision. Moreover, the backbone matrix Q is quantized to low- precision format - for instance, using BQ = 2 bits, coarsely captures the essence of the moderately decaying and low singular components of W. We approach this problem by approximately solving this following minimization problem:
min || || - (Q + L,L2)lle subject to Q Q,
Qdi xd2 L1 E Rdink and L2 E Rxxd2
Here, Qdhxx22 C Rd xd2 denote the lattice codebooks used to quantize Q, using BQ bits. This strategy leverages the approximate low rank structure inherent in weight matrix, where lesser contributing singular components can be pruned with minimal impact on the functionality of the matrix.
Discussion we provide the analyses of computation of average bit budget per-parameter: We assume that each transformer layer have seven weight matrices (i.e., query, key, value, output, gate, up and down) with dimensions of d di. where i (1,... 7). In general, if the backhone matrix Q has bit budget of Bo, the low rank factors both have full bit budget of BL, then the average number of bits per parameter Bavg
Bavg Σ
Ba di (B + d2))
d d
(3)
When ts < (1
(the detailed derivations can be found in Appendix B.5 the average bits Bavg is smaller than the full precision bits B For LLaMa3-8B models, the query
Bg Be
di +42))
E
Eis
4

---

Published as a conference paper at ICLR 2026
Smigullr values LiaM6-3 Quary metrices
4
Spatial Domain
Domain
Hugnttale of Smgalar volue
W
Q
Lospedion
IDFT
4,/2
4/2
dy/2
W
W-4
[d
Q
203
300
Hill periaaa
Putl predicco
Figure 3: Illustration of Fourier approximation. Left: Singular value spread in the spatial domain vs. frequency domain. Right: LLaVA-FA decomposes real-valued weight matrix W into a frequency-
low-rank component (L, and L2) that keeps the top-r singular values with b,. by bits, plus PolarQuant residual Q for the remaining spectrum. The de-correlation and conjugate symmetry of Fourier transform lets us store only half of the complex coofficients, yielding the same rank as spatial- truncation but with fewer parameters and smaller Frobenius error.
and output matrices are 4096 4096, the key and value matrices are 4096 1024, the gate and up projections are 14336 x 4096, and the down projection 4096 x 14336. For common LLMs configuration, Bavg B₁, apparently holds.
Algorithm 1: Fourier Approximation
Algorithm 2: ODC
Input
:omplex weight matrix W, Calibration mairix C, Target rank r, Amplitude
Input
:oompeex residual matrix R Q.
Calibration matrix Target rank
bitwidth by, Phase bitwidth by
if is None then
Initialize Q and 60 +
Perform Fourier SVD on complex matrix
for /. to T do
Ú. E, V <- FourierSVD(R))
# Optional Diagonal Calibration Decomposition
# Keep only top singular values
L. L2 ODC(W Q C,r) # Polar Quantization
Σ, x[1:r,1:r]
Ü, U:.1:
Q PolarQuant W Lr,
V < V
L Ü.VE, VE.V.
if C None then
# Weighted error
else
W - ( ا. ااااحا|
D + Row RlowAveage Avergger
Deat -CoAAveagee(C
else
Ü. E, V <- -ourienSVDDDD....u RD..)
# Calibration weighted error
Keep only top r singular values
VCo W ( [.
E. + r[
U, U:
end
4 +- (H(.1:r)
c
> c--1 then break
L D U.VE
end
La + VE.V
end
end
Output :Q L,, T2
Output :T,,, E
3.2 WEIGHT APPROXIMATION IN THE FREQUENCY DOMAIN
LLaVA-FA substitutes each weight matrix W a LMM, with its approximate low-rank plus quantized weight decomposition in the frequency domain. Our approach is motivated by the de- correlation of frequency domain transforms, which minimizes spectral redundancy and effective in compression since they concentrate most of the energy in few coefficients.
Lemma 3.1. Let 1W1, AW2 values (sorted descending), and r is the kept rank. If Ti,k < f2r every k 2 r + 1, then
be adapters of identical shape, 01,k, 02,k their k-th singular
AAW AW || < AAW, (
(4)
where AW(") denotes the rank-r truncation of AW.
5

---

Published as a conference paper at ICLR 2026
We provide detailed proof for the above lemma in Appendix [B6 As shown in Fig. observe that the weight matrix has a more compact spread of singular values in the frequency domain as compared to spatial domain. To this end, using Lemma 3.1 we can say that the accumulated error for low-rank approximation is smaller in the frequency domain compared to spatial domain with the same rank. At the same time, the conjugate symmetric property of Fourier transform implies that matrix in the real spatial domain with the shape [d1, de] can be mapped into the complex frequency domain with the compressed shape [d1, / without losing information. By leveraging this property, approximating matrix in the frequency domain can save nearly half learnable parameters. Therefore, we approach the optimization problem of Eq. [2 from the perspective of Fourier transform where we first convert the real weight matrix W E qddidd into the complex weight matrix W E C xd2/2 via 2D) Discrete Fourier transform:
B.6
M-IN-1
wu, Σ Σ
W (m, n) -2xxxxxx 4F
+
(5)
m=0 n=0
where u 0, 1..... ., M and v
are the deqquency-oomile indices, j is the imaginary unit. Then, our goal is to obtain a decomposition W ~ Q L2 in the frequency domain by approximately solving the minimization problem:
min VCCa (-(Q+L,
subject to Q P xddx2d//2 bo
L E Cdixr, and L E Crxda/2
Q.L,
(6) Here, VC Rdi xd2/2 is the calibration data that aims to preserve the Frobenius norm error of the compressed layer output activations. Following common practice in calibration- quantiza- tion (Nagel et al., 202I Dong et al. 2019), we construct C as the dementtwise squared activation expectations over a small subset of training samples, i.e., Cij = EzaDam (a)()))) where Deatib denotes the calibration dataset (usually 256-2048 held-out samples) and (x(x) is the activation of neuron under input x. @ denotes the Hadamard product. |. represents the modulus, iee. magnitude
of the complex matrix. To be note worthy, pdi xdz/² Cdh xda/2 denotes the designed PolarQuant
b,,bb
lattice codebooks used to quantize the complex matrix Q using amplitude bitwidth b, and phase bitwidth bg.
3.3 COMPLEX MATRIX OPTIMIZATION WITH OPTIONAL CALIBRATION DATA
Following iterative algorithms which have been shown to be effective for matrix approximation (Guo
et al.
2023 we heuristically solve Eq. 6 via alternating between optimizing L, L2, and Q:
(8) L < ODC(W- (tt))r)) 1,,,
(+)
arg min
W (Q (t-1))
La
(7)
Q ( PolarQuant W- I
W - (Q + (Q++[91 1
arg min
xdg/2
where (00) is initialized to 0. Unlike SVD for real matrix, FourierSVD operates on complex matrix, which decomposes any complex matrix R E Cd xd2/2 into ÜΣV", where Ú Cd xdy and V e dd dz// are both unitary, i.e., Ü Im. V Im. V denotes the conjugate transpose of V. Σ E dRdxxdd/2 is real, diagonal and non-negative, Σ diag(a),..., miin....))))..))) ak 2 0 the singular values). In this paper, we keep only top singular values to get L1 and Lg by leveraging the low-rank property of matrix, i.e., L, t U,VE,, L, + VE, V.. Although recent works (Saha et al., 2224]] Guo et al., (023)) demonstrate the importance of using calibration data for quantizing LLMs, the performance of these approaches highly relies on the availability and quality of calibration data, which may not always be accessible or representative. To mitigate this issue, we design optional diagonal calibration (ODC) scheme to optimize the complex matrix optimization problem. Unlike its unweighted counterpart, calibration-weightee problem is in general intractable and in fact NP-hard. It is typically addressed through approximate methods, we consider the calibration-awar aware version of this approach by using a diagonal approximation of the calibration information matrix to weight the reconstruction objective:
L Lg = arg min
VVCo
= arg min
D ED
(8)
L.
L, L.

---

Published as a conference paper at ICLR 2026
where given E W
Drow is diagonal matrix consists of row-means of vC, and (i+I,
Deal is a diagonal matrix consisting of the column-means vC, i.c.
Doow= [[avs(v (ovg(vCC................ avg( (Cd,,))
Desl= diag (Ing(vC..)..... avg( (C,
(9)
To this end, this problem can be solved by the FourierSVD algorithm:
U. t FourierSVD(D low RD cot). L1 f D-D U, Er, Le VE, V col
(10)
Here, we keep only top singular values of Σ, i.e., Σ, f - :r,1: U, - Üi: 1|::1:1] VH
[; r.. We provide a detailed theoretical and empirical justification of ODC in Appendix B.7 The detailed computational process of the proposed ODC algorithm is described Algorithm After obtaining the low-rank approximation L, and L2,
Algorithm 3: PolarQuant
the residual complex matrix R w L,L, still con-
tains non-negligible frequency-doman energy. To fur- ther compress this residual, we propose PolarQuant, polar-coordinate-bsse quantization method tailored for complex matrices. Specifically, PolarQuant first decom- poses each element of R into its real and imaginary components, i.e., X = Re(R), Y (m()). Then, converts each complex entry into its polar form, i.e.,
Input :Complex residual matrix
R = W L, Amplitude bitwidth by. Phase bitwidth by
Extract real and imaginary parts [m(R) Convert to polar coordinates for each element (i. do
X + Re(R),
T1. v X Y2,
0,,3 (X,,
riy
X2 +Y
0,g = atan2(X.j,. Xi),,
(11)
end
Compute quantization parameters Tmax f max(r:,))
where Tay and Ozy represent the amplitude and phase of the residual frequency component at location (t.j).
Ar 7max/(2fr- -
Dg 2=/(22w)
respectively. Next, compute uniform quantization step sizes for amplitude and phase,
Quantize amplitude and phuse for each element (i.j) do
9rais <- round(rig// /A) A.)
max(ra)) Ar 26. -1
2T
20.1. + roundd (Gi.) + =)/4))
AO
(12)
Fi.j Pr,ij Ar
20
6., + 40.1. Ag -
where by and by are the bit-widths for amplitude and phase, respectively. Each amplitude and phase value is then independently quantized, i.e., griij round(r.,//Aa Ar), (e.i. round(0,, + π 7/))). Fi-
end
Reconstruct complex matrix for each element (i.j) do
end Quj + Pt.j "..
nally, the quantized complex matrix Q is reconstructed QuanizzQuattized complex matrix Q
as Q = fr.j eسo.., This approach allows to repre- sent the residual matrix in a compact polar format while preserving the phase structure critical for cross-modal alignment. The entire process summarized in Algorithm 3 We employ a simple stopping criterion where we keep track of the error and terminate the algorithm if the error increases, i.e., 4 4-11 The proposed Fourier approximation algorithm is shown in Algorithm T
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Implementation Details. The proposed architecture employs pretrained CLIP-ViT-L/14 (Radford
al. 202T) as the vision encoder, which is post trained via the Fourier approximation. A two-layer
MLP that compressed via Fourier approximation functions as the VI.. For the I.M,, Qwen-2.5 Huu al. 2024) series models at different scales are utilized as the foundation for LLaVA-FA. Specifically, LLa FA- and LLaVA-FA 1B are derived through compression from Qwen-2.5-7B and Qwen- 2.5-3B, respectively. Moreover, LLa VA-FA -F--BB and -LLVA--F--3 are derived through compression from InterLLM-2-200 (Cai et al. 2024) and LLLMAA3-8B (Dubey al.) 2024), respectively. LLaVA- FA is trained using NVIDIA RTX 4090 GPUs. The detailed training strategy and hyperparameter
illustrated in Appendix C.I
Training Datasets. LLaVA-FA is trained on a diverse collection of lision-language datasets spanning pre-training (e.g., LLLVAA-Pea V--rrrraii (Liu et al. 2023b), ShareGPT4V (Chen al. 2024a)), visual
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with tfte-of-the-art LMMs the commonly-usee multimodal benchmarks for LMMs. #Sample: Training data sample. #Paara: Trainable parameters. SQA': ScienceQA test, VQA": TextVQA val. MME: MME Benchmark, normalized to percentage, MMB: MMBench dev, MMB CN MMBench- Chinese dev. The optimal result is shown in bold, and the sub-optimal result is underlined. Our LLa LLaVA-FA achieves the hest average result for both.
Method
LLM
#Sample #Param
GQA
VisWiz
SQA'
VQA"
MME
MMB
MMB
AVG
BLIP-2
Vicuna-133 13B
129M
41.0
19.6
61.0
42.5
64.7
VILA-7B
LLaMA- 7B
50M
62.3
57.8
68.2
64.4
76.7
68.9
61.7
65.7
CogVLM
Vicuna-7B
1500M
64.9
78.2
71.8
63.7
53.8
InstructBLIP
Vicuna-13B
130M
49.5
33.4
50.7
60.6
27
Owen-VL-Chait
Qwen-7B
1450M
57.5
38.9
68.2
61.5
74,4
60.6
56.7
56.7
Deepseck-VL-7B
DLLM-7B
2000M
61.3
49.9
74.0
64.7
73.4
74.1
72.8
67.2
LLaVA-1.5-78
Vicuna-1.5 1.57-
1.2M
62.0
50.0
66.8
58.2
75.5
64.3
58.3
62.1
LLaVA-NeXT
Vicuna-1.5 1.ana-1.5-13B
1.3M
65.4
60.5
73.6
67.1
78.7
70.4
64.4
68.5
LLaVA-FA-78 78
InternI. M--2-088
5M
68.5
62.0
76.0
68.0
74.5
74.5
69.5
70.4
Imp-3B
Phi-2-2.7B
1.6M
63.5
54.1
72.8
59.8
72.3
72.9
46.7
Bunny-318
Phi-2-2.718
2.7M
62.5
74.4
VILA-3B
LLaMA-2.7B
5IM
61.5
53.5
60.4
72.1
63.4
52.7
61.8
MobileVLM
MIL.MMA22.7B
1.3M
59.0
61.0
47.5
64.4
59.6
MobileVLMM
MIL.MMA-2.7B
3.6M
61.1
72.0
63.2
MoE-LLaVA-338
Phi-2-2.718
2.2M
61.4
43.9
68.5
51.4
71.1
65.2
41.8
57.6
MiniCPM-V
MiniCPM-2.4B
S70M
51.5
50.5
74.4
56.6
68.9
64.0
62.7
61.2
MiniCPM-V: V-2
MiniCPM-2.4B
570M
52,1
60.2
76.3
73.2
70.5
68.5
67.2
66.9
LLaVA-FA-3B
LLaMA-3-8B
5M
65.0
62.5
77.0
64.0
71.0
70.5
68.0
68.3
1mp-28
Owen-11.5-1.80
1.6M
61.9
39.6
66.1
54.5
65.2
63.8
61.2
58.9
Bunny-2B
Qwen-115-1.8B
2.7M
59.6
34.2
64.6
53.2
65.0
59.1
Mini Mini-Gemini-28
Gemma-2B
2.7M
60.7
67.0
57.1
2B
MoE-LLaVA-2B
Qwen-1.5-L.8B
2.2M
61.5
32.6
48.0
64.6
59.7
57.3
55.3
DeepSeck
VL--1.33
DLLM-1.3B
2000M
59.3
36.8
64.2
58.4
65.3
64.6
61.0
58.5
LLaVA-FA-2B
Qwen-2.5-7B
5M
62.1
41.7
68.2
59.3
66.6
66.7
61.7
60.9
SPHINX-Tiny
TILaMA-1.1B
15M
58.0
49.2
21.5
57.8
63.1
56.6
37.8
49.2
1B
LLAVA-FA-IB
Qwen-2.5-3B
5M
56.7
49.7
61.3
$7.9
63.3
58.3
49.4
56.7
question answering across general (GQA (Hudson Manning 2019). VQA (Goyal al. (017)). text-centric (OCR-VO) (Mishra et al.
2019). SynthDoG-EN (Kim et al. 2022)), and specialized domains (ChartQA (Masry al. 2022) 2. Doc VQA (Clark & Gardner (018)), visual reasoning and instruction-followin (MIMIC-TI (Li et al.) 2023a), LRV (Liu et al. 2023at). and acrge-scale web data (RefCOCO (Yu et al. 2016), Venome (Krishna et al. (017)). This comprehensive mixture covers natural images, documents, charts, diagrams, and synthetic visualizations, balancing general visual understanding with specialized capabilities in OCR and spatial reasoning.
Evaluation Benchmarks, conduct experiments three multimodal Denchmarks-MMM. (Fu et al. 2.24 MMB (Liu et al. (2024)), and MMB -to evaluate multimodal understanding and reasoning capabilities. Our evaluation also encompasses diverse VQA tasks across three domains: general visual understanding using VizWiz (Gurari et al. 2018) and GQA (Hudson & Manning 2019); text-oriented
recognition via TextVQA (Singh et al., (2019;; and scientific reasoning through ScienceQA (Lu et al. 2022). Additionally, assess model reliability using hallucination benchmarks including POPE (Li
et al. 2023b), Object HalBench (Yu et al. 2024), and MMHal-Bende (Sun et al. 2024).
4.2
MAIN RESULTS
This section evaluates LLaVA-FA across two critical dimensions: model performance and computa- tional efficiency. Performance assessed through comprehensive results on comprehension-orien benchmarks (Tab. and hallucination-oriiniee evaluations (Tab. 2). Efficiency analysis exam- ines training data requirements and model parameters. Furthermore, Appendix C.2 evaluates the real-world inference overhead introduced by DFT/IDFT and PolarQuant operations, demonstrat- ing that their latency contribution is minimal (3.2% % of TTFT) and more than compensated by compression gains.
8

---

Published as a conference paper at ICLR 2026
Comprehension--Oriette Benchmarks. As shown in Tab. I LLaVA-FA achieves competitive results across comprchension-ore -oriented benchmarks. With only 5M training samples, the 2B variant obtains average score of 60.8%, surpassing MoE-LLaVA-2B (Lin et al. 2024) by 5.5% and 2024b) by 3.7%. Despite its compact size, LLaVA- FA-2H approaches the Mini-Gemini-22 (Li et al. performance of LLa VA-1. 5-7B (Liu et al. 2024c) (62. 1 1% vs. 60.8%) while using fewer parameters. [Lu et al., 2024b) performance (49,4% vs. 49.2%) with substantially reduced training data (5M vs. 15M). Notably, LLaVA-FA's strong performance on VQA" and MMB benchmarks, demonstrating its effectiveness in multimodal understanding tasks. We further evaluate our method on substantially larger backbones:
The IB variant matches SPHINX-TinY
constructed from LLaMA-3- 8B (Dubey et al., 2024), and LLa LLaVA-FA-7B. constructed from InternLM-2-225 (Cai al.) 2024). Importantly both models are compressed using exactly the same Fourier approximation pipeline, ensuring strict fairness and demonstrating that our method generalizes without any modification or heuristic tuning. Experimental results show that Fourier-domain compression continues to yield strong performance across diverse benchmarks, with even smaller relative degradation at the 8B/20B scales. These results verify that our Fourier approximation method seamlessly extends to LLMs far beyond Qwen-2.5, maintaining both accuracy and robustness while preserving the same compression pipeline across all model sizes. This confirms that the proposed method is truly model-famen model-family agnostic and highly scalable.
Hallucination-O0 Ciiiiattio--Orenteed Benchmarks. As shown in Tab. LLaVA-FA demonstrates exceptional performance in mitigating hallucination, achieving competitive results despite its compact 2B pa- rameter size. It can be attributed to the effectiveness of our approach from two aspects: Firstly, by maintaining high factual accuracy with an Fl score of 87.5% on POPE, LLaVA-FA ensures reliable and grounded visual understanding. Secondly, by achieving remarkably low hallucination rates (11.2% response-level and 7.7% mention-level on Object HalBench), it significantly reduces the generation of incorrect or unfounded information. Notably, LLaVA-FA outperforms other 2B-scale models including MiniCPM-V-2 Yao et al. 2024a) by 3.3% in response-level hallucination rate and Mini-Gemini-22 (Li et al. 2024b) by 18.5%. Furthermore, it even surpasses several larger 7B models such as VCD (48.8% response-level) and POVID (48.1 1 response-level) by over 37% on Object HalBench, demonstrating that our approach enables efficient hallucination mitigation without requiring extensive model scaling. The strong performance on hallucination benchmarks can be explained by the properties of the proposed Fourier-domain decomposition. Specifically, applying the Fourier transform reduces cross modsll redundancy and produces more compact singular value distribution. As discussed and supported by Lemma 3.1] the low-rank truncation in the frequency domain introduces smaller approximation error than in the spatial domain under the same rank. This leads to more stable intermediate representations and reduces the error propagation that often triggers hallucination in multimodal reasoning (He et al. 2025 Sun al. 2024).
Floating Point Operations and Latency In practice, we evaluate the FLOPs (Floating Point Operations) and latency of LLa VA-FA under varying numbers of vision tokens using calflops and our inference benchmark, both conducted on eight RTX 4090 GPUs. Specifically, we measure FLOPs, Time First Token (TTFT), and KV cache usage within the same multi-GPU environment. As illustrated in Tab. and Fig. 4 VL-FA-FA achieves substantial reduction in computational cost compared to 1mp-2B. Furthermore, our method lowers KV cache usage by 83.3% and improves inference speed by 61.1% in TTFT, outperforming Imp-2B under equivalent visual token counts. These improvements highlight the efficiency of Fourier approximation approach, which is crucial for enabling real-time deployment of LMMs resource-constrainee devices.
Table 3: Flops and latency comparisons of LLaVA-FA.
#I represents the number of image tokens per image.
E...% Loss Cauba
Model
#1
FLOPs (T) Latency (ms)
1mp-2B
576
2.14
36.2
Bunny-2B
S76
2.81
40.1
Mini-Gemini-29
576
2.95
41.5
MoE-LLaVA-228
576
2.48
39.3
DeepSeek-VLL
VL-1.3
576
2.01
35.1
LA-VA-FA-22
S76
1.93
30.6
Figure Latency and KV cache usage LL.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison with state-of- ofte-offthheart LMMs the hallucination benchmarks. We compare proposed LLaVA-FA-2B with SFT-based works and RLHF-based works Hall: Hallucination Rate Resp: response- level hallucination rate, Ment: mention- -eviillleeeel hallucination rate. The optimal result is in bold, and the sub-optimal result is underlined.
Object HalBench
POPE
MMHal-Bench
Model
LLM
#Param
Resp
Ment 4
FI f
Score
Hall
Qwen- Owen-VL-Chal VL-Chal
Qwen-7B
20.7
LLaVA-1.5-7B
Vicuna-7B
53.6
25.2
86.1
2.36
51.0
VCD
Vicuna-1 1.5-B
48.8
24.3
84.5
2.12
54.2
OPERA
Vicuna- 1.5-7B
22.3
85.4
2.15
54.2
HA-DPO
Vicuma- 1. 1.5-7B
7B
39,9
19.9
86.8
1.98
60.4
POVID
Dicuma-1.5-77
7B
48.1
24.4
86.3
2.08
56.2
LLaVA-RLHF
Vicuna-1 1. 1.5-13B
13B
38.1
18.9
82.7
2.02
62.5
LURE
Vicuna-1 Bettna-1.55777 1.5-7B
7B
27.7
17.3
1.64
60.4
RLHF-V
Vicuna-
13B
12.2
7.5
86.2
2.45
51.0
RLAIF-
Vicuna-I.55 1.5-7B
7B
8.5
4.3
3.06
29.2
MiniCPM-V
MiniCPM-2.4B
2.8B
21.6
11.5
79.5
3.70
24.9
MiniCPM-V-2
MiniCPM-2.4B
2.8B
14.5
7.8
86.3
4,09
18.2
Mini-Gemini-2B
Gemma-2B
2B
29.7
21.1
85.6
2.83
18.8
Bunny-2B
Qwen-1 1,5-1
2.2B
50.2
23.4
85.8
2.72
19.3
LLaVA-FA-2B
owen-2.5-77
2.2B
11.2
7.7
87.5
2.79
17,5
4.3
ABLATION STUDY
Impact of Rank Choice. We explore the effect of rank choice in LoRA adaptation model performance across vision-language benchmarks. As shown Tab. increasing the rank from 64 to 256 consistently improves performance, with LLaVA-FA-2B showing stronger sensitivity (+1.9% average gain) compared to LLaVA-FA-IB (+1.3% average gain). These results indicate that higher rank values enable better adaptation capacity, especially for larger base models.
Choice of Bit-widths. Tab. shows that raising amplitude and phase bit-widths (by, ba) from 2 to bits improves accuracy by 4.6% while still achieving 11.2x compression and the lowest hallucination rate (11.28 (1... Beyond 4 bits the gains platcau. We therefore adopt b, = =b = 4 as the default, balancing quality, compression and inference speed.
Table 4: Performance comparison with different ranks.
Table Impact b, and by on model performance and compression ratio (CR).
Model
Rank
Avg Ace f
by
Avg Acc
CR
Hall +
64
55.3
LLaVA-FA-IB
128
56.3
2
56.3
14.5
18.7
256
56.6
3
59.1
12.8
13.4
60.9
11.2
11.2
64
58.9
5
61.0
10.1
10.9
LLaVA-FA-2B
128
59.7
4
60.9
10.8
11.0
256
60.8
61.2
8.9
10.5
CONCLUSION
In this paper, we present LLaVA-FA, a novel efficient LMM that compresses weight matrices via joint low-rank plus quantization approximation the frequency domain. By leveraging Fourier de-correlation and conjugate symmetry, method delivers compact yet accurate representations that eliminates the need for large-scale calibration data. Extensive experiments show that LLaVA-Fi outperforms existing efficient LMMs multiple benchmarks while maintaining minimal activated parameters and low computational costs, offering a practical solution for deploying LMMs in resource-constrained scenarios.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was partially supported by the National Natural Science Foundation of China under grant 62220106008, and 62572104.
REFERENCES
Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprume: Diversity- based visual token pruning for large multimodal models. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2392-9401, 2025.
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research, 6(9), 2005.
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. International Conference Machine Learning, pp. 557-565. PMLR, 2017.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. arXiv preprint arXiv:2403.17299 2024.
Lin Chen, Jinsong Li, Xiaoyi Dong. Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2024a.
Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milms. arXiv preprint arXiv:2401. 16160, 2024b.
Yongyong Chen, Shuqin Wang, Yin-Ping Zhao, and CL Philip Chen. Double discrete cosine transform-oriented multi-view subspace clustering. IEEE Transactions Image Processing, 33: 2491-2501, 2024c.
Yuxin Chen and Yuejie Chi. Spectral compressed sensing via structured matrix completion. In International conference on machine learning, pp. 414-422. PMLR, 2013.
Christopher Clark and Matt Gardner. Simple and effective multi-paragraph reading comprehension. In ACL. pp. 845-855 2018.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv preprint arXiv:2307.0880 007.09997,, 2023.
Mare Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong. Mathematics for machine learning. Cambridge University Press, 2020.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36: 111100088101111 10115, 2023.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. Proceedings of the IEEE/CVF international conference on computer vision, pp. 293-302, 2019.
Marco Duarte and Richard G Baraniuk. Spectral compressive sensing. Applied and Computational Harmonic Analysis, 35(1): 111-:1911111,,, 2013.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The laama 3 herd of models. arXiv e-prints, pp. arXiv 2407, 2024.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- frika, (3):211-218. 1936.
11

---

Published as a conference paper at ICLR 2026
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL 2306. 13394
Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming- Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, and Jiaping Zhao. Improving zero-shot generalization and robustness of multi-modal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1093-11100,, 2023.
Mohsen Gholami, Mohammad Akbari, Kevin Cannons, and Yong Zhang. Casp: Compression of large multimodal models based attention sparsity. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9372-9381, June 2025.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding visual question answering. In CVPR, pp. 6904-69 6913, 2017.
Robert M Gray al. Toeplitz and circulant matrices: A review. Foundations and Trends@ in Communications and Information Theory, 2(3):155-239, 2006.
Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. ICLR 2024, 2023.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Yixiao He, Haifeng Sun, Pengfei Ren, Jingyu Wang, Huazheng Wang, Qi Qi, Zirui Zhuang, and Jing Wang. Evaluating and mitigating object hallucination in large vision-languag models: Can they still see removed objects? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume
1: Long Papers), pp. 6841-6858, 2025.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7700-609,, 2019.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Giorgos lacovides, Wuyang Zhou, and Danilo Mandic. Towards Ilm-guided efficient and interpretable multi-linear tensor network rank selection. arXiv preprint arXiv:2410. 10728, 2024.
Yonghyun Jeong, Doycon Kim, Seungjai Min, Scongho Joe, Youngjune Gwon, and Jongwon Choi. Bihpf: Bilateral high-pass filters for robust deepfake detection. In Proceedings of the IEFE/VV Winter Conference on Applications of Computer Vision (WACV), pp. 48 38-57, January 2022.
Ayush Kaushal, Tejas Vaidhya, and Irina Rish. Lord: Low rank decomposition of monolingual code Ilms for one-shot compression. arXiv preprint arXiv:2309. / /402/, 2023.
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In ECCV, 2022.
12

---

Published as a conference paper at ICLR 2026
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36:21487-21506, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Matthew Leinhauser, Jeffrey Young. Sergei Bastrakov, René Widera, Ronnie Chatterjee, and Sunita Chandrasekaran. Performance analysis of picongpu: particle-in-cell on gpus using nvidias nsight systems and nsight compute. Technical report, Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States), 2021.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023a.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleav:: Tackling multi-image, video, and 3d in large multimodal models. CoRR, 2024a.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of modality vision language models, 2024b. URL httts:://aaxiiii org/ 2403. 18814
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint N.Nv:2305.10355. 2023b.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision- laiionla models. arXiv preprint 1xXx:24411119444 2024.
Chenxiao Liu, Zheyong Xie, Sirui Zhao, Jin Zhou, Tong Xu, Minglei Li, and Enhong Chen. Speak from heart: an amotion-guided Ilm-based multimodal method for emotional dialogue generation. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pp. 533-542, 2024a.
Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, et al. Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprint 2.Xiv:2402.05935,, 2024b.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 0..34892-34916 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVE conference computer vision and pattern recognition, pp. 66296-26306 2024c.
Shih-Yang Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposen low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024d.
Yikun Liu, Yajie Zhang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4015 4025, 2025.
13

---

Published as a conference paper at ICLR 2026
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?
In European conference on computer vision, pp. 216-233, Springer. 2024e.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2):
129-137, 1982.
Zijun Long, George Killick, Richard McCreadie, and Gerardo Aragon Camarasa. Multiway- adapater: Adapting large-scale multi-modal models for scalable image-text retrieval. arXiv preprint arXiv:2309.0155 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 039883 2016.
lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv: 171.557 2017.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Meng Luo, Bobo Li, Shanqing Xu, Shize Zhang, Qiuchan Chen, Menglu Han, Wenhao Chen, Yanxiang Huang, Hao Fei, Mong-Li Lee, et al. Unveiling the cognitive compass: Theory-of-mind- guided multimodal emotion reasoning. arXiv preprint arXiv:2602.0097) 2026.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 21702-21720. Curran Asso- ciates, Inc., 2023. URL http:/tpss/pprrceeasss..coo
pappprrrrppppppppppppeeeeeeeeeee haappp/ppscccpppppe_esssee///oo files/paper) pdf
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. ACL, pp. 2263-2279 2022.
Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty, Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pp. 947-952, 2019.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tij- men Blankevoort. A white paper on neural network quantization. arXiv preprint (xXx:210660829: 2021.
Minoru Nakagami. The m-distributiona general formula of intensity distribution of rapid fading. In Statistical methods in radio wave propagation, pp. 3-36. Elsevier, 1960.
Hongyi Pan, Diaa Badawi, and Ahmet Enis Cetin. Fast walsh-hadamard transform and smooth- thresholding based binary layers in deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4650 4650, 2021.
Matthew Plumlee and Roshan Joseph. Orthogonal gaussian process models. Statistica Sinica, pp. 601-619, 2018
ASYMPTOTIC OPTIMALITY OF UNIFORM QUANTIZING. Asymptotically efficient quantizing. 1964.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-87 8763. PmLR, 2021.
Mark Rudelson and Roman Vershynin. On sparse reconstruction from fourier and gaussian mea- surements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 1255) 2008.
14

---

Published as a conference paper at ICLR 2026
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Compressing large language models using low rank and low precision decomposition. Advances in Neural Information Processing Systems, 17:88981-890118 2024.
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. prameeere Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 2024.
Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-sclective rank reduction. arXiv preprint arXiv:2312.13558, 2023.
Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Crossget:
Cross-guided ensemble of tokens for accelerating vision-language transformers. In Forty-first International Conference on Machine Learning.
Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Upop: Unified and progressive pruning for compressing vision-language transformers. In International Conference Machine Learning, pp. 31292-31311 PMLR, 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:21 /tt::2//1..0005. 2021.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, al. Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguistics: 2024, pp. 13088-13110, 2024.
Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Frequency- aware deepfake detection: Improving generalizability through frequency space domain learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 5052-5060, 2024.
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. Advances in Neural Information Processing Systems, 37:61735-61199 2024.
Marinus T Vlaardingerbroek and Jacques A Boer. Magnetic resonance imaging: theory and practice. Springer Science Business Media, 2013.
Xinye Yang, Yuxin Yang, Haoran Pang, Aaron Xuxiang Tian, and Luking Li. Freqts: Frequency- aware token selection for accelerating diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 9310-9317, 2025.
7ai Yang and Lihua Xie. Exact joint sparse frequency recovery via optimization methods. IEEE Transactions on Signal Processing, 64(19):51455 5157, 2016.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng. Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 000000 2024a.
URL
wwwpsiiwivrrrreeoooooooeeoooeere abs 240..///2222..2 01800
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Exploring post-training quantization in Ilms from comprehensive study to low rank compensation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19377-19385, 2024b.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European conference computer vision, pp. 69-85. Springer, 2016.
15

---

Published as a conference paper at ICLR 2026
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- areeggraied correctional human feedback. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13807-13816, 2024.
Zhihang Yuan, Yuzhang Shang, Yue Song. Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation- aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05521, 2023.
Cheng Zhang, Jianyi Cheng, George A Constantinides, and Yiren Zhao. Lqer: Low-rank quantization error reconstruction for llms. arXiv preprint arXiv:2402.02244 2024.
Zhenhai Zhu and Radu Soricut. Wavelet-based image tokenizer for vision transformers. arXiv preprint 1.X::2405..86166 2024.
Zwartjes and Gisolf. Fourier reconstruction with sparse inversion. Geophysical prospecting, 55 (2):199-221, 2007.
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
In the preparation of this manuscript, the authors utilize LLMs, specifically GPT-4, to assist with polishing the writing. The LLM is used solely for improving grammar, clarity, and readability of the text, without altering the technical content, methodology, scientific contributions. All ideas, results, and interpretations presented this paper are solely those of the authors. No part of the paper, including the main text supplementary materials, is generated by the L.M..
B THEORETICAL ANALYSIS
B.1 MATHEMATICAL PROPERTIES OF 2D-DFT
In this section, we rigorously establish two key properties of the two-dimensional Discrete Fourier Transform (2D-DFT) that motivate its in LLaVA-FA: (1) de-correlation of frequency-domain representations and (2) conjugate symmetry for real-valued spatial matrices.
Proof for De-correlation
Let W E R4 xd2 be cero-mean
random field:
W = 0,
E m.n W Wm'mn m = Rm mnnnn''
(13)
We define its 2D-DFT by
d1-1da 1
j2 um
W u,0
Wm,m E
(14)
m==0
n=0
Theorem B.1. The frequency components asymptotically uncorrelated, i.e.,
Cov
Cov(Wn.p (Wu,w, Wwr di d2 au Pu,..
(15)
where S is the Kronecker delta, which equals to if = v' and otherwise. The power-spectral density is
Robb e 12 (2x
Pu,
(16)
b (d2-1)
Proof. By definition,
Cov W.. = W. W w'
(17)
=Σ Σ
E[W mn W m
-52m (2p + )h) (em(
(18)
m.n mn m
16

---

Published as a conference paper at ICLR 2026
Insert Rmmmm nn-x and change variables
m', b n²:
Σ 0,6
Ra.b 12am
(
+
Σ
f2n
'im! (w-u/)n
(19)
di 00
The inner sum equals d1d2 when (u, v) = (u', v²) and zero otherwise giving
Cov ( un w.r,)) did2 du de, PurPaun Pu,.-
(20)
This completes the proof. Theorem B.1 shows that energy compacted into mutually independent coefficients, legitimising aggressive low-rank truncation in LLaVA-FA: discarding small-variance components introduces minimal reconstruction error.
Proof for Faster Singular- Value Decay in the Frequency Domain
Here, we provide a rigorous justification that the singular values of a weight matrix decay faster after 2D>-DFT than in the spatial domain. Let W IRdixda xd2 be zero-mean weakly-stationne random field with autocorrelation
Ra, EW m,n W m+tantbl
(21)
and denote its 2D-DFT by
d1 d2-1
W
W m,n C f2x
um +2)
(22)
u,0
17=00
n=0
Theorem B.I states that the frequency components are asymptotically uncorrelated, i.e.
Cov Wa,,s Wwe,or di d2 du Pu,os
where
Pu,e
Da.b a.b
j2 ug +))
(23)
Hence the covariance matrix of vec (e) diagonal with entries proportional to the power-spectral density Pa... For natural images and learned convolutional kernels, Pu, is smooth and rapidly decreasing function of frequency, typically satisfying
Pu, S C1(1 x² -
a>
(24)
Here, a is the exponential decay rate in the frequency domain. C' > O is a positive constant. Ordering the diagonal entries of the covariance matrix gives the expected squared singular values (lo((W) (W).. By the Szegö theorem on Toeplitz-Fourier operators (Gray et al. 2006), the empirical spectral distribution of W converges to limit whose tails are governed by the decay rate of P Consequently.
Blo W) s C
(25)
Where the index for singular values (rank). contrast, the spatial- covariance matrix is not diagonal. Its eigenvalues decay
[[o 2 8
B x a/2,
(26)
because off- diagona correlations slow the decay. We take square roots yields the singular value bounds, i.e.
Elox (W) I V/1-
E[ox(W)] 2 Vek-8.
(27)
Therefore for any & 2 2, we have
E O
VC VC
(a/2220)
(28)
where the exponent a/2 under typical smoothness assumptions. This inequality shows that the fwequencc- domain singular values decay strictly faster than their spatial counterparts, enabling a lower-rank approximation for the same reconstruction error.
17

---

Published as a conference paper at ICLR 2026
Proof for Conjugate Symmetry
We give full derivation of conjugate symmetry in this section. Let W E Id xda be a real-valued matrix and its 2D-DFT can be represented as:
da-1
W u,E
Σ>
Σ
Wm.
um + 22
(29)
m== =2=0
Theorem B.2. For real-valued matrix W E h xd2 its 2D-DFT satisfies
W u W di do
(30)
for all u (0....,.1-1 -11 and E (0, ..... d2 1}. where (-) denotes complex conjugation.
Proof. Evaluate the coefficient at the complementary frequency:
-2 (4) uim
da m2m
W dd -2 WWm. m,n
41
d2
(31)
Σ Wm. (22m(m-n)
j2
(2x((2)
am wn
(32)
Since m, n Z, e (m22x(m+)) hence
Wd
Σ Wm.n e m,,
um t 2
(33)
Σ Wm,n
m.n
j2
21
dq
W
(34)
where the last equality uses W
Wm,n. m,n R. This completes the proof.
This symmetry implies that only half of the frequency coefficients are unique, the other half are redundant. Thus, when storing or approximating the frequency-domale representation, we can discard nearly 50% of the coefficients without any loss of information.
B.2 THEORETICAL JUSTIFICATION: LOWER RANK IN THE COMPLEX DOMAIN
Here, we provide a formal bound showing that for the same reconstruction error, the Fourier domain requires lower rank than the spatial domain.
Proposition 1. Let the real-valued weight matrix W E Rd xd2 be a zero-mean weakly-stationary random field with exponentially decaying covariance:
B.W m. Wm mm]] [mml]=pmmmae pm mmm H+|m-'),
0<p<l,
(35)
where p E (0,1) controls the spatial correlation strength: smaller implies faster decay and thus weaker spatial dependency. This exponential decay model widely used in spatial statistics and image modeling to characterize locally correlated structures (Gray et al. 2006).
Let W dd, xd2/2 denote the 2D-DFT of W. reduced by conjugate symmetry. Let (A(A) denote the k-th largest singular value of matrix A. Then for any truncation rank r 2 0,
min(d, (2) Σ
min(d d2))
1
(W) <
Σ
(W))
(36)
Proof. The 2D-DFT diagonalizes the covariance matrix of W, yielding uncorrelated frequency components whose variances are given by the power spectral density Pa,., where (u, v) denotes the 2D frequency index. For the given covariance model, have:
iPu.ol x p/u +|vl
(37)
18

---

Published as a conference paper at ICLR 2026
which decays exponentially in the frequency. Since singular values of W are asymptotically equal to the square root of these variances, the tail energy of W decays exponentially, while that of W decays only polynomially. Hence, the ratio of the tails is O(p²), yielding the bound. To achieve prescribed reconstruction error, the Fourier domain requires fewer components (smaller rank r)) than the spatial domain, i.c., Fireq
Vspatial.
B.3 THEORETICAL ANALYSIS OF THE ENERGY CONCENTRATION IN COMPLEX SVD
We emphasize that the energy-compaction property used in our method does not rely on the matrix being real, but on the diagonalization of the covariance structure afforded by the 2D-DFT. Below we give formal bound showing that the singular values of the complex matrix W decay strictly faster than those of the real matrix W, even though both matrices share the same generating process. xd2 Proposition 2. Let W E Rd Xd2 be the same zero-mean, weakly-stationary random field defined in the main text with covariance
B[Wm,, W Wmmmn'] -p (omm-m(+innny (4lnn'1,, 0 p<p<l.
(38)
Let W E Cdixdal2 be its 2D-DFT with conjugate-symmelry reduction. Then every
B[ox(WW) S -i-p= B[ox(W)).
(39)
Hence the expected singular value tail of W is dominated by dimension-free constant times the tuil of W. Since the singular values decay strictly faster, for any prescribed approximation error, the required rank in the complex domain is no larger and often smaller than in the real spatial domain, confirming that the low-rank structure is preserved under the complex-valle valued Fourier transform.
Proof. The 2D-DFT diagonalises the covariance operator of W (Theorem B.1 so the entries of W are uncorrelated complex Gaussians with variances
B B[[W.. -r = Pu, Σ 0,5
p jolt e (2d/duu///
<
+ pllttlll
(40)
1
Ordering these variances gives the expected squared singular values [loo(()) Using the Szeg theorem for stationary random fields (Gray al. 2006), obtain
Biox [W) S VC p/2 while Elax(W) 2 Vep², for universal constants C, c >> 0. Taking ratios yields the claimed inequality.
(41) L
The complex matrix W inherits the exponentially decaying power spectrum the original real field. Its singular values therefore decay faster than those of the real matrix W. Consequently, for any prescribed Frobenius error e, the Fourier domain requires smaller rank r than the spatial domain, even though the SVD is performed on a complex matrix. The low-rank assumption is therefore stable under the Fourier transform.
Furthermore, to empirically verify that the complex-valued more compact singular-valus value spectrum, we conduct a controlled apectrum-decal experiment. We apply identical low-rank plus quantization pipelines to both the spatial (real) and Fourier (complex) representations the Qwen-2.5-7B query layers and record the Frobenius reconstruction error at rank 256 as well as the smallest rank required to push the error below 1%. As shown in Tab. 6 the
Beqquency-oomare matrix admits
Table Singular-valle value decay comparison between spatial (real) and Fourier (complex) representations. Freq.- Complex exhibits lower error at the same rank and reaches the 1 % error threshold with fewer components.
Domain
Frobenius error rank 256
Min. rank for 1 error
Spatial (Real)
0.031
312
Freg-Complex (Ours)
0.018
217
complex- valued frequency matrix consistently yields smaller reconstruction error under the same rank and achieves the theoretical finding that the Fourier-domain SVD is indeed more amenable truncation.
preset accuracy target with significantly fewer singular vectors, corroborating
19

---

Published as a conference paper at ICLR 2026
B.4 INFORMATION ANALYSIS OF POLARQUANT
(1) Uniform amplitude quantization is rate-optimal for the heavy-tailed DFT envelope.
After the Fourier transform, the residual entries follow Nakagami-m envelope distribution NNak agami 1960). whose Fisher information is Jr) o 1/y². For such heavy-tailed scale parameters, the inverse-umiform partition the unique minimax solution the quant-design game
min max DKL. (p(r) (g(rr)
(42)
Q r20
where 12 the piece- wise constant density induced by the quantiser. The solution coincides with the uniform grid used in PolarQuant (step-size A, Tmax/ m2nx/(2h - 1)) and is asymptotically rate-optimal
the Gish-Pierce sense (QUANTIZING T964):
R(D) b, 1 (2 O(D),
(43)
and coincides with the empirical step size we adopted.
(2) The / phase step the minimum-Me MM lattice for a Von-Mises variable.
The phase residual is well modelled by zero-mean Von-Mises distribution 0 ~ VM x) with concentration K & rank truncation error (Banerjee et al. 2005). For a circular random variable the wrapped Lloyd-Mat conditions (Lloyd 1982) yield the uniform lattice on [-T,, x) with step 2 / as the unique minimum-Me MSSE solution, giving
((--))) 12 22hg (1+0
(2T)²
(44)
Since the phase residuals observed under our default (4-bit amplitude/phase, rank-256) setting are well fitted by zero-mean Von-Mises distribution with concentration x s 9-14, the resulting (1/xx) term is 0.04 and the phase-noise energy is provably bounded.
(3) Theoretical bound on phase-error propagation.
Let the reconstructed complex weight be
à = re 10 re (0++)0)
(45)
where Ae is the quantization error introduced by PolarQuant. The resulting complex error
| w 2 = 1² - 22 = 2 (A0/2) < y A2?
(46)
Taking expectation over the Von-Mises distribution of (concentration quantization error AO ~ U1 1111x/200 T/ 220 we obtain
12) and the uniform
B à
< B
22b 0.0013 E[r²] for by = 4.
(47)
Hence the normalized reconstruction error contributed by phase guantization is 0.13%, an order of magnitude smaller than the typical low-rank truncation error (+= 1.2%).
Empirical verification. We conduct an ablation study LLaVA-FA-2B in which we replace PolarQuant with a baseline that quantizes real/imaginary parts independently (QIM, 4 + 4 bits). Results shown in Tab. PolarQuant reduces average phase error by 36%, improves PSNR by 1.2
Table 7: Mon-Mises concentration for 4-bit phase residuals, bounding the chase-quantization error below 0.04
Scheme
Phase-error E|
Recon. PSNR
HalBench-Resh 4
MMB t
QIM (real/imag)
0.14 rad
38.9 dB
14.8%
65.2
PolarQuant (ours) 0.09 rad
40.1 dB
11.2 %
66.7
dB, and yields 3.6% lower hallucination rate, despite using the same bit budget. This confirms that the 2n /2 lattice not only controls phase noise but also translates into measurable gains in downstream quality. In short, phase quantization does not degrade reconstruction because the residual phase rightly concentrated and the uniform lattice 2T 2r/2 2 is the minimum- MSE solution under this concentration. Both theory and experiment show that PolarQuan reduces whase-erron rather than amplifyyng it.
20

---

Published as a conference paper at ICLR 2026
B.5 DERIVATION OF CONDITION 3.1
We derive the exact condition rank under which the average bit-width Bavg is strictly smaller than the full-precision budget Br.. Recalling the definition in $3.1
Ei= Bod d2 (r + d5)
Bagg
(48)
E._ di
we require Bavg Bt. Multiplying both sides by the positive denominator Ei= di
gives
Bodi de + Br + +5) Σ
Bt. i=1
didz.
(49)
Collecting terms proportional B₁, on the right-hand side,
B k:
di + )2)) < (BI- - -Bq) Σ did3.
(50)
Finally, dividing by the positive quantity B1. Σi= (d) + dz²) yields the concise condition
k < (1
Bq BL
El= didd
(51)
E/ (dj + d
Whenever this inequality holds, the low-rank plus quantized decomposition achieves average bit-width strictly below the full-precision baseline BL, validating the memory efficiency of the low rank plus quantized matrix method.
B.6 PROOF FOR LEMMA
Let AW, and A 2 be two low-rank adapters of identical dimensions, Denote their singular value decompositions by
AW1 U
AW, = U,
(52)
where Σ diag(o 1.1, .... ) and Eg diag(02.1 ........ with singular values sorted in non-increasing order. For fixed rank r, the best rank approximation in Frobenius norm is
Aw(r) U. U. with Σ/- diag(o..I....... 0,....................... O....).
(53)
The reconstruction error
E AW, - AW = U. V.
(54)
Following the Eckart- Eckart-Young theorem Eckart & Young 1936) and theorem 4.95 in Mathematics for Machine Learning (Deisenroth al. 2020), the value of the norm of reconstruction error is given as
min(man) Σ
E,
E.
(55)
k=++1
By assumption, 01,6 < 02.6 for all lo 2 r + 1. Hence
min(m.n)
min(m.n)
Σ
of
Σ
22
E < E
(56)
k=r+1
Therefore, the rank-r approximation of WW, incurs strictly less error than that of AW,, which completes the proof.
21

---

Published as a conference paper at ICLR 2026
B.7 THEORETICAL JUSTIFICATION OF ODC HEURISTIC (RoW/COLUMN AVERAGING)
The proposed ODC scheme replaces the full Hessian C E xd2/2 by two diagonal matrices, i.e.
D row- - -dia ( Cit.VCC.)... C1..)...., avgly Cdl.:) ; Doo diag ([avs( V.,)) avg( /C C de/2)
(57)
where avg(v C, and avg( / C. denote the i-row/j-colune means of /C. This exact whenever Cs diagonal, and remains spectrally close when C is diagonally dominant, property that has been repeatedly observed for the loss Hessian of deep Transformers (LeCun et al. 1989 Botev et al. 2017). off Formally, let off-diag( ( diag( [ where diag( C) denotes the diagonal matrix that keeps only the main-diagonal entries of C, while off diag(() C) is its complement, i... C with all diagonal entries set to zero. Thus quantifies how strongly C is diagonally dominant.
Lemma B.3. Let C
D jow CD col- Then
||C |C- S E ||C||
(58)
Hence the calibration matrix used by ODC an e-perturbation of the exact Hessian. The recon- struction error of the low-rank component inherits this bound with the same constant (proof uses [Drow l. [Deal] S 1 and standard matrix-perturbuin identities). In short, ODC is safe whenever E < 1, a condition that can be checked on-the-fly with one extra reduction operation.
C ADDITIONAL EXPERIMENTS
C.I TRAINING STRATEGY AND HYPERPARAMETERS
This section details the complete training recipe employed to produce LLaVA-FA-7B, LLaVA-FA- 3B, LLaVA--F--2 2B and LLaVA-FA-IB. As shown Tab. g the four variants are initialized from InternL.-2-2001 LLaMA-3-SB. Qwen-2.5-7B and Qwen-2.5-3B checkpoints, respectively. The vision encoder inherited from CLIP (ViT-L/14@ 336 px for all except the B model which uses ViT-B/224 px) (Radford et al., 2021). We keep the visual token sequence length fixed at 576 (256 for B) and cap the language-modelling context 2048 tokens. Optimisation is performed with AdamW (Loshchilov & Hutter 2017): 31 is set to 0.9 for all models; /3 is 0.99 for 7B/3B/2B and 0.98 for IB. A single-cycle cosine-decay schedule (Loshchilov & Hutter 2016) is adopted, with initial learning rates of le-5, 1.5e-5, 2c-5 and 5e-4 for 7B, 3B, 2B and 1B, respectively. Weight decay (Krogh & Hertz 1991) is applied only to the 7B/3B/2B models (0.1, 0.1, 0.2) and disabled for IB. The entire pre- tre-training stage spans 100 epochs with warm-up ratio of 3% and global batch text text pairs. No additional regularisation or data augmentation is introduced. Under
size 128 image- this configuration the models converge reliably after processing only five million samples eight RTX 4090 GPUs.
Table 8: Training hyper-parameters of LLa VA-FA variants.
Configuration
LLaVA-FA-7B
LLaVA-FA-3B
LLaVA-FA-2B
LLaVA-FA-IB
L.I. init.
Internl M---20B
L-M--33-88
Qwen-2.5.77 MLP
Qwen-2.5-38
VL Adaptor init.
MLP
MLP
MLP
VIT init.
LarP-Laree@336
CLIP--Lagee@03
CLIP- 60336
CLIP BaseBBBee@B 224 224 x 256 2048 Adam 0.9, 0.98
Image resolution
336 x 336
336 x 336 576
336 x 576
VIT sequence length
576
L.LM sequence length
2048
2048
20488
Optimizer
AdamW
AdamW
Adam 0.9, 0.99 4
Optimizer hyperparameter
8
0.9, 0.99
0.9. 0.99
&
Learning rate
le
6
1.5e
2e
6
Se 4
Learning rate schedule
Cosine decay
Cosine decay 0.1
Cosine decay 0.2
Cosine decay 0.0
Weight decay
0.1
Training epoch
100
100
100
100
Warm-up cum ratio
0.03
0.03
0.03
0.03
Batch size
128
128
128
128
Rank r
256
256
256
256
by, by
(4,4)
(4,4)
(4,4)
(4.4)
22

---

Published as a conference paper at ICLR 2026
C.2 IMPACT OF DFT/IDFT AND POLARQUANT ON WORLD INFERENCE LATENCY AND GPU MEMORY FOOTPRINT
Although LLa VA-FA reduces parameters and FLOPs, a natural concern is whether the additional DFT/IDFT and PolarQuant operations introduce non-negligible latency or memory overhead. To this end, we conduct controlled experiments to quantify these costs and compare them with the savings produced by our frequency domain compression. We measure end-to-end latency and peak GPU memory on single NVIDIA A100-40G with CUDA 12.1, PyTohc 2.2, torch. compile in mrode==reduce--ee
and Flastontent (Dao 2023 enabled. The input images are 336x336 images (576 visual tokens) and text prompts contains of 512 tokens on average. We report Time-to-First- Fio--Fi---Toonnnn (TTFT), peak memory during the full generation cycle, and throughput (tokens to compare real-world inferences. DFT/IDFT is implemented with batched cuFFT calls. PolarQuant de- de-quuatization is fused into the linear-layer CUDA kernel (no extra memory allocation). All timings are averaged over 500 warm runs with Nsight-Systems (Leinhauser et al., 202I) profiling isolate transformation cost. As shown in Tab. 9 LLLVA-FA- -B achieves the lowest TTFT and memory footprint among 2B-scale models. Profiling reveals that DFT + IDFT accounts for only 3.2% of TTFT (< 0.9 ms) and PolarQuant de-quantization for 0.7% (5 0.25 ms). Because frequency domann matrices are manjugae-syymeetre we store only half of the coefficients, yielding a net memory reduction despite the auxiliary tensors needed by CuFFT.
Table 9: Real-world inference latency and GPU memory foolprint. DFT/IDIT overhead is measured with Nsight Systems. PolarQuant de-quantization is fused into the GEMM kernel and micro-benchmarked independently.
Model TTFT (ms)
Peak Mem (GB))
DFT/IFFT (%)
PolarQuant (%)
MiniCPM-V-2
38.4
15.2
1mp-2B
36.2
14.6
Bunny-2B
40.1
15.8
LLa VA-FA-2B
30.6
12.1
3.2
0.7
Removing DFT/IDFT (i.e., keeping spatial low-rank plus quantization) reduces TTFT by only 0.9 ms but increases memory by 2.3 GB and FLOPPs by 34.6%. The overall throughput drops by 8.7 tok/s. This confirms that the tiny latency introduced by frequency transformation more than compensated by the compression gains. DFT/DDFT and PolarQuant introduce micro-second-tevel overhead, but the resulting parameter, memory, and FLOP savings yield faster and lighter inference practice. Therefore, VA-FA not only possesses theoretical compression benefits but also delivers improved deployment efficiency on resource-consie constrained GPUs.
C.3 SPATIAL DOMAIN vs. QLORA vs. FREQUENCY DOMAIN.
For a fair and reproducible comparison, we implement the spatial- -ontial-domiin counterpart of our method ""Spatial-LQ"), which follows the exact same optimization schedule and bit-width configuration except that SVD and quantization are applied directly on W instead of W. We conduct controlled doatial-domain baselines using the same training protocol as LLaVA-FA. Moreover, reproduce the exact QLORA (Dettmers et al. (023)) pipeline to compress the same Qwen-2.5-7B backbone, which we use to derive our FA--ABB model. This creates a "QLoRA-Qwen" baseline whose substantive difference from LLa VA-FA22B is that the low-rank plus quantization decomposition is performed in the spatial rather than the frequency domain. The results are shown in Tab. TO In terms of computational efficiency, LLaVA-FA achieves the lowest inference cost with 1.93 T FLOPs and 30.6 ms TTFT, outperforming both Spatial-L-Q (2. 14 T, 36.2 ms) and QLoRA-Qwen 10 T, 35.8 ms. Furthermore, frequency-domain approximation yields +2.6% better average accuracy and significantly lower hallucination. This supports our key claim: the singular spectrum is more compact and the low-rank truncation error is strictly smaller in the Fourier domain (Lemma 3.1. enabling better reconstruction under identical bit-width and rank. Compared to QLoRA (Dettmers et al. 2023), LLaVA-FA jointly optimizes low-rank and quantization in the frequency domain, leveraging Founer de-correlation and energy compaction for a more compact and accurate weight approximation, leading to better accuracy and lower hallucination.
23

---

Published as a conference paper at ICLR 2026
Table 10: Comparison of spatial versus frequency domain optimization. LLaVA-FA consistently outperforms the spatial baseline across metrics, validating the benefits spectral compactness.
Method
Domain
Avg Ace
POPE F1
HalBench Resp
FLOPs (T)
TTFT (ms) &
Spatial-LQ (ours)
Spatial
58.3
84.6
16.9
2.14
36.2
QLoRA-Qwen
Spatial
58.5
85.7
15.7
2.10
35.8
LLaVA-FA (ours)
Frequency
60.9
87.5
11.2
1.93
30.6
C.4 INFLUENCE OF CALIBRATION DATA SIZE
We also conduct ablation study the LLaVA-FA-28 model examine the influence of calibration data size. The results shown in Tab. [1 indiicate that the method is highly robust and that performance saturates quickly. Using more calibration samples does not provide noticeable improvements, confirming that only a small amount of data is sufficient for constructing a reliable calibration matrix.
Table Ablation study on the influence calibration data size. Performance saturates quickly, demonstrating that a small amount of calibration data is sufficient.
HCalibration Samples
GQA
WizWiz
SQA
VQA
MME
MMB
MMB
AVG
256
61.0
67.2
66.0
65.8
63.8
60.5
512
61.5
41.3
67.5
58.9
66.2
66.0
63.5
60.7
1024
62.1
41.6
68.2
59.4
66.6
66.7
61.7
60.9
2048
62.1
41.7
68.2
59.3
66.6
66.7
61.7
60.9
C.5 SENSITIVITY OF ODC UNDER DISTRIBUTION SHIFTS
Theoretical analysis in Appendix B.7 reveals that ODC is safe whenever & < 1. Therefore, evaluate the sensitivity of ODC under three levels of distribution shift on the LLaVA-FA-28 model. an in in-distribution baseline.
The first scenario retains the original LLVVA-Pretrain validation The second keeps all images identical but re-samples user prompts from an alternative template pool, introducing purely textual style shift (shift-A). The third replaces the images with an entirely new visual domain consisting of acrial-scene photographs and OCR-heav documents, thereby forcing the model cope with previously unseen visual statistics (shift-B), Each scenario is repeated 256 times, yielding stable estimates of and score drop. As shown in Tab. I2 as rises from 0.11 + 0.02 under the in m-distribution condition to 0.15 = 0.03 under prompt-style shift and further to 0.29 + 0.07 under the visual domain, the average benchmark score declines by -0.4% and -1.1% respectively. Once e exceeds the empirical guard-rail threshold of 0.35 (note that this value is observed from experiments rather than derived from theory), accuracy falls by more than 3% in 4% of the runs, corroborating the theoretical prediction and demonstrating that the on on-the-fly s-check reliably signals when the row/column averaging approximation is no longer safe.
Table 12: Sensitivity of ODC under distribution shifts. The results confirm that effectively tracks domain divergence, reliably signaling performance degradation as the shift severity increases.
Condition
(mean + std)
AScorel
Failure (e 0.35)
In Distribution
0.11 + 0.02
0%
Shift-A
0.15 + 0.03
-0.4%
0%
Shift-B
0.29 + 0.07
-1.1%
4%
C.6 COMPRESSION TIME COMPARISONS FOR MODELS OF DIFFERENT SIZES
In this Section, provide the wall-clock compression time for the four LLaVA-FA variants derived from Qwen-2.5, LLaMA-3, and InternLM22 models. All numbers are measured on eight NVIDIA RTX 4090 GPUs and include every step of our pipeline, Lee. DFT/IDFT, Fourier-SVDD PolarQuant, and
24

---

Published as a conference paper at ICLR 2026
the ODC. As shown in Tab. 13 compressing the 3B-base model into LLaVA-FA-IB IB requires about 1.1 hours, while compressing the BB-base LLM into LLaVA-FA-2B takes roughly 2.3 hours. The larger LLaVA-FA-3B, built from an 8B backbone, completes in approximately 3.8 hours, and even the largest LLaVA-FA- derived from a 20B model, finishes within 6.2 hours. These results exhibit
Table 13: Wall-clock compression times for LLaVA-FA variants. Measurements were conducted on eight NVIDIA RTX 4090 GPUs and include the complete pipeline (DFT, SVD, PolarQuant, and ODC).
Model
Base I.M. Size
Final FA Size
Layers Compressed
Compression Time
LLaVA-FA-IB
Qwen-2.5-3B
~ 1B
36
1.1 h
LLaVA-FA-2B
Qwen-2.5-7B
2.2B
48
~ 2.3 h
LLaVA-FA-3B
LLaMA-3-8B
3B
64
~ 3.8 6.2 h
LLaVA-FA-7B
InternL.M-20OB
~ 7B
80
an approximately linear scaling trend with respect to model depth and hidden dimension, because each layer only performs one forward and inverse DFT, a single complex SVD on a Fourier-shrunken matrix (benefiting from conjugate symmetry), and one PolarQuant pass. To be note worthy, once the one-off compression is finished the resulting checkpoints are loaded and used like any standard model, so the cost is amortised over the entire deployment lifetime.
C.7 THE EFFECT OF OPTIONAL DIAGONAL CALIBRATION (ODC)
As described in Section 3.3 ODC designed to approximate calibration-awat weighting without requiring a large calibration set, but it is not a mandatory component of the Fourier approximation pipeline. When ODC disabled, the model reverts to an unweighted FourierSVD decomposition, which optimizes reconstruction error in the standard Frobenius norm. We conduct the ablation experiment that compare "with ODC" and "without ODC" under identical rank and bit-width settings. The results in Tab. 14 show that ODC brings consistent improvements: improves accuracy by 0.8% on comprehension benchmarks and reduces hallucination rate by 0.6%- 0.8%, with the 2B variant improving from 60.1 1% to 60.9% (AVG) and hallucination rate from 11.8% to 11.2% on Object HalBench. This behavior aligns with the purpose of ODC, which stabilizes layer-wise reconstruction for layers whose sensitivity varies across rows/columns (Zhang al. 2024). Importantly, even without ODC, LLaVA-FA still outperforms existing efficient LMMs of similar size, demonstrating that the core Fourier low-rank plus quantization formulation is effective on its own.
average, enabling ODC
Table 14: Ablation study of Optional Diagonal Calibration (ODC). ODC consistently improves accuracy and reduces hallucination rates.
Model
ODC
GQA
VizWiz SQA!
VQA T
MME MMB MMB
AVG1
HalBench Resp..
LLaVA-FA-2B w/ ODC 62.1
41.7
68.2
59.3
66.6
66.7
61.7
60.9
11.2%
ILAVA-FA-2B w/o ODC
61.6
41.1
67.4
58.6
65.0
66.0
61.0
60.1
11.8%
LIAVA-FA-IB w/ ODC
56.7
49.7
61.3
57.9
63.3
58.3
49.4
56.7
18.1 %
LLAVA-FA-IB w/o ODC 56.0
49.0
60.5
57.0
62.6
57.4
48.8
55.9
18.9%
C.8 GENERALIZABILITY TO SMALL SAMPLES
Since every community model in Tab. is trained with its own released corpus, a perfectly controlled data-matched comparison is unfortunately impossible. We therefore cite the sample counts reported in their respective papers. To further verify that our gain is not produced by more examples, we re-train LLaVA-FA-2B with only M image-text pairs (one-fifth of our full set). As shown in Tab. 15 this model achieves 59.2% average accuracy on the same seven benchmarks. It achieves 0.3 higher score than Imp-2B 98, 1.6 M samples) and clearly ahead of Bunny-2B (56.3%, 2.7 M), Mini-Gemini-2B (57. %, 2.7 M) and MoE-LLaVA-2DD (55.3%, 2.2 M). When we scale back to the full M mixture the absolute score rises to 60.9%. yielding a wider margin over all baselines. The steady lead across two different training sizes shows that the proposed Fourier-domati compression is intrinsically more data-efficient. and the original results are not a consequence of simply ingesting more data.
25

---

Published as a conference paper at ICLR 2026
Image: (80xx883)
Instruction:
Image: (1988x737))
Instruction:
What types of flowers are on the top shelf?
How much Bay-Ban Meta W Wayfarer?
1mp-2B: ( 45ms There are puppl and rod flowers on the top shelf.
Imp-2B: @ 54ms The listed price Ray-Ban Meta ay is 299 USD.
Mini-Gemini-22 emini-28:: 2B: There is bouquet of pupple flower on the top sholf.
Mini-Gemini-22 3565 The price for the Ray- Ban Meta W Wayfarer is $259.
MoE-LLaV ts 19msThh flowers top shelf purple red.
MoE-LLLVA-22 60ms The image shows a peir of
Ray Ban glasses.
LLaVA-PA 11FA-FA-22.. 39ms The flowers the top shelf appear to be purple and red petunlas.
LLaVA-FA-2B. ( 50ms The Ray-Bat Meta W Wayfarer costs $299.
Figure 5: Qualitative comparison. LLaVA-FA 2B demonstrates superior precision efficiency on fine-gr fine-grained recognition (left) and OCR tasks (right). It correctly identifies specific details (e.g.. "petunias") ") and prices with the lowest latency among all baselines.
C.9 ABLATIONS ON DIFFERENT TYPES OF APPROXIMATION
In this Section, conduct ablation study that trains exactly the same low-rank plus quantization pipeline while replacing the 2-D DFT with four representative linear transforms:
(1) Hadamard (Pan) al. 2021), (2) DCT-II (Chen et al. 2024c), (3) Daubechies-4 wavelet (Zhu & Soricut 2024). and (4) a Gaussian random orthogonal projection (aaieti Plumlee & Joseph (2018). The results are unambigu- ous: LLaVA-FA (Fourier) achieves the lowest average Frobenius reconstruction error on the seven compressed layers (1.39 x 10 100 followed by Daubechies-4 (1.73 x 10 DCT-II (1.91 x 10 10-"), Hadamard (2.06 10 10-3), "), and random projection (2.48 x 10. More importantly, only the Fourier basis reduces the activated parameter count to 52% of the spatial baseline thanks to conjugate symme- try. The precise value is 52%, which slightly exceeds the ideal 50% saving due to the zero frequency (DC) component and small per layer calibration buffer (Saha et al. (022)). Every other transform retains 100% parameters, so their compression ratios are strictly worse. Downstream evaluation on the same seven benchmarks shows that Fourier obtains 60.9% average accuracy, outperforming Hadamard (57.4%), DCT (58.7%), wavelet (59.1%), and random projection (56,8%), while also exhibiting the lowest hallucination rate Object HalBench (11.2% vs. 2 15.8% for the rest). These experimental results demonstrate that the Fourier transform is not a neutral linear wrapper but the unique choice that simultaneously minimizes reconstruction error, maximizes parameter savings, and preserves task performance, thereby validating its optimality in the VL- pipeline.
Table 15: Comparison data efficiency. LLaVA-FA-22 outperforms baseline models even when restricted to only IM samples, which demonstrates its superior data efficiency.
C.10 QUALITATIVE ANALYSIS
Fig. 5 provides comparative visualization of the model predictions from Imp 2B, Mini- Gemini-2B, MoE-LLaVA-20 and LLaVA-FA- 2B across diverse reasoning examples. The left example demonstrates fine- fin-inained visual recog- nition capabilities, where the model is asked identify flower types a shelf. While all models correctly identify the purple and red colors. LLaVA-FA FA-2B provides the most pre- cise answer by specifying "purple and red petu- nias". whereas other models give more generic responses. Notably, LLLVA-FA-22 achieves this superior visual understanding while maintaining the lowest latency, representing a 13-28% speedup over competitors. The right example tests OCR and price-reading ability, requiring the model to accurately interpret text from product tag. Here, LLaVA-FA-28 correctly identifies both the product name ("Ray-Bl Ban Meta Wayfarer") and its price ("$299"), demonstrating strong text-centric visual comprehension. Other models either provide $259; Imp-2B: 299 USD without product context) or fail to extract the specifie product name (MoF-LLaVA-2B)) Again, LLLVA-FA -2B delivers the most
Model
Training samples
AVG
Imp-2B
1.6M
Bunny-2B
2.7M
56.3
Mini-Gemini-2BB
2.7M
57.1
MoE- MoE-LLaV
2.2M
55.3
DeepSeek-VL-1..
2000M
58.5
LLaVA-FA-2B
5M
60.9
LLaVA-FA-2B
2.5M
59.7
LLaVA-FA-2B
IM
59,2
incorrect pricing information
26

---

Published as a conference paper at ICLR 2026
k Interaction
W,W, Interaction
Spati Demain
Spatial Domain
Frequency Domain (Ours)
Frequency Domain (Ours)
Normalzze Magnitutc. (Lag)
Normalized Magmitutic (Log)
10:1
10
10-2
10:
10-3
10-3
10-1
10-4
0
250
500
750
1000
1250
1500 1750
2000
250
500
750
1000 1250 1500
1750
2000
Singular
Singular
Figure Singular value spectrum analysis the attention interaction WW,W7 and output projection (W.W. matrices. The frequency domain (Ours) exhibits a sharper decay compared to the spatial domain. This validates that the Fourier transform achieves superior energy compaction, where the majority of information is concentrated the top few ranks, thereby minimizing the reconstruction
caused by low-rank truncation.
Table 16: Comparison of different orthogonal bases for approximation. Fourier achieves the lowest Frobenius error and uniquely reduces activated parameters to 52%.
Basis
Avg Frobenius error
Activated params spatial
Avg Acc (%)
HalBench Resp
(x (x10-3)
(%)
Spatial
2.48
100%
56.8
20.4
Random projection
2.48
100%
56.8
20.1
Hadamard
2.06
100%
57.4
18.3
DCT-III
1.91
100%
58.7
16.9
Daubechies-4
1.73
100%
59.1
15.8
Fourier (LLaVA-FA)
1.39
52%
60.9
11.2
accurate response with the fastest inference time. Overall, LLaVA-FA-2B demonstrates exceptional performance terms of both model efficiency and reasoning capabilities.
C.11 SPECTRUM VISUALIZATION OF W, WT AND W,W,
To further investigate the mechanism behind the superior compression performance of VLa-A- we visualize the singular value spectrum of key interaction matrices in the Transformer architecture. Specifically, we analyze the attention interaction matrix (W, W and the output projection path (W.WW) from the last layer of the Qwen2.5-7B model. As illustrated in Fig. 6 we compare the singular value decay profiles the original spatial domain versus frequency domain. It is observed that the singular values in the frequency domain (red curve) decay faster than those in the spatial domain (blue curve). In the context of low-rank compression, this steep decay a highly desirable property known energy compaction. According to the Eckart- Young-Mirsky, theorem (Eckart & Young 1936), the reconstruction error of a rank-r approximation is determined by the sum of the squared singular values the discarded tail components (Ei-r+1 a.) The spatial domain exhibits "heavy tail" distribution, implying that truncating the matrix a low rank would result the loss of significant information (high energy). In contrast, the frequency domain concentrates the majority of the weight energy into the top few principal components. This results in a negligible tail energy, allowing LLaVA-FA to achieve much lower reconstruction error under the same rank constraint.
D LIMITATIONS AND FUTURE WORKS
LLaVA-FA currently targets static 2-D weight compression for text-image pairs. Future research explore how to generalize the Fourier approximation framework to higher- dimensional tensors in video, audio and 3-D voxel data. Moreover, the added DFT/IDFT and PolarQuant kernels still rely
27

---

Published as a conference paper at ICLR 2026
GPU-only CUDA paths, which limits deployment commodity ARM-NPU micro-controller boards. To enable frictionless edge rollout, a possible solution could be to provide lightweight CPU/NEON and 4-bit fixed-point reference kernels that trade small accuracy drop for orders-of- magnitude power savings, making LLaVA-FA runnable on everyday mobile and embedded devices without extra hardware.
28


Reviews:
[{"review_id": "wHrYNF5jmz", "reviewer": "Reviewer_xZPt", "summary": "This paper proposes LLaVA-FA, a compression framework for large multimodal models (LMMs) that addresses the limitations of existing methods, where decoupled low-rank decomposition and quantization often lead to compounded reconstruction errors. LLaVA-FA employs Fourier approximation to integrate low-rank decomposition and quantization within the frequency domain, leveraging two essential properties of the Fourier transform: de-correlation, which reduces spectral redundancy, and conjugate symmetry, which nearly halves parameter storage.To handle complex matrices in the frequency domain, the paper introduces PolarQuant—a polar-coordinate quantization method that discretizes amplitude and phase separately to preserve complex structure. It also proposes an Optional Diagonal Calibration (ODC) scheme, which approximates the full Hessian with row/column means to avoid reliance on large-scale calibration data.", "questions": "1. How does the proposed method perform when extended to models beyond Qwen-2.5?\n2. How is the calibration matrix C constructed in the paper, and has there been any ablation study on the amount of calibration data used?\n3. Could you elaborate on the compression time required for models of different sizes?\n4. The algorithm flow in the paper provides an option to disable ODC—how does the performance change when ODC is not used?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper proposes LLaVA-FA, a compression framework for large multimodal models (LMMs) that addresses the limitations of existing methods, where decoupled low-rank decomposition and quantization often lead to compounded reconstruction errors. LLaVA-FA employs Fourier approximation to integrate low-rank decomposition and quantization within the frequency domain, leveraging two essential properties of the Fourier transform: de-correlation, which reduces spectral redundancy, and conjugate symmetry, which nearly halves parameter storage.To handle complex matrices in the frequency domain, the paper introduces PolarQuant—a polar-coordinate quantization method that discretizes amplitude and phase separately to preserve complex structure. It also proposes an Optional Diagonal Calibration (ODC) scheme, which approximates the full Hessian with row/column means to avoid reliance on large-scale calibration data.", "weaknesses": "1. The paper proposes performing low-rank decomposition and quantization in the frequency domain via the Fourier transform, but the experimental section lacks comparisons with existing solutions in the spatial domain, limiting the credibility of its claimed competitiveness.\n2. The method shows limited generalization capability, as experiments are only conducted on 3B and 7B-scale LLMs (Qwen-2.5) without evaluation on larger-parameter models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OMTq1oGgzl", "reviewer": "Reviewer_tSjP", "summary": "The paper proposes LLaVA-FA, an integrated compression pipeline that merges low-rank and quantization in Fourier space, accompanied by a concrete algorithmic recipe and a complexity-aware design. More specifically, LLaVA-FA applies a 2D Discrete Fourier Transform (DFT), factorizes the largest spectral components with a low-rank complex SVD, quantizes the residual using a polar-coordinate codec (PolarQuant), and optionally weights the reconstruction objective with a diagonal calibration derived from row and column statistics (ODC). The experiments demonstrate favorable performance at small scales, accompanied by measured efficiency gains in latency, FLOPs, and KV-cache usage.", "questions": "1. Do the authors plan to release their models?\n\n2. The current results are mainly on small-scale models. Would the authors consider including results on larger models to demonstrate scalability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed LLaVA-FA is well-motivated, and supported by a clear theoretical framing.\n\n2. Efficiency evidence with concrete measurements. Specifically, latency, FLOPs, KV-Cache usage, and TTFT are reported, which aligns well with the goals of a model compression study.", "weaknesses": "1. Limited ablation for ODC and calibration choices. The paper mentions that ODC removes the need for large calibration sets, but no direct comparison or ablation is presented to isolate this effect. Adding such results would make the claim more convincing.\n\n2. There are some fairness concerns regarding the baseline comparisons in Table 1. The amount of training data varies across methods, and some baselines use fewer samples than LLaVA-FA while achieving comparable performance.\n\n3. Discussion on a few related works seem to be missing. For instance, in the line of LMM efficiency “CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers, ICML 2024,” and in the line of vision-language weight compression “UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers, ICML 2023.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qlCz9JlcVK", "reviewer": "Reviewer_YXsM", "summary": "This paper targets on efficient Large Multimodal Model (LMM). They bring Fourier approximation to decomposes the weight matrices into a\nlow-rank plus quantized weight, designing a efficient LMM framework called LLaVA-FA. They also design PolarQuant which is an amplitude-and-phase polar codec to quantize complex matrix, and an optional diagonal calibration (ODC) scheme to approximate Hessian Matrix. Extensive experiment results prove the effectiveness of the proposed method.", "questions": "See weaknesses, especially discussions about the limitation and future work.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is written in a very high quality, the figures analyze the problem and illustrate the idea very clearly, especially Figure 1 and 3. \n2. I think this paper targets on an important problem, the efficient LMM. \n3. The idea is interesting and reasonable. I am happy to see Fourier approximation can be applied to LMM since it really has some good characteristics. \n4. The experiments are abundant and clear, proving the effectiveness of the proposed method.", "weaknesses": "1. Just one discussion. This paper choose Fourier approximation, and can we consider other type of approximation? I am happy to see more comparison results. \n2. The authors can have more discussions about the limitations and future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "41OMfTiOhC", "reviewer": "Reviewer_kWwk", "summary": "This paper proposes an efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain to exploit de-correlation and conjugate symmetry, leading to more compact and accurate weight representations. Additionally, authors introduce PolarQuant that separately discretizes amplitude and phase in polar coordinates. Moreover, in order to eliminate the need for large-scale calibration data, authors derive an optional diagonal calibration scheme that approximated the Hessian with row/column means.", "questions": "- Referring to line 066 *\"we observe that the weight matrices of LMMs in the frequency domain have a more **compact** spread of singular values as compare to spatial domain.\"* , what is \"compact spread of singular values\"\n\n- The proposed method achieve exceptional performance in hallucination benchmark and can you elaborate the reason\n\n- Since important contribution of the paper is reducing the computational and memory costs, it would be better to move analysis to the main paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper, for the first time, integrates low-rank + quantization optimization in the fourier domain. \n\n- The proposed method using fourier approximation is elegant and has solid mathematical justification.\n\n- Clear pseudocode for algorithms are provided (line 233 - 252).\n\n- Superior performance achieved across different benchmarks.", "weaknesses": "- The novelty is limited somehow and while Fourier-domain decomposition is interesting, it may be seen as a direct extension of existing LoRA + quantization frameworks.\n\n- Comparison with related baselines (QLoRA[1]) is underdeveloped.\n\n- Some qualitative or visualization results may enhance the presentation.\n\n- The ODC heuristic (row/column averaging) lacks strong theoretical or empirical justification. It’s unclear when this approximation fails or how sensitive the method is to distribution shifts.\n\n[1] Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). *QLoRA: Efficient Finetuning of Quantized LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)