RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models
Authors: Zukang Xu
Xing Hu
Qiang Wu
Dawei Yang
Venue: NeurIPS 2025 poster ()
Keywords: Efficient AI
LLM Quantization
Large Language Model
Model Compression
Primary area: deep_learning

Abstract: Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing tasks. However, their exponentially increasing parameters pose significant challenges for deployment on resource-constrained devices.  Vector Quantization (VQ) shows great promise for low-bit quantization (e.g., 2 to 4 bits), but existing work faces two key challenges: unconstrained direction error and suboptimal bit allocation. In this paper, we propose RSAVQ, a novel VQ framework to enhance extremely low-bit quantization for LLMs. RSAVQ introduces two geometry-driven innovations that effectively mitigate above limitations: (1) Error Direction Sensitivity Guidance (EDSG), which leverages the Fisher information matrix (FIM)-induced Riemannian metric to project quantization errors onto low-sensitivity directions in the parameter space. Specifically, this projection is performed along the negative natural gradient direction, which effectively suppresses error expansion.  (2) Weight Channel Sensitivity Guidance (WCSG) , which constructs a channel-wise sensitivity metric via FIM curvature analysis to dynamically guide bit resource allocation. The approach facilitates a globally optimal quantization solution within prescribed bit constraints.  Experiments demonstrate that RSAVQ outperforms existing methods for LLMs. For example, in 2-bit quantization of LLaMA-3 8B, RSAVQ leads baselines like VPTQ and QuIP\# by 0.4 in perplexity (PPL) and 1.5 in zero-shot accuracy.  This work offers a practical solution for constrained environments and a theoretical bridge between information geometry and the quantization of neural networks, advancing efficient deep learning.

RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models
Zukang Xu" Houmo AI
Xing Hu" Houmo Al
Qiang Wu Houmo AI
Dawei Yang Houmo Al
Abstract
Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing tasks. However, their exponentially increasing parameters pose significant challenges for deployment resource- constrained devices. Vector Quantization (VQ) shows great promise for low-bit quantization (e.g.. 2 to bits), but existing work faces two key challenges: uncon- strained direction error and suboptimal bit allocation. In this paper, we propose enhance extremely low-bit quantization for LLMs. RSAVQ introduces two geometry-driven innovations that effectively miti- gate above limitations: (1) Error Direction Sensitivity Guidance (EDSG), which leverages the Fisher Information Matrix (FIM)-induced Riemannian metric to project quantization errors onto low-sensitivity directions in the parameter space. Specifically, this projection is performed along the negative natural gradient direc- tion, which effectively suppresses error expansion.
RSAVQ, novel VQ framework
(2) Weight Channel Sensitivity Guidance (WCSG). which constructs a channel-wise sensitivity metric via FIM curvature analysis to dynamically guide bit resource allocation. The approach facilitates globally optimal quantization solution within prescribed bit constraints. Experiments demonstrate that RSAVQ outperforms existing methods for L.LMs. For example, in 2-bit quantization of LLaMM-33 8B, RSAVQ leads baselines like VPTQ and QuIP# by 0.4 in perplexity (PPL) and 1.5 in zero-shot accuracy. This work offers practical solution for constrained environments and theoretical bridge between information geometry and the quantization of neural networks, advancing efficient deep learning.
Introduction
In recent years, large language models (LLMs) have achieved breakthrough results in natural language
processing (NLP), code generation, reasoning, and multimodal tasks [4 progress is impressive, comes the cost of dramatic increase in model size (c.g.. the LLaMA-3 70B (35] model demands around 140GB of memory in FP16 precision), posing significant barriers when deploying on resource-constraiind devices.
35| 29 [3]. While this
Post- tas-Training Quantization (PTQ) 25 has emerged as promising technique for reducing the resource footprint of LLMs by converting model weights into lower-bit bit fixed- point representations without the need for retraining. A typical strategy in PTQ Scalar Quantization (SQ), where each individual weight is quantized independently near-original model accuracy with bit quantization. However, due to the limitations of numerical representation, SQ struggles to maintain performance extremely low bits (e.g. 3 bit fewer), often leading to significant degradation in model accuracy.
lower-bit value. Recent work [88 28 24 47 22 has achieved
"Equal contribution.
'Corresponding author: davei, dww.i.yanggaa.... yang@houmo. ai
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1.0
Gradient Direction
Vegative Natural Gradient Direction
Incressed KL Loss Degres
0.5
Loos (MSE
0.
0.0
0.1
out channal
in chnnne
in channel
(b)
Figure I: Schematic diagram of loss per- turbation from isotropic weight perturba- tions: natural gradient direction yields maximum loss increment.
Figure 2: (a) Weight distribution of sampling patch in down-project layer of final block in LLaMA-3 8B model. (b) Change in loss after applying the same per-
turbation( + 0.05 per element) to the input channel of figure (a).
In contrast, Vector Quantization (0), another strategy in PTQ shows potential in ultra-low-br LLM quantization. VQ maps high-dimension vectors to a set of predefined lower-dimensional vectors and achieves more effective data compression than SQ by leveraging correlations and redun- dancies across different data dimensions. However, we identify that there two critical limitations in existing VQ-based methods [42 44 31 50]: (I) unconstrained direction error: these methods often overlook directional discrepancies between floating- Moating-point vectors and their quantized represen- tations, which significantly affect model performance. In particular, by treating quantization errors as isotropic perturbations under Euclidean assumptions, they fail capture the geometric relationship between the loss function's sensitivity and the directions of perturbation. As demonstrated Fig. [ emors along the negative natural gradient direction (0 - 135%) lead to loss reduction or significantly less loss increase compared to those along the natural gradient direction (0 = 45%). This highlights that error direction control is critical for preserving accuracy. (2) suboptimal bit allocation: these methods often assume uniform sensitivity across weight channels and assign equal bit-widths to each channel. However, shown in Fig. bb), applying identical perturbations on each channel results in varying effects on model accuracy, indicating that such a naive bit-allocation policy compromises quantization performance.
Relationship Between bits and Perplexity
Relationship Between bits and Accuracy
GPTO
vPTQ
QuiP
QuiPe
VPTQ
RSAVQ(ours)
50
QuIP
RSAVQ(ours)
7
40
2.0
2.5
3.0
3.5
4,0
4.5
5.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Figure 3: WikiText-2 PPL (left) and average zero-shot accuracy (right) for LLaMA-3 8B quantized at different bit-widths.
To address these limitations, we propose Riemannian Sensitivity-Aware Vector Quantization (RSAVQ), a novel VQ framework that leverages information geometry model the parameter space (i.e., the weights) of LLMS as Riemannian manifold with non-uniform crrvature-wheee the local geometry is described by Fisher Information Matrix (FIM)
27 RSAVQ employs FIM to characterize the local geometric structure of the parameter space, including inter-parameter correlations and manifold curvature, thereby enabling a precise quantification of how parameter perturbations along different directions affect the loss function. RSAVQ comprises two core com- ponents: EI)rrror Direction Sensitivity Guidance(EDDCC) projects inevitable quantization errors onto low- sensiivity directions (i.e., along the negative natural gradient directions on the Rieman- nian manifold), minimizing their adverse impact on model performance and effectively mitigating етгог accumulation. Weight Channel Sensitivity Guidance (WCSG) utilizes FIM to quantify each channel's sensitivity, identifying functionally critical channels and dynamically allocating bit resources to prioritize high-curvature (sensitive) channels while applying aggressive compression to low-sensitivity counterparts. This synergistic design enables RSAVQ to optimize the error direction of each quantized sub-vector through low-sensitivity projection and adaptively allocate bits based on

---

channel-wise geometric analysis, thus achieving both ultra-low-bie bit compression and robust accuracy preservation in large language models.
Experiments demonstrate that RSAVQ achieves state-of-the-art performance on L.Mss (see Fig. for detailed comparisons). Specifically, for the LLAMA-3 8B model, our method significantly outperforms existing approaches across diverse quantization bit-widths. In particular, under 2-bit quantization, RSAVQ outperforms baselines such VPTQ and QuIP# by 0.4 in PPL and 1.5 in zero-shot accuracy, Our main contributions are summarized as follows:
We identify and systematically analyze two critical limitations in vector quantiza- tion unnossncostrrinedd direction error and suboptimal bit allocation and demonstrate that they are key contributors of accuracy degradation ultra-low-bit settings.
We propose RSAVQ, a novel VQ framework grounded information geometry. RSAVQ uti- lizes FIM to guide both error direction projection and per-channel bit allocation, integrating these insights into a single holistic algorithm.
We validate RSAVQ on LLMs of various model sizes, demonstrating superior performance compared to existing quantization methods, particularly in extremely low-bit scenarios.
2 Related Work
Recent PTQ methods can be broadly categorized into scalar and vector quantization. These approaches typically assume a Euclidean parameter space, neglecting the intrinsic geometric structure of deep networks. Although geometric deep learning advances show that the parameter space is better modeled as non-uniformly curved Riemannian manifold, these insights have only been theoretically explored, with their practical value in extreme low-bit quantization remaining unexplored. Moreover, to date, no unified framework has integrated information geometry to jointly optimize bit allocation and project errors onto lowssensitvvity directions. This integration is crucial for minimizing performance degradation. Accordingly, we review related work from two angles: PTQ methods and geometric deep learning research.
2.1 PTQ Methods
Scalar Quantization (SQ), a classical quantization approach, maps weights to low-bit representa- tions using fixed scaling factors and zero points. It relies on two key assumptions: (I) isotropic errors, where quantization errors in all directions equally impact model performance; (2) uniform parameter sensitivity, treating all parameters as requiring the same bit precision. Methods like GPTQ[8]] and 28Q28 follow this paradigm, with GPTQ incorporating error compensation and AWQ adjust- ing outlier weights to stabilize quantization. Techniques such as Quarot OSTQuant (3) and MambaQuant[99 utilize Hadamard rotations to enhance weight distribution uniformity, yet these still operate under Euclidean assumptions. Critically, these SQ methods fail to model geometric sensitivities, leading to severe accuracy degradation in extreme low-bit scenarios (e.g. < 3-bit) where direction-specific and channel-wise sensitivities dominate error impacts.
Vector Quantization (VQ) 22 methods enhance weight compression by clustering weight vectors into shared codebooks, For example, GPTVQ[44) integrates error compensation with EM algorithms to optimize codebooks and indices, while VPTQ and AQLM adopt residual quantization to refine error fitting. CRVQ achieves 1-bit quantization by iteratively selecting critical channels for residual processing, and QuIP# [42] uses Hadamard rotations to preprocess weights before uniform codebook quantization. Despite these advancements, existing VQ techniques lack adaptive bit allocation guided by channel sensitivity and fail to constrain quantization errors to low-impact geometric directions, leading to suboptimal performance extreme low-bit scenarios.
2.2 Geometric Deep Learning
Geometric Deep Learning highlights the Euclidean nature of neural network parameter spaces, suggesting that they can be more accurately modeled as Riemannian manifolds endowed with FIM[7] 1[27] Fisher Information Matrix bbsee approaches (45) define a local Riemannian metric for the parameter space and demonstrate that natural gradient descent achieves faster convergence by adapting to the underlying manifold structure. To address the computational complexity of FIM, methods such as K-FAC[3] propose efficient Kronecker-factoree curvature approximations, making second-order optimization feasible for large-scale networks. Beyond optimization, manifold- aware methods [5] have shown that leveraging geometric structures can further unlock hidden information in deep models through Riemannian updates.
3

---

Explorations of Riemannian manifolds in quantization. Prior works, including CLRQ (man- ifold geodesic distances for clustering), GLRSQ[39 (strategies for symmetric positive definite matrices), MANIQUEANTIO (FIM-based gradient mismatch mitigation), FIT (information- geometric metries for distortion reduction), and PLRQ[40] (probabilistic vector quantization with manifold learning), have explored geometric approaches in quantization. However, these efforts focuses on isolated geometric properties (c.g., matrix symmetry, manifold
remain fragmented- metrics) individual quantization issues (e.g., gradient alignment, distortion unified framework to jointly optimize error direction and channel-wise bit allocation.
Preliminaries
To facilitate the presentation of our proposed method, we first introduce three key components: Information Geometry and Riemannian Manifolds, Natural Gradient, and Cartesian Product Vector Quantization. Specifically, these concepts form the theoretical foundation of our framework and provide a unified mathematical structure for subsequent sections.
3.1 Information Geometry and Riemannian Manifolds
The parameter space of deep neural networks is often assumed to be a high-dimensional Euclidean space. However, this assumption neglects the non-uniform sensitivity of different parameter directions. Recent studies [7] 27 2 demonstrate that the parameter space can be more accurately modeled as a Riemannian manifold endowed with the Fisher Information Metrie 45), This geometric view better captures the sensitivity of the model to parameter perturbations, providing a theoretical basis for analyzing quantization errors. Let M denote the differentiable manifold of network parameters W. The Fisher Information Matrix(FIM), which we denote by Fw, defines the metric tensor in the parameter space as:
a Dogg(</\)) Olog p(x W) aw,
Fo (W) E
ow;
where p(x[W) W) is the model's output distribution. This metric endows the parameter space with a Riemannian geometric structure, and the corresponding inner product induced as follows:
( AW2)w = AW, FW
This inner product characterizes the geometric distance of parameter perturbations on the manifold, directly relating parameter changes to the KL divergence of the model output distribution (for the derivation process, see Appendix A.5). practical applications, the channel-wise FIM sub-matrix Fe is used to approximate the global geometrie structure for fine-grained sensitivity analysis.
3.2 Negative Natural Gradient
In flat Euclidean space, the standard negative gradient indicates the direction of the steepest descent. However, on a Riemannian manifold with complex curvature, the optimal descent direction is given by the negative natural gradient[]]:
-C VV. 7 = W
geometrically corrects the Euclidean gradient through the FIM, ensuring that the parameter update descends along the shortest path (geodesic, defined in (p.endix(A6)) on the manifold. The negative natural gradient direction - L is the direction in which the loss function decreases the fastest, and it also the "low-sensitivity direction" that the quantization error should try to align with, in order to minimize the impact of the error on the model performance. The derivation of the natural gradient can be based on the Taylor expansion and the Lagrange multiplier method. For the specific derivation steps, refer to Appendix A.4
3.3 Product Vector Quantization
Product Quantization (66 extends vector quantization (VQ) by splitting high-dimensional vectors into sub-vectors and quantizing them independently, improving compression efficiency and scalability, Similar to VQ, the optimization process of PQ involves finding the closest cluster center Chee i-th codeword in the codebook) for an input vector u:
arg min ||u 1
(4)

---

For detailed comparison between PQ and VQ, refer to Appendix A.3 formulation as its foundation and introduces two key enhancements: specifically, (1) enror direction alignment via Riemannian projections, and (2) adaptive bit allocation guided by channel-wise curvature. These modifications jointly improve quantization accuracy under ultra-low-bit settings.
RSAVQ adopts this PQ
Method
4.. Traditional quantization methods simplify optimization in Eq. as a Euclidean problem, leading to suboptimal control of quantization error. Inspired by the Riemannian perspective in Preliminaries which models the parameter space (i.e. weights) as a curved manifold endowed with the Fisher information metric-we propose the Riemannian Sensitivity-Ae framework. RSAVQ unifies information geometry and vector quantization via two tightly coupled modules: (1) Error Direction Sensitivity Guidance(EDSG)) which projects clustering errors onto low-sensitivity (negative natural gradient) directions on the manifold; and (2) Weight Channel Sensitivity Guidance(Wea WCCGG) which measures each channel's local curvature to dynamically allocate bits and guide codebook updates. By integrating EDSG and WCSG, RSAVQ minimizes quantization distortion under extreme low-bit constraints while maintaining computational efficiency. For full algorithmic details, Appendix [A.3
Vector Quantization (RSAVQ)
4.1 EDSG: Error Direction Sensitivity Guidance
During model quantization, quantization error inevitably occurs, defined
(5) where C(-) denotes the gpeudo-uaanizzttinn operator. Traditional methods [31 typically rely on mean squared error (MSE) or other Euclidean based metrics to quantify quantization error and determine optimal coefficients. However, these approaches overlook the geometric sensitivity associated with the direction of the error. Consequently, errors can amplify along directions that are highly sensitive to model performance, resulting in significant accuracy degradation. Although existing techniques [44 attempt to mitigate such errors through post hoc compensation, their initial disregard for directional sensitivity fundamentally limits their effectiveness. To address this limitation, we propose an Error Projection algorithm based on information geometry. Specifically, we project the quantization error onto the negative natural gradient directions the loss function, which correspond to low-sensitivity directions on the Riemannian manifold. This projection enables a more precise assessment of the quantization error's 's impact on overall model performance.
E W C(W)
As illustrated in Fig. 4 our goal is to project the quantization error onto di- rections that have minimal influence on the loss function, thereby reduc- ing performance degradation from the perspective of information geometry,
Vector
Kmnana
Negative Natural Gradient Directon
Ours Erar
Kinean Intielizer
Error Projection
Vector quantization (VQ) has been demonstrated to be highly promising technique for large model compression[3] 43), largely due to its inherent flexibility in adjusting the quantization error directions. Building on this property, our objective is to design a codebook C that encourages the quantization error E to lie along "low-sensitivity" directions on the parameter manifold M, thereby minimizing the growth of the KL divergence induced by quantization. Through Lagrangian optimization (see Appendix A.44), we formally prove that projecting E onto the negative natural gradient direction VL -F VL minimizes the first-order increase in the loss function on the Riemannian manifold. Here, Fw denotes the Fisher information matrix, characterizing the local geometry of the parameter space. To enforce this projection during quantization, we define the negative natural gradient direction projection loss in the tangent space T as:
Figure 4:
Clustering process of error projection along nega- tive natural gradient direction.
4ppnect
E.
C
where || | denotes the norm induced by the Fisher metric, and À a hyperparameter that controls the trade-off between strict error projection and quantization flexibility. By minimizing Lpmject. RSAVQ encourages the quantization error F. to remain in low-sensitivity directions, thus preserving model performance even under extremely low-bit quantization.
5

---

Building upon the product quantization (PQ) strategy introduced in Preliminaries 3.3 we decompose the weight vectors into a set of sub-vectors {uz}. each independently quantized. During the PQ optimization process, we incorporate the projection loss Lemjrct in the negative natural gradient directions as an additional constraint, alternately optimizing both the codebooks and the assignment indices. This procedure ensures that the quantization error is geometrically projected along the negative natural gradient direction, fundamentally minimizes loss increase and performance degra- dation However, under constrained bit budget, different channels exhibit varying tolerances to quantization error. This observation motivates us to further exploit the local curvature information of the parameter manifold to dynamically allocate bit resources, thereby achieving a more fine- ine-grained trade-off between accuracy and compression.
4.2
WCSG: Weight Channel Sensitivity Guidance
Mean Value
Weight Matrix
First Derivative
Loss Change
0.5
0.025
0.00050
0.0
My MM
0.02
-0.5
0.000
0.00025
50
100
50
100
50
100
100
Channel Index
Channel Index
Channel Index
Channel Index
Figure 5: Weight channel sensitivity analysis and comparison.
Building on the channel sensitivity characterization via the Fisher Information Matrix, we propose a gcometry-aware bit allocation strategy. Our goal is to further minimize the global quantization distortion in addition to the error direction projection achieved in Method 4.1
In practice, different channels exhibit heterogeneous sensitivity to quantization perturbations. Thus, a naive uniform bit allocation is suboptimal, and an adaptive strategy is needed to allocate bits proportionally to each channel's 's importance, thereby minimizing the overall loss degradation.
Existing Euclidean-basee sensitivity metrics-such as the gradient magnitude used in SparseGPT[17 or the weight-activation product employed by Wanda[38] fail to capture the underlying curvature of the parameter space. As demonstrated in Fig. 5 simple statistics like (a) the average weight magnitude (b) the average gradient per channel show no clear correspondence with (c) the true loss sensitivity under perturbations. This mismatch highlights the necessity of curvature-aware measurement for accurate sensitivity estimation.
As introduced in Preliminaries 3.1 the negative natural gradient is defined as -VC traditional Euclidean space, the negative gradient - - C represents the direction of steepest descent. However, a Riemannian manifold with non-uniform curvature, the optimal descent direction must be corrected by local metric w. leading to the negative natural gradient formulation.
-F -F V. In
While the negative natural gradient addresses the geometrically optimal update direction, quantifying weight channel sensitivity requires scalar metric that integrates both gradient magnitude and local manifold curvature. To this end, we leverage the Riemannian norm of the negative natural gradient, which measures how parameter perturbations in each channel's tangent space affect the loss function. We decompose the weight tensor W channel-wise as {Wo). and for each channel c, we compute its Riemannian curvature energy Ic (full derivation in Appendix A.7) as:
I. 1 7..
1 2 W 2 (L.) F.V..
Here, F. is the FIM corresponding channel c. The acquisition of FIM is approximated through Kronecker decomposition. For detailed acquisition steps and principles, refer to A.9 Intuitively, Ie captures how sharply the loss landscape curves along the parameters of channel c: larger Is indicates higher sensitivity to perturbations and thus a greater need for precise quantization.
Under fixed total bit budget Bmax Σ be. how to allocate bits across channels becomes crucial for minimizing global quantization distortion. Based on the rate-distottion theory [21]. the quantization distortion scales exponentially with the number of bits D x 2- Considering the channel curvature sensitivity I, as an amplification factor for distortion, the global distortion objective can be formulated as:
Global Distortion = Σ 22-2222
(8)
Using Lagrangian optimization (details Appendix A. the optimal bit allocation rule is derived as: be x log, Le, ensuring that more sensitive channels receive greater bit allocation.

---

Channel importance
Channel importance
Channel
25
Channel Index
50
73
100
100
Channel Index
In
I1
Ie-1 L
1 1
Im
Jn
b; bg
du b.
sort
group
Figure 6: Channel sensitivity-die driven channel grouping and bit assignment.
In practice, for layer with C' channels, the bit assignment per channel is given by:
loge La
be Round
Bmax
(9)
Σe log2 Io
To enable codebook sharing (each group uses single codebook) while preserving weights channel sensitivity, we introduce sensitivity- -ordeiee channel grouping strategy. As shown in Fig. First, channels are sorted in descending order of their Riemannian curvature energy Ic. We then partition them into G uniform groups, where the group size is n [C/G] and the g-th group contains channels indexed by G, = [[g 1)n + 1, min(gn, C))l For each group, a unified bit-width b is assigned by averaging the dynamically allocated bits of its members, that is,
bg
Round
be cEGg
(10)
G
where be log2 Is under the total bit budget constraint. This strategy clusters sensitive channels (high Io) to receive more bits while aggressively compressing less sensitive groups, balancing distortion minimization with practical deployment constraints.
Experiments
Baseline. We focus on weight-only quantization and compare RSAVQ with several strong PTQ baselines, including GPTQ[I), GPTVQ(44), AQLM[14, DB-LLM QulPP QuIP#[42], and VPTQ[1]. Baseline results for these methods are cited from their original papers. For RSAVQ. we k-means-basse VQ approach similar to VPTQ, with the following settings: the vector length is set to 6, and weight matrices are divided into 4 groups, with each sharing its own codebook. Unless otherwise specified, all experiments are conducted on NVIDIA A100-80GB GPU,
Table The perplexity (ppl) of various quantization algorithms on the LLaMA-2 models when the dataset is Wikitext-2 and the sequence length is 4096, well as the test performance in various
zero-shor shot tasks.
LLaMA-2 7B
LLaMA-2 2
13B
LLaMA-2
70B
Methods
Bits
W21
O- Avgl
Bits
W21
O-shot Avgì
Bits
W2]
0-shot Avg
FPI6
5.12
4.57
67,82
16
3.12
70.21
GPTQ
50.75
39.16
43.84
43.72
2
59.18
GPTVQ
2.25
6.71
56.14
2.25
5.72
61.56
2.25
4.25
68.55
DB-LLM
2.01
7.23
55.12
2.01
6.19
59.41
2.01
4.64
65.83
AQLM
2.29
6.29
58.57
2,18
5.41
61.58
2.07
3.94
68.75
VPTQ
2.02
6.13
58.13
2.02
5.32
62.37
2.07
3.93
68.61
QulP#
2
6.19
58.22
2
5.35
61.96
2
3.91
68.94
RSAVQ
5.97
58.66
5.29
62.84
2
3.55
69.05
GPTQ
8.06
53.1
5.85
59.61
3
4.4
65.41
GPTVO
3.125
5.44
62.69
3.125
59.63
3.125
AQLM
5.46
3.03
63.49
3.01
3.36
69.86
VPTQ
3.02
5.43
61.72
3.03
4.79
64.21
3.01
3.34
69.58
QulPH
5.41
3
4.78
3
3.35
RSAVQ
3.01
5.26
62.7
3.01
4.74
66.12
3.01
3.25
70.42
GPTQ
5.49
60.64
4
4.78
63.87
4
3.35
69.25
GPTVQ
4.125
5.27
62.28
4.125
5.27
64.28
4.125
AQLM
4.04
5.21
62.54
3,94
4.65
65.12
4.14
3.19
69.93
VPTQ
4.01
5.26
61.98
4.02
4.64
64.89
4.01
3.19
69.8
QuIP
5.19
RSAVQ
5.22
63.62
4.01
4.72
66.82
4.01
70.34

---

Models and Datasets. We conduct experiments on LLaMA-2 7B, LLaMA-2 13B, LLaMA-2 4T) and LLaMA-3 8B. LLaMA-3 70B models evaluate the performance of our proposed method. The calibration dataset used in our experiments is sampled from the Red LajaPajaaa dataset [46].
To evaluate the performance of the baselines, we compute the perplexity (PPL.) of the models on the WikiTexx-2 dataset (34]. We evaluate the models by randomly sampling sequences from the dataset with the same length as the calibration data. Lower perplexity indicates better preservation of the original output distribution. For direct comparison with methods like VPTQ[3I] and Quip# we use the same sequence lengths during testing. Specifically, we test PPL with sequence lengths 4096 for the LLaMA-2 models and 2048 for the LLaMA-3 models.
Additionally, we evaluate generalization capability WinoGraad 37), HellaSwagl (22, PIQA[I, ARC-e 5l and cRC-c All evaluations are per- formed using the open-source LMM Harness [9] toolkit.
several zero-shot tasks, including
Main Experimental Results. Tab. [ presents the experimental results the LLaMA-2 series models. We evalu-
Table 2: PPL. on Wikitext-2(seq 1ikieext-2see__le=44099 and 0-shot
task accuracy of PTQ algorithms on LLaMA-3.
ate LLaMA-2 7B, LLaMA-2 13B, and
Methods
Bits
LLMMA-3BP 88
W2,
0-shot Avgl
Bits
LLaMA-3
W2I
0-shot Avg
LLaMA-2 70B under a 2-bit quantiza- tion configuration. For the LLaMA-2 7B model, RSAVQ achieves a perplexity of 5.97 at 2-bits, which is lower than VPTQ (6.13) and other methods, demonstrating more robust performance WikiTeext-2. 2. Furthermore, RSAVQ's zero-shot average accuracy comparable other methods and even exceeds them. For the LLaMA- 2 70B model, in the 2-bit configuration, RSAVQ's s performance is nearly identical to the FP16 baseline, with only a 0.4 PPL
FP16
16
6.14
68.66
16
2.9
75.32
GPTQ
210
36.16
11.9
45.42
QuIP
85.1
36.81
2
13
48.66
QuIP#
VPTQ
2.08
9:29
60.22
2.07
5.66
70.74
RSAVQ
8.79
61.72
5.6
71.3
GPTQ
QuIP
7,5
63.72
4.7
72,56
QuIP#
6.77
VPTQ
3.03
6.97
66.66
3.01
3.81
73.68
RSAVQ
3.01
6.34
66,38
3:01
3.69
74.26
GPTQ
74.88
QuIP
6,5
67.12
4
3,4
74.52
QulPa
6.34
3.21
VPTQ
4.03
6.42
68.14
4.05
3.15
74.7
RSAVQ
4.01
6.31
68.42
4.01
3.11
75.1
decrease and 1.2 accuracy drop, indicating that the model can achieve near original accuracy with
significantly reduced bitwidth. Tab. 2 shows the results on LLaMA-3 8B and LLaMA-3 70B models. Under the 2 bit quantization configuration, RSAVQ achieves 61.72 and 71.3 zero-shot performance for the LLaMA-3 8B and LLaMA-3 70B models, respectively. Compared to other methods, such as GPTQ and VPTQ, RSAVQ not only maintains lower perplexity but also ensures higher zero-shot task accuracy. These results validate the robustness of RSAVQ across different model scales.
Ablation Study. To validate the importance of each component in RSAVQ, performed ablation experiments to compare the performance of the k-means baseline method with the gradual addition of the two components: EDSG and WCSG.The test results are recorded in Tab. 3
Table 3: On the LLaMA-2 7B model, based on the Wikitext-2 dataset and (-short datasets, test the effect comparison after using method and method respectively on the original Kmeans basis.
LLaMA-2 7B
Bits
Methods
W2,
AC
AE
HE
QA
WI
Acc Avg
FP16
5.12
43.3
76.3
57.1
78.1
68.7
64.70
Kmeans
9.20
28.9
62.5
43.3
71.5
63.3
53.90
2bit
+EDSG
7.29 (-1.91)
31.5
66.0
46.6
73.3
63.6
56. 10 (+2.20)
+WCSG
5.81 (-3.39)
37.2
64.4
50.7
75.4
65.7
58.69 (+4.79)
Kmeans
7.25
35.0
68.6
47.7
73.4
64.6
57.86
3bit
+EDSG
5.63 (-1.62)
40.1
72.8
53.9
76.6
66.2
61.77 (+3.91)
+WCSG
5.26 (-1.99)
41.0
73.0
54.7
76.7
68.2
62.70 (+4.84)
On the LLaMA-2 7B model, ablation experiments various datasets (AC, AE, HE, QA, WT) showed: In the 2-bit quantization setting, the k-means baseline achieved an average accuracy of 53.90. Adding the Error Direction Sensitivity Guidance (EDSG) module improved metrics such as AC,HE, thereby lifting average accuracy to 56. 10. Introducing channel grouping further improved performance to 58.69. A similar trend in the 3-bit experiments confirmed that both modules reduce the quantization error and improve accuracy. The results validate their collaborative effect and the framework's effectiveness under extremely low-bit conditions.
8

---

We conducted an ablation experiment to evaluate the effect of the hyperparameter on the performance of RSAVQ. Specif-
LlaMa2-7B quantization
ically, we focused on the LLaMA-2 7B model with 2-bit quan- tization and analyzed the PPL. performance on the WikiText-2
wikitext2 ppl
dataset for various values of X. As shown in Fig. Z Our ex- periment showed that as À increased, the quantization accuracy first improved and then decreased. This indicates that À has an optimal range where the projection between quantization error and the natural gradient direction is most effective. Based on our experiments, we found that the optimal range for À lies be- tween 0.01 and Add.Aiiiional ablation studies across multiple models under 2-hit quantization are reported in Appendix (A.10) showing that À values in the range [0.01, 0.1] are robust across architectures. This result highlights the importance of tuning À for balancing the projection strength between quantization error and natural gradient direction, with the optimal value providing the best trade-off between error reduction and model accuracy. We further analyze the sensitivity of RSAVQ group size and codebook vector length Appendix A.I1
0.8
value
Figure 7: Optimal X Tuning in 2-bit quantized LLaMA-2 7B.
Speed and Efficiency Testing. In terms of hardware efficiency, conducted speed and memory usage tests on the LLaMA-2 7B and LLaMA-2 13B models running inference on a single NVIDIA A100 GPU. The results demonstrate the significant improvements in inference speed and memory efficiency achieved by RSAVQ under low-bit quantization.
As shown in Tab. 4 for the LLaMA-2 7B model. the inference speed with the FP16 model was 38.6 tokens/s, while under 2-bit quantization, the speed achieved 1.57x speedup. Similarly, the LLaMA- 2 13B model achieved comparable acceleration under 2-bit quantization. These results highlight that RSAVQ not only maintains low memory usage but also provides a significant speedup in
Table 4: Speed and Memory Testing
LLaMA-2 7B
[LaMA-2 13B
Bits
tokens's
memory(G)
tokens's
memory(G)
FPI6
38.60
13,16
24.29
25.42
60.60
2.28
44.21
4.03
50.57
3.83
29.15
5,56
inference, making it highly suitable for edge devices resource-constrained environments where both memory and computation resources are limited. Beyond inference efficiency, RSAVQ also reduces the offline quantization time, as detailed in AppendixA.11
RSAVQ achieves superior low-bit performance by integrating information geometry and channel bit allocation, outperforming traditional methods in PPL, zero-shot accuracy, and hardware efficiency across LLaMA models.
Conclusion
This paper introduces RSAVQ, a Riemannian geometry-driven VQ framework that addresses extreme low-bit quantization challenges LLMs. RSAVQ features two key innovations: (1) Error Direction Sensitivity Guidance (EDSG), which projects quantization errors onto low-sensitivity directions (negative natural gradient) using the Fisher Information Matrix (FIM), minimizing performance degradation; (2) Weight Channel Sensitivity Guidance (WCSG), which dynamically allocates bit resources based on FIM-derived curvature metrics to prioritize sensitive channels, balancing accuracy and compression efficiency.
Theoretically, RSAVQ bridges information geometry and neural network quantization by modeling the parameter space as Riemannian manifold, enabling geometrically informed eгror control and adaptive bit allocation. The proposed Riemannian curvature energy metric I. offers a principled way to quantify channel sensitivity, overcoming Euclidean-basee limitations. Practically, its hardware- friendly channel grouping strategy ensures efficient inference while achieving state-of-the-a 2-b compression.
Experimental results LLaMA models demonstrate RSAVQ's superiority, particularly in extreme low-bit scenarios (e.g., 2-bit quantization with minimal PPL degradation and high zero-shot accuracy). By unifying geometric insights with quantization techniques, this work provides robust solution for deploying L.LMs in resource-constrained environments and opens new avenues for integrating information geometry into model optimization. While RSAVQ exhibits strong empirical performance. its cawiooon--innnnnnnnn limited cross crosttarchitecture and cross-domain generality, unexplored scalar quantization applicability, and hardware-speciffc memory efficiency constraints are analyzed in detail in Appendix A.I

---

References
[1] Shun-Ichi Amari Natural gradient works efficiendl in learning. Neural computation, 10(2):251-276 1998.
[2] Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi. Fisher information and natural gradient learning in random deep networks. The 22nd International Conference on Artificial Intelligence and Statistics. pages 694-702 PMLR, 2019.
[3] Salch Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-fre free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37: 101000001002400, 2024.
[4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432-7439, 2020.
[5] Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj. Rajarshi Das, Andrew McCal- lum, Maria Chang. Achille Fokoue-Nkoutche Pavan Kapanipathi, Nicholas Mattei, al. A systematic classification of knowledge, reasoning, and context within the are dataset. arXiv preprint arXiv: / 1806 00358, 2018.
[6] Alice Brigant and Stéphane Pucchmorel. Optimal riemannian quantization with an application air traffic analysis, arXiv preprint
2018.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond cuclidean data. IEEE Signal Processing Magazine, 34(4):18-43 2017.
[8] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher language models with guarantees. Advances in Neural Information Processing Systems, 36:4396 4429, 2023.
Quip: 2-bit quantization of large
[9] Hong Chen, Chengtao Lv, Liang Ding. Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, et al. Db-lim: Accurate dual-binarization for efficient llms. arXiv preprint arXiv:2402. 19660 2024.
[10] Jun Chen, Hanwen Chen, Jiangning Zhang, Tianxin Huang, Yong Liu, et al. Riemannian manifold embeddings for straight-through estimator. OpenReview, 2022.
[11] Elwin Bruno Christoffel. Ueber die transformation der homogenen differentiallussriide zweiten grades. Journal für die reine und angewandre Mathematik, 1869.
[12] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint (rXiv:1803.000 05457, 2018.
[13] Alibaba Cloud. Qwen-vl: A versatile large language model for multimodal understanding, https /mmodelscope.c. cn/ wttp///ggaaaaiio.n....aaoooaaaae Qwen 2025. Available http:///ndeeecopcccmmaaaaaaanino
[14] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression large language models via additive quantization. arXiv preprint arXiv:2401.061 2024.
[15] Yanhong Fei, Yingjie Liu, Chentao Jia, Zhengyu Li, Xian Wei, and Mingsong Chen. A survey of geometric optimization for deep learning: from cuclidean space to riemannian manifold. ACM Computing Surveys, 57(5):1-37, 2025.
[16] Determine Filters' Importance. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 2016.
[17] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned oho-shot. In International Conference on Machine Learning, pages 10323 10337. PMLR, 2023.
[18] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post- post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2200. 173.2,33,,
17323, 2022.
[19] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPoti, Charles Foster, Laurence Golding. Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Zenodo, 2021.
10

---

[20] Allen Gersho. Asympiotically optimal block quantization. IEEE Transactions on information theory. 25(4):373-380, 1979.
[21] Allen Gersho and Robert M Gray. Vector quantization and signal compression, volume 159. Springer Science & Business Media, 2012.
[22] Xing Hu, Zhixuan Chen, Dawei Yang, Zukang Xu, Chen Xu, Zhihang Yuan, Sifan Zhou, and Jiangyong Yu. Moequant: Enhancing quantization for mixture-of-experts large language models via expert- sampling and affinity guidance. arXiv preprint arXiv:2505.0300 2025.
[23] Xing Hu, Yuan Cheng, Dawei Yang. Zukang Xu, Zhihang Yuan. Jiangyong Yu. Chen Xu, Zhe Jiang. and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. arXiv preprint ar X50: 13997, 2025.
[24] Xing Hu, Yuan Cheng, Dawei Yang. Zhihang Yuan, Jiangyong Yu, Chen Xu, and Sifan Zhou. I- IIm: Efficient integer-only inference for fully-quantized low-bit large language models. arXiv preprint arXiv:2405. 17849, 2024.
[25] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2715. 2018.
[26] Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-12, 2010.
[27] Isay Katsman, Eric Chen, Sidhanth Holalkere, Auna Asch, Aaron Lou, Ser Nam Lim, and Christopher M De Sa. Riemannian residual neural networks. Advances in Neural Information Processing Systems, 36:63502-633 -3002,,6314,220 2023.
[28] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[29] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 34916, 2023.
[30] Xue Liu, Weijie Xia, and Zhimiao Fan. A deep neural network pruning method based on gradient 11-norm. In 2020 IEFE 6th International Conference on Computer and Communications (ICCC), pages 2070-2074. IEEE, 2020.
[31] Yifei Liu, Jicheng Wen, Yang Wang. Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit bit vector post-training quantization for large language models. arXiv preprint arXiv:2409. 2024.
[32] James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1-76. 2020.
[33] James Martens and Roger Grosse Optimizing neural networks with kronecker-factoree approximate curvature. In International conference on machine learning, pages 2408 2417. PMLR, 2015.
[34] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint (ed::::09908.40 2016
[35] AI Meta. Introducing meta llama 3: The most capable openly available Ilm to date. Meta AI, 2(5).6, 2024.
[36] Dhananjay Saikumar and Blesson Varghese. Drive: Dual gradient-based rapid iterative pruning. arXiv preprint urXiv:2404.0366) 2024.
[37] Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
[38] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
[39] Fengzhen Tang, Mengling Fan, and Peter Tiño. Generalized learning riemannian space quantization: A case study on riemannian manifold of spd matrices. IEEE Transactions on Neural Networks and Learning Systems, 32(1):281-29 2020.
11

---

[40] Fengzhen Tang, Haifeng Feng, Peter Tino, Bailu Si, and Daxiong Ji. Probabilistic learning vector quantization manifold of symmetric positive definite matrices. Neural Networks, 142:105-118, 2021.
[41] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open foundation and line-tuned chat models. arXiv preprint urXiv:2307.09288 2023.
[42] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipi:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04 04396, 2024.
[43] Albert Tseng, Qingyao Sun, David Hou, Christopher M De Sa. Qti:: Quamtization with trellises and incoherence processing, Advances in Neurul Information Processing Systems, 37:59597-59620, 2024.
[44] Mart Van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for Ilm quantization. arXiv preprint 24X:::2402. 15319, 2024.
[45] DA Wagenaar. Information geometry for neural networks. Term paper for reading course with ACC Coolen, King's 's College London, 1998.
[46] Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao. Virginia Adams, et al. Redpajama: open dataset for training large language models. Advances in neural information processing systems, 37:1 16116-1116199999 2024.
[47] Chen Xu, Yuxuan Yue, Zukang Xu, Xing Hu, Jiangyong Yu, Zhixuan Chen, Sifan Zhou, Zhihang Yuan. and Dawei Yang. Rwkvquant: Quantizing the rwkv family with proxy guided hybrid of scalar and vector quantization. arXiv preprint arXiv:2505.0300 030000000, 2025.
[48] Yuzhuang Xu, Shiyu Ji, Qingfu Zhu, and Wanxiang Che. Crvq: Channel-relaxed vector quantization for extreme compression of Ilms. Transactions of the Association for Computational Linguistics, 13: 13:1488- 1506, 2025.
[49] Zukang Xu, Yuxuan Yue, Xing Hu, Zhihang Yuan, Zixu Jiang, Zhixuan Chen, Jiangyong Yu. Chen Xu, Sifan Zhou, and Dawei Yang. Mambaquant: Quantizing the mamba family with variance aligned rotation methods. arXiv preprint arXiv:2501. 13484, 2025.
[50] Yuxuan Yue, Zukang Xu, Zhihang Yuan, Dawei Yang. Jianlong Wu, and Liqiang Nie. Pedvq: En- hancing vector quantization for large language models via polar coordinate decoupling. arXiv preprint 07X44:2550.005432, 2025.
[51] Ben Zandonati, Adrian Alan Pol, Maurizio Pierini, Olya Sirkin, and Tal Kopetz. Fit: A metric for model sensitivity. arXiv preprint 2100022200008502, 2022.
[52] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentenee? arXiv preprint arXiv: 1905.07830 2019.
[53] Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shanghang Zhang Gradient- based parameter selection for efficient fine-tuning In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 28566-28577, 2024.
12

---

A Appendix
A.1 Limitations
Despite its strong empirical gains, RSAVQ has three main limitations that merit further study:
Domain and architecture generality. RSAVQ is validated primarily Transformer- based language models (e.g.. LLaMA), with limited empirical exploration in multimodal models, computer vision architectures, or reinforcement learning frameworks.
Quantization method scope. This work focuses exclusively applying information geometry to vector quantization (VQ), while its potential utility in scalar quantization (SQ) remains unexplored. Extending geometric sensitivity analysis to SQ could reveal optimization strategies for low-hit quantization.
Hardware deployment efficieicc.... Vector quantization's reliance on codebook lookups intro- duces memory access overhead, particularly on heterogeneous hardware (CPUs/GPUs/odge devices), with limited optimization for platform-specific memory systems (e.g., cache- friendly indexing, parallel computation).
Future work will explore crossarchitecctue generalization, scalar quantization extensions, and hardware-aware quantization optimizations to enhance RSAVQ's practical applicability.
A.2
Acknowledgments
We would like to acknowledge the assistance of large language models in improving the clarity and readability of the manuscript text. We emphasize that all experiments, analyses, and core methodological innovations presented in this paper were independently conceived and executed by the authors, without the use of large language models for generation.
We are also grateful the reviewers, the area chairs and the program chairs for their valuable time and constructive feedback during the review process, which helped us significantly improve the quality this work.
A.3 Vector Quantization vs. Product Quantization
codebook codabook.
M
N
Reshape
W
Kmeans
N
W
Reshape W
Kmeans
Index
index__1
index
Indox
(a) Quantization
(b) Quantization
Figure 8: Schematic diagram of standard vector quantization (a) and product vector quantization (b).
Vector Quantization (VQ). As shown in Fig. 8 a), given weight matrix W E R MrN, we first
reshape it into sequence of v-dimensional vectors W' E IR MN xe A codebook C E R is then built using k-means clustering.
Each vector in W is quantized by selecting the nearest centroid using:
min || C3 arg iE(L.....k
(11)
where C; denotes the i-th codeword. The quantized results are stored as an index array referencing the codebook entries.
13

---

Product Quantization (PQ). To scale quantization to high- vectors, PQ divides each o-dimensional vector into n equal-lengee sub-
v U1 U2 Upl.
U/ E RR//
(12)
Each sub-vector o2 independently quantized using a dedicated codebook C; R producing sub-indices. These indices are concatenated into single composite index that jointly encodes the original vector.
A.4 Principle of Negative Natural Gradient
Referring to 32], the derivation process is as follows:
Let the small change of the loss function L(W) at point W be AL. According to the first-order Taylor expansion, we have:
(13) On the Riemannian manifold, hope find direction AW so that the loss function decreases the fastest under the given Riemannian metric. According to the Lagrange multiplier method, introduce the constraint condition (AW, AW)w 1, and construct the Lagrange function:
VL
I(AW, A) VL X(AW" FwAW -)
(14)
Take the partial derivative of AW and set it to zero to obtain:
VL 2X AW 0
(15)
Get AW F VL. Let -VL FF VL, which is the negative natural gradient.
A.5
Detailed Derivation of KI. Divergence Approximation
The following presents the detailed derivation of the KL divergence approximation formula, involving Taylor expansion, the definition of the Fisher Information Matrix (FIM), and differential properties of probabilistic models:
For parameterized probability distributions p(z|W) and p(x W + AWW, the KL divergence is defined as:
DKL (p(x)/)) [[ + AW) Be-p(=/W) log p(=[W)
(16)
p(x:W + AW)
Expanding the logarithmic term:
DRL. E [[ogp(<[W)) log b [oW
(17)
Let f(W) = log ( For a sufficiently small parameter perturbation AW, the second-order Taylor expansion of f(W + AW) around W (neglecting higher-order terms) is:
f(W AW) se f(W) + VS(W) AW + AW T HWWWAAW,
(18)
where: Vf(W) @logp(2|W) W)
is the gradient (score function), HH((W)
8 log p(x) W)
is the
BW WWW.
Hessian matrix.
Substituting into the KL divergence expression:
DKL se [[(W) (F(W)) 1
1 AW WWTTH,AW) = - V AW H,AW
-
(19)
By the property of the score function (regularity condition of probability distributions):
E V logp(z|W)) = / p(=|W)- V log log)(z)W)) dr = V p(=lWW =1=
= 0.
(20)
Thus, the expectation of the first-order term vanishes:
E V AW] -W [ 0.
(21)
14

---

Using the Information Matrix Equality:
E (W)) = E [V ( log logp(z|W)"] = -FW,
(22)
where Fw is the Fisher Information Matrix (FIM), defined as:
F Fw
V log
(23)
The expectation of the pecond-order term becomes:
AW H,AW]
1 - AW T E[WHH = - AWWFFWA FwAW.
(24)
Under the Riemannian metric defined in the paper, the inner product is:
(AW,, AW2) AW, Aw2,,
(25)
so the second-order term can be expressed as:
AW F Fw 2 (AW, AW),....
(26)
Neglecting higher-order infinitesimal terms, the KL divergence approximates to:
DKL (p(x)W) p(z|W + +AW) (AW,AW)w:
(27)
This approximation reveals the direct relationship between parameter perturbations in the Riemannian manifold (measured by FIM) and changes in the model's output distribution (KL divergence).
In practical applications. to reduce computational overhead or enable fine-grained analysis, block-wise approximations (such as diagonal blocks or K-FAC approximation (33] are often used to decompose the global Fisher matrix into channel-level (or jubnetwork-levee submatrices Fe. These submatrices provide local geometric information for each channel, laying the foundation for subsequent sensitivity analysis and bit allocation.
A.6 Geodesic
In Riemannian manifolds, the shortest path between two points is called a geodesic [[], and its length is defined as the geodesic distance. For two points p, q E M on the manifold, their geodesic distance definition is:
d(p, ạ) = 1 ||((t]] dt.
(28)
Here, yt) is the geodesic that connects and q. and (t) is the geodesic velocity vector. Geodesic distance an important measure of the relationship between points on a manifold. Especially in quantitative error analysis, it can help us evaluate the impact of quantization error on model performance. For example, in our method, the direction of quantization error is projected onto a low-sensitivity direction to minimize its contribution to the geodesic distance. If the manifold is compared the surface of the earth, then the geodesic distance the length of the great circular are between two points. In the parameter space of the neural network, the geodesic distance on the manifold based on the Fiv measure in Preliminaries 3.1 reflects the actual impact the parameter changes on the performance of the model.
A.7 Riemannian Curvature Energy
For each channel c, we aim to measure the impact of parameter perturbations on the loss function. In deep learning, methods relying on the magnitude of weight gradients to determine weight importance have long dominated, as the gradient magnitude directly reflects the dynamic contribution of weights to the loss function. Early studies [16] proposed using the L2 norm of weights to assess importance, but subsequent work revealed that gradient information can more accurately capture the real-time impact of weights. For example, R0 53 introduced the use of the L1 norm of gradients to filter important weights, and (36) further incorporated gradient direction information by fusing I.1 and 1.2 norms for weight scoring. However, directly using the Euclidean norm of gradients || [ Lell overlooks
15

---

critical fact: the parameter space of deep neural networks is inherently a Riemannian manifold. Therefore, instead of directly using gradients, we employ the Riemannian norm of the negative natural gradient:
|| -
(29)
where the negative natural gradient - L. = - VL. projects the Euclidean gradient onto the langent space of the Riemannian manifold. Substituting the negative natural gradient into the norm yields:
-|- -c. (F.' F. (F.
(VLc) F.
(30)
We thus define the Riemannian curvature energy as:
- - (V
(31)
Physical and Geometric Interpretation The formula can be interpreted through two key perspectives: (I) Balance between Gradient and Metric Scaling: The numerator VL. represents the gradient magnitude of channel indicating the local sensitivity of the loss function to parameter changes. The matrix F, the inverse of the Fisher information matrix, characterizes the local metric scaling of the parameter space. When the eigenvalues of F, are large in a particular direction (signifying high local curvature or strong parameter correlations), the corresponding components of F=1 are small, which suppresses the gradient contribution in that direction. This mechanism avoids misjudgments of importance caused by geometric distortion in the parameter space. In contrast, the Fuclidean norm implicitly assumes F. = (the identity matrix), ignoring the actual geometric structure of the manifold.
(2) Impact of Unit Geometric Perturbations: The Riemannian norm of the negative natural gradient measures the effect of unit-length perturbations in the Riemannian manifold on the loss function. By normalizing with F=! perturbations in different parameter directions are converted to the manifold's intrinsic geometric scale. In high-cun curvatureeeee directions (where F, has large eigenvalues), the same Fuclidean-distance perturbation corresponds to smaller effective geometric distance in the manifold, leading to a weaker impact on the loss. This property enables I, to accurately capture the dynamic importance of channel parameters in the true geometric space, overcoming the scale biases inherent in the Euclidean framework.
A.8 Bit Allocation Principle
Following the inference from [2I], the quantization error can be approximated uniformly dis- tributed, and the quantization distortion D has an exponential relationship with the number of quantization bits b, expressed as:
D x 2-
(32)
In our framework, for each channel We, the quantization distortion influenced not only by the number of quantization bits but also by the sensitivity of the channel to the loss function. This sensitivity is measured by Ta 2 1 (where we ensure non-negativity by shifting T +- Ta + 1). Intuitively, Te reflects that even with the same magnitude of quantization error, the impact on the final loss can vary: a larger Le indicates that the channel has stronger gradients and higher curvature, meaning the quantization error is amplified and causes greater loss degradation; conversely, smaller la suggests the channel is less sensitive to errors, resulting in smaller loss impact.
Therefore, the quantization distortion for channel can be formulated as:
De(be) Ic 2-20.
(33)
Under the global bit budget constraint Σg b. = Bmax. where Bmax denotes the total bit allocation across all channels (c.g.. Bmaax = 2 x C for 2-bit quantization), we aim to minimize the overall distortion. To this end, we construct the following Lagrangian:
(((be)))) Σ
L. 2-20 20
+
Eb
b
Bmux
(34)
Taking the partial derivative of L with respect to each by and setting it to zero (ignoring discretization effects), we obtain: ØL = -2nn I 2 26. + y = 0. ab.
(35)
16

---

which leads to:
2 2-26. 2 2ln 1
(36)
Taking the logarithm on both sides, we further derive:
be x log2 (Te).
(37)
Considering the overall bit budget B max. the final bit allocation formula is:
loga Te
bc Bmax
(38)
Σ
log2 Le
A.9 Approximation of FIM
Given the extremely high complexity directly computing the full FIM (scaling with the square of the weight dimension), adopt Kronecker factorization approximation (instead of diagonalization) to decompose the FIM into the tensor product of two low-dimensional matrices: F Fo & F₁, where Fo E IR x (output channel FIM) and F₁ Rax (input channel FIM). This decomposition implemented as follows:
Gradient definition: Vw l denotes the gradient of the loss function with respect to the weight W, computed for single sequence
2. Calculation of Fy: Input channel statistics are estimated via the expectation of the outer product of gradients:
1 E.DD (Vwl)) (Vw()]
Fj
m
3. Calculation of Fo: Output channel statistics are estimated via the expectation of the outer product of gradients:
Fo .END [Vwwe) -V (Vw.)"].
A.10 Additional Experiments on the Projection Hyperparameter À
We conducted further ablation studies to examine the sensitivity of the projection hyperparameter across different model architectures under the 2-bit quantization setting on the Wikitext2 test set. Specifically, we tested three representative models: LLaMA2-7B, LLaMA3-8B, and Qwen2.5-7B. The results summarized in Table
Overall, we find that À values in the range [0.01, 0.1] consistently yield strong performance across all tested models. Based on these results, recommend setting À = 0.05 as robust default choice.
Table 5: Ablation study of projection hyperparameter À under 2-bit quantization Wikitext2. Numbers denote perplexity (lower is better).
Model
0
0.001
0.01
0.05
0.1
0.2
0.4
0.6
0.8
LLaMA2-7B
9.20
7.51
5.94
5.81
5.84
5.87
6.03
6.90
13.10
LLaMA3-8B
13.32
11.19
9.17
8.79
8.78
8.99
14.86
17.63
20.92
Qwen2.5-7B
13.17
10.37
8.75
8.77
9.02
11.27
15.90
19.71
36.74
A.11 Ablation Studies Group Size and Codebook Vector Length
We provide additional ablation studies to analyze the sensitivity of RSAVQ to two critical implemen- tation parameters: the number of groups WCSG and the codebook vector length. These results complement the main text and justify chosen default configurations.
17

---

Effect of group size We conducted experiments on the LLaMA2-7B model with WikiTex12 (sequence length 4096, 2-bit quantization, vector length 6). Results in Thol that performance improves as the number of groups increases, but the gain diminishes after 4 groups. Specifically, perplexity (ppl) drops significantly from 6.03 to 5.81 when increasing groups from 4, while further increases (6-10 groups) yield stable results around 5.78-5.79. This indicates moderate sensitivity to group size, with stable performance at 2 4 groups. We thus adopt 4 groups as the default configuration to balance performance and efficiency.
Table Perplexity (ppl) of LLaMA2-7B with different WCSG group sizes under 2-bit quantization.
Groups
2
3
4
6
8
10
ppl
6.03
5.88
5.81
5.79
5.78
5.78
Effect of codebook vector length Vector length plays a central role in vector quantization. We analyzed its impact using LLaMA2-7B WikiText2 (sequence length 4096, 2-bit quantization, 2 groups for product quantization, 4 groups for WCSG). shows that longer vector lengths yield slight performance gains (e.g., ppl decreases from 5.81 length 6 5.62 at length 14). However, increasing vector length also raises storage and quantization costs. For example, at 2-bit quantization, vector length growth from 6 14 increases the average bit count from 2.0 2.875, with corresponding bandwidth costs.
This demonstrates that RSAVQ's sensitivity vector length lies in the performance-com trade-off: longer vectors offer better performance but incur higher costs. To balance accuracy and efficiency, use vector length of 6 in main experiments.
Table 7: Impact of codebook vector length on perplexity (ppl) and average bit count under 2-bit quantization (LLaMA2-77, WikiText2).
Vector length
8
10
12
14
Avg. bits
2.00
2.00
2.00
2.04
2.19
2.88
ppl
5.97
5.81
5.81
5.81
5.75
5.62
The above results confirm that RSAVQ is moderately sensitive to group size and vector length. Performance saturates after groups, and vector length offers tunable trade-off between accuracy and cost. The chosen defaults (group size 4, vector length 6) provide balanced configuration for our main experiments.
A.12 Quantization Time Comparison
We have supplemented data on the efficiency of the offline quantization process, comparing RSAVQ with mainstream PTQ methods (VPTQ, GPTVQ) under 2-bit configuration on 80GB A100 GPU. Our method adopts a product quantization scheme, which not only reduces quantization time and codebook size compared conventional vector quantization, but also achieves quantization latency comparable to GPTVQ, where the vector length 1.
Table 8: Quantization time comparison of different methods LLaMA2.
LLaMA2-7B
LLaMA2-13B
LLaMA2-70B
VPTQ
2 GPU hours
3.5 GPU hours
19 GPU hours
GPTVQ RSAVQ
1 GPU hours
1.8 GPU hours
8 GPU hours
1.2 GPU hours
GPU hours
10 GPU hours
18

---

A.13 Algorithm
Algorithm 1 RSSAV: Riemannian Sensitivity-Awar Vector Quantization
Input: Original weight matrix W E RMMx total bit budget Bmax, sub-vector length v, iterations T. small positive constants X, number of groups G, Fisher matrices { [F.)e=)
Output:
Quantized codebook C, index matrix I, original channel positions P
CWSG: Channel-wise Sensitivity Guidance
1: Initialize: P(c] = c (preserve original channel order)
2: Compute channel importance scores using Riemannian metric:
3: for c= 1 to C' do
4: Compute natural gradient: VL, V--VL.
Eq paper
5: Importance score: Le
F.
Fisher-weighted sensitivity, Eq paper
6: Store Le in Iimportance (c]
7: end for
8: Sort channels by Zimportance (descending) and reorder W, P
9: Allocate bits proportionally sensitivity:
10: [[c] = [[c] +
for c= = 1 to do
12: blc] = Round Bmax
[[c] logg [ec]
Eq in paper
13: end for
14: Channel grouping: Divide sorted channels into G' groups with equal size
15: [C/G]
16: for g = to G do
Group channels: G, l(g 1)n + 1, min(gn, C)]
Average bits per group: by Round Eoro b(e]
18:
Bq 10 in paper
G
19:
end for
EDSG: Error Direction Sensitivity Guidance
20: for
to G' do
21:
Extract group weights and reshape into sub-vectors: W, E M (Una {R}
22: Initialize codebook Cg via K-means with 2bu19) cluster centers
23:
for 1 T do
24:
for each sub-vector Ug do
25:
Quantize: i. arg min; || - Cg
26:
Compute error: Eg.1 = Ugg1 Cg.
27:
Project error to low-sensitivity direction:
28:
Cproject f- || + À VCI
Eq in paper
29:
Update C, via gradient descent Proojett end for
30:
31: Store index Iig. [] 'g.1
32:
end for
33: end for
34: Reconstruct W. from C and
35: return C,I.P
19

---

A.14 Additional Experiments
In this section we report additional experimental results for LLaMA-2 models 41 and LLaMA-3 models[35]
Table 9: Perplexity on Wikitext-2(Sequence Length=4096) and Zero-Shot Task Accuracy of Various
Quantization Algorithms on LLaMA-2 7B.
LlaMA-2 7B
bit
W24
AC
AE
HE
QA
WI
Avgt
seglen=4096
FPI6
5.12
43.3
76.3
78.1
68.7
64.7
GPTQ
50.75
20.9
34.9
30.5
57.2
52.3
39.16
GPTVQ
2.25
6.71
31.2
66.3
46.4
72.4
64.4
56.14
DB-LLM
2.01
7.23
33,53
45.2
61.98
73.18
61.72
55.12
AQLM
2.29
6.29
34.9
66.5
50.88
74.92
65.67
58.57
VPTQ
2.02
6.13
35.24
63.8
52.08
75.19
64.33
58.13
QulP#
6.19
51.91
75.1
64.9
58.22
RSAVQ(OUTS)
5.81
37.2
64.4
50.71
75.4
65.74
58.6
GPTQ
3
8.06
31.1
58.5
45.2
71.5
59.2
53.1
GPTVQ
3,125
5,44
39.93
74.07
54.21
76.17
69.06
62.69
AQLM
3.04
5,46
38.4
68.06
54.12
76,88
66.93
60.88
VPTQ
3.02
5,43
39.3
69.1
54.9
77.3
68
61.72
QulP#
5.41
68.4
77.3
66.5
RSAVQ(OuTS)
3.01
5.26
54.7
76.7
68.2
62.7
GPTQ
5.49
36.8
66.2
55.4
76.6
68.2
60.64
GPTVQ
4,125
5.27
42.83
75.17
56.41
77.37
69.61
62.28
AQLM
4.04
5.21
41
70.2
56
78.2
67.3
62.54
VPTQ
4.01
5.26
39.7
69
56
78.1
67.1
61.98
QulP#
5.19
40.5
69.1
78.4
67.6
RSAVQrours)
4.01
5.22
76.9
68.2
63.62
Table 10: Perplexity on Wikitext-2(Sequenne Length=4096) and Zero-Shot Task Accuracy of Various Quantization Algorithms on LLaMA-2 13B.
LLaMA-2 13B
bit
W21
QA
Avgt
40q
FP16
T6
4.57
48.2
79.5
60.1
79.1
72.2
67.82
GPTQ
43.84
23.3
43.3
36
61.3
54.7
43.72
GPTVQ
2.25
5.72
38.7
73.6
51.6
75.4
68.5
61.56
DB-LLM
2.01
6.19
38. 14
51.64
08.04
75.14
64.09
59.41
AQLM
2.18
5,41
39.42
69.15
54.68
76.22
68.43
61.58
VPTQ
2.02
5.32
40.02
71.55
56.18
77.26
66.85
62.37
QuIPf
2
5.35
39.5
69.3
56.01
77.3
67.7
61.96
RSAVQ(OUES)
2
5.29
41.4
72.5
56.3
75.1
68.9
62.84
GPTQ
3
5.85
38.48
65.66
53.47
76.5
63.93
59.61
GPTVQ
3.125
4.8
44.45
77.23
58.18
77.8
71.98
59.63
AQLM
3.03
4.82
42.58
70.88
58.3
77.26
68.43
63.49
VPTQ
3.03
4.79
42.32
73.99
58.42
77.64
68.67
64.21
QulP#
3
4.78
44
72.5
78.4
69.1
RSAVQ(OUES)
3.01
4.74
44.9
77.7
57.5
78.1
72.4
66.12
GPTQ
4
4.78
42,49
70.45
58.67
77.75
70.01
63.87
GPTVQ
4.125
5.27
42.83
75.17
56.41
77.37
69.61
64.28
AQLM
3.94
4.65
44.8
73.32
59.27
78.35
69.85
65.12
VPTQ
4.02
4.64
44.37
73.19
59.37
77.75
69.77
64.89
QulP#
4.63
45.5
73.9
78.9
69.9
RSAVQ(Ours)
4.01
4.72
46.5
78.8
59
78.3
71.5
66.82
20

---

Table 11: Perplexity on Wikitext-2Seequence Length=4096) and Zero-Shot Task Accuracy of Various Quantization Algorithms on LLaMA-2 70B.
LLaMA-2 70B
bit
W24
AC
AE
HE
QA
WI
Avgt
409
FP16
16
3.12
51.11
77.74
63.97
81.12
77.11
70.21
GPTQ
NaN
35.8
67
51.8
74.6
66.7
59.18
GPTVQ
2.25
49.4
80.47
58.26
79.4
75.2
68.55
DB-LLM
2.01
4.64
44.45
55.93
76.16
79.27
73.32
65.83
AQLM
2.07
3.94
47.93
77.68
61.79
80.43
75.93
68.75
VPTQ
2.07
3.93
47.7
77.1
62.98
80.3
74.98
68.61
QuIP
3.91
48.7
77.3
62.49
80.3
75.9
68.94
RSAVQ(OUIS)
3.55
48.03
77.56
62.43
80.19
77.03
GPTQ
4.4
44.11
72.73
60
78.4
71.82
65.41
AQLM
3.01
3.36
50
77.61
63.23
81.28
77.19
69.86
VPTQ
3.01
3.34
48.89
77.06
63.52
80.9
77.51
69.58
QulPu
3.35
50.9
77.7
81.4
76.4
RSAVQ(OurS)
3.25
50.8
78.5
63.7
77.5
70.42
GPTQ
3.35
49.15
76.81
63.47
81.23
75.61
69.25
AQLM
4.14
3.19
50,68
77.31
63.69
81.5
76.48
69.93
VPTQ
4.01
3.19
49.57
78.16
63.71
81.18
76.4
69.8
QulPH
4
3.18
50.6
78.1
81.4
77.1
RSAVQ(OUrS)
4.01
50.9
78.7
76.9
70.34
Table 12: Perplexity Wikitext-2 Length=2048) and Zero-Shot Task Accuracy of Various Quantization Algorithms on LLaMA-3 8B.
3LaMA-3 8B seqlen=2048
bit
W21
AC
AE
HE
QA
WI
Avgì
FP16
16
6.14
50.3
80.1
60.2
79.6
73.1
68.66
GPTQ
2
210
19.9
28.8
27.7
53,9
50.5
36.16
DB-LLM
2
13.6
28.2
59.1
42.1
68.9
60.4
51.74
QulP
2
85.1
21.3
29
29.2
52.9
51.7
36.81
QulP#
9.11
39.2
72.9
75.6
68.2
VPTQ
2.08
9.29
36.9
71
52.2
75.1
65.9
60.22
RSAVQ(Our))
2
8.79
40.9
74
51.9
75.7
66.1
61.72
GPTQ
8.2
37.7
70.5
54.3
74.9
71.1
61.7
QulP
7.5
41
72.9
55.4
76.8
72.5
63.72
QulP#
6.77
46.4
77.4
77.9
72.9
VPTQ
3.03
6.97
45.8
77.5
58.4
78.2
73.4
66.66
RSAVQ(Our))
3.01
6.34
46
76.
57.7
78.2
73.9
66.38
GPTQ
6.5
47.7
78.8
78.4
72.6
67.3
QulP
4
6.5
47.4
78.2
58.6
78.2
73.2
67.12
QulP#
6.34
50.2
80.1
79.7
72.9
VPTQ
4.03
6.42
49.1
78.8
59.3
78.7
74.8
68,14 14
(SAVQ(Our))
4.01
6.31
48.5
79.7
59.8
78.9
75.2
68.42
Table 13: Perplexity on Wikitext-2 Length=2048) and Zero-Shot Task Accuracy Various
Quantization Algorithms LLaMA-3
70B.
LLaMA-3 70B scqlen=2048
bit
W2,
AC
AE
QA
WI
Avg
16
2.9
87
66.3
82.4
80.8
75.32
GPTQ
11.9
24.6
38.9
41
62.7
59.9
45.42
QulP
2
13
26.5
48.9
40.9
65.3
61.7
48.66
QulP#
2
18.3
32.2
54.7
68.9
VPTQ
2.07
5.66
83.6
61.8
80.1
70.74
RSAVQ(ours)
2
54.4
83.1
61.7
80.2
77.1
71.3
GPTQ
3
52.1
79.6
63.5
80.6
77.1
70.58
QuIP
3
54,9
83.3
63.9
82.3
78.4
72.56
QulP#
3.8
31.1
36.6
58.8
76.4
VPTQ
3.01
3.81
57.3
84.7
65.5
81.7
79.2
73.68
RSAVQ(our))
3
3.69
58.1
85.2
67
81.1
79.9
74.26
GPTQ
3.3
58.4
86.3
66.1
82.9
80.7
74.88
QulP
4
3,4
58.7
86
65.7
82.5
79.7
74.52
QuIP#
4
3,21
35
67.3
71.9
76.7
VPTQ
4.05
3.15
59
86.1
66.2
82.4
79.8
74.7
(SAVQ(OurS)
4.01
3.11
59,2
86,4
66.1
83.4
80.4
75.1
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide both theoretical and empirical results supporting the main claim of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The Appendix A.l highlights some of the limitation of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
All statements are provided with proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We hope to provide hyperparameters and experimental details in the Section 5 and provide code to reproduce the experiments.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example:
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[No]
Justification:
We aim to collect the code as soon as possible in Git repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.com wttpcccpppbccccceec guwde//ddeeeeee Cooeeooodeesssseeeeeee http:///d///uuuiidd/doo////ea/e/
ssioislicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide all experimental details in the main text, as well as the appendix.
Guidelines:
(a) The answer NA means that the paper does not include experiments.
(b) The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
(c) The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:[Ne [No]
Justification: Many of the experiments were performed large scale models foundation models, rendering the computation of multiple seeds unrealistic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specified the experimental equipment required for experiments in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////waaaaescccccccc//ccacaa
Answer: [Yes]
Justification: The paper should be considered a theory and/or conceptual paper. We discussed implication for efficient LLM deployment in the main text, and can not anticipate that the presented results can not conform any aspect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: here is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No data and models realese.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We clearly cite all code packages and datasets.
Guidelines:
existing assets. The answer NA means that the paper does
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
27

---

Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components
Guidelines:
The answer NA means that the core method development in this research does not non Please refer to our LLM policy fattps://neune cwwpouuipp..cccoofeeeeeeeeeeeeee /tte/oonfeeeeces/2222222 for what should or should not be described.
involve LLMs as any important, original, non non-standard components.
28


Reviews:
[{"review_id": "nqVCyM6ZM1", "reviewer": "Reviewer_RNyf", "summary": "The paper provide a weight quantization method based on product vector quantisation. They extend VQ qunatization in two direction first they take a step further and instead of only minimising the each vector error they also add the negative natural gradient to the error since perturbation in different direction can have different effect on model performance. Second they take the advantage of the fisher information to provide per-channel bit allocation which provide dynamic bit quantization for different group of channel.", "questions": "See **Weaknesses and Questions**", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper is generally well-written and logically structured.\n- The idea of using natural gradients for error correction is interesting and worth further exploration.\n- The use of Fisher Information Matrix (FIM) for bit allocation appears promising.\n\n**Weaknesses and Questions:**\n\n- A key piece of related work, **QTIP**[1], is missing from the discussion. The paper does not provide a comparison with QTIP, which is currently the state-of-the-art method in vector quantization. A comparison would be essential to properly assess the contribution.\n- Important implementation details, such as **codebook vector size** and **group size**, are not justified. Additionally, these parameters are not explored in the ablation study. How sensitive is the RSAVQ to changes in vector size and group size? It would be helpful to include an ablation study using different settings for these parameters.\n- **Figure 7** presents an ablation study for the projection hyperparameter **λ** on a single model. However, it is unclear how sensitive the results are to the choice of λ across different model architectures and bit-widths. Could the authors provide further experiments or analysis to assess this sensitivity? (is it a hyperparameter that have be tuned for different model or between 0.01 and 0.1 work for all models?)\n- Based on the results reported in **Table 3**, **WCSG** appears to significantly impact performance. Could the authors explore applying the same WCSG approach to other methods, such as GPTVQ, to evaluate whether it yields similar improvements?\n- All the results are limited to the LLaMA model. A study on other models, such as Mistral or Qwen, could demonstrate RSAVQ generalizes across different models.\n\n[1] QTIP: Quantization with Trellises and Incoherence Processing NeurIPS 2024", "was_revised": true, "final_justification": "The authors have provided additional experimental results comparing their method with previous approaches, and they have addressed some of my concerns. However, the proposed method does not demonstrate a significant improvement in performance nor a substantial theoretical novelty. Therefore, I only raise my score to borderline accept.", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.  \n***\n> Q1:A key piece of related work, QTIP[1], is missing from the discussion. The paper does not provide a comparison with QTIP, which is currently the state-of-the-art method in vector quantization. A comparison would be essential to properly assess the contribution.\n- Thank you for your valuable feedback. We have conducted extensive comparisons with QTIP on various models (Llama2 7B, 13B, and 70B) under both 2-bit and 3-bit settings\nTests were conducted on the Wikitext2 dataset (sequence length 4096) for perplexity (ppl) and average accuracy across four zero-shot tasks (AC, AE, QA, WI). The results demonstrate that our RSAVQ method outperforms QTIP in nearly all metrics.\n- If you have further questions or suggestions, we would be happy to conduct additional experiments.\n|Bits|Method|LLaMA2-7B||LLaMA2-13B||LLaMA2-70B||\n|---|---|---|---|---|---|---|---|\n|||W2|0-shot Avg|W2|0-shot Avg|W2|0-shot Avg|\n|16|FP16|5.12|66.6|4.57|69.75|3.12|71.77|\n|2|QTIP|**5.86**|60.47|5.35|63.45|3.7|69.9|\n||RSAVQ|5.97|**60.69**|**5.29**|**64.48**|**3.55**|**70.7**|\n|3|QTIP|5.28|63|**4.69**|66.08|3.26|71.53|\n||RSAVQ|**5.26**|**64.72**|4.74|**68.28**|**3.25**|**72.1**|\n***\n> Q2:Important implementation details, such as codebook vector size and group size, are not justified. Additionally, these parameters are not explored in the ablation study. How sensitive is the RSAVQ to changes in vector size and group size? It would be helpful to include an ablation study using different settings for these parameters\n- Regarding the rationale for the vector length and group size of the codebook, as well as the ablation studies, we have supplemented the following findings and revised the latest version of the manuscript accordingly:\n  - Ablation study on group size\n    - We conducted a detailed ablation study on the number of groups for the WCSG method using the LLaMA2-7B model (experimental setup: WikiText2 dataset, sequence length 4096, 2-bit quantization, vector length 6). The results show:  \nThe number of groups has a certain impact on model performance, with an overall trend of \"increased number of groups → improved performance\", though the improvement gradually slows as the number of groups increases.\n    - A performance inflection point appears around 4 groups: when the number of groups increases from 2 to 4, the perplexity (ppl) decreases significantly from 6.03 to 5.81; when exceeding 4 groups (e.g., 6, 8, 10 groups), the ppl stabilizes at 5.78-5.79, indicating saturated improvement.\n    - This demonstrates that RSAVQ has moderate sensitivity to the number of groups, with performance stabilizing at 4 or more groups. Thus, we chose 4 groups as the default configuration in main experiments to balance performance and computational efficiency.\n|WCSG_group|2|3|4|6|8|10|\n|---|---|---|---|---|---|---|\n|ppl|6.03|5.88|5.81|5.79|5.78|5.78|\n  - Ablation study on codebook vector length\n    - Vector length is a critical parameter in vector quantization. We analyzed its impact through ablation studies (experimental setup: WikiText2 dataset, LLaMA2-7B model, sequence length 4096, 2 groups for product quantization, 4 groups for WCSG). The results show:\n    - Performance-wise: Increasing vector length yields certain performance improvements (e.g., when vector length increases from 6 to 14, perplexity (ppl) decreases from 5.81 to 5.62).\n    - Cost-wise: Longer vector lengths lead to increased codebook storage and significantly prolonged quantization time. **For instance, under 2-bit quantization, increasing vector length from 6 to 14 raises the average bit count from 2 to 2.875, with corresponding increases in bandwidth costs**.\n    - This indicates that RSAVQ's sensitivity to vector length primarily manifests in the 'performance-cost trade-off': longer vector lengths may be used for higher performance, while moderate lengths (e.g., 6-8) can achieve favorable results when storage and time costs need control. This aligns with RSAVQ's design goal of 'dynamic codebook adaptation'—maximizing precision retention under limited resources. Therefore, we selected a vector length of 6 as the balanced configuration in main experiments.\n|vector length|4|6|8|10|12|14|\n|---|---|---|---|---|---|---|\n|bits|2|2|2|2.04|2.19|2.875|\n|ppl|5.97|5.81|5.81|5.81|5.75|5.62|\n    - \n***\n>  Q3:Figure 7 presents an ablation study for the projection hyperparameter λ on a single model. However, it is unclear how sensitive the results are to the choice of λ across different model architectures and bit-widths. Could the authors provide further experiments or analysis to assess this sensitivity? (is it a hyperparameter that have be tuned for different model or between 0.01 and 0.1 work for all models?)\n- Thank you for your valuable question. We've conducted ablation studies across multiple models to evaluate the generality and sensitivity of λ. Specifically, under the 2-bit quantization setting (Wikitext2 test set), we conducted λ ablation experiments on three models(LLaMA2-7B, LLaMA3-8B, and Qwen2.5-7B). The results show that λ values in the range [0.01, 0.1] work well for all models, and we recommend λ=0.05.\n- If you have further questions or suggestions, we would be happy to conduct additional experiments.\n|λ|0|0.001|0.01|0.05|0.1|0.2|0.4|0.6|0.8|\n|---|---|---|---|---|---|---|---|---|---|\n|LLaMA2-7B|9.2|7.51|5.94|5.81|5.84|5.87|6.03|6.9|13.1|\n|LLaMA3-8B|13.32|11.19|9.17|8.79|8.78|8.99|14.86|17.63|20.92|\n|Qwen2.5-7B|13.17|10.37|8.75|8.77|9.02|11.27|15.9|19.71|36.74|\n***\n> Q4:Based on the results reported in Table 3, WCSG appears to significantly impact performance. Could the authors explore applying the same WCSG approach to other methods, such as GPTVQ, to evaluate whether it yields similar improvements?\n- Thank you for your valuable suggestion. We've supplemented comparative experiments combining WCSG with two mainstream quantization methods (GPTVQ and VPTQ) to verify its generality and performance improvement effects. As a channel-wise mixed-precision scheme, WCSG's core advantage lies in its flexible adaptation to different quantization frameworks, enhancing performance via dynamic channel-wise precision allocation—a mechanism generalizable to diverse quantization frameworks.\n- Under approximately 2-bit settings on the LLaMA2-7B model, we combined WCSG with GPTVQ and VPTQ respectively, conducting tests on the Wikitext2 dataset and five zero-shot tasks (AC, AE, HE, QA, WI). Results demonstrate that WCSG's performance improvement is not limited to a single quantization method; instead, when integrated with different quantization frameworks (VPTQ, GPTVQ), it stably enhances model performance in language modeling (PPL) and zero-shot tasks while maintaining roughly the same bit count.\n- This further verifies the generality and effectiveness of WCSG as a mixed-precision scheme—its core value lies in providing performance gains for various quantization algorithms through channel-wise precision allocation optimization, rather than relying on specific base methods.\n|Method|Bits|LLaMA2-7B|||||||\n|---|---|---|---|---|---|---|---|---|\n|||Wikitext2|AC|AE|HE|QA|WI|Acc Avg|\n|FP16|16|5.12|43.3|76.3|57.1|78.1|68.7|64.7|\n|VPTQ|2.02|6.13|35.2|63.8|52.1|75.2|64.3|58.2|\n|VPTQ+WCSG|2.04|5.93|37.1|64.1|51.7|74.8|65.9|58.7|\n|GPTVQ|2.25|6.71|31.2|66.3|46.4|72.4|64.4|56.1|\n|GPTVQ+WCSG|2.25|6.28|33.5|67.1|48.8|72.9|65.2|57.5|\n***\n> Q5:All the results are limited to the LLaMA model. A study on other models, such as Mistral or Qwen, could demonstrate RSAVQ generalizes across different models.\n- Thank you for your suggestion. We have supplemented experimental results of RSAVQ on Qwen series models (Qwen2.5-7B and Qwen2.5-14B) to verify its generalization capability across non-LLaMA architectures. The results indicate that RSAVQ still outperforms mainstream quantization methods (GPTQ, GPTVQ, VPTQ) on Qwen2.5 models. These findings confirm that the advantages of RSAVQ are not limited to the LLaMA architecture; it stably surpasses existing quantization methods on Qwen series models, validating its generalization across different model architectures. This generalization stems from the error control mechanism of RSAVQ guided by vector-level Fisher (which does not rely on the assumption of the weight distribution of a specific model architecture). Detailed experimental data are presented in the table below:\n|Model|Method|Bit|W2↓|AC|AE|HE|QA|WI|AVG↑|\n|---|---|---|---|---|---|---|---|---|---|\n|Qwen2.5-7B|FP16 |16|7.46|49.15|79.84|60.93|78.62|70.64|67.84|\n||GPTQ|2.125|110.02|25.48|34.37|33.63|44.92|44.01|36.48|\n||GPTVQ|2.25|22.83|29.52|57.37|37.81|62.95|56.75|48.88|\n||VPTQ|2.02|9.8|48.46|78.87|56.22|73.71|67.03|64.86|\n||RSAVQ(ours)|2|8.77|48.98|77.52|57.51|74.52|67.87|65.28|\n|Qwen2.5-14B|FP16|16|5.7|60.58|85.44|65.63|81.5|75.06|73.64|\n||GPTQ|2.125|56.45|35.58|63.05|40.9|66.43|61.64|53.62|\n||GPTVQ|2.25|83.43|23.38|49.2|28.61|57.51|51.78|42.1|\n||VPTQ |2.02|7.96|53.07|82.45|59.08|79.49|74.27|69.67|\n||RSAVQ(ours)|2|7.31|58.91|83.23|62.99|77.47|73.92|71.3|\n***\nWe have incorporated additional results and detailed analyses addressing your concerns into the revised version of the manuscript. We hope these supplements adequately address your questions and improve the clarity and robustness of our work. Please do not hesitate to let us know if you require further discussion, additional experiments, or any clarifications—we are happy to provide more details to address your concerns comprehensively."}, {"review_id": "KxAzoopNib", "reviewer": "Reviewer_DQR5", "summary": "This paper introduces RSAVQ, a novel post-training quantization (PTQ) framework for large language models (LLMs) that leverages information geometry to perform extremely low-bit vector quantization (VQ). The authors identify two key limitations in existing VQ methods: (1) unconstrained quantization error, which is treated isotropically despite the model's non-uniform sensitivity to perturbations in different directions, and (2) suboptimal bit allocation, which often assumes uniform sensitivity across model channels.\n\nTo address these issues, the paper makes two primary contributions, both grounded in a Riemannian view of the parameter space using the Fisher Information Matrix (FIM) as the metric:\n*   **Error Direction Sensitivity Guidance (EDSG):** A mechanism that projects the inevitable quantization error onto low-sensitivity directions. This is achieved by aligning the error vector with the negative natural gradient, thereby minimizing its impact on the model's loss function.\n*   **Weight Channel Sensitivity Guidance (WCSG):** An adaptive bit allocation strategy that assigns bits based on a channel's \"Riemannian curvature energy,\" a principled, FIM-derived metric. This allocates more bits to more sensitive channels, moving beyond simple heuristics like weight or gradient magnitude.\n\nThe authors deliver a cohesive framework that integrates these two components. They provide extensive experimental results on LLaMA-2 and LLaMA-3 models, demonstrating that RSAVQ consistently outperforms strong VQ and PTQ baselines in terms of perplexity and zero-shot accuracy, particularly in the challenging 2-bit and 3-bit quantization regimes.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1.  To the best of my knowledge, the application of Riemannian geometry (via the FIM) to guide VQ error direction and bit allocation is a novel idea.\n2.  RSAVQ demonstrates state-of-the-art results, consistently outperforming strong baselines, particularly in the ultra-low-bit (2-bit) regime where progress is most needed.\n3.  The paper is very well-written, with clear explanations and informative figures that make complex ideas accessible.\n4.  The comprehensive experiments, including a strong ablation study, provide convincing evidence for the effectiveness of each proposed component.\n\n**Weaknesses:**\n1.  The paper completely omits the computational overhead of the RSAVQ quantization process itself. This is a critical piece of information for a method aimed at efficiency.\n2.  The practical details of how the FIM is approximated are not sufficiently described, which impacts the reproducibility and full understanding of the method. In other words, some key details are missing:\n*   How is the expectation over the data distribution in the FIM definition (Eq. 1) approximated? How many samples are used?\n*   Are there further approximations made to the FIM block `F_c` (e.g., diagonal)?\n3.  Results are reported without error bars, making it difficult to assess the statistical significance of the improvements over baselines.", "was_revised": true, "final_justification": "I have carefully read the rebuttal and other comments. The authors provided additional results and addressed my concerns. I decided to raise my rating.", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.  \n\n***\n> Q1:The paper completely omits the computational overhead of the RSAVQ quantization process itself. This is a critical piece of information for a method aimed at efficiency.\n- We would first like to clarify that RSAVQ is a Post-Training Quantization (PTQ) method, and its quantization process is entirely offline. This means the computational overhead of quantization occurs only during the pre-deployment phase (i.e., when converting a pre-trained model to its quantized version) and has no impact on the inference speed or latency during actual model deployment—a key metric for efficiency in practical applications. The overhead in question here relates solely to the convenience of the offline quantization process.\n- With this in mind, we have supplemented data on the efficiency of the offline quantization process, comparing RSAVQ with mainstream PTQ methods (VPTQ, GPTVQ) under a 2-bit configuration on an 80GB A100 GPU:\n|  | LLaMA2-7B | LLaMA2-13B | LLaMA2-70B |\n|---|---|---|---|\n| VPTQ | 2 GPU hours | 3.5 GPU hours | 19 GPU hours |\n| GPTVQ | 1 GPU hours | 1.8 GPU hours | 8 GPU hours |\n| RSAVQ | 1.2 GPU hours | 2 GPU hours | 10 GPU hours |\n- These results show that:\n  - RSAVQ's quantization overhead exhibits reasonable linear growth with model size, aligning with its efficiency-oriented design goal.\n  - It achieves a favorable balance between performance and cost—outperforming VPTQ across all model scales while maintaining competitiveness with GPTVQ, particularly for larger models (LLaMA2-70B).\n- Further details on implementation optimizations contributing to these efficiency characteristics are provided in the revised manuscript. Please let us know if you require additional clarification.\n\n***\n> Q2:The practical details of how the FIM is approximated are not sufficiently described, which impacts the reproducibility and full understanding of the method. In other words, some key details are missing.\nHow is the expectation over the data distribution in the FIM definition (Eq. 1) approximated? How many samples are used?\nAre there further approximations made to the FIM block (e.g., diagonal)?\n- Thank you for your attention to the key details of the FIM approximation process. We provide supplementary details below to address your two questions:\n- Regarding the approximation method and sample size for the expectation of data distribution  \n  - The expectation over the data distribution ( $E_{s∼D}$ ) involved in the FIM definition in Equation (1) is approximated via finite-sample estimation: specifically, 256 sequence samples (each with a length of 4096) from the RedPajama dataset are used for calculation, balancing estimation accuracy and computational cost. These randomly sampled samples cover the typical characteristics of the data distribution, ensuring the reliability of the expectation estimation. Meanwhile, gradient computation is based on individual complete sequences (instead of token-level independent processing) to preserve contextual dependencies within sequences, which aligns with the core design of the FIM definition that relies on \"sequence-level gradient statistics.\"\n- Regarding further approximation of FIM blocks  \n  - Given the extremely high complexity of directly computing the full FIM (scaling with the square of the weight dimension), we adopt Kronecker factorization approximation (instead of diagonalization) to decompose the FIM into the tensor product of two low-dimensional matrices: $F \\approx F_O \\otimes F_I$, where $F_O \\in \\mathbb{R}^{m \\times m}$ (output channel FIM) and $F_I \\in \\mathbb{R}^{n \\times n}$ (input channel FIM). This decomposition is implemented as follows:  \n    - Gradient definition: $\\nabla_{W} \\ell$ denotes the gradient of the loss function with respect to the weight $W$, computed for a single sequence $s$. \n    - Calculation of $F_I$: Input channel statistics are estimated via the expectation of the outer product of gradients:  \n    $$F_I = \\frac{1}{m} \\cdot \\mathbb{E}_{s \\sim \\mathcal{D}} \\left[ \\left( \\nabla_W \\ell \\right)^T \\cdot \\left( \\nabla_W \\ell \\right) \\right]$$ \n    - Calculation of $F_O$: Output channel statistics are estimated via the expectation of the outer product of gradients:  \n    $$F_O = \\frac{1}{n} \\cdot \\mathbb{E}_{s \\sim \\mathcal{D}} \\left[ \\left( \\nabla_W \\ell \\right) \\cdot \\left( \\nabla_W \\ell \\right)^T \\right]$$\n\n***\n> Q3:Results are reported without error bars, making it difficult to assess the statistical significance of the improvements over baselines.\n- Thank you for highlighting the importance of statistical significance and error bars for the results. We have supplemented experimental results under different random initialization conditions to verify the stability of RSAVQ and quantify the range of performance fluctuations, as follows:\n- Under the 2-bit quantization setting, we have conducted repeated experiments with multiple random seeds (0, 10, 100, 1000, 10000) on the PIQA dataset for the LLaMA2-7B and LLaMA3-8B models. The results show:\n  - Model performance is minimally affected by initialization: the accuracy fluctuation range is [75.2, 75.4] for LLaMA2-7B and [75.5, 75.7] for LLaMA3-8B, with an overall fluctuation within ±0.2.\n  - This stability stems from our optimization of k-means clustering in vector quantization: although different initializations may lead to minor differences in clustering results, when the number of k-means iterations is set to 120 or more, the convergence of the algorithm is significantly improved, and the error bars (fluctuation range) of the final results are effectively compressed to a negligible level.\n- The supplementary experimental data are shown in the table below:\n| Random Seed | 0 | 10 | 100 | 1000 | 10000 | Avg |\n|---|---|---|---|---|---|---|\n| LLaMA2-7B | 75.4 | 75.3 | 75.2 | 75.4 | 75.3 | 75.3 |\n| LLaMA3-8B | 75.5 | 75.6 | 75.5 | 75.7 | 75.7 | 75.6 |\n***\nWe have incorporated additional results and detailed analyses addressing your concerns into the revised version of the manuscript. We hope these supplements adequately address your questions and improve the clarity and robustness of our work. Please do not hesitate to let us know if you require further discussion, additional experiments, or any clarifications—we are happy to provide more details to address your concerns comprehensively."}, {"review_id": "sikE6IIgx4", "reviewer": "Reviewer_Mrg3", "summary": "This paper presents a method for non-uniform vector quantization of LLMs by leveraging the Fisher Information Matrix (FIM). The proposed approach consists of two components: EDSG, which uses FIM to guide vector-level sensitivity, and WCSG, which applies channel-wise FIM for mixed-precision allocation. The method is further combined with product quantization for compact weight representation. The paper is well-written, and the proposed ideas are grounded in established theory. While the design is principled and the overall structure is clear, the empirical improvements over existing baselines appear relatively modest, and the evaluation setup may benefit from more carefully disentangled comparisons.", "questions": "- Could you clarify how many times each experiment was run, and whether the performance gains are consistent across random seeds or sample variations?\n\n- Given that the reported improvements are modest, could some of the observed gains be attributable to data sampling rather than the method itself?\n\n- Can you provide EDSG-only results with fixed bit-width per channel, to enable fair comparison with other uniform-bit quantization methods?\n\n- Since WCSG is a mixed-precision scheme, would it be possible to compare it directly with other mixed-precision methods like MP-GPTQ or SpQR?\n\n- What is the individual impact of product quantization on the final performance? An ablation would help quantify its contribution.", "limitations": "- Gains may be within the natural variance of evaluation benchmarks, especially without multiple trials or error bars.\n- The combination of EDSG + WCSG makes it difficult to assess each component’s standalone merit.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "(+) The method is modular and extensible, incorporating both sensitivity-aware quantization and mixed precision.\n\n(+) The presentation is polished, and the mathematical formulations are clearly explained.\n\n(+) The ablation between EDSG and WCSG helps illuminate component-level contributions, at least partially.\n\n(-) The reported improvements (PPL -0.4, zero-shot accuracy +1.5%) are relatively small, especially in low-bit regimes where prior work already performs strongly. It is not clear how many runs were conducted to confirm the reported gains — without confidence intervals or statistical testing, it’s hard to judge significance.\n\n(-) The current evaluation combines EDSG and WCSG, making it difficult to isolate the benefit of EDSG alone. Since mixed-precision is arguably orthogonal, comparing WCSG-enhanced results to existing baselines may overstate the total improvement.\n\n(-) While the use of Fisher Information Matrix is methodologically sound, it has been previously applied in scalar quantization settings such as BRECQ and Guided Quant. The paper does not sufficiently clarify what new insights or advantages arise from extending it to vector quantization, making the novelty somewhat unclear.\n\n(-) There is no ablation study on the impact of product quantization, which could account for part of the reported gains.", "was_revised": true, "final_justification": "The authors clarified the conceptual distinction between RSAVQ and prior FIM-based quantization methods such as GuidedQuant, emphasizing their use of Riemannian geometry and global Fisher construction. Empirical comparisons under consistent baselines (e.g., QTIP + EDSG vs QTIP + GuidedQuant) supported the claimed advantages, especially in low-bit regimes.\nThe applicability to INT formats and possible extensions to uniform quantization were also addressed with preliminary experiments and discussion. I still believe the contribution could be strengthened by citing final versions of related works and releasing source code to enhance transparency and usability.", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.\n***\n> W1.The reported improvements (PPL -0.4, zero-shot accuracy +1.5%) are relatively small, especially in low-bit regimes where prior work already performs strongly. It is not clear how many runs were conducted to confirm the reported gains — without confidence intervals or statistical testing, it’s hard to judge significance.\n- In terms of the average accuracy of zero-shot reasoning tasks, our method not only reaches the current state-of-the-art (SOTA) level, but also achieves a significant performance improvement. Specifically, the 3-bit quantization results of LLaMA2-13B in Table 6 of the manuscript clearly show that compared with the previous optimal method, the scheme we proposed also leads in terms of the performance improvement range, which fully demonstrates the substantial value of the improvement.\n|Current Method|Previous SOTA|Increase|\n|---|---|---|\n|GPTVQ|GPTQ|0.32%|\n|Quip#|GPTVQ|1.9%|\n|AQLM|Quip#|1.96%|\n|VPTQ|AQLM|0.72%|\n|RSAVQ(ours)|VPTQ|1.91%|\n- To verify the stability of the results, we have supplemented rigorous experimental design and statistical analysis (added in the revised manuscript): All results are based on the mean of 5 independent experiments with different random seeds, where the quantization codebook was initialized with a distinct random seed for each experiment to eliminate the influence of accidental factors. As shown in the accuracy comparison of 2-bit quantized models on the PIQA dataset, the impact of initialization is minimal, with accuracy fluctuations within [-0.3,0.3].\n|Random Seed|0|10|100|1000|10000|Avg|\n|---|---|---|---|---|---|---|\n|LLaMA2-7B|75.4|75.3|75.2|75.4|75.3|75.3|\n|LLaMA3-8B|75.5|75.6|75.5|75.7|75.7|75.6|\n***\n> W2: The current evaluation combines EDSG and WCSG, making it difficult to isolate the benefit of EDSG alone.\n- Same as question 3 (Q3), it will be answered in Q3.\n***\n> W3: Fisher Information Matrix, though valid, is already used in scalar quantization (e.g., BRECQ, Guided Quant). The paper does not clarify new insights/advantages in its extension to vector quantization, leaving novelty unclear.\n- We appreciate noting that FIM has been used in scalar quantization (e.g., BRECQ, Guided Quant). However, RSAVQ does more than merely extend FIM to vector quantization: it rethinks FIM’s application in the vector domain to unlock capabilities scalar methods cannot achieve, with advancements:\n  - In scalar quantization, methods like BRECQ and Guided Quant primarily compute FIM at the channel level, where sensitivity is averaged across all parameters within a channel. This results in a single \"global\" sensitivity score per channel, limiting the ability to distinguish local variations in parameter importance within the same channel.\n  - By contrast, RSAVQ decomposes weights into low-dimensional vector units, computing FIM for each vector—sinking control from channel level to vector level. It identifies not just high-importance channels but also critical vectors within them. This finer granularity enables targeted codebook allocation (e.g., finer clustering for critical vectors), aligning error distribution more closely with model sensitivity.\n***\n> W4:There is no ablation study on the impact of product quantization, which could account for part of the reported gains.\n- Same as question 5 (Q5), it will be answered in Q5.\n***\n> Q1:Could you clarify how many times each experiment was run, and whether the performance gains are consistent across random seeds or sample variations?\n- Thank you for your attention to experimental reproducibility and result stability. We have supplemented a systematic analysis regarding the impact of random seeds and sample size on the results, as detailed below:  \n  - Each experiment was run 5 times, with results reported as the average of these 5 runs.\n  - With 120 k-means iterations, random seed effects are negligible (accuracy fluctuations ≤ ±0.3; PIQA-validated, see weakness 1 table), showing RSAVQ's robustness.\n  - For Fisher information matrix calculation, sample size tests (32 to 4096) show stabilization at ≥256 samples (performance fluctuation <0.1 PPL). Hence, 256 samples are used by default to balance efficiency and stability.\nLLaMA2-7B, 2bit, sequence length=4096\n|Samples|64|128|256|1024|4096|\n|---|---|---|---|---|---|\n|PPL|6.41|5.98|5.81|5.8|5.81|\n- The results of the supplementary ablation experiments (such as the relationship between different random seeds, calibration sample sizes and performance) have been compiled in the appendix, further verifying the consistency of performance gains.\n***\n> Q2:Given that the reported improvements are modest, could some of the observed gains be attributable to data sampling rather than the method itself?\n- To verify the reliability of the improvements and the effectiveness of the method itself, we conducted validation through controlled variable experiments and cross-setting consistency analysis, with the specific results as follows:\n  -  First, as highlighted in our response to Weakness 1, our method not only achieves state-of-the-art (SOTA) performance but also delivers substantial improvements. Critical to ruling out sampling effects, repeated experiments across 5 distinct random seeds showed performance fluctuations within ±0.2, confirming that data sampling randomness has negligible impact on results.\n  - Second, consistent improvements across multiple quantization frameworks directly validate the method’s efficacy. As shown in the table below, combining RSAVQ with baseline methods (PQ and VQ) yields significant gains:\n|Method|W2|0-shot Avg|\n|---|---|---|\n|FP16|6.14|70.1|\n|PQ|24.41|45.1|\n|PQ+RSAVQ|8.79(**-15.62**)|64.2(**+19.1**)|\n|VQ|23.17|47.9|\n|VQ+RSAVQ|8.21(**-14.96**)|64.9(**+17**)|\n  - Notably, in LLaMA3-8B experiments (2-bit quantization, sequence length 2048), RSAVQ combined with PQ/VQ boosted zero-shot average accuracy by nearly 20%—demonstrating that improvements persist regardless of the baseline’s initial performance level.\n  - Such consistent improvements across different quantization frameworks and experimental setups conclusively indicate that performance improvements originate from RSAVQ’s core mechanism (e.g., vector-level Fisher-guided error control) rather than accidental data sampling bias.\n***\n> Q3:Can you provide EDSG-only results with fixed bit-width per channel, to enable fair comparison with other uniform-bit quantization methods?\n- Thank you for your suggestion to supplement the results of EDSG acting alone (with fixed channel bit width) to support a fair comparison with other uniform bit quantization methods. To clearly demonstrate the effect of EDSG as an independent module, we have added comparative experiments combining EDSG with various uniform quantization methods (Kmeans, GPTVQ, VPTQ) under fixed bit width settings, with the results as follows:\n- The core design of EDSG is to improve the clustering loss function, making vector quantization results more aligned with the sensitivity of the model's output loss. Its mechanism is orthogonal to other quantization methods and can be directly applied to uniform bit quantization frameworks. On the Qwen2.5-7B model, under 2-bit fixed width settings, we compared the performance differences between the base methods and \"base methods + EDSG\", specifically as follows:\n|Method|Qwen2.5-7B||\n|---|---|---|\n||Wikitext2↓|0-shot Avg↑|\n|BFP16|7.46|67.84|\n|Kmeans|49.82|39.98|\n|Kmeans+EDSG|16.34(**-33.48**)|56.51(**+16.53**)|\n|GPTVQ|22.83|48.88|\n|GPTVQ+EDSG|14.92(**-7.91**)|59.03(**+10.15**)|\n|VPTQ|9.8|64.86|\n|VPTQ+EDSG|9.33(**-4.7**)|65.89(**+1.03**)|\n***\n> Q4:Since WCSG is a mixed-precision scheme, would it be possible to compare it directly with other mixed-precision methods like MP-GPTQ or SpQR\n- Thank you for suggesting a direct comparison of WCSG with other mixed-precision methods (e.g., MP-GPTQ, SpQR). Below are our supplementary explanations and experimental results addressing this:\n- Regarding MP-GPTQ, we have not yet retrieved relevant literature in major academic databases. If detailed sources or technical specifics of this method can be provided, we would be happy to supplement comparative experiments in future work.\n- For SpQR (a quantization method using a mix of FP16 and integer types), we have added direct comparative experiments on the LLaMA3-8B model. Results show that WCSG retains performance advantages under similar bit budgets.\n|Bits|Method|LLaMA3-8B||\n|---|---|---|---|\n|||W2|0-shot Avg|\n|16|FP16|6.14|68.66|\n|3|SPQR(2.96bit)|8.81|64.02|\n||WCSG(3bit)|7.88(**-0.93**)|65.5(**+1.48**)|\n|4|SPQR(3.96bit)|7.12|68|\n||WCSG(4bit)|6.5(**-0.62**)|68.12(**+0.12**)|\n***\n> Q5:What is the individual impact of product quantization on the final performance? An ablation would help quantify its contribution.\n- To clarify PQ's role in our framework, we analyzed its independent effects via comparisons with Vector Quantization (VQ):\n- PQ's core advantage lies in significantly reducing codebook storage (especially in multi-channel grouping) by decomposing high-dimensional vectors into quantized low-dimensional sub-vectors, though its basic expressiveness is slightly weaker than VQ due to independent sub-vector encoding.\n- Experiments on LLaMA3-8B (2-bit, seq 2048, zero-shot: AC,AE,QA,WI): Basic PQ lags VQ slightly, but with RSAVQ, both improve comparably, achieving near-identical performance while retaining PQ's codebook efficiency.\n|Method|W2|0-shot Avg|\n|---|---|---|\n|FP16|6.14|70.1|\n|PQ|24.41|45.1|\n|PQ+RSAVQ|8.79(**-15.62**)|64.2(**+19.1**)|\n|VQ|23.17|47.9|\n|VQ+RSAVQ|8.21(**-14.96**)|64.9(**+17**)|\n***\nAdditional results/analyses on your concerns are in the revised manuscript. We hope these address your questions. Let us know if you need further discussion, experiments, or clarifications—happy to provide more details."}]

Author rebuttal:

Meta-review: This paper introduces RSAVQ, a quantization framework combining vector-level error direction sensitivity (EDSG) and channel-wise mixed precision allocation (WCSG) guided by the Fisher Information Matrix. The reviewers initially raised concerns regarding missing baselines (e.g., QTIP), unclear hyperparameters, statistical robustness, and computational overhead. The authors provided extensive rebuttal experiments, including ablations on λ, vector size, group size, and multi-seed stability, as well as comparisons across diverse models (LLaMA, Qwen) and methods (GPTVQ, VPTQ, GuidedQuant). These additions clarified reproducibility, demonstrated generalization, and improved empirical support. Reviewers acknowledged the responses and raised their scores, though some felt the contribution remains incremental and the improvements modest. Overall, the work is technically solid, well-written, and proposes a principled geometric approach with meaningful though not groundbreaking gains, making it a valuable addition to the quantization literature.
Decision: Accept (poster)