DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
Authors: Yuantian Shao
Yuanteng Chen
Peisong Wang
Jianlin Yu
Jing Lin
Yiwu Yao
Zhihui Wei
Jian Cheng
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Quantization
Rotation
Efficient inference
Primary area: deep_learning

Abstract: Quantization plays a crucial role in accelerating the inference of large-scale models, and rotational matrices have been shown to effectively improve quantization performance by smoothing outliers. However, end-to-end fine-tuning of rotational optimization algorithms incurs high computational costs and is prone to overfitting. To address this challenge, we propose an efficient distribution-aware rotational calibration method, DartQuant, which reduces the complexity of rotational optimization by constraining the distribution of the activations after rotation. This approach also effectively reduces reliance on task-specific losses, thereby mitigating the risk of overfitting. Additionally, we introduce the QR-Orth optimization scheme, which replaces expensive alternating optimization with a more efficient solution. In a variety of model quantization experiments, DartQuant demonstrates superior performance. Compared to existing methods, it achieves 47$\times$ acceleration and 10$\times$ memory savings for rotational optimization on a 70B model. Furthermore, it is the first to successfully complete rotational calibration for a 70B model on a single 3090 GPU, making quantization of large language models feasible in resource-constrained environments.

DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
Yuantian Shao
Yuanteng Chen Peisong Wang2.3t Jianlin Yu Yiwu Yao Zhihui Wei Jian Chenga3 Nanjing University Science and Technology. DI., Institute of Automation, Chinese Academy of Sciences,
Jing Lin
School of Artificial Intelligence, University of Chinese Academy of Sciences, ZZhongguancun Academy, Huawei Technologies Co., Ltd.
Abstract
Quantization plays crucial role in accelerating the inference of large-scale mod- els, and rotational matrices have been shown to effectively improve quantization performance by smoothing outliers. However, end-to-end fine-tuning of rota- tional optimization algorithms incurs high computational costs and is prone to overfitting. To address this challenge, we propose efficient distribution-awann rotational calibration method, DartQuant, which reduces the complexity of rota- tional optimization
constraining the distribution of the activations after rotation. This approach also effectively reduces reliance on task-specific losses, thereby mitigating the risk of overfitting. Additionally, mization scheme, which replaces expensive alternating optimization with more efficient solution. In a variety of model quantization experiments, DartQuant demonstrates superior performance. Compared to existing methods, it achieves 47x acceleration and 10x memory savings for rotational optimization model. Furthermore, it is the first to successfully complete rotational calibra- tion for 70B model on single 3090 GPU, making quantization of large lan- guage models feasible in resource-constraingg https://githhu www/CAS
introduce the QR-Orth opti-
70B
environments. Code is available at CAS CLab/ CaA.AAS--LaDDDDattaaaaaaaa http//aattttttttttaeeaa
Introduction
Large Language Models (LLMs) 3 have been a key breakthrough in natural language processing. demonstrating exceptional language understanding and generation capabilities through training vast datasets with numerous parameters. These models perform exceptionally well on multiple tasks, including text generation, translation, and question-answering systems [4] 5). However, the high computational and memory demands of LLM inference severely limit their deployment in cssurrc-constrrained environments 6
9
Current methods for reducing computational cost and improving the inference efficiency of deep learning models and LLMs include model pruning, knowledge distillation, parameter sharing, and
quantization [10 MT 12 1 44 15). Among these, post-training quantization (PTQ) stands out as crucial technique to reduce computational costs due its advantage of bypassing complex training processes, making it highly practical for real-world deployment [6
"Equal contribution.
I Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In LLM quantization, activations pose greater challenge than weights due to the frequent presence of extreme outliers, which can significantly degrade model accuracy [2T]. To address this issue, various outlier-handling techniques have been proposed. For example, high-bit protection mechanisms preserve the precision of outliers, while diagonal matrices help smooth extreme values in activations [22 23]. Recent studies have shown that rotation matrices and affine transformations are highly effective in reducing outliers in activations, significantly improving quantization performance [24]. Rotation matrices are invertible, preserve vector norms, and can be seamlessly integrated into model architectures without introducing additional inference costs, making them a mainstream approach for quantization [25]_ Although random Hadamard rotations can improve performance to some extent, they are not optimal. SpinQuant demonstrates that training rotation matrices further enhances quantization performance [26].
However, existing methods (e.g.. SpinQuant 26).. OSTQuant (27]) treat the rotation matrices as network parameters and fine-tune them end- to-end, which incurs substantial computational and memory costs associated with quantization. As shown in Figure [] optimizing the rotation matrices for a 70B model requires hundreds of GiB of GPU memory and tens of gpu hours of computation, which conflicts with the fast deployment goals of PTQ algorithms. More- over, end-to-end fine-tuning of rotation matri- presents unique challenges due to the com- plexity of optimizing on the rotation manifold. Specifically, rotation matrices must be carefully optimized to preserve orthogonality, which ne- cessitates the use of specialized techniques such as Cayley or Riemannian SGD [28 29]. These
7b
13b
70b
Accuracy
SpinQuant
OSTQuant
%
DartQuant
Time (GPU hour)
Figure 1: Comparison of computational costs across different rotation optimization methods. methods are computationally intensive and incur significant time overhead. Additionally, using small sample sizes for end-to-end fine-tuning poses a substantial risk of overfitting [27). which would
worsen the optimization process.
To address these challenges, we propose DartQuant, a distribution- based rotation matrix calibration method that eliminates the need of end-to-end fine-tuning, significantly reducing the resource de- mands of rotation optimization while achieving higher accuracy. To mitigate overfitting, we redefine the rotation optimization problem from the perspective of distribution calibration, i.e., transform the activations into the distribution most suitable for quantization. Based on the distribution trans- formation function that expands small value ranges and compresses large value ranges, we design the Whip loss. Unlike others that directly constrain outliers, Whip loss optimizes the activation distribution, making it more uniform and reducing the impact of outliers, thus lowering quantization errors. Finally. we introduce QR-Orth, an optimization method that applies orthogonal constraints using QR decomposition, avoiding complex projection calculations and significantly reducing the computational complexity of orthogonal optimization, thereby enhancing calibration efficiency.
Our contributions are summarized follows:
We introduce fast LLM Quantization with rotational distribution calibration framework, avoid- ing the excessive computational and memory costs of end-to-end fine-tuning paradigm. Based on this, the Whip loss is designed, which drives rotated activations toward a uniform distribution, effectively reducing quantization error and improving calibration efficiency.
We present the QR-Orth optimization scheme, which ensures orthogonality of rotations via QR decomposition. eliminating the need for complex orthogonal optimizers. This reduces computa- tional complexity and further enhances calibration efficiency.
The proposed DartQaunt framework achieves superior quantization performance while significantly accelerating the rotation matrix calibration. For the 70B model, it delivers 47x speedup in terms of GPU hours and reduces memory usage by 10x compared to existing methods. Notably, DartQuant enables rotation calibration of the 70B model on single 3090 GPU in ~3 hours, greatly reducing calibration costs.

---

2 Related Work
2.1 Challenges in LLM Quantization
Large language models (LLMs) face challenges in quantization due to activation outliers, which take up most of the quantization range and reduce aceuracy. To address this, researchers have proposed various strategies. Early methods used mixed precision, applying different precisions to weights and activations reduce errors. However, mixed precision methods are complex and hinder inference speed and memory efficiency.
2.2 Outlier Handling through Scaling
To address outliers in activation quantization, scaling- baaiin methods have been proposed. SmoothQuant [23] transfers outliers from activations to weights using scale invariance, reducing activation quantization errors. Outlier Suppression+ [2I] addresses the asymmetric distribution of ac- tivations across channels by applying channel-wise scaling and shifting. OmniQuant [19] introduces learnable weight clipping and fine-tunes quantization errors using blockwise error minimization. Although scaling methods reduce outliers in activations, they often shift the quantization difficulty to weights. This does not fully solve the problem, especially in the presence of extreme outliers [30] Efficiently handling outliers without complicating weight quantization remains key challenge.
2.3 Outlier Handling through Rotation
Recent research shows that rotation matrices offer unique advantages in handling outliers in activation quantization. QulP [31] first introduces incoherent processing to reduce the impact of outliers in both weight and activation spaces. QulP (32 further improves speed by using randomized Hadamard transforms, which also have better theoretical properties. Building this, QuaRot [25] combines the outlier suppression ability of rotation matrices with invariance transformations, applying it to large models like LLaMA, significantly improving PTQ performance. QuaRot also finds that Hadamard transforms outperform random orthogonal transformations in quantization. SpinQuant [26] extends the rotation matrix trainable parameter and employs the Cayley optimizer (28) for end-to-end fine- tuning. QServe 277 combines rotation and scaling techniques, using random Hadamard transforms and scaling methods suppress outliers across different modules, enhancing performance under low-bit quantization. Further, OSTQuant (27 treats both rotation and scaling as trainable parameters and employs KL-top loss for end-to-end fine-tuning, achieving better quantization accuracy.
Existing end-to-end fine-tuning methods for optimizing rotation matrices, while relatively simple to implement, typically incur significant optimization costs and are prone to overfitting. These methods also require high-quality calibration samples. Furthermore, the orthogonal optimizers used during optimization computationally expensive, they need to perform optimization complex manifolds, adding additional challenges. In contrast, DartQuant significantly improves the efficiency of rotation matrix calibration. While maintaining comparable accuracy, it achieves calibration speed that is more than 47 x faster than existing methods.
VAlUe
Vikee
Original.
Hadamard.
(c) SpinQuant.
(d) OSTQuant.
(c) DartQuant.
Figure 2: Effects of different transformations on activation distribution.
3 Preliminaries and Difficulty
The Transformer architecture, commonly used in large language models (LLMs), consists of multi- head self-attention modules and feedforward network modules, both primarily composed of linear layers. Let the output of linear layer be represented as Y = XWT, where X E R XCn denotes the input activation, and W Caa represents the weight matrix. Based on rotational invariance, we can insert orthogonal transformation R into the linear layer without altering the output, yielding

---

Table I: Impact of overfitting: Calibration on different data distribution on LLaMA models.
Sor
Darthvar
Model
Datasets
WiKi
PTB
C4
Baseline
5.47
37.91
7.26
Models
Slddess
2 7b
WiKi
5,94
45.13
8.13
PTB
6.02
38.24
8.13
(a) The number of outliers.
(b) Quantization error.
C4
6.05
44.99
8.02
Figure Effects of different transformations on 1000 activations in layer 20 for various mod- els. The rotation matrix optimized by DartQuant achieves the lowest number of outliers and the smallest quantization error.
Baseline
4.88
50.94
6.73
2 13b
WiKi
5.21
58.39
7.40
PTB
5.33
49.14
7.41
C4
5.33
60.59
7.32
Y = (XI)(R WT), where R E RCxxC. is an orthogonal matrix satisfying RR = I. By combining R with the previous weight matrix and R with the current layer's weight matrix, we can rotate the activation vector without introducing any additional computational cost.
Similarly, under the condition that the model's ' output remains unchanged. we can insert four orthog- onal matrices R1, R2, R3, R4 within the Transformer block, as outlined in [26] and Appendix A Specifically, by multiplying R₁ on the right side of Wg. Wk. W.., Wup. W Wgate. and multiplying Rj on the left side of W our and down: an equivalent transformation is achieved. Similarly, R2 can be inserted between Wa and Won. Rg can be inserted between the rotated encodings of Q and K. Ra can be inserted before W Wdown Finally, W Wembedding is multiplied on the left by Rf and Wm, head is multiplied on the right by R1, completing all the equivalent transformations. This process referred to as the "Computational Invariance"
(33
QuaRot 25] demonstrates that using random Hadamard rotation can achieve good results; however, this is not optimal, as shown in Figure 2 the rotation matrix R as learnable network parameters and fine-tune them in an end-to-end manner with
Methods like SpinQuant [26) and OSTQuant [27] treat
inserted, resulting in better quantization performance. Although end-to-end fine-tuning is simple implement, the complex computational process faces resource challenges and optimization difficulties. In Figure 3a| we present the number of outliers among 1000 activations in the 20th layer of various models after different transformations. Figure 33b shows the average quantization error of these samples (statistics for activations from other layers are in Appendix (F). It is evident that the transformations in end-to-end tne-tuning do not significantly reduce the number of outliers in the activations, nor do they notably lower the quantization error, highlighting the limitations of end-to-end fine-tuning.
Moreover, end-to-end fine-tuning based on calibration sets not only consumes considerable compu- tational resources but also tends lead overfitting on the calibration set [34 (7]. As shown in [ fine-tuning methods exhibit significant performance improvement on the corresponding test sets, with the improvement being particularly pronounced on the PTB dataset. A possible explanation is that the limited and relatively simple calibration data often fail to fully cover the parameter space of large models, causing the model to overfit to a narrow feature distribution, thereby limiting its generalization and emergent capabilities (27).
In addition, optimizing the rotation matrix requires the use of orthogonal optimizers to ensure the matrix's 's orthogonality. These optimizers are based on Riemannian optimization on the Grassmanian or Stiefel manifold, involving complex projection computations 28 29 As a result, their computational time is approximately twice that of standard optimizers. This, in turn, further increases the cost of rotation optimization and slows down the optimization process.
Method
In this section, we provide detailed description of the proposed DartQuant method. DartQuant is comprised of three key components: the rotational distribution calibration, the Whip loss function, and QR-Orth optimization. Each of these components addresses the primary challenges outlined earlier. [igure/4illustrenn the overall framework, highlighting the flow and interactions between these components.

---

rotational distribution calibration
calibration results
soobeddling
q8 deconpprtton
large outtlers
R
MIHSA
forward
smalll and fewer outllers
activatioa(X)
O=X&R
backward
MBR
avoid ovarfitting compatatiinn (w/o end
almost outtiers
daR
los==whip(O))
L. Read
25/75 percentile
minn/max percentile
ont-skot colbbation data collection
itorative rotation coliratiin
2/40 percentlle
Figure 4: Left: The DartQuant implementation process, with Z representing the latent parameters in QR-orth and R as the applied rotation matrix. Right: The change in rotation matrix before and after calibration.
4.1 Rotational Distribution Calibration
End-to-end fine-tuning typically requires more data, and the optimization of rotation matrices depends on the task -aee-fpec loss, which significantly increases the risk of overfitting. For LLMs, end-to end-to-end fine-tuning also entails substantial computational and memory overheads. To address these challenges, we redefine the rotation optimization problem and propose rotational distribution calibration.
Specifically, we revisit the optimization objective of the rotation matrix from the perspective of feature distribution transformation. We redefine the problem finding a rotation matrix that transforms the activations into the distribution most suitable for quantization. This approach reduces the reliance on task-specific loss during calibration, thereby mitigating the risk of overfitting.
Previous studies have shown that outliers are the primary cause of activation quantization loss. Therefore, we constrain the activation distribution after rotation by minimizing the number of outliers in the transformed activations, i.e.
GLa
min T(((R))d > r)
R
The function I(-) represents the indicator function, and J is the threshold used to identify outliers. Although this problem cannot be directly solved using standard stochastic gradient descent, we can resort to approximation methods for calibration. In statistics, variance is commonly used to measure the dispersion of data; however, using variance as an optimization objective is not ideal. Due to the symmetric distribution of activations 34 the variance of activations typically corresponds to constant multiple of the activation vector's norm square. Furthermore, the norm-invariance property of the rotation matrix introduces significant challenges when directly optimizing using variance (as shown in Figure 7a). In addition to variance, kurtosis frequently used to measure the heaviness of the distribution's 8 tails, making it a suitable alternative objective. However, since the rotated activations are already close to a Gaussian distribution with relatively few outliers, optimizing with kurtosis is slow (as shown in Figure 7ap. Therefore, there is an urgent need for better optimization objective to constrain the activation distribution.
4.2 Activation Uniformity via Whip Loss
To better reduce outliers, we propose a new optimization objective that constrains the activation distribution to approach uniform distribution, thereby effectively reducing the number of outliers in the rotated activations.
As shown in Figure 2a and Appendix G Assuming that the activation tokens follow a Laplace distribution with mean ja = and scale parameter b, the probability density function (PDF) is given by:
the activation tokens exhibit distribution near Laplace.
1
a
f(x)
exp(
)
(2)
26
b
5

---

Rotate
(x)
outlier
outlier
outlier
outlier
peak
disperse
compress
Probability
flatten
compress
disperse
gather
gather
hard to quantize
easy to quantize
Figure 5: Intuition behind the distribution transformation: Ux (x)) transforms the Laplace distribution into a uniform distribution by flattening the peak and aggregating the outliers.
In statistics, cumulative distribution functions (CDFs) are often used to transform one distribution into another [35]. To convert æ ~ Laplace b) to a uniform distribution over the interval [-T, T]. the transformation function is the following:
[
(x = 2T
exp(
dt b
-o0 2b
2
rlexp( : T[L [[ll-epp(-1
1].
xΣ 0,
}] 0.
As shown in Figure 5 the left side presents the function graph of (z), where the intervals the origin are expanded, while those further from the center are compressed. The right side visually illustrates the impact of this transformation concentrated around the center over a wider range, thus smoothing the peak of the distribution. Meanwhile, outliers farther from the center are gathered together, shrinking the overall distribution range, ultimately resulting in a uniform distribution within the interval [-r,
the distribution. Ux (x) spreads values originally
Inspired by the mechanism of Ux (x), we propose the Whip loss function:
Whip Σ exp(-1x;1).
E CRC.. denotes the activation vector. Clearly, the Whip function continuously differentiable, and has larger gradients near zero. When used as loss function, smaller values in the rotated activation vector are pushed away from zero. other words, the Whip function encourages the rotation to smooth the sharp central peak of the Laplace distribution, producing a more uniform distribution. As the magnitudes of several small-value channels the activation vector increase, the outliers suppressed due to the norm-invariance constraint. This results in an "aggregation" " effect the activation values. As a result, the overall activation distribution tends to converge toward uniform distribution within smaller interval, thereby effectively reducing the quantization error.
Here, = =[x1, (2.,,
4.3 Enforcing Orthogonality with QR-Orth
To satisfy the orthogonality constraint, the rotation matrix must be optimized the Grassmannian or Stiefel manifold, which necessitates the use of specialized optimizers, such as the Cayley SGD
28 used in SpinQuant [26] Unlike gradients in Euclidean space, gradients manifolds require
complex projection operations, resulting in significantly higher computational costs. To avoid the computational complexity of orthogonal optimizers, we propose the QR-Orth optimization method.
Specifically, we can obtain an orthogonal matrix E Baxn and an upper triangular matrix E R by performing QR decomposition any matrix Z pax", Based on this relationship. design

---

the rotational distribution calibration method with QR-Orth optimizer shown in Algorithm [ We use the orthogonal matrix R. obtained from the QR decomposition, as the rotation matrix for the actual computation. The latent parameter Z is treated as a optimization parameter and is discarded after calibration. By optimizing the latent matrix Z, we indirectly optimize the ro- any
Algorithm Rotational Distribution Calibration with QR-Orth Optimizer
1: Input: LLM model LLM, calibration se- quence S, initial latent parameter Zo E RR**, iterations T, learning rate n).
2: Output: Rotational matrix R E IR
3: X < LLM(S)
tation matrix R. In this way, optimizer to optimize the rotation matrix.
4: X + token_ Sikte__smmling({X
5: Z s Zo
6: for k = 0 I do
When the matrix size becomes large, the Cayley optimizer introduces a computational overhead
7:
& gr qr_ decomposition (Z)
8:
O t X@R
of approximately 6n compared to standard op- timizers. contrast, QR-Orth only incurs the cost of the QR decomposition, with a compu- tational complexity on the order of fn² (see Appendix B for detailed complexity deriva- tion). Although QR decomposition usually requires iterative calculations, the significant reduction in overall computational load has brought about 1.4x acceleration effect. In practice, QR-Orth can casily integrate with various optimizers such as SGD or Adam, making it highly adaptable. This flexibility makes QR-Orth promising solution for optimizing orthogonal matrices.
9:
L t Whip(O)
Z + Z 1az
DC
11: end for
5 Experiment
Model and Dataset. We evaluate method on the Llama series models, including Llama-2 (7B/13B/70B) [ and Llama-3 (8B/70B). Moreover, we also provide results on two popular MoE models: Mixtral-8xx xx (36] and Deepseck- Moo (37]. We report perplexity (PPL) scores the WikiText2 [3), C4 [39, and PTB [40]. Additionally, we assess model performance on nine zero- WinoGrande [4].
shot evaluation tasks, including LAMBADA (41), HellaSwag 442, PIQA
43
OpenBookQA
45 SIQA 46 MMLU
47 ARC-E, and ARC-C
48
Baselines and Implementation Details. In addition to the basic RTN method, we compare
approach with several other methods, including SmoothQuant (23), GPTQ 49,, OmniQuant [1)].
and current state-of-the-arr methods such as Quarot 255 SpinQuant 26 and OSTQuant (27 for weight and activation quantization.
In the main results, we apply GPTQ to reconstruct the weights. To do so, we use 128 samples from WexiText2, with sequence length of 2048 tokens, as the calibration set for GPTQ, following the standard GPTQ setup. All activations are quantized using per-token asymmetric quantization. We optimize all orthogonal matrices using SGD combined with QR-Orth. During the orthogonal matrix calibration phase, we use 128 samples from WikiText2, each with token length of 2048.
5.1 Main Results
Table evallatess six models across four common bit-width settings, offering practical guidance for selecting appropriate rotation schemes. DartQuant utilizes learned rotation matrices Rú and R2 which can be fused into the model weights during inference, eliminating any additional computational overhead. In contrast, online Hadamard rotations (Ra and R4) leverage fast Hadamard kernels for efficient inference computation [32]. As shown in Table 2 when both weights and activations quantized to 8 bits, the performance differences among methods minimal. However, when weights are quantized to 4 bits and activations to 8 bits, methods like SmoothQuant and OmniQuant experience significant performance degradation. This is primarily due to SmoothQuant's 's design, which complicates weight quantization and increases quantization errors, leading to a substantial drop in performance. In contrast, other methods generally maintain the model accuracy.
Although rotation transformation methods improve quantization performance when activations are quantized to bits, the additional computational cost associated with R3 and R4 makes this approach less efficient. When activations are quantized to 4 bits, omitting the rotation matrix results significant performance drop. Furthermore, DartQuant, SpinQuant, and OSTQuant, which optimize
7

---

Table 2: Comparison of the average Perplexity Scores across three datasets and the average accuracy on nine Zero-shot Common Sense Reasoning tasks. The results for all comparison methods were obtained using their publicly available codebases. Full results can be found in the Appendix C
Bits (W-A-KV)
Method
Llama-2 2 7B
Llama-2 13B
Llama-2 2 70B
Llama:5 SD
Llama-3 70B
PPL. J
O-shor"
P
0-chor"
PPL L
0-chot"
PPL J
O-shor"
PPI. 1
0-shor"
16-16-16
PlozzingPoint
16.88
61.16
20.85
64.28
11.09
69.53
8.92
66.04
6.19
72.70
12.38
SmoothQuant
332.17
30.97
1510.66
29.89
180.96
38.36
112.46
31.94
544.68
33,95
GPTQ
6977.62
60.03
20.76
63.70
11.90
69.03
10.27
61.79
6.89
69.44
4-3-16
OmniQuant
426.53
59.15
20.74
62.95
14.06
67.18
10.48
62.72
14.95
59.94
QuaRot
9.59
6.92
SpinQuant
17.85
60.10
21
63.53
11.29
69.57
9.48
65.01
6.63
71.76
DarqQuant
17.69
60.17
20.93
63.77
11.18
69.30
9.49
65.58
6.66
71.82
2523
SmoothQuant
$278.95
29.48
4366647
29:05
1636.53
29.40
2216.28
29:55
6242.66 6222.622
29.
GPTQ
1529.13
31.05
1554.72
29.85
68684
29.23
270.19
33:47
14201.63
32.53
4-4-16
OmniQuant
202.95
40.18
107.01
42.98
109:27
41,08
186,02
31.25
380094
28,
QuaRot
SpinQuunt
19.90
57.85
22.58
62.32
11.70
68.59
10.67
62.29
9.61
66.06
OSTQuant
19.24
57.94
22.33
62.38
11.98
68.29
10.66
62.18
7.67
67.94
DurQQuant
18.53
58,05
22.44
62.64
11.51
69.02
10.58
62,80
7.99
69.39
2535.13 13
29.53
63772.42
178003.40
30.41
GPTQ
1813.34
29.97
1929 93
29.53
78362 25
29.15
496.93
31,60
17361.71
32.9%
4-44
QuaRot
27.01
57.03
24.9%
39.87
11.49
67.41
12.29
57.32
11.38
61.50)
SpinQuant
10.17
OSTQuant
19.74
57.88
23.83
62.31
11.67
68.11
10.66
61.57
7.76
67.84
DarQQaatt
19.14
$7.96
22.64
62.46
11:55
68.22
10.78
62.38
8.13
69.05
rotation matrices, clearly demonstrate the necessity of rotation matrix optimization, outperforming QuaRot. Additionally, while SpinQuant and OSTQuant perform well in reducing perplexity, their performance in zero-shot tasks is poor, which highlights the potential overfitting risks associated with end-to-end fine-tuning methods. In contrast, DartQuant, with its novel calibration strategy, generates rotation matrices that effectively compress the activation distribution range, resulting outstanding performance in 0-shot tasks. Notably, we achieve performance loss of only 0.5% on Llama 2-70b under w4a4kv16 setting. For Llama 3-70b which is more difficult to quantize [50], we manage to limit the average performance loss to 3.31% %, outperforming SpinQuant and OSTQuant by 3.33% and 1.45% respectively.
Besides dense LLMs, we also extend our research to recent popular MoE-LLMs, and further experimental results can be found in the Appendix H
Table 3 presents a comparison of the optimization time and memory con- sumption of SpinQuant, OSTQuant, and DartQuant
Table 3: Comparison of Rotation Matrix Optimization Cost.
Cost
Method
7B
13B
70B
A800 GPU
server. DartQuant simplifies the cal- ibration framework, leading to sig- nificant reductions in resource over- head across various models. In par- ticular, for the 70B model, DartQuant completes the calibration in 30 min- utes using a single GPU, achieving a speedup of 47x in training and 10x in memory savings compared to Spin-
SpinQuant
0.30
0.70
42.90
Time
OSTQuant
0.30
0.80
44.00
(GPU hour)
DartQuant
0.14
0.23
0.91
DartQuant
0.43
0.70
2.90
SpinQuant
19.98
33.73
238.89
Memory (GiB)
OSTQuant
42.25
239.16
583.86
DartQuant
17.41
21.40
23.47
DartQuant
17.41
21.40
23.47
Quant and OSTQuant. Moreover, DartQuant is the first to optimize the rotation matrix of the 70B model on a single 3090 GPU, with calibration time of ~~3 hours. This development substantially reduces the cost of rotation matrix optimization and enhances its practical value.
5.2 Ablation Studies
We compared the effectiveness of four optimization objectives: quantization loss, variance, kurtosis, and the Whip function. As shown in Figure 7a| the change in activation quantization loss over iteration steps is presented for each optimization objective. It is clearly observed that when using quantization loss, variance, kurtosis as the optimization objective, the activation quantization loss shows minimal variation. However, when the Whip function used as the optimization objective, the quantization loss curve decreases significantly within fewer iterations and converges rapidly.
8

---

Count
Caunt
Count
Value
Value
Value
(a) Original.
(b) Hadamard.
(c) Quant.
Count
Count
Count
50-
Value
(d) Variance.
(c) Kurtosis.
(f) Whip.
Figure 6: Histograms of Activation Distributions After Rotation by Different Rotation Matrices. The region outside the red dashed line represents the outliers.
5.2.1 Optimization objectives
By comparing the effects different optimization objectives on the activation distribution (as shown in Figure(6) (),
clearly observe the substantial changes induced by the Whip function. Figure 6aa shows the histogram of the original, unrotated activation distribution. From the range on the x- axis, is evident that the original distribution contains significant outliers. Figure 6b displays the histogram of the activation distribution after a random Hadamard matrix rotation. After the Hadamard rotation, the activation range notably compressed, although some outliers remain untreated. The rotation matrices trained with quantization loss and variance as optimization objectives show little improvement, resulting an activation distribution nearly identical that obtained by random Hadamard rotation. Although kurtosis optimization slightly improves the distribution, its effect limited. In contrast, the histogram after Whip optimization shows significant improvement (as shown in Figure (í): this method not only effectively addresses the outlier problem but also disperses the activation points, initially concentrated around zero, across other regions. The resulting distribution is the closest to uniform distribution. This outcome aligns closely with our design goals and further validates the effectiveness of our approach. More ablation studies on different optimization objectives under zero-shot tasks and perplexity metrics are provided in Appendix
5.2.2 Optimizer Comparison
Figure 7b presents the comparison of the convergence curves between the Cayley optimization and our proposed QR-Orth optimization, both using Whip loss under identical settings. evident that QR-Orth demonstrates faster convergence and lower final loss, regardless of whether combined with SGD or Adam. As shown in Table 4 QR-Orth achieves 1.4x speedup over Cayley optimization for the same number of iterations. Due to its faster convergence, QR SGD achieves the same result as Cayley SGD after 100 steps just 6 steps, yielding an overall acceleration factor of 41 x. This significantly improves the efficiency of orthogonal optimization.
Table Comparison of Time Taken for 100 Iterations Across Different Orthogonal Optimization Schemes.
Method
Cayley
QR-Orth
Speed up
SGD
8.2h
5.7h
1.44x 1.42x
Adam
8.1h
5.7h

---

0,023
Quant
2140
Adam 36D
Vaniace Nurtosss 14hp
Activatin Quantaation LoSS
0.024
Q8-Orth Adam
QR Orth SGD
w/hp Loss
23000
0.093
2000
6.000
6.199
3030
6010
40
Epoch
60
80
3000
Epoch
Loss comparison.
(b) Cayley QR-Orth.
Figure 7: Comparison of activation quantization loss and convergence performance using different optimization methods.
5.3 Results on Different Datasets
To investigate the sensitivity of DartQuant different training datasets, we sample training data from three datasets: WIKITEXT2, PTB, and C4. These datasets are used to optimize RI and R2 separately, and the impact of the dataset on the performance of the quantized LLM compared. As shown in 55 the results of all three experiments are largely consistent. This demonstrates that DartQuant robust calibration datasets and does not negatively affect the generalization ability of the LLM. Further analysis on the impact of sample size on performance is provided in Appendix D
Table 5: Comparison of LLM Performance with Different Calibration Datasets Using DartQuant.
Model
Dataset
WikiText2
PTB
C4
Avg
Baseline
5.47
37.91
7.26
16.88
2 7b
WikiText2
5.92
42.63
7.99
18.85
PTB
5.91
42.78
8.01
18,90
C4
5.92
42.99
8.00
18.97
Baseline
4.88
50.94
6.73
20.85
13b
WikiText2
5.25
58.29
7.3
23.61
PTB
5.24
58,46
7.33
23.68
C4
5.28
58.18
7.31
23.59
6 Conclusions
In this paper, we introduce DartQuant, an innovative method for LLM quantization that efficiently handles activation outliers. By constraining the activation distribution, DartQuant simplifies rotation matrix calibration and avoids overfitting risks in end-to-end fine-tuning. The QR-Orth optimization method climinates the need for complex orthogonal optimization, further speeding up the process. DartQuant achieves state-of-the-ar results in 4-bit quantization while significantly reducing costs. Notably, it successfully quantizes a 70B model on single RTX 3090, advancing LLM deployment in eecource-constraingg environments.
7 Acknowledgments
This work was supported in part by the National Key R&D Program China (No. 0025ZD0122000 the Science and Technology Major Special Program of Jiangsu (No. Bo2244224))8) Beijing Natural Science Foundation (I.244046), the Jiangsu Key Research and Development Plan (No. BE2023016), and the CCF-Baidu Open Fund.
10

---

References
[1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[2] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 0tti:220050000666 2022.
[3] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arOiv:2103. 10360, 2021.
[4] Jacob Devlin Ming-Wai Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page Minneapolis, Minnesota, 2019.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-8788 1766. PMLR, 2021.
[6] Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
[7] Zhilin Yang. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv: 190.69996687777 2019.
[8] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems, 35:30318 15:30318-30333 8-30332, 2022.
[9] Yuanteng Chen, Yuantian Shao, Peisong Wang, and Jian Cheng. EAC-MoE: Expert-selection aware compressor for cexxuur--of-xxpeeee large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22942-12963 July 2025.
[10] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint ar.in:1103..v0.. 2015.
[11] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffiman coding, arXiv preprint arXiv:1510.00149 2015.
[12] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
[13] Wei Huang. Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. An empirical study of Ilama3 quantization: From Ilm milms. Visual Intelligence, 2(1):36, 2024.
[14] Peisong Wang, Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, and Jian Cheng. Two-step quantization for low-bit neural networks. In Proceedings of the IEEE Conference on computer vision and pattern recognition, pages 1337643844 2018
[15] Weihan Chen, Peisong Wang, and Jian Cheng. Towards mixed-precistoe quantization of neural networks via constrained optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5350-5359, 2021.
[16] Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In International Conference on Machine Learning, pages 9847-9856. PMLR, 2020.

---

[17] Peisong Wang, Weihan Chen, Xiangyu He, Qiang Chen, Qingshan Liu, and Jian Cheng. Optimization based post-training quantization with bit-split and stitching. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(2):2119-22 11-22119-2181 2022.
[18] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeticonnf inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018
[19] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantiza- tion for large language models. arXiv preprini arXiv:2308. /3137, 2023.
[20] Chen Tianqi, Yuanteng Chen, Weixiang Xu, Zeyu Zhu, Peisong Wang, and Jian Cheng. Q- mamba: Towards more efficient mamba models via post-training quantization, 2025.
[21] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, arXiv preprini arXiv:2304.09) 2023.
[22] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier- aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13355-13364, 2024.
[23] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[24] Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[25] Salch Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00455 2024.
[26] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruy Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant- quantization with learned rotations. arXiv preprint arXiv:2405. 16406, 2024.
[27] Xing Hu, Yuan Cheng, Dawei Yang, Zhixuan Chen, Zukang Xu, Jiangyong Yu, XUCHEN, Zhi- hang Yuan, Zhe jiang, and Sifan Zhou. OSTQuant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. In The Thirteenth International Conference on Learning Representations, 2025.
[28] Jun Li, Li Fuxin, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. arXiv preprint arXiv:2002.01 2020.
[29] Gary Bécigneul and Octavian-Fure Fugen Ganea. Riemannian adaptive optimization methods. arXiv preprint arXiv: 1810.00760, 2018.
[30] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llss. arXiv preprint arXiv:2406.0122, 2024.
[31] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
[32] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipi:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint 24X.:22402.44366 2024.
12

---

[33] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprini arXiv:2401. 15024, 2024.
[34] Janghwan Lee, Jiwoong Park, Jinseok Kim, Yongjik Kim, Jungju Oh, Jinwook Oh, and Jung- wook Choi. Amxfp4: Taming activation outliers with asymmetric microscaling floating-point for 4-bit II inference. arXiv preprini arXiv:241 1.09909, 2024.
[35] Yucui Liu and Tomasz J Kozubowski. A folded laplace distribution. Journal of Statistical Distributions and Applications, 2:1-17, 21111,2015. 2015.
[36] Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401. 04088, 2024.
[37] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-experts language models. arXiv preprint arXiv:2401.066 1.06666 2024.
[38] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint (tt1::6600078880 2016.
[39] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 21(140):1-61 2020.
[40] Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
[41] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
[42] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07880 2019.
[43] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
[44] Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9))99-106, 2021.
[45] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.022 880990278, 2018.
[46] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv: 100:.99440000 19040998 2019.
[47] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[48] Michael Boratko, Harshit Padigela, Divyendra Mikkilineni. Pritish Yuvraj, Rajarshi Das, An- drew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, et al. A systematic classification of knowledge, reasoning, and context within the are dataset. arXiv preprint arXiv: /806 00358, 2018.
[49] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17325 2022.
13

---

[50] Minghai Qin. The uniqueness of Ilama3-70b with per-channel quantization: An empirical study. arXiv e-prints, pages arXiv-2408, 2024.
[51] Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Quik: Towards end-to-end 4-bit inference generative large language models. arXiv preprint arXiv:2310.09259, 2023.
[52] Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving, Proceedings of Machine Learning and Systems, 6: 19-9,,, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims in the abstract and introduction accurately reflect the paper's contributions and scope. They align with both the theoretical and experimental results, and the motivations and assumptions are clearly stated.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are discussed in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: The paper includes comprehensive hypotheses, derivations, and experimental validations, with additional detailed discussions provided in Appendices B G and J
Appendix provides a detailed analysis of the computational complexity of QR-Orth and Cayley-based orthogonal optimization.
Appendix G presents activation distributions across different models along with statisti- cal metrics such as kurtosis variance and mean, supporting the validity of Laplacian modeling.
Appendix D proves the norm-invariant property of the rotation transformation and thoroughly explains the mechanism of the Whip function.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The code required to reproduce the experiments is provided in the supplemen- tary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
16

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code will be packaged into Zip file and uploaded the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/gust watcc//guiis//eeeee Sttee//Suuinsiioooooooo.oooooeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The experimental setup is described in Section 5, and Appendix K contains the detailed hyperparameter settings for reproducing the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
17

---

Answer: [Yes]
Justification: All experimental results in the paper were obtained using open- pourcs bench- mark frameworks and datasets. The detailed experimental setup is provided in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard crror of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experimental section of the paper provides details on the types of servers used for the experiments, as well as the memory and time requirements for the computations.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wrrrpp.ccoaaaaaacccccaaa
Answer: [Yes]
Justification: We have carefully reviewed the NeurlPS Code of Ethics and ensured that our research complies with its guidelines all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impact of this work is discussed in Appendix M
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimesss considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
19

---

Justification:
All existing assets used in this work are explicitly cited in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssii thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code required to reproduce the results in this paper will be included the supplementary materials, along with detailed README guide the experiments.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
The LLM is used only for writing, editing and formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:://aaaurpsse.cc///ofeeeeeep for what should or should not be described.
21

---

A Computational Invariance in Transformers
Attentac
TEN
W2
MISA
Myw
RM5Korm
RMSNam
Figure 8: Flowchart of the transformer block used in most language models, including pre- pe-RRMNoom, multi-head self-attention (MHSA), and the gated feedforward network (FFN). The solid arrows represent the data flow during training, pre-filling, and inference for each token. In RMSNorm, the input signal is normalized by norm and rescaled by the parameter a. In MHSA, the RoPE module computes the relative position embeddings, and the dashed arrows indicate access to the KV-Cache during generation. In the FFN, the activation function a is applied to the gated signal, and the two signals are combined element-wisc.
The weights and activations between blocks in the transformer can be transformed using orthogonal matrices without altering the model's output. Specifically, if a rotation transformation R₁ is applied to the input activations, the effect can be offset by left-multiplying the weight matrices on the left side of the transformer block (such as Wa. k, W., Wup gaate in Figure 8 by the orthogonal matrix R.T To ensure the correctness of the residual computation, R2 is right-multiplied with the output matrices (such W. and W Woown in Figure
8 Although RMSNorm is applied between two blocks, this transformation remains valid as long as there is no rescaling within RMSNorm (in practice, any rescaling is typically absorbed into the adjacent weight matrices). Theoretically. this is because RMSNorm only normalizes the activations, and applying a rotation transformation does not alter the norm of the activations. As result, the commutative property RMSN or Rt) RM N orm (X Ri can be established 33
Atterbion
PPN
RMSSNom
RMSNNm
AToow,
ROW
M Jo
To
R100 W.
Figure 9: Transformer applied in DartQuant. The RMSNorm scaling factor (a) has been absorbed into the weight matrices. The black section represents the flow in FP16 format, while the gray section indicates the flow INT4 format, and the dashed line shows the flow in and out the KV buffer. The hidden state X has been rotated by R1, which is offset by R.T and absorbed into the weight matrices Wg. WR, W., Wep: gate- Rí is also incorporated into W and W Wonen to ensure correct residual calculation. R2 in W., cancels with RT in W.. R3 and R4 are random Hadamard matrices computed online: Rg cancels out during the attention computation, and R2 cancels with R3 absorbed in W Woown. All weights are stored in INT4 format, and all activations prior to the weights are also quantized to INT4. The result of the matrix multiplication between INT4 weights and INT4 activations on TensorCore is INT32, which is immediately converted (and scaled) to FP16
22

---

In addition to R1 the Transformer block can also incorporate Rg, Rs, and R4 as shown in Figure 2 R2 operates within the multi-head attention mechanism by being applied to each attention head. absorbed into W, to rotate the input of W.. To counterbalance the rotation effect introduced by W.. the transpose of R2, denoted R2 left-multiplied to W.-
R3 is absorbed into the KV cache to alleviate the quantization loss within the KV cache. Due to the presence of RoPE, directly incorporating R3 into the weight matrices is challenging, so R& is designed as an online Hadamard transform.
Finally, R4 is rotation matrix used to smooth the input to the down Nown-projection. Given the gating mechanism, its rotational component cannot be fused into Wup or W Wgate, thus it also designed as an online Hadamard transform. However, the inverse transformation can be combined with W Wdown avoiding the computational overhead of the inverse operation.
It is important to emphasize that the inference framework of DartQuant (Figure is identical to the framework proposed by SpinQuant [26), resulting in the same level of inference acceleration.
B
Comparison of Calculation Amount
B.1 QR Decomposition
The Householder QR decomposition for matrix A can be expressed follows:
Algorithm 2 Householder QR Decomposition
1: Input: Matrix E R
2: Output: Orthogonal matrix Q E Rnx", Upper triangular matrix R R2xx
3: Initialize = A
4: for k =
do
5: Set the vector R
6: Compute the Householder vector
sign(z1)llz[lee,
7: Normalize U: U Tell
8: Update R: R = Rem.kk 20u Rkmn,k
9: Update the orthogonal matrix Q: Q Qumm.kn = Qum 200 RouT"Qkm,kkr Qkmmkkn
10: end for
11: Rn, is the upper triangular matrix, Q is the orthogonal matrix.
Let us analyze the computational complexity of each step in the above pseudocode:
Step 6&7:
Computing the Householder vector ut:
The operation involves computing the norm ||>]||, which takes (n k 1). Computing involves adding two vectors and normalizing, which is also (n k + 1).
Step 8: Updating R:
The update of R requires multiplying vector by matrix slice Rian which takes
(n k + 1)².
Step 9: Updating Q:
Updating Q involves similar computation to updating R, so it also takes (n- k 1)2.
We now aggregate the computational complexity of all steps across the iterations. The overall computational complexity is given by:
Total complexity
2[(n - k 1)2 + (n - k + 1)]
(5)
k=1
2n(n + 1)(2n 1)
+ +nnn + 1)
6
4
3
3
23

---

B.2 Cayley SGD
A provides the detailed computational process the Cayley SGD. Compared to standard
SGD, it introduces additional computations in steps 5 through 12. These computations are primarily
composed of matrix multiplications.
Algorithm Cayley SGD with Momentum
1: learning rate I, momentum coefficient 3, e 10 1 =0.5, s =
2: Initialize X₁ as an orthonormal matrix; and M1
3: for k 0 to T do
4: M&+1 -- 3Mx g(Xx)
5: Wi - M (X, Mx+1X,)
6: WE < Wx- W
7:
MR+1 f W
8:
a minfl, (iinfl,2//100 ([lWl + e)}
9:
Initialize y t X + aMa+1
10:
for to do
11: y' X + (Xx + yy-i))
12: end for
13: Update X4+1 t Y
end for
Step 5: Computing the auxiliary matrix Wi
The first part involves computing M&+1 XT, which requires a matrix multiplication with complexity n².
The second part involves the term (X M ). To optimize this computation, the matrix multiplication X Mx requires n²,, and multiplying this with Xx results in a total complexity of 2n due to the nested operations.
Step 7:
Momentum projection update
The update Mc+1 W. is matrix multiplication, which takes n².
Step 9 to 11: Iterative Cayley transform
Each iteration involves matrix addition and matrix multiplication, which has plexity n
Since there iterations, the total complexity is 2n
In conclusion, Cayley SGD incurs approximately an additional 6n in computational complexity compared to standard SGD.
C Complete Results of Main Result Table
In Tables tables to 15
we present the full results of Table We report perplexity (PPL) scores on the
WikiText2 [38], C4
39 and PTB 40). Additionally, we assess model performance on nine zero-shot
evaluation tasks: LAMBADA 41, HellaSwag [42], PIQA 43), WinoGrande [44), OpenBookQA
45). SIQA 46 MMLU 47 ARC-E, and ARC-C 48 We compare our approach with several other
methods, including SmoothQuant [23], GPTQ 44), OmniQuant [19], and current ofate-of-the-art
methods such Quarot (25 and SpinQuant (26 for weight and activation quantization.
24

---

Table 6: Comprehensive comparison of the average accuracy of LLaMA-2 7B on nine Zero-Shot Commonsense Reasoning tasks.
(W-A-XV)
Bis
Method
WG
SIQA
PIQA
OBQA
LAMB
[IS
ARC-E
ARC-C
MMLU
Arg
16-16-16
Full Procision
09.06
46.16
79.05
44.2
73.9
76.02
74.54
46.33
41.21
61.15
SmaethhQamt
50.15
34.14
55.22
26.40
0.82
32.09
31.61
24.91
22.80
30,97
GPTQ
69.53
45.55
78.75
42.80
72.29
74,80
73:40
44.11
39.21
60.03
4-8-16
OmniQQuant
SpinQuant
67.96
44.83
78.78
45.60
72.95
74.80
73.61
444 50.00
39, 55
60.10
DarQuuant
68.75
45.55
78.75
42.60
73.03
75.15
73.65
64.02
38.81
0U.17
SmouthQuami
49:25
35.36
49.89
0.00
25.60
27.23
28.62
24.61
29
GPTQ
50.28
35.47
$3.05
27.40
3,50
29.17
32.03
25.76
23.22
31.05
4-4-16
QmniQQunt
51.85
$7.67
63.55
32,60
28.22
49,43
46.04
27.47
24.81
40.18
SpinQuant
43.71
72.17
73.20
34.70
OSTQuant
66.69
45.60
76.71
43.60
70.62
73.64
69.36
40.96
34.32
$7.94
DariQuam
67.17
44.93
76,93
19.00
71.65
73.26
70,96
42.41
35.66
58.05
49:38
GPTO 50.91
33.98
52.39
24.60
1.92
28.21
30.51
25.63
23.61
29.97
4-4-4
Qualkor
66.27
43.65
77.04
40.60
70.25
72.81
68.43
41.64
33.56
57.05
SpinQuant
$7.55
OSTQuant
77,57
71.24
57.88
DuriQuant
06.69
44.08
77:48
38.60
70.95
73.81
69.87
44.20
35.32
37.96
Table 7: Comprehensive comparison of Perplexity scores for LLaMA-2 7B across three datasets.
Bits
Method
Wiki
PTB
C4
Avg
(W-A-KV)
16-16-16
Full Precision
5.47
37.91
7.26
16.88
RTN
5.91
40.64
7.89
18.15
SmoothQuant
250.33
500.08
246.09
332.17
GPTQ
7.69
20917.14
8.04
6977.62
4-8-16
OmniQuant
5.7.
126661
7.7.5
426.53
QuaRot
5.63
42.00
18.41
SpinQuant
5.61
40.41
7.53
17.85
DartQuant
5.60
39,95
7.52
17.69
RTN
487.93
758.10
759.48
668.50
SmoothQuant
2999.11
2426.98
4410.75
3278,95
GPTQ
995.88
2497.44
1094.07
1529.13
4-4-16
OmniQuant
17.06
566.12
25.68
202.95
QuaRot
6.02
47.64
8.24
20.63
SpinQuant
5.89
45.66
8.14
19,90
OSTQuant
5.89
43.76
8.06
19.24
DartQuant
5.88
41.72
7.99
18.53
RTN
731.05
819.7
1010.3
853,68
GPTQ
1905.56
2254.27
1280.19
1813.34
4-4-4
QuaRot
66.41
27.01
SpinQuant
5.99
61.03
8.34
25.12
OSTQuant
5.94
45.16
8.13
19.74
DartQuant
5.93
43.41
8.08
19.14
25

---

Table Comprehensive comparison of the average accuracy of LLaMA-2 13B on nine Zero-Shot Commonsense Reasoning tasks.
(W-A-XV)
Bis
Method
WG
SIQA
PIQA
OBQA
LAMB
[IS
ARC-E
ARC-C
MMLU
Arg
16-16-16
Full Procision
72.14
47.39
80.52
45,20
76.73
79.38
77.44
49.15
50.53
64,28
71.67
SmaethhQamt
$1.38
34.54
53.65
26.20
0.35
27.21
28.20
24.57
22.95
29.89
GPTQ
72.30
46.57
80.09
45.60
76.25
78.13
76.73
48.29
49.32
63.70
4-8-16
OmniQQuant
46.11
47.56
48.
62,95
SpinQuant
71.11
46.21
79.76
44.80
76.85
78.30
76.56
49.98
49.2
63.55
DarQuuant
72.06
40.88
79.95
44.40
76.81
78.25
76.85
48.98
49.25
63.77
SmouthQuami
49:41
34.08
24.20
0.00
25.77
27.39
23.
29 05
GPTQ
$1.30
34,60
$2.56
23,60
1.26
27.29
30.68
23,
23.97
20 85
4-4-16
QmniQQunt
53.67
90:51
67.68
33.00
30.59
54.90
53.58
29.60
24.42
42.98
SpinQuant
74.36
77.50
47.
OSTQuant
69.46
46.37
78,89
44.00
74,62
77.13
75.21
47.95
47.79
62,38
DariQuam
71.11
46.
79.27
44.20
75.18
78.04
75.38
47.6
46 80
62.64
20.12
GPTO
50.04
34.19
50.53
25.00
0.55
27.87
29.34
24.06
23.75
29.55
4-4-4
Qualkor
69.30
44.27
78.24
42,80
65.44
75.75
72.64
45.00
44.52
59.87
SpinQuant
OSTQuaat
75.20
47.19
Duriuuat
7L.11
45:29
79.27
43.00
74.93
77.57
76.01
47.53
46.59
02:46
Table 9: Comprehensive comparison of Perplexity scores for LLaMA-2 13B across three datasets.
Bits
Method
Wiki
PTB
C4
Avg
(W-A-KV)
16-16-16
Full Precision
4.88
50.94
6.73
20.85
RTN
5.23
52.95
7.12
21.77
SmoothQuant
1884.63
920.66
1726.69
1510.66
GPTQ
5.00
50.39
6.90
20.76
4-8-16
OmniQuant
5.07
50.13
7.02
20.74
QuaRot
5.02
54.08
6.96
22.02
SpinQuant
4.99
51.56
6.91
21.15
DartQuant
4.98
50.89
6.91
20.93
RTN
1595.89
3016.34
2959.29
2523.84
SmoothQuant
4811.32
4299.06
3989.04
4366.47
GPTQ 914.38
1856.09
1893.68
1554.72
4-4-16
OmniQuant
15.39
282.92
22.73
107.01
QuaRot
5.35
59,52
7.46
24.11
SpinQuant
5.19
56.07
7.37
22.88
OSTQuant
5.21
54.46
7.33
22.33
DartQuant
5.22
54.82
7.28
22.44
RTN
1793.38
2845.81
2966.21
2535.13
GPTQ
1367.96
2758.47
1663.36
1929,93
4-4-4
QuaRot
5.51
61.78
7.65
24.98
SpinQuant
5.30
57.3
7.51
23.37
OSTQuant
5.25
55.87
7.37
22.83
DartQuant
5.26
55.32
7.34
22.64
26

---

Table 10: Comprehensive comparison the average accuracy of LLaMA-2 70B on nine Zero-Shot Commonsense Reasoning tasks.
(W-A-XV)
Bis
Method
WG
SIQA
PIQA
OBQA
LAMB
[IS
ARC-E
ARC-C
MMLU
Arg
16-16-16
Full Procision
72.98
40.13
82.75
46.8
79.58
83.8
81,02
57.51
65.2
69.53
11.39
SmaethhQamt
52.35
34.34
69.50
23.60
9,70
41.60
50.34
28.41
24.10
28.35
GPTQ
77.24
47.70
82.75
19.00
79.57
82.83
81.27
56.74
63.68
69.03
4-8-16
OmniQQuant
38.66
67.18
73.27
SpinQuant
78.22
48.93
82.48
48.60
80.25
83.90
81.52
57.76
64.44
69.57
DarQuuant
78.14
48.82
82.81
47,80
$0.13
83.4
80.6
57.51
64,49
69.30
SmouthQuami
48.93
33.83
0.52
27.21
27.57
24.66
24.83
29 40
GPTQ
49.90
34,80
50.82
25.00
0.12
25.72
26.85
25.76
24,
20.23
4-4-16
QmniQQant
$2.88
$7.62
63.71
33.20
25.00
$1.89
50.76
50.80
23.76
41.08
SpinQuant
79.53
62.
68.
OSTQuant
76.24
47.95
82.52
47.20
79.20
82.52
80.09
56.31
62,78
68.29
DariQuam
77.58
48.52
82.70
48.20
79.99
82.62
81.93
57.00
62.60
69
23:02
29.15
GPTQ 50.59
34.03
49.46
25.60
0.16
25.88
26.39
26.25
23.96
29.15
4-4-4
Qualkor
76.48
46.26
81.28
46.00
79.00
81.82
79.46
55.63
60.78
67.41
SpinQuant
OSTQuant
79.58
DuriQuant
76.40
48.93
82.81
47.00
79.76
82.55
78.38
53.38
62.55
08.22
Table 11: Comprehensive comparison of Perplexity scores for LLaMA-2 70B across three datasets.
Bits
Method
Wiki
PTB
C4
Avg
(W-A-KV)
16-16-16
Full Precision
3.32
24.25
5.71
11.09
RTN
3.75
27.72
6.11
12.53
SmoothQuam
95.26
318.96
128.66
180.96
GPTQ
3.50
26.33
5.87
11.90
4-8-16
OmniQuant
4.01
31.78
6.40
14.06
QuaRot
3.42
24.23
11.15
SpinQuant
3.40
24.68
5.79
11.29
DartQuant
3.41
24.34
5.78
11.18
RTN
65680.98
60069.34
64182.98
6331T.10
Smooth:uant
1868.60
1906.48
1134.50
1636.53
GPTQ
69089,76
68684.35
79127.77
72300.63
4-4-16
OmniQuant
27.29
265,00
35.53
109.27
QuaRot
3.78
24.16
6.10
11.35
SpinQuant
3,67
25.38
6.04
11.70
OSTQuant
3,65
26.28
6.01
11.98
DurtQuant
3.64
24.90
5.99
11.51
RTN
66102.31
60357.61
64857.35
63772.42
GPTQ
73603.62
75839.09
85644.05
78362.25
4-4-4
QuaRat
3.81
24.53
11.49
SpinQuant
3.69
25.52
6.07
11.76
OSTQuant 3.67
25.3
6.03
11.67
DartQuant
3.67
24.98
6.01
11.55
27

---

Table 12: Comprehensive comparison of the average accuracy of LLaMA-3 8B on nine Zero-Shot Commonsense Reasoning tasks.
(W-A-XV)
Bis
Method
WG
SIQA
PIQA
OBQA
LAMB
[0S
ARC-E
ARC-C
MMLU
Arg
16-16-16
Full Procision
73.32
47.19
80.95
45.00
75.63
79.14
77.82
53.24
62.06
66.04
SmaethhQamt
$1,70
34.39
55.71
24.20
8.97
33.30
33.84
22.10
23.27
31.94
GPTQ
72.45
46.57
79.56
46.10
73:47
77.60
76.43
51.31
59.70
64.79
4-8-16
OmniQQuant
70.64
68.53
38.
62.72
SpinQuant
74.19
45.91
79.92
44.80
75.74
74.81
77.30
51 538
60.74
65.01
DarQuuant
73.24
40.26
80.41
44.80
76.09
78.17
78.24
52.05
60.57
63.58
SmonthQuami
49:41
34.60
50.87
27.20
0.06
26.6
28
23.77
29.55
GPTQ
49.41
34.54
55.53
78.80
13.55
36.14
35.44
25.76
22.93
33.47
4-4-16
QmniQQunt
49:57
33.47
53.92
25.411
4,04
36.64
30.00
24.15
22.97
31,25
SpinQuant
44.73
72.77
47.61
55.
OSTQuant
68.43
44.78
77.91
41.40
72.68
75.49
75.63
48.55
$4.74
62.18
DariQuam
70.96
45.34
79.16
43,40
72.39
75.81
74,45
48.21
55/46
62,80
23.29
GPTO 49.17
34.70
54.02
27.00
7.18
31.48
33.92
25.21
23.14
31.60
4-4-4
Qualkor
06.77
44.28
73.61
41.00
68.91
71.69
66.33
40.02
46.77
57.32
SpinQuant
OSTQuant
77.58
71.95
DuriQuant
70.68
44.68
78.18
43.00
72.58
75.28
15.32
48.03
54.31
02.38
Table 13: Comprehensive comparison of Perplexity scores for LLaMA-3 8B across three datasets.
Bits
Method
Wiki
PTB
C4
Avg
(W-A-KV)
16-16-16
Full Precision
6.14
11.18
9.45
8.92
RTN
7.20
12.65
11.19
10.35
SmoothQuant
84.37
139.98
113.02
112.46
GPTQ
7.22
12.88
10.71
10.27
4-8-16
OmniQuant
7.17
12.68
11.6
10.48
QuaRot
6.55
11.74
10.48
9.59
SpinQuant
6.49
11.63
10.31
9.48
DartQuant
6.50
11.68
10.29
9.49
RTN
179.40
244.36
177.91
200.56
SmoothQuant
2623.41
2213.76
1811.66
2216.28
GPTQ
137.12
258.71
415.63
270.49
4-4-16
OmniQuant
124.75
248.78
184.54
186.02
QuaRot
8.05
13,99
13.19
11.74
SpinQuant
7.22
12.88
11.90
10.67
OSTQuant
7.26
12.81
11.9
10.66
DartQuant
7.32
12.51
11.81
10.58
RTN
354.37
372.86
333.09
353.44
GPTQ
282.29
308.06
900.45
496.93
4-4-4
QuaRot
8.40
14.68
13.79
12.29
SpinQuant
7.41
13.33
12.23
10,99
OSTQuant
7.26
12.81
1.90
10.66
DartQuant
7.43
12.89
12.02
10.78
28

---

Table 14: Comprehensive comparison the average accuracy of LLaMA-3 70B on nine Zero-Shot Commonsense Reasoning tasks.
(W-A-XV)
Bis
Method
WG
SIQA
PIQA
OBQA
LAMB
[0S
ARC-E
ARC-C
MMLU
Arg
16-16-16
Full Procision
s0.82
50.61
84.44
48.40
79.31
84.99
85.95
64.25
75.47
72,70
SmaethhQamt
49.25
34.95
61.92
27.20
2,02
39.39
40.49
27.39
22.95
33.95
GPTQ
79.48
48.76
82.37
16.60
75.30
83.18
80.30
57.08
72.25
69.44
4-8-16
OmniQQuant
79:48
72.92
SpinQuant
s0.27
49.90
84.22
49.20
78.58
84,42
83.54
61.77
73.97
71.76
DarQuuant
80.35
49.80
83.35
48.40
78.79
84.89
85.05
61.18
73.96
71,82
SmouthQuami
51.62
33.42
52.12
27.40
0.00
25.66
26.01
24.15
23.28
29. 39.30
GPTQ
48.07
34.70
72
25.40
3.53
28.77
39.81
26.96
24.77
32.55
4-4-16
QmniQQunt
49.57
$2.91
49.51
27.60
00.00
25.04
25.08
22.70
22,95
28.37
SpinQuant
73.56
74.13
79.42
06
OSTQuant
73.64
46.32
81.56
45.40
75.61
83.19
80.13
57.17
69.29
67.94
DariQuam
77.27
47.54
83.08
48.00
76.44
83.61
81.57
58.00
68.96
69.39
24.01
GPT) 50.36
34.19
58.60
30.00
3.45
28.98
39.27
27.82
24.11
32,98
4-4-4
Qualkor
70.24
43.96
79.00
39.60
69.90
75.56
71.04
48.12
36.04
61.50
SpinQuant
43.81
64.76
OSTQuant
75.78
DuriQuant
77.19
47.13
81.88
47.40
70.36
83.53
80.81
58.70
08:44
09:05
Table 15: Comprehensive comparison of Perplexity scores for LLaMA-3 70B across three datasets.
Bits
Method
Wiki
PTB
C4
Avg
(W-A-KV)
16-16-16
Full Precision
2.86
8.53
7.17
6.19
5.29
10.81
21.05
12.38
SmoothQuant
429.54
691.87
512.64
544.68
GPTQ
3.56
8.99
8.11
6.89
4-8-16
OmniQuant
8.09
21.83
14.93
14.95
QuaRot
3.78
9.05
7.94
SpinQuant
3.46
8.76
7.66
6.63
DarQQuant
3,45
8,84
7.69
6.66
RTN
28070.21
4096.69
20005.66
17390.85
SmoothQuant
5328.88
7907.32
$491.67
6242.62
GPTQ 14637.05
3119.77
24848.08
14201.60 63
4-4-16
OmniQuant
363.42
462.06
317.33
380.94
QuaRot
6.51
12.97
12.70
10.73
SpinQuant
5.92
11.66
11.25
OSTQuant
9.58
8.93
DartQuant
4.83
9.80
9.35
7.99
RTN
31807.71
3742.77
17859.72
17803.40
GPTQ
15320.82
2741.99
34022.33
17361.71
4-4-4
QuaRot
6.79
14.01
13.33
11.38
SpinQuant
6.15
12.1
12.26
10.17
OSTQuant
4.55
9.71
9,03
7.76
DariQuant
4.92
9.92
9.55
8.13
D Sample Size Analysis
Table 16 investigates the impact of sample size DartQuant's performance. All experiments are performed with token sampling ratio 10%. The results show that DartQuant's calibration performance remains robust even with small datasct.
29

---

Table 16: Comparison of DartQuant Calibration Results with Different Sample Sizes.
Model
Sample
WikiText2
PTB
C4
Avg
32
5.91
42.61
8.03
18.85
64
5.88
43.33
8.00
19.07
2 7b
128
5.92
42.63
7.99
18.85
256
5.92
42.41
8.04
18.79
32
7.30
12.66
11.79
10.58
64
7.30
12.77
11.79
10.62
3 88
128
7.29
12.71
11.85
10.62
256
7.41
12.83
11.99
10.74
E Comparison with Mixed Precision Quantization Methods
For more comprehensive comparison, we selected two mixed precision quantization algorithms, QUIK [5T] and Atom [52], and compared them with DartQuant under the 4-4-16 bit setting, It is important to note that, for fair comparison, we preserved QUIK's feature of protecting the first 256 outlier channels, while quantizing the down projection to 4 bits to ensure consistency with our method, Atom was tested using its default settings. The specific experimental results are shown in Tables 17 and
Table 17: Comparison of accuracy with mixed precision quantization methods on zero-shot tasks.
Model
Method
WG
SIGA
PIQA
OBQA
LAMI
H15
ARC-E
ARL
MMLU
AVG
QUIK
27.76
Atom
76.04
69.76
70.17
41.25
DunQuant
67.17
44.93
76.93
39.00
71.65
73.76
70.96
42.41
35.66
$8.05
QUIK
74.86
41.40
62.27
72.88
Atom
71.94
73.94
DanQuant
71.11
46.16
79.27
44.20
75.18
78.04
75.38
47.61
46,80
62.64
QUTK
68.67
44.22
77.58
42.30
64.12
74.32
68.98
46,42
46.33
59 79
Atom
DunQQuatt
77.58
48.52
$2.70
48.20
79.99
82.62
$1.93
57,00
62.60
69.02
QUIK
59.50
30.00
65.78
35.00
42.81
58.45
52.53
34.98
29.01
46.35
DanQuaat
79.16
72.39
75.81
QLIK
50.83
20.54
71.33
34.30
35.07
70.11
62.16
38.82
31.87
51.33
3-706
Atum
74.16
48.95
78.61
45.54
72.17
76.88
74.25
51.78
61.62
61.11
Table 18: Comparison of perplexity with mixed precision quantization methods the WikiText2, PTB, and C4 datasets.
Model
Method
WIKI
PTB
C4
AVG
QUIK
8.05
51.97
10.12
23.38
2-7b
Atom
6.03
46.77
8.25
20.35
DartQuant
5.88
41.72
7.99
18.53
QUIK
7.29
65.72
9.23
27.41
2-13b
Atom
5.26
52.95
7.33
21.85
DurtQuant
5.22
54.82
7.28
22.44
QUIK
6.36
35.28
8.77
16,80
2-70b
Atom
3,68
28.21
6.05
12.65
DanQuant
3.64
24.90
5.99
11,51
QUIK
18.01
37.42
14.72
23.38
3-8b
Atom
7.57
16.67
13.28
12.50
DarQQuant
7.32
12.51
11.81
10.58
QUIK
10.32
21.48
17.24
16.35
3-70b
Atom
5.23
13.00
11.48
9.90
DartQuant
4.83
9.80
9.35
7.99
30

---

Even though DartQuant strictly quantizes all activations and weights to 4 bits (resulting in lower average bit-width compared to QUIK and Atom), still achieves accuracy improvements on most datasets. This underscores the effectiveness of our method.
SpinQunnt
Average Quant Loss
SptnQuret
Outller Count
X:30
1,00
1200
Lian 2-18
Lsnn 2-330
LSanaa 2-108
Liana 2:6
Liana 3-108
Linra 2-75
Lioee 2:300
Licne 2-105
Liera 2-5
Models
Models
(a) Layer 0
(b) Layer
3003
Outlier Count
27500
SpinQuant DartQuant
Average Quant Loss
2.00
SptnQuret
Ganas 2.70
Lans 2:100
Varta 27100
Usoa 3.00
Uimn
Lana 2.79
Usme 2:300
Usit 2/709
Liana 3.50
Usea 122
Models
Models
(c) Layer 10
(d) Layer 10
Madamare
Hadamard
2009
Spinquant COTCUaaE
Average Quant Loss
SptnOuant
Outller Count
lQuantt
CSTOuant DartQluant
012
Lavaa 2.70
Usos 2:30
Uarra 20100
Ulena 3.50
Lana 2.70
Lame 2:30
Usni 2010
Limna 2.50
Unoa 123
Models
Models
(e) Layer 30
( Layer 30
Madamare
Spinquant
Avcrage Quant Loss
Sptrouaet CSTOuont DartQuannt
Outller Count
COTCUarE
(Quant
Li006 2-18
Ldass 5-10
Uiata 2-108
Viana 2.d
Ltons 3-106
Lams 5-78
Uann 2.399
Lavis 5.70
Llato 3-58
Utoss 9.106
Models
(g) Last Layer
(h) Last layer
Figure 10: Impact of different transformations on 1000 activations across layers in different models.
31

---

(a) 2-7b layer 0
(b) 2-7b layer 9
(c) 2-7b layer 19
(d) 2-7b layer 29
(e) 2-7b layer 31
(f) 2-13b layer 0
2-13b layer
(h) 2-13b layer 19
(i) 2-13b layer 29
() 2-13b layer 39
-5
140
(k) 2-70b layer
(1) 2-70b layer 19
(m) 2-70b layer 39
(n) 2-70b layer 69
(o) 2-70b layer 79
(p) 3-8b layer 0
(q) 3-8b layer 9
(r) 3-8b layer 19
(s) 3-8b layer 29
(1) 3-8b layer 31
(u) 3-70b layer
(v) 3-70b layer
(w) 3-70b layer
(x) 3-70b layer
(y) 3-70b layer
Figure 11: Activation distribution histograms for different layers of various models. The x-axis represents activation values, while the y-axis denotes the channel count.
32

---

Effects of Different Transformations on Activation Outliers and Quantization Errors
We extracted 1000 activation samples from different layers of Llama-2 (7B/13B/70B) and Llama-3 (8B/70B) models and analyzed the number of outliers and the average quantization error after applying different transformation methods. The statistical results are shown Figure TO As observed, the activations after DartQuant transformation exhibit significant reduction in outhiers, with marked decrease in quantization loss, further validating the effectiveness DartQuant large-scale language model quantization.
G
Activation Distribution
We conducted an in-depth investigation into the distribution characteristics of activations in LLMs. First, we randomly sampled 1,000 activation samples from each model and computed their mean, variance and kurtosis. The detailed statistical results are presented Table 19 As observed, the mean of activations is close to zero, the variance is approximately and the activations exhibit high kurtosis (whereas the kurtosis of a Gaussian distribution is O). These statistics indicate that most activation values concentrated around zero and exhibit significant heavy-tailed properties.
Figure presents the activation distribution histograms across different layers of various models. As shown, apart from few outliers, the overall activation distribution is symmetric around zero. These distribution characteristics closely align with those Laplacian distribution. Therefore, we model the activation distribution simplified Laplacian distribution.
Table 19: Statistics of each model activation.
Model
Kurtosis
Mean
Variance
Llama 2-7b
87.69
1.18e-02
9.97e-01
Llama 2-13b
58.99
3.17e-03
9.98e-01
Llama 2-70b
245.10
-4.88e-03
9.97e-01
Llama 3-8b
44.32
-2.92e-05
9.91e-01
Llama 3-70b
37.35
4.64e-03
9.80e-01
H
Performance on MoE
Expert
Ww
RM5Nomm
Weights by Rouler
Expect
Figure 12: MoE applied in DartQuant. The fusion method of the MHSA block is the same as that of dense models 9
33

---

Table 20: Zero-shot and perplexity evaluation results of Mixtral-7x8B using DartQuant.
Biis
SIQA
PIQA
ORQA
ARC-C
MMLII
AVG
Baseline
FP16
76.24
29.30
83.57
47.60
78.05
$4.03
83.07
59.30
67.29
09:15
3.84
25.60
24.66
Quarct
GPTQ
4-4-66
69.46
45.45
34.70
79.60
52.88
44.00
27.40
74.27
1.89
29.81
79.01
76.47
13.63
25.34
53.41
23.30
59.42
64.57
31.03
219.61
4.97
DurQQuant
71.90
46.09
45,30
75,02
77.31
$4,10
59.93
68.58
4.75
GPTQ
4-4-4
49.17
33.31
54.03
26.40
1.69
29.26
32.79
24.05
23.29
30.59
239,90 239.59
Quarce
67.32
45. 45
80.14
42.40
71.98
78.74
75.76
$2.05
58.44
63.59
5.03
DanQQuant
69.85
Table 21: Zero-shot and perplexity evaluation results of DeepScek- SersSeckk--o--leee 16b-base using DartQuant.
Method Bits
SIQA
PIQA
OBQA
MMLU
AVG
Biseline
IP16
69.53
45.56
30.52
43.20
72.59
77.43
13.19
47.53
35,115
00.99
6,51
52.17
35.93
31.60
33.70
24.13
42.211
251,42
4-4-16
23.95
Quarce
66.64
45.55
78.40
42.60
70.81
74.68
70.34
44.11
34.05
58.58
7.02
DanQQuatt
67.72
45.70
78.89
43.00
72.44
75.97
71,13
45.25
35.04
59.46
6,88
Quarct
GPTQ
4-4-
66.32
52.25
44.83
35.62
78.16
65:40
42.32
30.20
70.53
29.23
74.68
54.25
70.10
43.94
30.72
24.41
33.54
41.00
58.27
127.74
7.07
DunQuuan
67.24
45.30
78.07
42.80
71,46
75.08
44.34
35,06
59.05
To demonstrate the adaptability of DartQuant to different model architectures, we also applied it to two MoE models (Mixtral-7xtee and DeepSeek-MoE-16B3 The rotation matrix is integrated with the MoE architecture as shown in Figure 12] As shown in Table 20 and 21 DartQuant continues to demonstrate outstanding performance on the MoE architecture, with improved accuracy compared to QuaRot, which uses random rotations.
I Additional Ablation Analysis of Whip Loss
To further demonstrate the effectiveness of the Whip loss, evaluated PPL and zero-shot accuracy under different loss functions. As shown in Table 22] the results highlight the advantages of Whip loss in both PPL and zero-shot accuracy.
Table 22: Comparison of zero-shot accuracy and perplexity across different loss functions.
model
loss
WG
ARC-E
ARC-C
MMLU
WIKI
PTB
C4
Quant
66.30
68.90
41.72
33.73
6.03
45.70
8.17
Variance
66.85
68.52
41.30
34.70
6.03
44.96
8.20
Llama 2-7b
Kurtosis
66.22
69.87
42.15
35.64
5,99
46.57
8.14
Whip
67.17
70.96
42.41
35.66
5.90
42.94
8.01
Quant
68.19
71.46
46.33
54.08
7.64
13.17
12.32
Llama 3-8b
Variance
69.61
72.52
46.93
53.30
7.66
12.92
12.32
Kurtosis
69.93
71.72
45.99
54.16
7.64
13.09
12.37
Whip
70.96
74.45
48.21
55.46
7.29
12.71
11.85
J The Mechanism of Whip Loss
As shown in Figure the smoothing of matrix peaks is not solely due to the amplification of small values near zero; it also involves broader redistribution of the activation values.
Specifically, the Whip Loss designed pushes small activation values away from zero, making them more evenly distributed, which in turn helps reduce quantization error. Under the norm- norm-invane cermiivariiace constraint imposed by the rotation transformation, activation values effectively redistributed: high-peak regions in the distribution histogram are compressed, while low-value regions are filled, leading to overall "peak smoothing" effect.
Thus, the smoothing of matrix peaks is not merely a result of amplifying small values but also stems from the uniformization of the overall distribution of the key objectives of DartQuant.
34

---

The property of norm invariance originates from the mathematical characteristics of rotation matrices. Let W be rotation matrix (i.e., an orthogonal matrix) satisfying WTW For any input vector æ, the transformed norm follows:
[
(W") Wr = x² || 1|
(6)
This confirms that rotation preserves the Euclidean norm of the vector.
illustrate this effect, we provide four-dimensional example explaining why amplifying small values under the norm-invariance constraint leads to outlier reduction. Consider a vector = [2], 22, 23, Ta]. where æ æ2 and æ3 have absolute values close zero, while æ4 is an outlier. Introducing perturbation ( 1,2,3, 4.) to each component, the new norm of the vector becomes:
||ح|| - +2
(7)
= =V (21 +61) + (22 +22 + (as + Ex)2 + (I4 + E2)
(8)
2 + + + 1
Clearly, if E3, > 0, it must follow that < 0. DartQuant effectively leverages this constraint to suppress outliers.
K
Hyperparameter Settings
The specific hyperparameter settings for DartQuant are shown in Table 23 It is important to note that the latent parameter Zo is initialized using random Hadamard matrix.
Table 23: Comparison of zero-shot accuracy and perplexity across different loss functions.
Rotation
Model
LR
Epoch
Optimizer
BS
2-7b
2.00E-03
10
SGD
64
2-13b
1.00E-02
10
SGD
64
R1
2-70b
1.00E-03
10
SGD
64
3-8b
8.00E-03
10
SGD
64
3-70b
3.00E-03
10
SGD
64
R2
All
1.00E-03
10
SGD
64
L Limitation
DartQuant is tailored for uniformly distributed integer formats. Its effectiveness on alternative formats, such as FP4, other non-uniform numerical representations, remains to be further explored and validated. Furthermore, Whip assumes that activations are approximately zero-mean, which generally holds true for most transformer layers. However, in rare cases where the activation mean significantly deviates from zero, the effectiveness Whip may degrade.
In the future, we can explore wider range of distribution transformation methods to better accom- modate various numerical formats.
M Impact Statement
This work introduces DartQuant, an efficient rotational distribution calibration method for LLM quantization. DartQuant simplifies the optimization rotation matrices while avoiding the overfitting risks associated with end-to-end training. It successfully quantizes a 70B model on single RTX 3090, marking a significant advancement. This progress not only improves the efficiency of LLMs in practical applications but also provides practical solution for large-scale model quantization, contributing to the accessibility and scalability of Al technologies. However, if misused for harmful purposes, it could have negative social impact.
35


Reviews:
[{"review_id": "XXRvjExo3X", "reviewer": "Reviewer_sxB6", "summary": "This paper introduces DartQuant, a post-training quantization method for large language models. The method aims to significantly reduce the computational cost associated with calibrating rotation matrices. DartQuant builds upon the computationally invariant framework and optimizes rotation matrices to improve uniform quantization. It consists of two key components:\n1. Whip Loss – a loss function designed to reshape activation distributions toward uniformity.\n2. QR-Orth – an optimization scheme using QR decomposition to enforce orthogonality of the rotation matrices.\n\nThe authors report a 47× speedup and 10× memory reduction, enabling the calibration of a 70B model on a single consumer GPU.", "questions": "1. Can you explain the better convergence rate of QR-orth method compare to Caley optimization?\n2. Can you compare to [5] which should be easily doable within the rebuttal period, and [3,4] if you have additional time? (See strengths  and weaknesses)\n3. Can you add more models (Mistral, Qwen, or Phi) and mathematical reasoning tasks such as GSM8K, MathQA?", "limitations": "The only limitation mentioned is \"tailored for uniformly distributed integer formats\". Is there no other limitations?", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper addresses a highly practical and relevant challenge: reducing the cost of optimizing rotation matrices during LLM quantization.\nCompared to prior methods such as SpinQuant, DartQuant offers substantial speed and memory benefits, making it more accessible for users with limited hardware. The paper provide a strong result on extreme qunatization scenario like 4 bit qunatization.\n\nOne of the biggest weakness is that all experiments are conducted exclusively on LLaMA models. The generality of DartQuant should be evaluated on other architectures like Mistral, Qwen, or Phi to demonstrate broader applicability. Moreover, adding experiments on mathematical reasoning tasks such as GSM8K, MathQA or longbench would strengthen the empirical evaluation and show the method’s performance on more challenging domains.\n\nAnother big weakness is the missing comparisons with closely related quantization methods, including some mixed-precision quantization method, e.g. at least [3, 4].\n\nAnother big weakness is the unsubstantiated dismissal of kurtosis-based methods. Reference [5] demonstrates the effectiveness of kurtosis as a quantization signal. Dismissing it without a rigorous comparison weakens the argument for Whip Loss. A head-to-head evaluation and deeper justification for the superiority of Whip Loss are needed. This comparison is key since with KurTail [5] we can also train the rotation matrices single consumer GPU similar to DartQuant.\n\nThe justification for the “Whip Loss” function is vague. It is said to be “inspired by” transforming Laplace to uniform distributions, yet the specific formulation, ∑exp(−|xᵢ|), is heuristic and lacks theoretical derivation. The paper does not explore alternative formulations or explain why this one is particularly effective at promoting uniformity.\n\nThe paper reports a 1.4× per-iteration speedup (Table 4) but also mentions a 41× total speedup (line 375) and a 47× training speedup (line 332). I suggest the authors to provide a section and ablation study to clarify and justify each speed up.\n\nThe QR-Orth optimization appears more effective than methods such as Cayley optimization, which contradicts [1]. The paper does not justify its choice or acknowledge this performance gap. The slower convergence rate in Caley methods may be due to suboptimal hyperparameter selection. Moreover, QR-Orth is not a novel method. The QR-Orth scheme is well-known in manifold optimization. The current presentation can be potentially misleading and be read as a novel contribution. The authors should cite foundational works,\nsuch as Optimization Algorithms on Matrix Manifolds [2].\n\nMinor comment: The exact optimization process for rotation matrices is unspecified. The details of how the rotation matrices are optimized are under-explained. For instance, the specific optimization process for Rotations is not described. Further the paper just claimed that they can optimise llama 70B on 1 GPU but running a llama 70B in fp16 needs 120GB for inference. Do they use a larger gpu to save the activation or they used methods like offloading to run their experiment clarifying the optimization process would help paper clarity.\n\nFigure 11 presents the distribution before rotation is applied. Including the post-rotation distribution would help demonstrate the effectiveness of DartQuant more convincingly.\n\nReferences:\n1. Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley\nTransform, ICLR 2020\n2. Optimization Algorithms on Matrix Manifolds, Princeton University Press, 2009.\n3. ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank\nResiduals, ICML 2025\n4. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger\nQuantized LLMs (Neurips 2024)\n5. KurTail: Kurtosis-based LLM Quantization, SLLM workshop ICLR 2025", "was_revised": true, "final_justification": "The authors addressed all of my concerns, hence I'm increasing my score.", "rebuttal": "We thank the reviewer for their valuable comments and suggestions. We have thoroughly reviewed each point and provide our detailed responses below. \n\n**Weakness1&Qusetion3** *More model & task*\n\nIn **Appendix H**, we experimentally demonstrate the effectiveness of our method on two MoE models (Mistral-8x7B and Deepseek-moe-16B). Following your suggestion, we also conducted comparative experiments on Qwen, Phi, and Mistral models across 5 zero-shot tasks, as well as additional evaluations on math reasoning datasets GSM8K and MathQA. \n\n- Our method significantly outperforms QuaRot in perplexity, zero-shot accuracy, and math reasoning on all three models.\n\n- Notably, on the math reasoning task GSM8K, our method achieves a remarkable improvement over QuaRot (e.g., a 16.52% increase on Qwen 2.5 3B), indicating promising advantages on tasks sensitive to quantization complexity.\n\n|Method|Model|WIKI|ARC_C|ARC_C|OBQA|PIQA|WG|5 AVG|GSM8K|MQA|MMLU|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|baseline|mistral 7B|5.25|53.92|79.75|43.80|81.99|74.03|66.70|37.75|35.94|58.71|\n|quarot||5.86|45.73|76.60|41.40|77.64|70.80|62.43|26.84|30.95|50.72|\n|Ours||5.57|49.91|77.81|42.00|80.68|71.58|64.40|30.85|33.46|53.96|\n|baseline|qwen2.5 3B|8.03|47.44|73.31|42.60|78.94|68.19|62.10|75.89|37.28|65.09|\n|quarot||9.85|43.51|67.55|38.80|74.04|62.74|57.33|40.56|34.17|55.51|\n|Ours||9.03|43.17|71.38|40.20|75.46|63.93|58.83|57.08|36.31|58.98|\n|baseline|phi 3 mini|6.02|56.65|78.57|47.80|80.90|73.95|67.57|80.21|39.59|70.15|\n|quarot||8.09|46.50|73.40|41.80|74.86|63.72|60.06|50.34|34.07|57.45|\n|Ours||7.64|49.23|75.17|44.40|76.93|67.71|62.69|59.21|34.50|59.94|\n\nDue to cost, we only ran LongBench on LLaMA 3 8B, where our method nearly halves the performance degradation compared to QuaRot. Different subsets of Longbench use different metrics (including F1 score, Rouge, etc.). For simplify, we used the average of these scores to evaluate the overall performance. The comparison of scores for each subset can also be seen in the following table.\n\n|Method|**longbench avg**|2wikimqa|hotpotqa|musique|multifieldqa_en|multifieldqa_zh|narrativeqa|qasper|triviaqa|gov_report|qmsum|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|baseline|**27.91**|11.50|9.55|6.12|20.55|21.20|5.04|18.56|90.78|27.50|23.90|\n|quarot|**23.52**|10.44|8.10|5.00|20.17|15.50|5.17|18.66|81.90|19.03|20.27|\n|ours|**25.61**|11.80|9.47|5.86|23.54|19.46|6.21|19.30|87.47|21.30|21.93|\n\n|Method|vcsum|dureader|lcc|repobench_p|passage_retrieval_en|passage_retrieval_zh|passage_count|trec|lsht|multi_news|samsum|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|baseline|6.67|13.16|71.98|68.40|9.17|8.02|2.00|75.00|34.50|17.65|44.84|\n|quarot|4.18|13.88|58.25|57.44|3.31|5.17|2.55|65.00|26.75|12.41|40.74|\n|ours|4.85|13.34|63.68|60.00|4.24|6.88|3.17|67.00|28.25|15.79|44.33|\n\n**Weakness2&Qusetion2** *Mixprecision comparison*\n\nIn **Appendix F**, we provide detailed comparisons with two classical mixed-precision quantization methods (QUIK and Atom). In line with your advice, we also compare with [3] and [4]. Using the publicly available code from [3,4], after aligning quantization settings, we performed the comparisons. Note that [3] does not support Mistral, and [4] doesn't support Qwen, so these results are omitted. The results are as follows:\n\n|Method|Model|WIKI|ARC_C|ARC_C|OBQA|PIQA|WG|AVG|\n|-|-|-|-|-|-|-|-|-|\n|baseline|qwen2.5-3B|8.03|47.44|73.31|42.60|78.94|68.19|62.10|\n|[3]||8.89|42.75|68.77|40.80|76.77|64.64|58.75|\n|Ours||9.03|43.17|71.38|40.20|75.46|63.93|58.83|\n|baseline|3 8B|6.14|53.24|77.82|45.00|80.96|73.32|66.07|\n|[3]||7.08|49.49|74.79|42.80|78.67|69.14|62.98|\n|[4]||8.13|44.62|72.14|40.80|75.52|68.03|60.22|\n|Ours||7.43|48.63|75.52|43.00|78.18|70.68|63.20|\n|baseline|mistral-7B|5.25|53.92|79.75|43.80|81.99|74.03|66.70|\n|[4]||5.78|45.82|75.33|42.20|79.43|69.93|62.54|\n|Ours||5.57|49.91|77.81|42.00|80.68|71.58|64.40|\n\nFrom the result, DartQuant performs comparably to [3] and better than [4]. Notice that, [3] achieves higher accuracy by combining rotation with high-bit outlier protection, while DartQuant improves low-bit quantization precision through optimizing the rotation matrix. The two methods are algorithmically orthogonal and can be combined for further improvements.\n\n**Weakness3** *vs [5]*\n\nWe have reported Whip vs. kurtosis results under DartQuant settings in **Appendix I**, demonstrating Whip’s superiority. To further validate this, we compared Whip to the method in [5]. As [5]’s code is unavailable and the authors have not yet responded to our inquiries, we re-implemented their approach strictly following their published settings, using 512 calibration samples (128 samples in paper for efficiency). The following results (with [5]’s paper results as reference) show that, with equal calibration samples, our Whip loss achieves significantly better performance on two different models. This aligns with Figures 6 (lines 395-362) and Appendix I, confirming that Whip pushes activations closer to uniformity, outperforming other losses.\n\n|Method|Model|WIKI|ARC_C|ARC_C|OBQA|PIQA|WG|AVG|\n|-|-|-|-|-|-|-|-|-|\n|baseline|2 7B|5.47|46.33|74.54|44.20|79.05|69.06|62.64|\n|[5]||5.90|43.10|72.00|41.20|76.60|66.80|59.94|\n|Ours||5.72|45.09|72.72|43.40|78.18|68.98|61.67|\n|baseline|3 8B|6.14|53.24|77.82|45.00|80.96|73.32|66.07|\n|[5]||7.20|48.20|75.40|43.60|78.40|70.00|63.12|\n|Ours||6.84|51.02|76.47|44.00|79.27|71.67|64.49|\n\n**Weakness4** *Why whip*\n\nThe Whip loss formulation follows the transformation in Eq. 3. As shown in Fig. 5, Eq. 3 expands the near-zero region of the original distribution to a larger range. The Whip gradient near zero is much larger than that at the tails (due to the exponential function), which is key to Whip’s efficiency, as it rapidly pushes small values up in the rotated data. Under norm-invariant constraints, outliers are quickly suppressed.\n\nWe also use Llama 3 8B tested two functions with similar shapes but gentler gradients than Whip:\n\n- $L1= \\sum ReLU(1-|x_i|/\\alpha)$, where $\\alpha$ is set to three times the median (outlier threshold)\n\n- $L2= \\sum 1/(|x_i|+1)$\n\n|Loss|WIKI|ARC_C|ARC_C|OBQA|PIQA|WG|AVG|\n|-|-|-|-|-|-|-|-|\n|baseline|6.14|53.24|77.82|45.00|80.96|73.32|66.07|\n|quarot|8.40|40.02|66.33|41.00|73.61|66.77|57.55|\n|L1|7.56|48.48|72.93|42.80|77.58|68.03|61.96|\n|L2|7.65|46.25|72.85|41.20|76.39|70.01|61.34|\n|whip|7.43|48.63|75.52|43.00|78.18|70.68|63.20|\n\nWith identical hyperparameters, these alternative losses show inferior performance compared to Whip, demonstrating Whip’s advantage.\n\n**Weakness5** *Speedup detail*\n\nWe clarify that QR-Orth achieves a 1.4× per-iteration speedup over Cayley (Table 4), a 41× “total speedup” to converge to the same loss (line 375), and a 47× speedup in rotation matrix training compared to SpinQuant (line 332).\n\n**Weakness6&Qusetion1** *QR-Orth vs Cayley*\n\nSection 5 of [1] points out that the Cayley transform’s iterative form is a contraction mapping, with Theorem 1 proving convergence only if step size $\\alpha < 2 / \\|W\\|$. Otherwise, updates may diverge or leave the manifold, limiting Cayley SGD’s step size and slowing convergence.\n\nQR-Orth, however, projects updated parameters back onto the manifold without this constraint, allowing larger update steps and faster convergence.\n\n|optim|Delta|\n|-|-|\n|QR-Orth|1.20E-04|\n|Cayley|8.95E-05|\n\nIn the same hyperparameter settings, we measured the average parameter update magnitude over 10 iterations: QR-Orth’s updates are 1.2× larger than Cayley’s, consistent with the reasoning above.\n\nNotably, the Cayley implementation follows SpinQuant’s original hyperparameter configuration in paper, which has been tuned and validated for this task. To rule out suboptimal hyperparameter effects, we tested multiple learning rates. Across all tested settings, QR-Orth achieved consistently lower losses than Cayley, confirming our claims.\n\n|Optim|LR|Loss|Optim|LR|Loss|\n|-|-|-|-|-|-|\n|Cayley|0.01|1577.49|QR-Orth|0.01|1530.68|\n||0.1|1565.52||0.1|1537.05|\n||1|1565.71||1|1527.65|\n||2|1566.04||2|1526.59|\n||5|1565.53||5|1531.07|\n\nFinally, we sincerely thank the reviewer for pointing out the lack of citation. We acknowledge that QR-based orthogonalization is a well-established tool in manifold optimization [2], and we do not claim novelty in the algorithm itself. Our contribution lies in being the first to adapt QR-Orth for rotation-matrix training in LLM post-training quantization. We will revise the final version to clarify this point and avoid any overstatement.\n\n**Weakness7** *Memory usage*\n\nAs Fig. 4 illustrates, DartQuant’s process has two stages. First, the LLM runs once on the calibration set to collect activations stored locally. Then rotation matrices are trained offline on this stored data, so no large model loading is needed during rotation optimization. For large models such as LLaMA 70B, we adopt a layer-wise loading strategy allowing inference to be performed on a single GPU. We further clarify these details in **Appendix K** to avoid misunderstandings.\n\nFinally, we plan to release our code to facilitate reproducibility within the community.\n\n**Weakness8** *Visualization*\n\nThank you for your suggestion. We will add histograms of post-rotation activation distributions in the revision to better demonstrate DartQuant’s effectiveness.\n\n**Limitations**\n\nIn addition to being tailored for uniform quantization formats, we acknowledge a potential limitation regarding the distributional assumption in the Whip loss. Specifically, Whip assumes that activations are approximately zero-mean, which generally holds true for most transformer layers. However, in rare cases where the activation mean significantly deviates from zero, the effectiveness of Whip may degrade. As a remedy, we suggest applying mean-centering or zero-mean normalization prior to optimization, and we will explicitly include this consideration in the revised version. We also plan to explore adaptive variants of Whip that are robust to non-zero-mean activations in future work."}, {"review_id": "KmCbmmRV5z", "reviewer": "Reviewer_SzJU", "summary": "This paper propose DartQuant, a post-training weight-activation-kv cache quantization method for LLMs, that propose a new layer-wise loss to learn the rotation matrices to flatten the activation distribution without end-to-end fine-tuning. This method reduces the quantization time (computational complexity) but achieves SoTA model performance.", "questions": "1. Could the author clarify the quantization format for activations? I assume the weight matrix is quantized using groupwise scales and integer quantization. What about activations?\n2. The Whip loss is impressive. Does the Whip loss assume the activation distribution is symmetric over zero? What if the activation mean is not zero?\n3. In Algorithm 1, what does the function \"Row_sampling(X)\" mean?\n4. Could the author give more information about the objective in Algorithm 1 like variance across $Z$ initialized with different random seeds and learning rates? Is the problem easy to solve for gradient descent?", "limitations": "As stated in the limitation section of the paper, DartQuan did not explore the number formats that may have native hardware support like FP4. INT4 activation is not natively supported by most of GPUs. Other concerns are reflected by my questions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- A good summary of related work on rotation-based PTQ methods for LLMs. This serves as a good background intro for readers who are not very familiar with this area.\n- A clear description of the method supported by sufficient explainations. The whip loss design is simple but effective.\n- Extensive experiments and ablation study to demostrate the effectiveness of the method\n\n## Weaknesses\nI did not see partitical weaknesses though I have a few questions about the details (See Question Section). \n\nIf will be helpful if the author show some inference latency/throughput/memory results even though it may be the same as other ratation-based methods like SpinQuant.", "was_revised": true, "final_justification": "Thanks the author for clarification. All of my questions have been answered. I remain positive with this work.", "rebuttal": "We sincerely thank the reviewer for the highly positive feedback and your thoughtful questions. We deeply appreciate your recognition of our work. We provide our point-by-point responses to the raised questions below.\n\n**Weakness:** *If will be helpful if the author show some inference latency/throughput/memory results even though it may be the same as other ratation-based methods like SpinQuant.*\n\n As suggested, we evaluated the inference performance of DartQuant (W4A4KV4) using LLaMA 2-7B on single A800 GPU under various batch sizes and input lengths using randomly sampled prompts and compared it with FP16 baseline. Specifically, we report prefill time, decoding speed (fixed 50-token generation), and peak memory usage (torch.cuda.max_memory_allocated()). Results are provided below:\n\n|Batch|SeqLen|Prefill(ms)DartQuant|Prefill(ms)FP16|Decode(tok/s)DartQuant|Decode(tok/s)FP16|Mem(GB)DartQuant|Mem(GB)FP16|\n|--|--|--|--|--|--|--|--|\n|1|2048|209|281|20.43|14.08|3.89|13.66|\n|4|2048|611|907|81.75|52.71|4.72|16.79|\n|8|2048|1138|1452|163.7|104.32|5.81|20.95|\n|1|1024|108|141|20.5|12.99|3.77|13.16|\n|4|1024|306|482|81.87|47.29|4.19|14.75|\n|8|1024|567|752|167|109.32|4.74|16.89|\n|1|512|61|83|21.28|12.43|3.70|12.9|\n|4|512|163|257|82.91|46.55|3.92|13.74|\n|8|512|288|395|168.35|112.33|4.21|14.85|\n\n**Question1:** *Could the author clarify the quantization format for activations? I assume the weight matrix is quantized using groupwise scales and integer quantization. What about activations?*\n\nThank you for raising this. As stated in Line 258,  all activations are quantized using per-token asymmetric  integer quantization.\n\n**Question2:** *The Whip loss is impressive. Does the Whip loss assume the activation distribution is symmetric over zero? What if the activation mean is not zero?*\n\nThis is an excellent question that highlights a potential limitation of the current Whip design. Indeed, Whip implicitly assumes that the activation distribution is approximately zero-mean. While this assumption often holds in practice, it is not strictly guaranteed.\n\nWhen the activation mean deviates significantly from zero, the effectiveness of Whip may be reduced. In such cases, applying zero-mean normalization to the activations before computing the loss could yield more stable optimization. We will add a discussion of this assumption and its implications in the revised version. We sincerely thank the reviewer for this insightful observation.\n\n**Question3:** *In Algorithm 1, what does the function \"Row_sampling(X)\" mean?*\n\n\"Row_sampling(X)\" refers to randomly sampling a subset of activation tokens from the full matrix for gradient computation in each optimization step. This strategy improves efficiency by reducing computation without sacrificing performance.\n\nTo avoid potential confusion caused by the name, we will rename this function to \"Token_sampling(X)\" in the revised version for better clarity.\n\n**Question4:** *Could the author give more information about the objective in Algorithm 1 like variance across \n initialized with different random seeds and learning rates? Is the problem easy to solve for gradient descent?*\n\nThank you for this thoughtful question. As the optimization involves random sampling, the process is inherently non-deterministic and not exactly reproducible across different random seeds.\n\nNonetheless, we conducted multiple independent runs under varying random conditions (implicitly covering different seeds and sampling paths). As shown below, the final loss consistently converges to similar values across runs, and with comparable iteration counts. This empirical evidence suggests that the optimization problem in **Algorithm 1** is well-conditioned and tractable via gradient descent, without suffering from instability or high variance.\n\n|ID|LR|FinalObjective|Iterations|\n|-|-|-|-|\n|1|1.00E-03|2004.9094|138|\n|2|1.00E-03|2004.9535|141|\n|3|1.00E-03|2004.9191|140|\n|4|2.00E-03|2004.9507|131|\n|5|5.00E-03|2004.8560|123|"}, {"review_id": "LoI6ifFgBA", "reviewer": "Reviewer_gzqM", "summary": "The paper proposes a new approach for finding orthogonal transformations, which is needed for reducing the activation outliers during post-training quantization.", "questions": "I have some questions and comments on this paper:\n\n1. I found the idea interesting. The issue with the current scheme is that the improvement is a bit marginal even when we compare against QuaRot which doesn't do anything (no training). However, this is nothing with the paper, but more with the PTQ schemes. It seems that we hit the wall, considering the limits of the 4-bit compression. That's why we cannot expect large improvements in all these kinds of work.\n\n2. Considering the above fact, I think the improvement would be even closer (compare to something like QuaRot) in more recent 4-bit formats. For this, can you please provide an experiment where you compare DartQuant with QuaRot (I do not suggest SpinQuant as I know that will take 4-5 hours to run) on NvFP4? I would improve my score if we get non-trivial improvement over that format.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n- The paper is well-written and easy to follow\n- The idea is interesting\n- The experiment setup is proper\n\nWeaknesses\n\n- The improvement is somehow marginal (please see the Question section)", "was_revised": true, "final_justification": "After a relatively long discussion with the authors, they provided additional data clarifying aspects of the work that were previously unclear to me. As a result, I am convinced to raise my score for this paper.", "rebuttal": "We sincerely thank the reviewer for the encouraging comments, and we are glad to hear that you found our idea interesting. Below we provide detailed responses to each of your comments and concerns.\n\n**Question1:** *I found the idea interesting. The issue with the current scheme is that the improvement is a bit marginal even when we compare against QuaRot which doesn't do anything (no training). However, this is nothing with the paper, but more with the PTQ schemes. It seems that we hit the wall, considering the limits of the 4-bit compression. That's why we cannot expect large improvements in all these kinds of work.*\n\nWe appreciate your valuable observations. While QuaRot indeed serves as a strong baseline for 4-bit PTQ quantization, we believe the improvements introduced by rotation matrix optimization are still meaningful—particularly for more recent and challenging models such as LLaMA 3 8B/70B, 3.2B, or 1B, and in even lower-bit quantization settings.\n\nFor instance, in the 4-4-4 setting (Table 2), DartQuant improves the performance of LLaMA 3 8B from QuaRot’s 12.29/57.32 (PPL/Zero-Shot) to 10.78/62.38, and LLaMA 3 70B from 11.38/61.50 to 8.13/69.05. These are non-trivial improvements.\n\nWe would also like to emphasize that, in the original paper, we prioritized extreme optimization efficiency by using a very small number of calibration samples. However, if one trades some speed for better accuracy, DartQuant benefits significantly from larger calibration sets (512 samples same as [1], SpinQuant use 800 samples). Detailed results are as follows:\n\n|Method|Model|WIKI|ARC_C|ARC_E|OBQA|PIQA|WG|AVG|\n|-|-|-|-|-|-|-|-|-|\n|baseline|2 7B|5.47|46.33|74.54|44.20|79.05|69.06|62.64|\n|quarot| |6.17|41.64|68.43|40.60|77.04|65.27|58.60|\n|Ours(paper)| |5.93|44.20|69.87|38.60|77.48|66.69|59.37|\n|Ours(512)| |5.72|45.09|72.72|43.40|78.18|68.98|61.67|\n|baseline|3 8B|6.14|53.24|77.82|45.00|80.96|73.32|66.07|\n|quarot| |8.40|40.02|66.33|41.00|73.61|66.77|57.55|\n|Ours(paper)| |7.43|48.63|75.52|43.00|78.18|70.68|63.20|\n|Ours(512)| |6.84|51.02|76.47|44.00|79.27|71.67|64.49|\n\nAs shown, under larger calibration sets, DartQuant consistently outperforms QuaRot on both models. In this light, we respectfully suggest that: \"Yes, it is possible to expect substantial improvements from our method.\"\n\n[1] KurTail: Kurtosis-based LLM Quantization, SLLM workshop ICLR 2025\n\n**Question2:** *Considering the above fact, I think the improvement would be even closer (compare to something like QuaRot) in more recent 4-bit formats. For this, can you please provide an experiment where you compare DartQuant with QuaRot (I do not suggest SpinQuant as I know that will take 4-5 hours to run) on NvFP4? I would improve my score if we get non-trivial improvement over that format.*\n\nTo further illustrate DartQuant’s advantages in more difficult scenarios, we conducted additional experiments on LLaMA 3.2B and 1B under 4-bit and 3-bit quantization. In line with your suggestion, we also included evaluations under the NvFP4 format. \n\nSpecifically, we implemented NvFP4 quantization using the FP4 codebase from the microsoft/microxcaling repository. Due to the use of complex secondary quantization in NvFP4, the inference speed is substantially reduced. Given limited time, we report results only on the two smaller models mentioned above. The quantitative results are presented below.\n\n|Method|Model|WIKI|ARC_C|ARC_E|OBQA|PIQA|WG|AVG|\n|-|-|-|-|-|-|-|-|-|\n|3.2 3B|baseline|7.81|46.07|71.63|43.40|77.52|69.37|61.60|\n||quarot|11.70|36.18|58.46|34.40|69.53|59.91|51.70|\n||our|9.65|41.13|66.25|39.40|74.21|61.72|56.54|\n||quarot+nvfp4|9.90|39.85|64.31|38.00|73.50|64.64|56.06|\n||ours+nvfp4|8.91|42.06|65.82|40.00|74.92|64.88|57.54|\n||quarot 3bit|105.36|23.72|30.68|23.00|52.56|49.41|35.87|\n||ours 3bit|16.04|30.12|48.82|29.60|64.20|56.27|45.80|\n|3.2 1B|baseline|9.75|36.26|60.47|37.20|74.53|60.61|53.81|\n||quarot|19.75|26.71|45.33|30.40|62.57|52.17|43.44|\n||our|13.58|30.89|50.63|31.80|69.59|54.54|47.49|\n||quarot+nvfp4|13.88|31.57|51.01|31.60|68.06|56.27|47.70|\n||ours+nvfp4|12.84|31.74|55.77|34.20|70.89|56.99|49.92|\n||quarot 3bit|255.90|23.46|30.64|23.80|53.59|52.64|36.83|\n||ours 3bit|27.54|25.09|39.73|27.80|60.45|52.17|41.05|\n\nThe results show that for these more quantization-sensitive models, there remains considerable room for improvement over QuaRot. In the 3-bit setting, the benefits of rotation matrix optimization become even more pronounced. This supports our hypothesis that the more aggressive the compression, the more valuable such optimization becomes.\n\nRegarding NvFP4, although the Whip loss was originally designed for uniform quantization and may not be optimal for non-uniform formats like FP4, we find that the distribution tightening it induces still benefits the overall quantization. As a result, DartQuant + NvFP4 yields measurable gains over QuaRot + NvFP4. \n\nLooking ahead, we plan to explore alternative loss functions that are more tailored to the characteristics of FP4 and other non-uniform quantization formats, to further improve performance in such settings.\n\nIn summary, we believe that for more difficult quantization scenarios—especially low-bit formats and challenging models—exploring rotation matrix optimization remains a promising and meaningful direction."}, {"review_id": "vHnF2IkJG4", "reviewer": "Reviewer_uEAv", "summary": "DartQuant is a method for efficient rotational calibration in LLM quantization. By replacing costly end-to-end fine-tuning of rotation matrices with a distribution-aware calibration approach, DartQuant reduces the risk of overfitting and dramatically lowers computational and memory costs during compression.", "questions": "1. What is the metrics of outlier in Figure 3? It should be included in the article.\n2. Which baseline is used in Table 1?  It should be stated in the article.\n3. Line 283 states \"In the main results, we apply GPTQ to rotate the weights.\", but GPTQ doesn't have rotate. I guess it's \"reconstruct\"?\n4. Why could DartQuant avoid overfitting? It still use calibration data for both rotate and reconstruct (if above is true).\n5. Line 25 is wrong. lora doesn't improve inference efficiency.", "limitations": "Limitation in appendix.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1. The overall design of Whip loss makes sense. Because the norm is constant, reducing small-value would also reduce outliers. \n2. The use of autograd qr decomposition makes sense, which avoids the time-consuming Cayley SGD but still guarantee the orthogonality. But I'm not sure if the gradient of qr decomposition is stable or not.\n\nWeakness:  \n1. The main weakness is that the improvement is not significant. The performance is very marginal compared to SpinQuant. The improvement on compression time and memory is large. But I doubt whether compression time is an important objective in quantization, compared with time and memory during inference, or performance.\n2. In four rotation matrix R1 ... R4, only R1 and R2 are learned by DartQuant, while R3 and R4 use Hadamard rotations like previous work, which limits the value of this method. By the way, the Hadamard matrix of R3 and R4 is random, right?\n3. See \"Questions\".", "was_revised": true, "final_justification": "From my perspective, the contribution is not compelling. The primary improvement is compression time, which is relatively minor given existing fast PTQ methods. For PTQ, the key metrics are (i) performance, (ii) inference speed, (iii)inference memory, and (iv) compression speed, with the last being least important. In addition, the manuscript contains several minor errors. Overall, I do not find the work strong enough for acceptance.", "rebuttal": "We are grateful for your review and positive feedback. We will address each of the weaknesses and questions raised (in a combined way) in detail below. We will be happy to answer any further questions.\n\n**Strengths2:** *But I'm not sure if the gradient of qr decomposition is stable or not.*\n\n Thank you for the comment. Regarding the gradient stability of QR decomposition, we provided empirical analysis in our reply to SzJU Question4. Under various initializations and hyperparameters, the autograd-based QR-Ortho consistently converged and showed strong robustness to learning rate. Hence, it is stable in practice.\n\n|ID|LR|FinalObjective|Iterations|\n|-|-|-|-|\n|1|1.00E-03|2004.9094|138|\n|2|1.00E-03|2004.9535|141|\n|3|1.00E-03|2004.9191|140|\n|4|2.00E-03|2004.9507|131|\n|5|5.00E-03|2004.8560|123|\n\n**Weakness1:** *The main weakness is that the improvement is not significant. The performance is very marginal compared to SpinQuant. The improvement on compression time and memory is large. But I doubt whether compression time is an important objective in quantization, compared with time and memory during inference, or performance.*\n\nThanks for the feedback. First, we would like to highlight the practical importance of compression time in post-training quantization (PTQ), especially as model sizes and deployment scales continue to grow. Our primary contribution is improving the efficiency of rotation matrix optimization, a point that has also been positively noted by reviewers sxB6 and SzJU.\n\nFor example, SpinQuant requires over 40 GPU hours (8×A800) to optimize LLaMA 2-70B. As models continue to scale (e.g., LLaMA 3.1 405B), and development cycles accelerate, optimization cost becomes an increasingly important factor in the applicability of PTQ techniques.\n\nThis trend is also observable in practice: although some existing methods can achieve strong accuracy, their relatively high computational cost can limit practical adoption. In contrast, methods with lower overhead (e.g., AutoGPTQ) have seen broader use in the community, due to their balance between speed and effectiveness. Similarly, while SpinQuant provides strong accuracy, its higher resource cost may affect its ease of use in large-scale or time-sensitive settings.\n\nAnother point we would like to emphasize is that,  DartQuant supports further accuracy improvements by increasing calibration data size, especially when extreme efficiency is not the primary concern. For instance, using 512 calibration samples on 7B/8B models, the optimization time remains moderate (about 24.97 minutes), approaching the runtime of SpinQuant while offering higher accuracy. The experimental results are as follows:\n\n|Method|Model|WIKI|ARC_C|ARC_E|OBQA|PIQA|WG|AVG|\n|-|-|-|-|-|-|-|-|-|\n|Baseline|2 7B|5.47|46.33|74.54|44.20|79.05|69.06|62.64|\n|SpinQuant||5.99|42.32|70.50|40.60|76.44|64.17|58.81|\n|Ours(paper)||5.93|44.20|69.87|38.60|77.48|66.69|59.37|\n|Our(512)||5.72|45.09|72.72|43.40|78.18|68.98|61.67|\n|Baseline|3 8B|6.14|53.24|77.82|45.00|80.96|73.32|66.07|\n|SpinQuant||7.41|47.95|73.48|41.40|77.26|67.56|61.53|\n|Ours(paper)||7.43|48.63|75.52|43.00|78.18|70.68|63.20|\n|Our(512)||6.84|51.02|76.47|44.00|79.27|71.67|64.49|\n\nFrom the results, it can be observed that our method achieves a significant accuracy improvement over SpinQuant when the optimization time is aligned. In summary, we believe that DartQuant is more practical for deployment with a small calibration set due to its high optimization efficiency. On the other hand, in scenarios where optimization time is not a major concern, our method can achieve outstanding accuracy. If you have any further questions regarding this, please do not hesitate to let us know.\n\n**Weakness2:** *In four rotation matrix R1 ... R4, only R1 and R2 are learned by DartQuant, while R3 and R4 use Hadamard rotations like previous work, which limits the value of this method. By the way, the Hadamard matrix of R3 and R4 is random, right?*\n\nIn current experiments, we apply DartQuant to R1 and R2, and use random Hadamard for R3 and R4 to minimize inference latency, following SpinQuant and OSTQuant.\n\nHowever, DartQuant is also compatible with R3/R4 optimization. Below is the accuracy improvement when optimizing R4 with DartQuant.\n\n|Method|Model|WIKI|ARC_C|ARC_E|OBQA|PIQA|WG|AVG|\n|-|-|-|-|-|-|-|-|-|\n|baseline|2 7B|5.47|46.33|74.54|44.20|79.05|69.06|62.64|\n|dartquant||5.93|44.20|69.87|38.60|77.48|66.69|59.37|\n|dartquant+r4||5.92|44.11|71.04|42.60|77.64|67.80|60.64|\n|baseline|3 8B|6.14|53.24|77.82|45.00|80.96|73.32|66.07|\n|dartquant||7.43|48.63|75.52|43.00|78.18|70.68|63.20|\n|dartquant+r4||7.35|49.57|75.84|43.40|78.94|72.85|64.12|\n\n**Question1:** *What is the metrics of outlier in Figure 3? It should be included in the article.*\n\nOutliers in Fig.3 follow PrefixQuant[1]: values >3×median (before rotation). We will clarify this definition in the appendix.\n\n[1] Chen, Mengzhao, et al. \"Prefixquant: Static quantization beats dynamic through prefixed outliers in llms.\" (2024).\n\n**Question2:** *Which baseline is used in Table 1? It should be stated in the article.*\n\nThe baseline in Table 1 is consistent with Table 2 and detailed in Appendix C. We will clarify this in Table 1 to avoid confusion.\n\n**Question3:** *Line 283 states \"In the main results, we apply GPTQ to rotate the weights.\", but GPTQ doesn't have rotate. I guess it's \"reconstruct\"?*\n\nYou are absolutely correct that Line 283 contains a misstatement. GPTQ does not involve weight rotation; rather, it reconstructs weights via quantization-aware optimization. We will revise the wording to accurately reflect that GPTQ applies reconstruction, not rotation.\n\n**Question4:** *Why could DartQuant avoid overfitting? It still use calibration data for both rotate and reconstruct (if above is true).*\n\nDartQuant uses calibration data, but unlike end-to-end methods, it optimizes a proxy target: distribution alignment via Whip loss, not final quantization error. This indirect optimization reduces overfitting to calibration data. As shown in Table 5, performance remains stable across calibration sizes, with no “diagonal effect.” Thus, the distribution-alignment design helps DartQuant avoid overfitting seen in SpinQuant.\n\n**Qusetion5:** *Line 25 is wrong. lora doesn't improve inference efficiency.*\n\nWe acknowledge the inaccuracy in Line 25: while LoRA is effective in reducing fine-tuning cost and memory footprint, it does not bring improvements to inference efficiency. We will amend this reference to prevent any misunderstanding.\n\nWe appreciate your attention to detail and your valuable suggestions, which help us improve the clarity and accuracy of our work."}]

Author rebuttal:

Meta-review: This paper presents DartQuant, an efficient method for calibrating rotation matrices in large language model post-training quantization. The method introduces a "Whip Loss" to reshape activation distributions for better quantization and a "QR-Orth" scheme for efficient optimization. The paper’s key strength, lauded by reviewers sxB6 and SzJU, is its massive improvement in efficiency, i.e., up to 47x speedup and 10x memory reduction, which makes rotation-based quantization for 70B models accessible on consumer hardware. Initial weaknesses, noted by sxB6 and gzqM, included limited experimental scope and marginal accuracy gains in some settings. However, the authors provided a comprehensive rebuttal with extensive new experiments across more models (Qwen, Phi) and tasks (GSM8K), convincingly demonstrating strong performance and scalability. This effort successfully swayed reviewers gzqM and sxB6 to support the paper. Although reviewer uEAv remained concerned about the focus on compression time, the consensus is that the dramatic reduction in computational cost is a significant and practical contribution. Therefore, I recommend acceptance.
Decision: Accept (poster)