Title: DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Quantization

Rotation

Efficient inference
Abstract: Quantization plays a crucial role in accelerating the inference of large-scale models, and rotational matrices have been shown to effectively improve quantization performance by smoothing outliers. However, end-to-end fine-tuning of rotational optimization algorithms incurs high computational costs and is prone to overfitting. To address this challenge, we propose an efficient distribution-aware rotational calibration method, DartQuant, which reduces the complexity of rotational optimization by constraining the distribution of the activations after rotation. This approach also effectively reduces reliance on task-specific losses, thereby mitigating the risk of overfitting. Additionally, we introduce the QR-Orth optimization scheme, which replaces expensive alternating optimization with a more efficient solution. In a variety of model quantization experiments, DartQuant demonstrates superior performance. Compared to existing methods, it achieves 47$\times$ acceleration and 10$\times$ memory savings for rotational optimization on a 70B model. Furthermore, it is the first to successfully complete rotational calibration for a 70B model on a single 3090 GPU, making quantization of large language models feasible in resource-constrained environments.

DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization

Abstract

Quantization plays a crucial role in accelerating the inference of large-scale models, and rotational matrices have been shown to effectively improve quantization performance by smoothing outliers. However, end-to-end fine-tuning of rotational optimization algorithms incurs high computational costs and is prone to overfitting. To address this challenge, we propose an efficient distribution-aware rotational calibration method, DartQuant, which reduces the complexity of rotational optimization by constraining the distribution of the activations after rotation. This approach also effectively reduces reliance on task-specific losses, thereby mitigating the risk of overfitting. Additionally, we introduce the QR-Orth optimization scheme, which replaces expensive alternating optimization with a more efficient solution. In a variety of model quantization experiments, DartQuant demonstrates superior performance. Compared to existing methods, it achieves 47 × acceleration and 10 × memory savings for rotational optimization on a 70B model. Furthermore, it is the first to successfully complete rotational calibration for a 70B model on a single 3090 GPU, making quantization of large language models feasible in resource-constrained environments. Code is available at https://github.com/CAS-CLab/DartQuant.git .

1 Introduction

Large Language Models (LLMs) [1, 2, 3] have been a key breakthrough in natural language processing, demonstrating exceptional language understanding and generation capabilities through training on vast datasets with numerous parameters. These models perform exceptionally well on multiple tasks, including text generation, translation, and question-answering systems [4, 5]. However, the high computational and memory demands of LLM inference severely limit their deployment in resource-constrained environments [6, 7, 8, 9].

Current methods for reducing computational cost and improving the inference efficiency of deep learning models and LLMs include model pruning, knowledge distillation, parameter sharing, and quantization [10, 11, 12, 13, 14, 15]. Among these, post-training quantization (PTQ) stands out as a crucial technique to reduce computational costs due to its advantage of bypassing complex training processes, making it highly practical for real-world deployment [16, 17, 18, 19, 20].

In LLM quantization, activations pose a greater challenge than weights due to the frequent presence of extreme outliers, which can significantly degrade model accuracy [21]. To address this issue, various outlier-handling techniques have been proposed. For example, high-bit protection mechanisms preserve the precision of outliers, while diagonal matrices help smooth extreme values in activations [22, 23]. Recent studies have shown that rotation matrices and affine transformations are highly effective in reducing outliers in activations, significantly improving quantization performance [24]. Rotation matrices are invertible, preserve vector norms, and can be seamlessly integrated into model architectures without introducing additional inference costs, making them a mainstream approach for quantization [25]. Although random Hadamard rotations can improve performance to some extent, they are not optimal. SpinQuant demonstrates that training rotation matrices further enhances quantization performance [26].

However, existing methods (e.g., SpinQuant [26], OSTQuant [27]) treat the rotation matrices as network parameters and fine-tune them endto-end, which incurs substantial computational and memory costs associated with quantization. As shown in Figure 1, optimizing the rotation matrices for a 70B model requires hundreds of GiB of GPU memory and tens of gpu hours of computation, which conflicts with the fast deployment goals of PTQ algorithms. Moreover, end-to-end fine-tuning of rotation matrices presents unique challenges due to the complexity of optimizing on the rotation manifold. Specifically, rotation matrices must be carefully optimized to preserve orthogonality, which necessitates the use of specialized techniques such as Cayley or Riemannian SGD [28, 29]. These

7b

13b

70b

70

600

89

- 500

66

- 400

64

62

09

SpinQuant

58

OSTQuant

-100

56

DartQuant

0

2-3

2-2

2-1

20

21

22

23

24

25

Time (GPU hour)

methods are computationally intensive and incur significant time overhead. Additionally, using small sample sizes for end-to-end fine-tuning poses a substantial risk of overfitting [27], which would worsen the optimization process.

To address these challenges, we propose DartQuant, a distribution-based rotation matrix calibration method that eliminates the need of end-to-end fine-tuning, significantly reducing the resource demands of rotation optimization while achieving higher accuracy. To mitigate overfitting, we redefine the rotation optimization problem from the perspective of distribution calibration, i.e., to transform the activations into the distribution most suitable for quantization. Based on the distribution transformation function that expands small value ranges and compresses large value ranges, we design the Whip loss. Unlike others that directly constrain outliers, Whip loss optimizes the activation distribution, making it more uniform and reducing the impact of outliers, thus lowering quantization errors. Finally, we introduce QR-Orth, an optimization method that applies orthogonal constraints using QR decomposition, avoiding complex projection calculations and significantly reducing the computational complexity of orthogonal optimization, thereby enhancing calibration efficiency.

Our contributions are summarized as follows:

We introduce fast LLM Quantization with rotational distribution calibration framework, avoiding the excessive computational and memory costs of end-to-end fine-tuning paradigm. Based on this, the Whip loss is designed, which drives rotated activations toward a uniform distribution, effectively reducing quantization error and improving calibration efficiency.

We present the QR-Orth optimization scheme, which ensures orthogonality of rotations via QR decomposition, eliminating the need for complex orthogonal optimizers. This reduces computational complexity and further enhances calibration efficiency.

The proposed DartQaunt framework achieves superior quantization performance while significantly accelerating the rotation matrix calibration. For the 70B model, it delivers a 47 × speedup in terms of GPU hours and reduces memory usage by 10 × compared to existing methods. Notably, DartQuant enables rotation calibration of the 70B model on a single 3090 GPU in ∼ 3 hours , greatly reducing calibration costs.

2 Related Work

2.1 Challenges in LLM Quantization

Large language models (LLMs) face challenges in quantization due to activation outliers, which take up most of the quantization range and reduce accuracy. To address this, researchers have proposed various strategies. Early methods used mixed precision, applying different precisions to weights and activations to reduce errors. However, mixed precision methods are complex and hinder inference speed and memory efficiency.

2.2 Outlier Handling through Scaling

To address outliers in activation quantization, scaling-based methods have been proposed. SmoothQuant [23] transfers outliers from activations to weights using scale invariance, reducing activation quantization errors. Outlier Suppression+ [21] addresses the asymmetric distribution of activations across channels by applying channel-wise scaling and shifting. OmniQuant [19] introduces learnable weight clipping and fine-tunes quantization errors using blockwise error minimization. Although scaling methods reduce outliers in activations, they often shift the quantization difficulty to weights. This does not fully solve the problem, especially in the presence of extreme outliers [30]. Efficiently handling outliers without complicating weight quantization remains a key challenge.

2.3 Outlier Handling through Rotation

Recent research shows that rotation matrices offer unique advantages in handling outliers in activation quantization. QuIP [31] first introduces incoherent processing to reduce the impact of outliers in both the weight and activation spaces. QuIP # [32] further improves speed by using randomized Hadamard transforms, which also have better theoretical properties. Building on this, QuaRot [25] combines the outlier suppression ability of rotation matrices with invariance transformations, applying it to large models like LLaMA, significantly improving PTQ performance. QuaRot also finds that Hadamard transforms outperform random orthogonal transformations in quantization. SpinQuant [26] extends the rotation matrix as a trainable parameter and employs the Cayley optimizer [28] for end-to-end finetuning. QServe [27] combines rotation and scaling techniques, using random Hadamard transforms and scaling methods to suppress outliers across different modules, enhancing performance under low-bit quantization. Further, OSTQuant [27] treats both rotation and scaling as trainable parameters and employs a KL-top loss for end-to-end fine-tuning, achieving better quantization accuracy.

Existing end-to-end fine-tuning methods for optimizing rotation matrices, while relatively simple to implement, typically incur significant optimization costs and are prone to overfitting. These methods also require high-quality calibration samples. Furthermore, the orthogonal optimizers used during optimization are computationally expensive, as they need to perform optimization on complex manifolds, adding additional challenges. In contrast, DartQuant significantly improves the efficiency of rotation matrix calibration. While maintaining comparable accuracy, it achieves a calibration speed that is more than 47 × faster than existing methods.

(a) Original.

(b) Hadamard.

(c) SpinQuant.

(d) OSTQuant.

(e) DartQuant.

90

9.5

0.5

0.5

Probability

0.5

Probability

Probability

Probability

0.4

Probability

0.4

0.3

0.3

E'0

0.2

0.2

0.2

0.2

0.2

6.1

0.1

0.1

0.1

0.1

0.0

10

Value

0Z

30

=1

Value

0.0

=2

-1

Value

0

1

0.0

-1

Value

1

E=

=1

Value

3 Preliminaries and Difficulty

The Transformer architecture, commonly used in large language models (LLMs), consists of multihead self-attention modules and feedforward network modules, both primarily composed of linear layers. Let the output of a linear layer be represented as Y = XW ⊤ , where X ∈ R T × C in denotes the input activation, and W ∈ R C out × C in represents the weight matrix. Based on rotational invariance, we can insert an orthogonal transformation R into the linear layer without altering the output, yielding

(a) The number of outliers.

(b) Quantization error.

Y = ( XR )( R ⊤ W ⊤ ) , where R ∈ R C in × C in is an orthogonal matrix satisfying RR ⊤ = I . By combining R with the previous weight matrix and R ⊤ with the current layer's weight matrix, we can rotate the activation vector without introducing any additional computational cost.

Similarly, under the condition that the model's output remains unchanged, we can insert four orthogonal matrices R 1 , R 2 , R 3 , R 4 within the Transformer block, as outlined in [26] and Appendix A . Specifically, by multiplying R 1 on the right side of W q , W k , W v , W up , W gate, and multiplying R ⊤ 1 on the left side of W out and W down, an equivalent transformation is achieved. Similarly, R 2 can be inserted between W v and W out . R 3 can be inserted between the rotated encodings of Q and K . R 4 can be inserted before W down. Finally, W embedding is multiplied on the left by R T 1 , and W lm_head is multiplied on the right by R 1 , completing all the equivalent transformations. This process is referred to as the 'Computational Invariance' [33].

QuaRot [25] demonstrates that using random Hadamard rotation can achieve good results; however, this is not optimal, as shown in Figure 2. Methods like SpinQuant [26] and OSTQuant [27] treat the rotation matrix R as learnable network parameters and fine-tune them in an end-to-end manner with pseudo-quantizers inserted, resulting in better quantization performance. Although end-to-end fine-tuning is simple to implement, the complex computational process faces resource challenges and optimization difficulties. In Figure 3a, we present the number of outliers among 1000 activations in the 20th layer of various models after different transformations. Figure 3b shows the average quantization error of these samples (statistics for activations from other layers are in Appendix F). It is evident that the transformations in end-to-end fine-tuning do not significantly reduce the number of outliers in the activations, nor do they notably lower the quantization error, highlighting the limitations of end-to-end fine-tuning.

Moreover, end-to-end fine-tuning based on calibration sets not only consumes considerable computational resources but also tends to lead to overfitting on the calibration set [34, 27]. As shown in Table 1, fine-tuning methods exhibit a significant performance improvement on the corresponding test sets, with the improvement being particularly pronounced on the PTB dataset. A possible explanation is that the limited and relatively simple calibration data often fail to fully cover the parameter space of large models, causing the model to overfit to a narrow feature distribution, thereby limiting its generalization and emergent capabilities [27].

In addition, optimizing the rotation matrix requires the use of orthogonal optimizers to ensure the matrix's orthogonality. These optimizers are based on Riemannian optimization on the Grassmanian or Stiefel manifold, involving complex projection computations [28, 29]. As a result, their computational time is approximately twice that of standard optimizers. This, in turn, further increases the cost of rotation optimization and slows down the optimization process.

4 Method

In this section, we provide a detailed description of the proposed DartQuant method. DartQuant is comprised of three key components: the rotational distribution calibration, the Whip loss function, and QR-Orth optimization. Each of these components addresses the primary challenges outlined earlier. Figure 4 illustrates the overall framework, highlighting the flow and interactions between these components.

calibration results

x

x

@R_init

x

@R_dart

large and

numerous

outliers

small and

fewer

outliers

almost

w/o

outliers

25/75 percentile

1/99 percentile

min/max percentile

rotational distribution calibration

embedding

1.

avoid overfitting

2.

decrease computation

(w/o end2end)

MHSA

FFN

LM head

Z

loss=whip(O)

forward

backward

QR decomposition

O=X@R

R

···

···

one-shot calibration data collection

iterative rotation calibration

4.1 Rotational Distribution Calibration

End-to-end fine-tuning typically requires more data, and the optimization of rotation matrices depends on the task-specific loss, which significantly increases the risk of overfitting. For LLMs, end-to-end fine-tuning also entails substantial computational and memory overheads. To address these challenges, we redefine the rotation optimization problem and propose a rotational distribution calibration.

Specifically, we revisit the optimization objective of the rotation matrix from the perspective of feature distribution transformation. We redefine the problem as finding a rotation matrix that transforms the activations into the distribution most suitable for quantization. This approach reduces the reliance on task-specific loss during calibration, thereby mitigating the risk of overfitting.

Previous studies have shown that outliers are the primary cause of activation quantization loss. Therefore, we constrain the activation distribution after rotation by minimizing the number of outliers in the transformed activations, i.e.

The function I ( · ) represents the indicator function, and τ is the threshold used to identify outliers. Although this problem cannot be directly solved using standard stochastic gradient descent, we can resort to approximation methods for calibration. In statistics, variance is commonly used to measure the dispersion of data; however, using variance as an optimization objective is not ideal. Due to the symmetric distribution of activations [34], the variance of activations typically corresponds to a constant multiple of the activation vector's norm square. Furthermore, the norm-invariance property of the rotation matrix introduces significant challenges when directly optimizing using variance (as shown in Figure 7a). In addition to variance, kurtosis is frequently used to measure the heaviness of the distribution's tails, making it a suitable alternative objective. However, since the rotated activations are already close to a Gaussian distribution with relatively few outliers, optimizing with kurtosis is slow (as shown in Figure 7a). Therefore, there is an urgent need for a better optimization objective to constrain the activation distribution.

4.2 Activation Uniformity via Whip Loss

To better reduce outliers, we propose a new optimization objective that constrains the activation distribution to approach a uniform distribution, thereby effectively reducing the number of outliers in the rotated activations.

As shown in Figure 2a and Appendix G, the activation tokens exhibit a distribution near Laplace. Assuming that the activation tokens follow a Laplace distribution with mean µ = 0 and scale parameter b , the probability density function (PDF) is given by:

hard

to quantize

easy

to quantize

outlier

outlier

Probability

0

0.5

0.5

0

Rotate

flatten

gather

gather

outlier

outlier

peak

0

𝜏𝜏

-

𝜏𝜏

0

𝜏𝜏

-

𝜏𝜏

𝑈𝑈

𝑋𝑋

(

𝑥𝑥

)

0

c

-c

disperse

compress

disperse

compress

In statistics, cumulative distribution functions (CDFs) are often used to transform one distribution into another [35]. To convert x ∼ Laplace (0 , b ) to a uniform distribution over the interval [ -τ, τ ] , the transformation function is the following:

As shown in Figure 5, the left side presents the function graph of U X ( x ) , where the intervals near the origin are expanded, while those further from the center are compressed. The right side visually illustrates the impact of this transformation on the distribution. U X ( x ) spreads values originally concentrated around the center over a wider range, thus smoothing the peak of the distribution. Meanwhile, outliers farther from the center are gathered together, shrinking the overall distribution range, ultimately resulting in a uniform distribution within the interval [ -τ, τ ] .

Inspired by the mechanism of U X ( x ) , we propose the Whip loss function:

Here, x = [ x 1 , x 2 , . . . , x C in ] ∈ R C in denotes the activation vector. Clearly, the Whip function is continuously differentiable, and has larger gradients near zero. When used as a loss function, smaller values in the rotated activation vector are pushed away from zero. In other words, the Whip function encourages the rotation to smooth the sharp central peak of the Laplace distribution, producing a more uniform distribution. As the magnitudes of several small-value channels in the activation vector increase, the outliers are suppressed due to the norm-invariance constraint. This results in an 'aggregation' effect in the activation values. As a result, the overall activation distribution tends to converge toward a uniform distribution within a smaller interval, thereby effectively reducing the quantization error.

4.3 Enforcing Orthogonality with QR-Orth

To satisfy the orthogonality constraint, the rotation matrix must be optimized on the Grassmannian or Stiefel manifold, which necessitates the use of specialized optimizers, such as the Cayley SGD [28] used in SpinQuant [26]. Unlike gradients in Euclidean space, gradients on manifolds require complex projection operations, resulting in significantly higher computational costs. To avoid the computational complexity of orthogonal optimizers, we propose the QR-Orth optimization method.

Specifically, we can obtain an orthogonal matrix R ∈ R n × n and an upper triangular matrix U ∈ R n × n by performing a QR decomposition on any matrix Z ∈ R n × n . Based on this relationship, we design

the rotational distribution calibration method with QR-Orth optimizer shown in Algorithm 1. We use the orthogonal matrix R , obtained from the QR decomposition, as the rotation matrix for the actual computation. The latent parameter Z is treated as a optimization parameter and is discarded after calibration. By optimizing the latent matrix Z , we indirectly optimize the rotation matrix R . In this way, we can use any optimizer to optimize the rotation matrix.

When the matrix size becomes large, the Cayley optimizer introduces a computational overhead of approximately 6 n 3 compared to standard optimizers. In contrast, QR-Orth only incurs the cost of the QR decomposition, with a computational complexity on the order of 4 3 n 3 (see Appendix B for a detailed complexity deriva-

Algorithm 1 Rotational Distribution Calibration with QR-Orth Optimizer

1: Input: LLM model LLM , calibration sequence S , initial latent parameter Z 0 ∈ R n × n , max iterations T , learning rate η .

2: Output: Rotational matrix R ∈ R n × n .

3: X ← LLM ( S )

4: X ← token _ sampling ( X )

5: Z ← Z 0

6: for k = 0 to T do

7: R ← qr _ decomposition ( Z )

8: O ← X @ R

9: L ← Whip ( O )

10: Z ← Z -η ∂ L ∂Z

11: end for

tion). Although QR decomposition usually requires iterative calculations, the significant reduction in overall computational load has brought about a 1.4x acceleration effect. In practice, QR-Orth can easily integrate with various optimizers such as SGD or Adam, making it highly adaptable. This flexibility makes QR-Orth a promising solution for optimizing orthogonal matrices.

5 Experiment

Model and Dataset. We evaluate our method on the Llama series models, including Llama-2 (7B/13B/70B) [1] and Llama-3 (8B/70B). Moreover, we also provide results on two popular MoE models: Mixtral-8x7B [36] and Deepseek-MoE [37]. We report perplexity (PPL) scores on the WikiText2 [38], C4 [39], and PTB [40]. Additionally, we assess model performance on nine zeroshot evaluation tasks, including LAMBADA [41], HellaSwag [42], PIQA [43], WinoGrande [44], OpenBookQA [45], SIQA [46], MMLU [47], ARC-E, and ARC-C [48].

Baselines and Implementation Details. In addition to the basic RTN method, we compare our approach with several other methods, including SmoothQuant [23], GPTQ [49], OmniQuant [19], and current state-of-the-art methods such as Quarot [25], SpinQuant [26] and OSTQuant [27] for weight and activation quantization.

In the main results, we apply GPTQ to reconstruct the weights. To do so, we use 128 samples from WikiText2, with a sequence length of 2048 tokens, as the calibration set for GPTQ, following the standard GPTQ setup. All activations are quantized using per-token asymmetric quantization. We optimize all orthogonal matrices using SGD combined with QR-Orth. During the orthogonal matrix calibration phase, we use 128 samples from WikiText2, each with a token length of 2048.

5.1 Main Results

Table 2 evaluates six models across four common bit-width settings, offering practical guidance for selecting appropriate rotation schemes. DartQuant utilizes learned rotation matrices R 1 and R 2 , which can be fused into the model weights during inference, eliminating any additional computational overhead. In contrast, online Hadamard rotations ( R 3 and R 4 ) leverage fast Hadamard kernels for efficient inference computation [32]. As shown in Table 2, when both weights and activations are quantized to 8 bits, the performance differences among methods are minimal. However, when weights are quantized to 4 bits and activations to 8 bits, methods like SmoothQuant and OmniQuant experience significant performance degradation. This is primarily due to SmoothQuant's design, which complicates weight quantization and increases quantization errors, leading to a substantial drop in performance. In contrast, other methods generally maintain the model accuracy.

Although rotation transformation methods improve quantization performance when activations are quantized to 8 bits, the additional computational cost associated with R 3 and R 4 makes this approach less efficient. When activations are quantized to 4 bits, omitting the rotation matrix results in a significant performance drop. Furthermore, DartQuant, SpinQuant, and OSTQuant, which optimize

rotation matrices, clearly demonstrate the necessity of rotation matrix optimization, outperforming QuaRot. Additionally, while SpinQuant and OSTQuant perform well in reducing perplexity, their performance in zero-shot tasks is poor, which highlights the potential overfitting risks associated with end-to-end fine-tuning methods. In contrast, DartQuant, with its novel calibration strategy, generates rotation matrices that effectively compress the activation distribution range, resulting in outstanding performance in 0-shot tasks. Notably, we achieve a performance loss of only 0.5% on Llama 2-70b under w4a4kv16 setting. For Llama 3-70b which is more difficult to quantize [50], we manage to limit the average performance loss to 3.31%, outperforming SpinQuant and OSTQuant by 3.33% and 1.45% respectively.

Besides dense LLMs, we also extend our research to recent popular MoE-LLMs, and further experimental results can be found in the Appendix H.

Table 3 presents a comparison of the optimization time and memory consumption of SpinQuant, OSTQuant, and DartQuant on an A800 GPU server. DartQuant simplifies the calibration framework, leading to significant reductions in resource overhead across various models. In particular, for the 70B model, DartQuant completes the calibration in 30 minutes using a single GPU, achieving a speedup of 47 × in training and 10 × in memory savings compared to Spin-

Quant and OSTQuant. Moreover, DartQuant is the first to optimize the rotation matrix of the 70B model on a single 3090 GPU , with a calibration time of ∼ 3 hours . This development substantially reduces the cost of rotation matrix optimization and enhances its practical value.

5.2 Ablation Studies

We compared the effectiveness of four optimization objectives: quantization loss, variance, kurtosis, and the Whip function. As shown in Figure 7a, the change in activation quantization loss over iteration steps is presented for each optimization objective. It is clearly observed that when using quantization loss, variance, or kurtosis as the optimization objective, the activation quantization loss shows minimal variation. However, when the Whip function is used as the optimization objective, the quantization loss curve decreases significantly within fewer iterations and converges rapidly.

(a) Original.

(b) Hadamard.

(c) Quant.

(f) Whip.

300

300

300

250

250

250

Count

200

Count

200

Count

200

150

150

150

100

100

100

50

50

50

Value

10

15

20

25

Value

D

Value

0

300

300

300

250

250

250

Count

200

Count

200

Count

200

150

150

150

100

100

100

50

50

50

Value

0

Value

0

Value

Q

5.2.1 Optimization objectives

By comparing the effects of different optimization objectives on the activation distribution (as shown in Figure 6), we can clearly observe the substantial changes induced by the Whip function. Figure 6a shows the histogram of the original, unrotated activation distribution. From the range on the xaxis, it is evident that the original distribution contains significant outliers. Figure 6b displays the histogram of the activation distribution after a random Hadamard matrix rotation. After the Hadamard rotation, the activation range is notably compressed, although some outliers remain untreated. The rotation matrices trained with quantization loss and variance as optimization objectives show little improvement, resulting in an activation distribution nearly identical to that obtained by random Hadamard rotation. Although kurtosis optimization slightly improves the distribution, its effect is limited. In contrast, the histogram after Whip optimization shows a significant improvement (as shown in Figure 6f): this method not only effectively addresses the outlier problem but also disperses the activation points, initially concentrated around zero, across other regions. The resulting distribution is the closest to a uniform distribution. This outcome aligns closely with our design goals and further validates the effectiveness of our approach. More ablation studies on different optimization objectives under zero-shot tasks and perplexity metrics are provided in Appendix I.

5.2.2 Optimizer Comparison

Figure 7b presents the comparison of the convergence curves between the Cayley optimization and our proposed QR-Orth optimization, both using Whip loss under identical settings. It is evident that QR-Orth demonstrates faster convergence and lower final loss, regardless of whether combined with SGD or Adam. As shown in Table 4, QR-Orth achieves a 1.4 × speedup over Cayley optimization for the same number of iterations. Due to its faster convergence, QR SGD achieves the same result as Cayley SGD after 100 steps in just 6 steps, yielding an overall acceleration factor of 41 × . This significantly improves the efficiency of orthogonal optimization.

(a) Loss comparison.

(b) Cayley vs QR-Orth.

0.025

Quant

2140

CayleySGD

0.024

Whip

Kurtosis

2120

QR-Orth Adam

QR-OrthSGD

Quatization

0.023

2100

0.022

Whip

2080

Activation

0.021

2060

0.020

2040

0.019

2020

0.018

20

40

Epoch

60

80

100

20

40

Epoch

60

80

100

5.3 Results on Different Datasets

To investigate the sensitivity of DartQuant to different training datasets, we sample training data from three datasets: WIKITEXT2, PTB, and C4. These datasets are used to optimize R1 and R2 separately, and the impact of the dataset on the performance of the quantized LLM is compared. As shown in Table 5, the results of all three experiments are largely consistent. This demonstrates that DartQuant is robust to calibration datasets and does not negatively affect the generalization ability of the LLM. Further analysis on the impact of sample size on performance is provided in Appendix D.

6 Conclusions

In this paper, we introduce DartQuant, an innovative method for LLM quantization that efficiently handles activation outliers. By constraining the activation distribution, DartQuant simplifies rotation matrix calibration and avoids overfitting risks in end-to-end fine-tuning. The QR-Orth optimization method eliminates the need for complex orthogonal optimization, further speeding up the process. DartQuant achieves state-of-the-art results in 4-bit quantization while significantly reducing costs. Notably, it successfully quantizes a 70B model on a single RTX 3090, advancing LLM deployment in resource-constrained environments.

7 Acknowledgments

This work was supported in part by the National Key R&D Program of China (No. 2025ZD0122000), the Science and Technology Major Special Program of Jiangsu (No.BG2024028), Beijing Natural Science Foundation (L244046), the Jiangsu Key Research and Development Plan (No. BE2023016), and the CCF-Baidu Open Fund.

References

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360 , 2021.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT , volume 1, page 2. Minneapolis, Minnesota, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.

Zhilin Yang. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 , 2019.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems , 35:30318-30332, 2022.

Yuanteng Chen, Yuantian Shao, Peisong Wang, and Jian Cheng. EAC-MoE: Expert-selection aware compressor for mixture-of-experts large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 12942-12963, July 2025.

Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 , 2015.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems , 36:21702-21720, 2023.

Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. An empirical study of llama3 quantization: From llms to mllms. Visual Intelligence , 2(1):36, 2024.

Peisong Wang, Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, and Jian Cheng. Two-step quantization for low-bit neural networks. In Proceedings of the IEEE Conference on computer vision and pattern recognition , pages 4376-4384, 2018.

Weihan Chen, Peisong Wang, and Jian Cheng. Towards mixed-precision quantization of neural networks via constrained optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5350-5359, 2021.

Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In International Conference on Machine Learning , pages 9847-9856. PMLR, 2020.

Peisong Wang, Weihan Chen, Xiangyu He, Qiang Chen, Qingshan Liu, and Jian Cheng. Optimization-based post-training quantization with bit-split and stitching. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(2):2119-2135, 2022.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2704-2713, 2018.

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137 , 2023.

Chen Tianqi, Yuanteng Chen, Weixiang Xu, Zeyu Zhu, Peisong Wang, and Jian Cheng. Qmamba: Towards more efficient mamba models via post-training quantization, 2025.

Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145 , 2023.

Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlieraware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 13355-13364, 2024.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087-38099. PMLR, 2023.

Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. In The Twelfth International Conference on Learning Representations , 2024.

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456 , 2024.

Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant-llm quantization with learned rotations. arXiv preprint arXiv:2405.16406 , 2024.

Xing Hu, Yuan Cheng, Dawei Yang, Zhixuan Chen, Zukang Xu, JiangyongYu, XUCHEN, Zhihang Yuan, Zhe jiang, and Sifan Zhou. OSTQuant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. In The Thirteenth International Conference on Learning Representations , 2025.

Jun Li, Li Fuxin, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. arXiv preprint arXiv:2002.01113 , 2020.

Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. arXiv preprint arXiv:1810.00760 , 2018.

Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. arXiv preprint arXiv:2406.01721 , 2024.

Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems , 36, 2024.

Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396 , 2024.

Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024 , 2024.

Janghwan Lee, Jiwoong Park, Jinseok Kim, Yongjik Kim, Jungju Oh, Jinwook Oh, and Jungwook Choi. Amxfp4: Taming activation outliers with asymmetric microscaling floating-point for 4-bit llm inference. arXiv preprint arXiv:2411.09909 , 2024.

Yucui Liu and Tomasz J Kozubowski. A folded laplace distribution. Journal of Statistical Distributions and Applications , 2:1-17, 2015.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024.

Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066 , 2024.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 , 2016.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994 , 1994.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789 , 2018.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728 , 2019.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, et al. A systematic classification of knowledge, reasoning, and context within the arc dataset. arXiv preprint arXiv:1806.00358 , 2018.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 , 2022.

Minghai Qin. The uniqueness of llama3-70b with per-channel quantization: An empirical study. arXiv e-prints , pages arXiv-2408, 2024.

Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Quik: Towards end-to-end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.09259 , 2023.

Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems , 6:196-209, 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims in the abstract and introduction accurately reflect the paper's contributions and scope. They align with both the theoretical and experimental results, and the motivations and assumptions are clearly stated.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our work are discussed in Appendix L.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper includes comprehensive hypotheses, derivations, and experimental validations, with additional detailed discussions provided in Appendices B, G and J.

Appendix B provides a detailed analysis of the computational complexity of QR-Orth and Cayley-based orthogonal optimization.

Appendix G presents activation distributions across different models along with statistical metrics such as kurtosis variance and mean, supporting the validity of Laplacian modeling.

Appendix J proves the norm-invariant property of the rotation transformation and thoroughly explains the mechanism of the Whip function.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code required to reproduce the experiments is provided in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be packaged into a Zip file and uploaded to the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setup is described in Section 5, and Appendix K contains the detailed hyperparameter settings for reproducing the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experimental results in the paper were obtained using open-source benchmark frameworks and datasets. The detailed experimental setup is provided in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The experimental section of the paper provides details on the types of servers used for the experiments, as well as the memory and time requirements for the computations.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully reviewed the NeurIPS Code of Ethics and ensured that our research complies with its guidelines in all respects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The societal impact of this work is discussed in Appendix M.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing assets used in this work are explicitly cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code required to reproduce the results in this paper will be included in the supplementary materials, along with a detailed README to guide the experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The LLM is used only for writing, editing and formatting purposes.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Computational Invariance in Transformers

x

𝑥

𝑑𝑖𝑎𝑔(α)

𝐖𝑞

𝐖𝑘

𝐖𝑣

RoPE

MHSA

𝑤𝑜

x

𝑥

𝑑𝑖𝑎𝑔(α)

𝐖𝑔𝑎𝑡𝑒

𝐖𝑢𝑝

𝜎

𝐖𝑑𝑜𝑤𝑛

⊗

KV

Cache

RMSNorm

Attention

RMSNorm

FFN

𝐗

𝐘

The weights and activations between blocks in the transformer can be transformed using orthogonal matrices without altering the model's output. Specifically, if a rotation transformation R 1 is applied to the input activations, the effect can be offset by left-multiplying the weight matrices on the left side of the transformer block (such as W q , W k , W v , W up , W gate in Figure 8) by the orthogonal matrix R ⊤ 1 . To ensure the correctness of the residual computation, R 1 is right-multiplied with the output matrices (such as W o and W down in Figure 8). Although RMSNorm is applied between two blocks, this transformation remains valid as long as there is no rescaling within RMSNorm (in practice, any rescaling is typically absorbed into the adjacent weight matrices). Theoretically, this is because RMSNorm only normalizes the activations, and applying a rotation transformation does not alter the norm of the activations. As a result, the commutative property RMSNorm ( XR 1 ) = RMSNorm ( X ) R 1 can be established [33].

𝐗

x

𝑥

𝐑𝟏

T (α)

𝐖𝑞

𝐑𝟏

T (α)

𝐖𝑘

𝐑𝟏

T

α

𝐖𝑣𝐑𝟐

RoPE

MHSA

𝐑𝟏

T (α)𝐖 𝒈𝒂𝒕𝒆

𝐑𝟏

T

α 𝐖𝑢𝑝

𝜎

𝐑𝟒

T 𝐖𝑑𝑜𝑤𝑛𝐑𝟏

⊗

𝐗

KV

Cache

𝐗

𝐗

RMSNorm

Attention

RMSNorm

FFN

𝐗𝐑𝟏

𝐘𝐑𝟏

𝐑𝟐

T 𝐖𝑜𝐑𝟏

𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒

𝑑𝑒𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒

x

𝑥

𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒

𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒

𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒

𝐑𝟒

𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒

𝐑𝟑

𝐑𝟑

In addition to R 1 , the Transformer block can also incorporate R 2 , R 3 , and R 4 as shown in Figure 9. R 2 operates within the multi-head attention mechanism by being applied to each attention head. It is absorbed into W v to rotate the input of W o . To counterbalance the rotation effect introduced by W v , the transpose of R 2 , denoted R ⊤ 2 , is left-multiplied to W o .

R 3 is absorbed into the KV cache to alleviate the quantization loss within the KV cache. Due to the presence of RoPE, directly incorporating R 3 into the weight matrices is challenging, so R 3 is designed as an online Hadamard transform.

Finally, R 4 is a rotation matrix used to smooth the input to the down-projection. Given the gating mechanism, its rotational component cannot be fused into W up or W gate , thus it is also designed as an online Hadamard transform. However, the inverse transformation can be combined with W down , avoiding the computational overhead of the inverse operation.

It is important to emphasize that the inference framework of DartQuant (Figure 9) is identical to the framework proposed by SpinQuant [26], resulting in the same level of inference acceleration.

B Comparison of Calculation Amount

B.1 QR Decomposition

The Householder QR decomposition for an n × n matrix A can be expressed as follows:

Algorithm 2 Householder QR Decomposition

1: Input: Matrix A ∈ R n × n

2: Output: Orthogonal matrix Q ∈ R n × n , Upper triangular matrix R ∈ R n × n

3: Initialize R = A

4: for k = 1 to n -1 do

5: Set the vector x = R k : n,k

6: Compute the Householder vector v = x + sign ( x 1 ) ∥ x ∥ 2 e 1

7: Normalize v : v = v ∥ v ∥ 2

8: Update R : R k : n,k : n = R k : n,k : n -2 vv T R k : n,k : n

9: Update the orthogonal matrix Q : Q k : n,k : n = Q k : n,k : n -2 vv T Q k : n,k : n

10: end for

11: R n,n is the upper triangular matrix, Q is the orthogonal matrix.

Let us analyze the computational complexity of each step in the above pseudocode:

Step 6&7: Computing the Householder vector v :

-The operation involves computing the norm ∥ x ∥ 2 , which takes ( n -k +1) .

-Computing v involves adding two vectors and normalizing, which is also ( n -k +1) .

Step 8: Updating R :

-The update of R requires multiplying a vector v by a matrix slice R k : n,k : n , which takes ( n -k +1) 2 .

Step 9: Updating Q :

-Updating Q involves a similar computation to updating R , so it also takes ( n -k +1) 2 .

We now aggregate the computational complexity of all steps across the iterations. The overall computational complexity is given by:

B.2 Cayley SGD

Algorithm 3 provides the detailed computational process of the Cayley SGD. Compared to standard SGD, it introduces additional computations in steps 5 through 12. These computations are primarily composed of matrix multiplications.

Algorithm 3 Cayley SGD with Momentum

1: learning rate l , momentum coefficient β , ϵ = 10 -8 , q = 0 . 5 , s = 2 2: Initialize X 1 as an orthonormal matrix; and M 1 = 0 3: for k = 0 to T do 4: M k +1 ← βM k -G ( X k ) 5: ˆ W k ← M k +1 X ⊤ k -1 2 X k ( X ⊤ k M k +1 X ⊤ k ) 6: W k ← ˆ W k -W ⊤ k 7: M k +1 ← W k X k 8: α ← min { l, 2 q/ ( ∥ W k ∥ + ϵ ) } 9: Initialize Y 0 ← X + αM k +1 10: for i = 1 to s do 11: Y i ← X k + α 2 W k ( X k + Y i -1 ) 12: end for 13: Update X k +1 ← Y s 14: end for

· Step 5: Computing the auxiliary matrix ˆ W k

-The first part involves computing M k +1 X ⊤ k , which requires a matrix multiplication with complexity n 3 .

-The second part involves the term X k ( X ⊤ k M k +1 X ⊤ k ) . To optimize this computation, the matrix multiplication X ⊤ k M k +1 X ⊤ k requires n 3 , and multiplying this with X k results in a total complexity of 2 n 3 due to the nested operations.

· Step 7: Momentum projection update

-The update M k +1 = W k X k is a matrix multiplication, which takes n 3 .

Step 9 to 11: Iterative Cayley transform

-Each iteration involves matrix addition and matrix multiplication, which has a complexity of n 3 .

-Since there are s iterations, the total complexity is 2 n 3 .

In conclusion, Cayley SGD incurs approximately an additional 6 n 3 in computational complexity compared to standard SGD.

C Complete Results of Main Result Table

In Tables tables 6 to 15, we present the full results of Table 2. We report perplexity (PPL) scores on the WikiText2 [38], C4 [39], and PTB [40]. Additionally, we assess model performance on nine zero-shot evaluation tasks: LAMBADA [41], HellaSwag [42], PIQA [43], WinoGrande [44], OpenBookQA [45], SIQA [46], MMLU [47], ARC-E, and ARC-C [48]. We compare our approach with several other methods, including SmoothQuant [23], GPTQ [49], OmniQuant [19], and current state-of-the-art methods such as Quarot [25] and SpinQuant [26] for weight and activation quantization.

D Sample Size Analysis

Table 16 investigates the impact of sample size on DartQuant's performance. All experiments are performed with a token sampling ratio of 10%. The results show that DartQuant's calibration performance remains robust even with a small dataset.

E Comparison with Mixed Precision Quantization Methods

For a more comprehensive comparison, we selected two mixed precision quantization algorithms, QUIK [51] and Atom [52], and compared them with DartQuant under the 4-4-16 bit setting. It is important to note that, for a fair comparison, we preserved QUIK's feature of protecting the first 256 outlier channels, while quantizing the down projection to 4 bits to ensure consistency with our method. Atom was tested using its default settings. The specific experimental results are shown in Tables 17 and 18.

Even though DartQuant strictly quantizes all activations and weights to 4 bits (resulting in a lower average bit-width compared to QUIK and Atom), it still achieves accuracy improvements on most datasets. This underscores the effectiveness of our method.

(a) Layer 0

(b) Layer 0

(c) Layer 10

(d) Layer 10

(e) Layer 30

(f) Layer 30

(g) Last Layer

(h) Last layer

12000

三

SpinQuant

Hadamard

0.35

SpinQuant

Hadamard

Outlier Count

10000

OSTQuant

DartQuant

Quant

0.30

0.25

OSTQuant

DartQuant

8000

0.20

6000

Average

0.15

4000

0.10

2000

0.05

0.00

Llama 2-7B

Llama 2-13B

Llama 2-70B

Llama 3-8B

Llama 3-70B

Llama 2-7B

Llama 2-13B

Llama 2-70B

Llama 3-8B

Llama 3-70B

Models

Models

20000

Hadamard

SpinQuant

0.35

Hadamard

SpinQuant

Count

15000

17500

OSTQuant

DartQuant

Quant

0.30

OSTQuant

DartQuant

0.25

12500

0.20

10000

Average

0.15

7500

5000

0.10

2500

0.05

0

0.00

Llama 2-13B

Llama 2-70B

Llama 3-70B

Llama 2-13B

Llama 2-70B

Llama 3-8B

Llama 3-70B

Models

Models

Hadamard

0.35

Hadamard

20000

SpinQuant

Loss

SpinQuant

OSTQuant

0.30

OSTQuant

Outlier Count

15000

DartQuant

Quant I

0.25

DartQuant

0.20

10000

Average

0.15

0.10

5000

0.05

Q

0.00

Llama 2-7B

Llama 2-13B

Llama 2-70B

Llama 3-8B

Llama 3-70B

Llama 2-13B

Llama 2-70B

Llama 3-8B

Llama 3-70B

Models

Models

Hadamard

0.35

Hadamard

20000

SpinQuant

SS07

SpinQuant

OSTQuant

0.30

OSTQuant

Count

15000

DartQuant

Quant

0.25

DartQuant

0.20

10000

Average

0.15

0.10

5000

0.05

Q

0.00

Llama 2-13B

Llama 2-70B

Llama 3-8B

Llama 3-70B

Llama 2-7B

Llama 2-13B

Llama 2-70B

Llama 3-8B

Llama 3-70B

Models

Models

Llama 2-7B

Llama 2-7B

Llama 3-8B

Llama 2-7B

Llama 2-7B

(a) 2-7b layer 0

(b) 2-7b layer 9

(c) 2-7b layer 19

(d) 2-7b layer 29

(e) 2-7b layer 31

(f) 2-13b layer 0

(g) 2-13b layer 9

(h) 2-13b layer 19

(i) 2-13b layer 29

(j) 2-13b layer 39

(k) 2-70b layer 0

(l) 2-70b layer 19

(m) 2-70b layer 39

(n) 2-70b layer 69

(o) 2-70b layer 79

(p) 3-8b layer 0

(q) 3-8b layer 9

(r) 3-8b layer 19

(s) 3-8b layer 29

(t) 3-8b layer 31

(u) 3-70b layer 0

(v) 3-70b layer 19

(w) 3-70b layer 39

(x) 3-70b layer 69

(y) 3-70b layer 79

350

350

350

009

-008

300

300

300

500

700

250

250

250

009

400

200

200

500

150

150

150

300

400 -

100

100

100

200

300

200

50

50

50

100

100

0

-8

Q

2

0

5.02.50.02.5

5.0

7.5

10.0

12.5

7.5

5.02.5

0'0

2.5

5.0

7.5

10.0

0

-5

0

5

10

15

20

25

-10

0

10

20

OE

800

160

140

400

700

500

1000

120

600

400 -

800

100

300

500

300

600

80

60

200

300

400

200

400

40

100

200

100

200

20

100

Q

0

7.5 -5.0-2.50.02.55.07.510.012.5

-15

-10

-5

5

10

15

20

0

5

10

15

0

10

0

10

20

30

250

800

800

700

1750

1500

1200

200

600

600

1250

1000

500

800

150

1000

100

400

300

400

750

600

50

200

100

500

250

200

400

-10

-5

0

5

10

15

10

-5

O

10

0

-10

0

10

20

30

40

10

15

20

25

30

1200

250

300

009

009

1000

200

250

500

500

800

200

400

400

150

600

150

300

300

100

400

100

200

200

200

50

50

100 -

100

D

-20

-15

-10

-5

5

10

15

20

Q

6

7.5

5'2-0'5-

0.0

2.5

5.0

0

25

-20

-15

-10

5

0

-25

-20

-15

-10

5

0

5

600

1600

500

008

1400

800

500

400

1200

1000

600

300

400

009

500

700

300

200

800

009

400

200

400 -

300

100

200

400

200

100

200

100

0

-6

0

2

口

20

10

10

20

15

10

5

0

5

10

-10

-8

口

2

4

Q

10

5

0

10

F Effects of Different Transformations on Activation Outliers and Quantization Errors

We extracted 1000 activation samples from different layers of Llama-2 (7B/13B/70B) and Llama-3 (8B/70B) models and analyzed the number of outliers and the average quantization error after applying different transformation methods. The statistical results are shown in Figure 10. As observed, the activations after DartQuant transformation exhibit a significant reduction in outliers, with a marked decrease in quantization loss, further validating the effectiveness of DartQuant in large-scale language model quantization.

G Activation Distribution

We conducted an in-depth investigation into the distribution characteristics of activations in LLMs. First, we randomly sampled 1,000 activation samples from each model and computed their mean, variance and kurtosis. The detailed statistical results are presented in Table 19. As observed, the mean of activations is close to zero, the variance is approximately 1, and the activations exhibit high kurtosis (whereas the kurtosis of a Gaussian distribution is 0). These statistics indicate that most activation values are concentrated around zero and exhibit significant heavy-tailed properties.

Figure 11 presents the activation distribution histograms across different layers of various models. As shown, apart from a few outliers, the overall activation distribution is symmetric around zero. These distribution characteristics closely align with those of a Laplacian distribution. Therefore, we model the activation distribution as a simplified Laplacian distribution.

H Performance on MoE

𝐗

𝐗

RMSNorm

෩

𝐗𝐑𝟏

x

𝑥

𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒

Expert 0

𝐑𝟏

T (α)𝐖 𝒈𝒂𝒕𝒆

𝟎

𝐑𝟏

T

α 𝐖𝑢𝑝

𝟎

𝜎

𝐑𝟒

T 𝐖 𝑑𝑜𝑤𝑛

𝟎

𝐑𝟏

⊗

𝐑𝟒

𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒

𝐑𝟏

T

α 𝐖𝑟𝑜𝑢𝑡𝑒𝑟

𝐑𝟏

T (α)𝐖 𝒈𝒂𝒕𝒆

𝒏

𝐑𝟏

T

α 𝐖𝑢𝑝

𝒏

𝜎

𝐑𝟒

T 𝐖 𝑑𝑜𝑤𝑛

𝐧

𝐑𝟏

⊗

𝐑𝟒

𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒

Expert n

···

𝐘𝐑𝟏

𝐘𝐑𝟏

⊗

Weights

by Router

To demonstrate the adaptability of DartQuant to different model architectures, we also applied it to two MoE models (Mixtral-7×8B and DeepSeek-MoE-16B). The rotation matrix is integrated with the MoE architecture as shown in Figure 12. As shown in Table 20 and 21, DartQuant continues to demonstrate outstanding performance on the MoE architecture, with improved accuracy compared to QuaRot, which uses random rotations.

I Additional Ablation Analysis of Whip Loss

To further demonstrate the effectiveness of the Whip loss, we evaluated PPL and zero-shot accuracy under different loss functions. As shown in Table 22, the results highlight the advantages of Whip loss in both PPL and zero-shot accuracy.

J The Mechanism of Whip Loss

As shown in Figure 5, the smoothing of matrix peaks is not solely due to the amplification of small values near zero; it also involves a broader redistribution of the activation values.

Specifically, the Whip Loss we designed pushes small activation values away from zero, making them more evenly distributed, which in turn helps reduce quantization error. Under the norm-invariance constraint imposed by the rotation transformation, activation values are effectively redistributed: high-peak regions in the distribution histogram are compressed, while low-value regions are filled, leading to an overall "peak smoothing" effect.

Thus, the smoothing of matrix peaks is not merely a result of amplifying small values but also stems from the uniformization of the overall distribution-one of the key objectives of DartQuant.

The property of norm invariance originates from the mathematical characteristics of rotation matrices. Let W be a rotation matrix (i.e., an orthogonal matrix) satisfying W T W = I . For any input vector x , the transformed norm follows:

This confirms that rotation preserves the Euclidean norm of the vector.

o illustrate this effect, we provide a four-dimensional example explaining why amplifying small values under the norm-invariance constraint leads to outlier reduction. Consider a vector x = [ x 1 , x 2 , x 3 , x 4 ] , where x 1 , x 2 and x 3 have absolute values close to zero, while x 4 is an outlier. Introducing a perturbation ϵ i ( i = 1 , 2 , 3 , 4 . ) to each component, the new norm of the vector becomes:

Clearly, if ϵ 1 , ϵ 2 , ϵ 3 > 0 , it must follow that ϵ 4 < 0 . DartQuant effectively leverages this constraint to suppress outliers.

K Hyperparameter Settings

The specific hyperparameter settings for DartQuant are shown in Table 23. It is important to note that the latent parameter Z 0 is initialized using a random Hadamard matrix.

L Limitation

DartQuant is tailored for uniformly distributed integer formats. Its effectiveness on alternative formats, such as FP4, or other non-uniform numerical representations, remains to be further explored and validated. Furthermore, Whip assumes that activations are approximately zero-mean, which generally holds true for most transformer layers. However, in rare cases where the activation mean significantly deviates from zero, the effectiveness of Whip may degrade.

In the future, we can explore a wider range of distribution transformation methods to better accommodate various numerical formats.

M Impact Statement

This work introduces DartQuant, an efficient rotational distribution calibration method for LLM quantization. DartQuant simplifies the optimization of rotation matrices while avoiding the overfitting risks associated with end-to-end training. It successfully quantizes a 70B model on a single RTX 3090, marking a significant advancement. This progress not only improves the efficiency of LLMs in practical applications but also provides a practical solution for large-scale model quantization, contributing to the accessibility and scalability of AI technologies. However, if misused for harmful purposes, it could have a negative social impact.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDartQuant is a method for efficient rotational calibration in LLM quantization. By replacing costly end-to-end fine-tuning of rotation matrices with a distribution-aware calibration approach, DartQuant reduces the risk of overfitting and dramatically lowers computational and memory costs during compression.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \n1. The overall design of Whip loss makes sense. Because the norm is constant, reducing small-value would also reduce outliers. \n2. The use of autograd qr decomposition makes sense, which avoids the time-consuming Cayley SGD but still guarantee the orthogonality. But I'm not sure if the gradient of qr decomposition is stable or not.\n\nWeakness:  \n1. The main weakness is that the improvement is not significant. The performance is very marginal compared to SpinQuant. The improvement on compression time and memory is large. But I doubt whether compression time is an important objective in quantization, compared with time and memory during inference, or performance.\n2. In four rotation matrix R1 ... R4, only R1 and R2 are learned by DartQuant, while R3 and R4 use Hadamard rotations like previous work, which limits the value of this method. By the way, the Hadamard matrix of R3 and R4 is random, right?\n3. See \"Questions\".\n\nQUESTIONS:\n1. What is the metrics of outlier in Figure 3? It should be included in the article.\n2. Which baseline is used in Table 1?  It should be stated in the article.\n3. Line 283 states \"In the main results, we apply GPTQ to rotate the weights.\", but GPTQ doesn't have rotate. I guess it's \"reconstruct\"?\n4. Why could DartQuant avoid overfitting? It still use calibration data for both rotate and reconstruct (if above is true).\n5. Line 25 is wrong. lora doesn't improve inference efficiency.\n\nLIMITATIONS:\nLimitation in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-03"}

comments: SUMMARY:
The paper proposes a new approach for finding orthogonal transformations, which is needed for reducing the activation outliers during post-training quantization.

STRENGTHS AND WEAKNESSES:
Strengths

- The paper is well-written and easy to follow
- The idea is interesting
- The experiment setup is proper

Weaknesses

- The improvement is somehow marginal (please see the Question section)

QUESTIONS:
I have some questions and comments on this paper:

1. I found the idea interesting. The issue with the current scheme is that the improvement is a bit marginal even when we compare against QuaRot which doesn't do anything (no training). However, this is nothing with the paper, but more with the PTQ schemes. It seems that we hit the wall, considering the limits of the 4-bit compression. That's why we cannot expect large improvements in all these kinds of work.

2. Considering the above fact, I think the improvement would be even closer (compare to something like QuaRot) in more recent 4-bit formats. For this, can you please provide an experiment where you compare DartQuant with QuaRot (I do not suggest SpinQuant as I know that will take 4-5 hours to run) on NvFP4? I would improve my score if we get non-trivial improvement over that format.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 4
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose DartQuant, a post-training weight-activation-kv cache quantization method for LLMs, that propose a new layer-wise loss to learn the rotation matrices to flatten the activation distribution without end-to-end fine-tuning. This method reduces the quantization time (computational complexity) but achieves SoTA model performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- A good summary of related work on rotation-based PTQ methods for LLMs. This serves as a good background intro for readers who are not very familiar with this area.\n- A clear description of the method supported by sufficient explainations. The whip loss design is simple but effective.\n- Extensive experiments and ablation study to demostrate the effectiveness of the method\n\n## Weaknesses\nI did not see partitical weaknesses though I have a few questions about the details (See Question Section). \n\nIf will be helpful if the author show some inference latency/throughput/memory results even though it may be the same as other ratation-based methods like SpinQuant.\n\nQUESTIONS:\n1. Could the author clarify the quantization format for activations? I assume the weight matrix is quantized using groupwise scales and integer quantization. What about activations?\n2. The Whip loss is impressive. Does the Whip loss assume the activation distribution is symmetric over zero? What if the activation mean is not zero?\n3. In Algorithm 1, what does the function \"Row_sampling(X)\" mean?\n4. Could the author give more information about the objective in Algorithm 1 like variance across $Z$ initialized with different random seeds and learning rates? Is the problem easy to solve for gradient descent?\n\nLIMITATIONS:\nAs stated in the limitation section of the paper, DartQuan did not explore the number formats that may have native hardware support like FP4. INT4 activation is not natively supported by most of GPUs. Other concerns are reflected by my questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DartQuant, a post-training quantization method for large language models. The method aims to significantly reduce the computational cost associated with calibrating rotation matrices. DartQuant builds upon the computationally invariant framework and optimizes rotation matrices to improve uniform quantization. It consists of two key components:\n1. Whip Loss – a loss function designed to reshape activation distributions toward uniformity.\n2. QR-Orth – an optimization scheme using QR decomposition to enforce orthogonality of the rotation matrices.\n\nThe authors report a 47× speedup and 10× memory reduction, enabling the calibration of a 70B model on a single consumer GPU.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses a highly practical and relevant challenge: reducing the cost of optimizing rotation matrices during LLM quantization.\nCompared to prior methods such as SpinQuant, DartQuant offers substantial speed and memory benefits, making it more accessible for users with limited hardware. The paper provide a strong result on extreme qunatization scenario like 4 bit qunatization.\n\nOne of the biggest weakness is that all experiments are conducted exclusively on LLaMA models. The generality of DartQuant should be evaluated on other architectures like Mistral, Qwen, or Phi to demonstrate broader applicability. Moreover, adding experiments on mathematical reasoning tasks such as GSM8K, MathQA or longbench would strengthen the empirical evaluation and show the method’s performance on more challenging domains.\n\nAnother big weakness is the missing comparisons with closely related quantization methods, including some mixed-precision quantization method, e.g. at least [3, 4].\n\nAnother big weakness is the unsubstantiated dismissal of kurtosis-based methods. Reference [5] demonstrates the effectiveness of kurtosis as a quantization signal. Dismissing it without a rigorous comparison weakens the argument for Whip Loss. A head-to-head evaluation and deeper justification for the superiority of Whip Loss are needed. This comparison is key since with KurTail [5] we can also train the rotation matrices single consumer GPU similar to DartQuant.\n\nThe justification for the “Whip Loss” function is vague. It is said to be “inspired by” transforming Laplace to uniform distributions, yet the specific formulation, ∑exp(−|xᵢ|), is heuristic and lacks theoretical derivation. The paper does not explore alternative formulations or explain why this one is particularly effective at promoting uniformity.\n\nThe paper reports a 1.4× per-iteration speedup (Table 4) but also mentions a 41× total speedup (line 375) and a 47× training speedup (line 332). I suggest the authors to provide a section and ablation study to clarify and justify each speed up.\n\nThe QR-Orth optimization appears more effective than methods such as Cayley optimization, which contradicts [1]. The paper does not justify its choice or acknowledge this performance gap. The slower convergence rate in Caley methods may be due to suboptimal hyperparameter selection. Moreover, QR-Orth is not a novel method. The QR-Orth scheme is well-known in manifold optimization. The current presentation can be potentially misleading and be read as a novel contribution. The authors should cite foundational works,\nsuch as Optimization Algorithms on Matrix Manifolds [2].\n\nMinor comment: The exact optimization process for rotation matrices is unspecified. The details of how the rotation matrices are optimized are under-explained. For instance, the specific optimization process for Rotations is not described. Further the paper just claimed that they can optimise llama 70B on 1 GPU but running a llama 70B in fp16 needs 120GB for inference. Do they use a larger gpu to save the activation or they used methods like offloading to run their experiment clarifying the optimization process would help paper clarity.\n\nFigure 11 presents the distribution before rotation is applied. Including the post-rotation distribution would help demonstrate the effectiveness of DartQuant more convincingly.\n\nReferences:\n1. Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley\nTransform, ICLR 2020\n2. Optimization Algorithms on Matrix Manifolds, Princeton University Press, 2009.\n3. ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank\nResiduals, ICML 2025\n4. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger\nQuantized LLMs (Neurips 2024)\n5. KurTail: Kurtosis-based LLM Quantization, SLLM workshop ICLR 2025\n\nQUESTIONS:\n1. Can you explain the better convergence rate of QR-orth method compare to Caley optimization?\n2. Can you compare to [5] which should be easily doable within the rebuttal period, and [3,4] if you have additional time? (See strengths  and weaknesses)\n3. Can you add more models (Mistral, Qwen, or Phi) and mathematical reasoning tasks such as GSM8K, MathQA?\n\nLIMITATIONS:\nThe only limitation mentioned is \"tailored for uniformly distributed integer formats\". Is there no other limitations?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}