MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization
Authors: Aozhong Zhang
Naigang Wang
Yanxia Deng
Xin Li
Zi Yang
Penghang Yin
Venue: NeurIPS 2024 poster (2024)
Keywords: post-training quantization
model compression
$\ell_\infty$-regularization
Primary area: other

Abstract: In this paper, we present a simple optimization-based preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post-training quantization. For each linear layer, we adjust the pre-trained floating-point weights by solving an $\ell_\infty$-regularized optimization problem. This process greatly diminishes the maximum magnitude of the weights and smooths out outliers, while preserving the layer's output. The preprocessed weights are centered more towards zero, which facilitates the subsequent quantization process. To implement MagR, we address the $\ell_\infty$-regularization by employing an efficient proximal gradient descent algorithm. Unlike existing preprocessing methods that involve linear transformations and subsequent post-processing steps, which can introduce significant overhead at inference time, MagR functions as a non-linear transformation, eliminating the need for any additional post-processing. This ensures that MagR introduces no overhead whatsoever during inference. Our experiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models. For example, we achieve a Wikitext2 perplexity of 6.7 on the LLaMA2-70B model for per-channel INT2 weight quantization without incurring any inference overhead.

MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization
Aozhong Zhang' Naigang Wang² Yanxia Deng' Xin Li' Zi Yang' Penghang Yin
University at Albany, SUNY IBM J. Watson Research Center (azhang3, ydeng5, x1i48, zyang8, pyin)0albany..co edu
nwanguus ibm. com
Abstract
In this paper, we present a simple optimization-baser preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post- training quantization. For each linear layer, adjust the pre-trained floating-point weights by solving a channel-wise -eeruallard optimization problem. This process greatly diminishes the maximum magnitude of the weights and smooths out outliers, while preserving the layer's s output. The preprocessed weights exhibit reduced range, which facilitates the subsequent quantization process. To implement Magk, we address the la regulllaariit by employing an efficient proximal gradi- ent descent algorithm. Unlike existing preprocessing methods that involve linear transformations and subsequent post-processing steps, which can introduce signifi- cant overhead at inference time, MagR functions as a non-linear transformation, eliminating the need for any additional post-processing. This ensures that MagR introduces no overhead whatsoever during inference. Our experiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models. For example, we achieve a Wikitext2 perplexity of 5.95 on the LLaMA2- 70B model for per-channel INT2 weight quantization without incurring any inference overhead. The code is available at bttps://gthbbboom///oooo//////g/
Introduction
Large language models (LLMs) have achieved outstanding performance across a broad range of applications, demonstrating remarkable success. However, their unprecedented model size has led to many computation operations and substantial memory footprints, becoming significant barriers to their practical deployment and adoption in production environments. Accordingly, it is highly desirable to develop efficient model compression techniques for LLMS so they can be more widely deployed in resource-limited scenarios. Among the various techniques
compress and accelerate deep neural networks (DNNs), low oo-precciioon quantization has proven to be highly effective across numerous application domains and is widely adopted for accelerating DNNs. For LLMs, the inference runtime dominated by the token generation process, where output tokens are produced sequentially, one at time. This process is known to be memory bandwidth bound B 19). As result, the quantization of LLMs has primarily focused on reducing the bit-width of model weights, with the dual goals of lowering the model's footprint to enable deployment resource- cow.rreennntttiieddddeeeeeeeeedd devices and decreasing the memory bandwidth requirements to improve computational efficiency and accelerate inference.
The enormous computational demands for pre-training and fine-tuning Large Language Models (LLMs) have led to the emergence of Post- Bost-Trainin Quantization (PTQ) [4] 15
22
24 27 B1
51 52 53 43 40), promising solution for quantizing these models. Unlike Quantization
Aware Training (QAT) 7 12 1
21
23 46 47 48 49), which designed to minimize a global
training loss for quantization parameters, PTQ directly applies low-precision calibration to a pre-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

trained full full-prrcision model using a minimal set of calibration samples. By aiming to identify an optimal quantized model locally through the minimization of a simplified surrogate loss, PTQ offers computational savings and resource efficiency compared to QAT. However, PTQ often lags behind QAT in accuracy, particularly for ultra-low precision lower than 4-bit. Thus, remains an open problem to achieve an improved balance between cost and performance for P'QQ-based approaches.
Motivation. To achieve state-of-the thrt performance, the latest advances in PTQ 8 25
26
36 42
have proposed applying a linear transformation to process the pre-trained weights within a linear layer. This strategy of linear transformation aims to make the weights more suitable for the subsequent
quantization procedure by reducing their magnitudes and suppressing outliers. In a nutshell, given the features X and weights W. one constructs linear transformation T such that TW is better conditioned than W in terms of being quantization-ffient matrices (so-called channel-wise scaling) 25 36 42), random transformations g 39] and finite
Such designs of T include diagonal
frames 13.. Then, quantization is performed on TW instead of the original weights W. To preserve the layer's output, however, the inverse transformation T has to be in turn applied to the features X, namely,
XW = (XT (TW) = (XT ()(W))
with Q(TW) being the quantized weights. PTQ done this way requires modifications on the original neural architecture, which involves additional computations of XT and extra memory storage for T=! at inference time. As a result, these steps introduce overhead that offsets the benefits provided by quantization. This raises natural question:
Can we effectively process the weights at the preprocessing stage to facilitate quantization without introducing inference overhead?
To address this problem, we propose a simple optimization-enn -based technique called Weight Magnitude Reduction (MagR). MagR functions non-linear transformation on weights without altering the original features/activatioos.. The optimization program is designed to find new weights with minimal maximum magnitude, i.c., the &o norm, while preserving the layer's outputs.
Contributions. We propose a non-linear approach, MagR, based on channel-wist foo -eggulauar least squares, to reduce the quantization scale without compromising the performance of pre- trained model, facilitating subsequent weight quantization while requiring no post-processing or inference overhead. See Figure [ for comparing weight magnitudes before and after applying MagR. To address the -ee-reeulaiiiztiinn problem, we develop an efficient and parallelizable proximal gradient descent algorithm that involves computing -ball projections at each iteration. Specifically, MagR preprocessing on single Nvidia A 100 GPU takes merely 15 min for LLaMA2-7B and 3.5 hr for the 70B model. Our results on INT weight-quanntzztioon demonstrate that MagR can significantly boost the performance in the sub-4bit regime when combined with fast gradient-fres methods for layer-wise PTQ, such as rounding-to-nearee (RTN) [30] and OPTQ [6). This approach achieves performance for weight quantization at least comparable to state-of-the ofehe-----e-arr art PTQ methods on natural language processing (NLP) tasks, including gradient-basee based methods using block-wise reconstruction.
2 Related Work
Recently, as the sizes of language models are exploding, there has been growing interest in developing
post-training quantization (PTQ) methods [8] 16 25 26
36
44 45 for large-scale AI models like
large language models (LLMs) to reduce the model sizes and accelerate inference by representing weight matrices in low precision. PTQ methods directly find the low-precision representation of the model without re-training, thereby preferred by extreme large-scale AI models. The OPTQ [16) uses approximate second-order information to calibrate the quantization. The method successfully compresses LLMs into 3 or 4 bits and can achieve reasonable accuracy in bits. Researchers have found that the extreme values and the distribution of the weight entries highly affect the quantization errors and the quantized model quality. The original weight can be converted into a more quantization- friendly one by linear transformations, The approach can significantly reduce the quantization errors while bringing more time overhead during inference because of the linear transformation. OmniQuant (36] proposes learnable weight clippings and equivalent transformations to avoid the influence of extreme values. AWQ (25) searches for the most significant entries in the weight by looking at the activation and selects the scales that protect these entries. SmoothQuant 44] passes the difficulty

---

Layer 42
Layer 119
Layer 210
0:2
Columr arise Max Magritude After
0.25
Colunrr-wiis Max Magnitude After
Coluirreette Maqritude
0.33
0.10
0.10
0.30
0.05
0.0
0.05
0.0
015
0.00
0.5 320
0.00
0.03
Columnn-wse Max Magnituda Betore Magit
Column-wiss Max Magnitude Betore Magi
Calumn-wise Max Magnituda MagR
effectively reduce the magnitude of weights the preprocessing stage. Each point denotes the maximum magnitude before (x-coordinate) and after (y-coordinate) applying MagR within sampled channel (or column) of the weight matrix from three random layers of LLaMa2-7B [38] These column-wise maximum magnitudes are typically more than halved through Magk.
Figure 1: Motivation behind MagR:
in activation quantization weights by equivalent linear transformation. QulP R AffineQuant (26 and FrameQuant [ apply a linear transformation hefore quantization to make the transformed These approaches achieve high performance for extreme bits, like bits, but introduce additional inference overhead though the transformation is carefully designed to be efficient. OmniQuant (36] and AffineQuant (26) can be adopted for weight-activation quantization by considering the activations in the proposed methods. The work [5] introduces a low-rank compensation method on top of other quantization methods, which employs low-rank matrices to reduce quantization errors with minimal increase in model size. By modeling the quantization 2 proposes applying l penalty on the gradient of loss
weight coantizaatio-frrennne
residual as an & bouunde perturbation, to enhance quantization robustness.
The works most closely related to ours constrain the weight range to a smaller scale. The Range Regularization (R²) method (20] applies an 6oo penalty or variants to the conventional network loss regularize the weight range during end-to-end model pre-training, optimized via SGD. However, this approach becomes practically infeasible for lcage models. In (27). layer-wise pre-processing technique is proposed, which involves solving an intractable Eu mi-mimimization problem while constraining the foc -nom of weights.
20 and 27 both utilizing foe norm regularize
3 Background
First, we clarify the mathematical notations that will be used throughout this paper:
Notations, We denote vectors by bold small letters and matrices by bold capital ones. For a positive integer n, [n]] = {1, (1.2.... n denotes the set containing all positive integers up to For any two vectors x, y _8" (x. y) Σ Jzy is the inner product. We denote by || || v /x, x)
V E 2 the Euclidean norm; | := Ei= |x | is the & no-mo ||x||c := maxişisn [xr]] the t. -nomm. For any matrix X E |Rmx" X E Ruxm is the transpose. We denote the spectrum norm of X by X o maxx X). which equals its maximum singular value. Its Frobenius norm given
by X || E Σラ X Moreover, for vectors and y, ==
R
denotes the Hadamard or element-win wise product, and likewise for two matrices.
Layerwise PTQ. Post-training quantization via layerwise reconstruction calls for solving least squares problem with a discrete constraint. For the pre-trained weights W within a linear layer, we aim find the quantized weights W, that minimize the following function
min W,EQ
[XW, XWI
where X E (bbl))xm is the feature matrix associated with batch of calibration data consisting of samples stacked together, and each data sample represented by an x m sub-matrix. Q C m an appropriate set of all feasible quantized weights.

---

The most straightforward PTQ technique, known as RTN, involves directly rounding the weight matrix W without utilizing any additional data. An improvement over RTN was introduced by AWQ
25
which enhances the quantization process by incorporating channel-wis scaling on W.
Thanks the simplicity of the layer-wise formulation ① several efficient gradient-free algorithms 4
56 [3 have been recently proposed to address layer-wise quantization, including OPTQ. Built on top OPTQ, QulP subjects X and W random orthogonal transformations produce "incoherent" weight and Hessian matrices, leading to superior accuracy with sub-4bit quantization. However, this advantage comes with trade-off: during inference, QulP requires random orthogonal transformations on the feature inputs of linear layers, rendering noticeably slower throughput compared to OPTQ.
Uniform Quantizer. Given set of points
R" the commonly used (asymmetric) uniform (esflwwwmmiill min(w) and zero-point 2 min( and it
quantizer [9] defines the quantization step ò max w
2
quantizes onto the scaled integer grids Q {= 6, (= ....., (2 + (2 1)) 6) follows: w 0, 0. -)) +2)
w4 = 5 (clamp (1=²
In per-channel (or per-group) PTQ, the quantization step is conventionally calculated based on the channel-wise (or group-wise, respectively) minimum and maximum values of the pre-trained weights W, defined above, and remains constant throughout the quantization procedure.
4 The Proposed Method
In this section, we present the Weight Magnitude Reduction (MagR) method based on & norm regularization, which is applied just before the quantization step within each linear layer. The intuition behind MagR is based on the following simple estimate of the layer-wise quantization error. Given the feature/activation matrix X, the quantizer 2, and any pre-trained weights Ihm have:
min Xw, - Xwi < [ - w))l < X () - w| S Tmax (X) (XVm M
5,
2
where 5 = max(w)
is the quantization step size. This shows that reducing the range of
2-
weights helps to suppress the quantization error. With this in mind, MagR preprocessing is designed achieve two key effects:
First, effectively reduces the channel-wise (or column- wise) maximum magnitude of the weights, illustrated by Figure
Second, it preserves the model's original performance with minimal accuracy loss after preprocessing. Table demonstrates that MagR preprocessing maintains the perplexity of the pre-trained models, with only minor degradation.
Table 1: comparison perplexity (PPL) for the original pre-trained and the LLaMA2 models.
Model
Method
Wikitext2 (PPL.)
C4 (PPL))
Original
5.47
6.97
LLaMA2-7B
MagR
5.52
7.04
Original
4.88
6.46
LLaMA2-13B
MagR
4.92
6.52
Original
3.31
5.52
LLaMA2-70B
Magk
3.35
5.56
4.1 Approximately Rank-Deficient Feature Matrix
To illustrate the idea behind the proposed MagR method, let us consider pre-trained weight vector ii e Rm of a linear layer and the associated feature input matrix X. MagR leverages the fact that the feature matrix X across all layers of LLMs is approximately rank-deficient. Specifically, if X is exactly rank-deficient, the linear system modeling the layer's output, X w X i with variables w, generally has infinitely many solutions. That is, for any v in the non-trivial kernel space of X, we have that ur = it + preserves the layer's output. Among all solutions, MagR aims identify the weight vector w with the smallest extreme value in magnitude.

---

Table 2: The statistics of (approxinate) fraction ranks in percentage (%) of feature matrix X across all layers of LLaMA models. All feature matrices are approximately rank-deficient with fraction rank less than 100%. Some of them are highly low-rank with fraction rank 2 1%.
Model
Min
Max
Mean
25% Percentile
75% Percentile
LLaMAI
7B
0.2
99.07
70.41
65.09
81.80
LLaMAI-II 13B
1.42
99.90
83.85
75.07
96.71
LLaMA1-300 1-30
0.73
99.85
84.40
79.76
99.46
LLaMA1-65B
1.17
99.90
83.11
82.76
98.71
LLaMA2-7B
0.1
99.95
76.83
67.71
91.02
1MMA2113B
0.44
99.76
78.30
66.54
98.58
LLaMA2-70B
99.71
81.55
74.90
99.56
In [8] the authors empirically observed that the Hessian matrix X X is approximately low-rank across all layers in open pre-trained (OPT) models [54]. Here we examined the feature matrix of
LLaMA models [7 38 Our approximate fraction rank of the feature matrix X is defined the fraction of singular values X such that o (X) > 0.01 - max (X) Table illustrates that all feature
matrices extracted from LLaMA models are indeed rank-deficient according this definition.
4.2 MagR via Lo0 Regularizzaiin
Let us consider the quantization of a weight vector for simplicity. Given pre-trained weight vector w, would like to find new set of weights with the smallest maximum magnitude, such that the layer output is preserved up to a small error s > 0, i.c.,
min |00|
subject to
| - X S.
To efficiently implement MagR, we consider the following mathematically equivalent fx- regularization problem instead:
min wER
Xw Xwol + al/2w|1
(2)
where a > 0 serves as the regularization parameter, balancing fidelity against the foo regularizer. To maintain the output of the layer, a should typically be minimizer of (2). we have that the lg error of the layer's output introduced by MagR is O(va):
small value. Indeed, let wn be the
Xw"" Xwi S v X w"" X w + 2a w 1oo
S / X X w 1|2 2a
Wa//w// = /2ae ( 1oo1
where ||00|/00 is a constant independent of a, and the second inequality that w" is the minimizer.
Proximal Gradient Descent. Note that -orm is convex but non-differentiable function. In theory, the optimization problem (2) can be simply solved by a subgradient algorithm, but it is significantly slower than the more sophisticated proximal gradient algorithm which matches the convergence rate of standard gradient descent.
With the step size vj O, proximal gradient descent
32 takes the following iteration:
w k+1 = prox ya b0
4 - V w
X - X x
=prox |-
nh 17 X
X w))
(3)
where proxt]|- with the scalar > O is the (scaled) proximal operator foc -norm function, defined
proxe]J-llas (v) == arg min 2ER マ 1 w1
+||0=/ooo
To ensure the convergence of 3 it is sufficient to choose the step size
11 X (X x)
5

---

where Amax (X X) is the maximum eigenvalue of X x X.
Proximal Operator of -Norm. It remains to determine the proximal operator of t:c -nrmm. turns out we can compute it by leveraging the celebrated Moreau decomposition 29 for any t> 0,
| 1 (v) = - proj H
(4)
That is, computing the proximal operator of &oo norm amounts to evaluating the projection onto I ball, which defined as
proj = arg min || v/ subject
||/|| 1
#ER
Fortumately, computing projection onto the & ball is established task, and there are several efficient algorithms available. For example, see [11] and the references therein. Here we adopted a simple algorithm of (m log m) time complexity implementation for the projections of a batch of weight vectors, i.e., a weight matrix, as will be described in the next subsection. The implementation mainly involves sorting and soft-threshodding [50]; see Algorithm 3 and its derivation in Appendix A.] for the details.
in [14], which supports parallelizable vectorized
MagR for Weight Matrix. In practical implementation of MagR, we preprocess the entire weight matrix W [w...... wn E R TM x within each linear layer. For per-channel quantization (or per-column quantization in setting), the lo penalty is imposed column- wise the weight matrix to reduce the quantization scale of each channel. That is, MagR amounts to solving
min WER
XW - XW +a
W
2
In this case, take the following iteration:
w+++1 = prox 1-
( - X xW*
where the proximal operator prox and the corresponding projection proji || s1 in (4) are applied column-wise the matrix input. Hereby we summarize MagR for processing one linaa layer in Algorithm with the column-wise -hall projection as detailed in Algorithm which generalizes
Algorithm Appendix A. by handling matrix inputs (or batches vectors).
Algorithm 1
Per-channel Magk for one linear layer.
Input: Pre-trained weight matrix W e Rm Hessian matrix H X X E I mx max iteration number K; step size Xmax. Hi penalty parameter a 0.
Output: Preprocessed weights W E R
1: Initialize V = W.
2: for k O....,
3: V W - 17. H(W* * W)
gradient descent step
4: Wh+ Vk na projiilisi
V
proj (h 1 is described in Alg.
7a
5: end for
6: return W W
Extension to Per-Group Quantization. By using more float scaling factors, per-group quantization becomes a preferred strategy for mitigating accuracy loss at extremely low bit-widths. In this approach, weight vector R" is segmented into groups of weights, each containing d elements, with all weights within a group sharing a common scaling factor for quantization. Here, per-group MagR applies fx penalty to each vector of grouped weights. Consequently. the -all projection is independently performed on these vectors, while maintaining the gradient descent step unchanged. We note that the group-wise balll projection can be easily done using Algorithm with an additional reshaping of the input V e mx into dx -n)..

---

Algorithm 2 Column-wise projection onto the unit E1-ball.
Input: Matrix V E Rm the radius of & ball, =
Output: W E IRmx* such that all columns ||æl S e, V. E [n].
1: Create binary mask M E Rmxx" filtering out the columns of V with ||5l|| sc.
2: Sort [V] column-wise in descending order into U.
3: Find index Pj = max i E [m]] u. > 1 EEi= uns - Vj
4: Define 0g = 1 (Σi= Whi - vjj E [a]
Pj
5: Tile 0 R into O E gmxx along the row.
6: Compute W = (1 - M) @ V + M @ VV) @ | -,,0}
7: return W
5 Experiments
Overview. We tested the proposed Magk for INT4, INT3, and INT2 weight quantization. In our notations, the weight and activation bits are denoted by 'W' and 'A', respectively. Additionally, we implemented group-wise weight quantization with the group size denoted by 'g'. For example, W2A16g 128 signifies INT2 weight and FP16 activation (i.e., INT2 weight-only quantization) with group size of 128.
We employed our MagR processing approach on top of the two gradient-free PTQ methods in main text, RTN and OPTQ [16]. to quantize the LLaMAl (7B-65B) 33 and LLaMA2 (7B-70B) (38) model families. In the Appendix A.2 we extend MagR with QulP (8) QMagR+QuIP) to quantize LLaMA2 (7B-70B) model families. By applying MagR on top of RTN (MagR+RTN), we achieved better results than AWQ [25 for per-channel INT3 and INT4 weight quantization. Additionally, Magk combined with OPTQ (MagR+OPTQ) achieved tate-offthhe-rr performance for INT3 and INT4 qunatization. To enhance the per-channel INT2 quantization, we ran 30 additional iterations of coordinate descent algorithm 51 on top of OPTQ. which we denote by MagR+OPTQ'. It turns out MagR+OPTQ' is superior to both Omniquant (36] and QuIP g in terms of perplexity (Table 9. and falls just short of QulP zero-shot tasks for 13B and 70B models (Table 4 Note that QulP uses random orthogonal transformations (so-called Incoherence Processing) to process both the weights and features, resulting in 1.5x slower throughput than OPTQ. In contrast, MagR- based method does not introduce any overhead whatsoever compared with OPTQ.
In conclusion, our MagR- based PTQ method intuitive yet effective in compressing models into extreme bit-widths, while maintaining performance without introducing any inference overhead.
Datasets and Evaluation. Following the previous work [66 2536) evaluate the quantized model on language generation tasks on WikiText2 (28) and C4 33 Additionally, we test its performance on zero-shot tasks, including PIQA (5). ARC (Easy and Challenge) [10, and Winogrande (35] For the language generation experiments, our implement is based on the OPTQ's [6] repository, which built using PyTorch. For executing all zero-shot tasks, we adhere to the an-eval-harnee [17].
Baseline: For the language generation task, we compare our method with RTN, OPTQ [66,, AWQ 255 and OmniQuant (36] on LLaMAI and LLaMA2 models. In addition to the aforementioned methods, we also conduct a comparison with QulP g on the LLaMA2-70B model. In the zero-shot task, we focus on four individual tasks and compare the average accuracy across all four tasks with Omniquant [36].
Implementation details. We utilized the HuggingFace implementations of the LLaMA1 and LLaMA2 models and perform quantization on a single NVIDIA A A100 GPU with 80GB of memory. Following the OPTQ method, we load one block consisting of 7 linear layers into GPU memory at a time. In line with previous work [8] [6), the input matrix X is obtained by propagating the calibration data through the quantized layers.
The choice of parameters. To ensure that the MagR processde layer output XW is faithful to the original X W. we need to use a tiny penalty parameter a in (2). For per-channel quantization, a was fixed to be 10-3 in our experiments, but we did find that setting it to smaller value of 5 x 10-1 or 10-4 can sometimes slightly improve the perplexity (with relative change of < 1% ppl). Similarly for per-group quantization, we set to 10-1, while reducing it to 5 x 10 10-5 could
7

---

sometimes also slightly improve the perplexity. An ablation study on a is provided in the Appendix
A.2
Furthermore, we used a multiplicative scalar 3 < to decay the standard quantization step å = maxt wr min(w) (or equivalently, the quantization scale) of the quantizer. In other words, our d B
100000 m
minn(w) (w)
It has been shown in existing works 21 that, optimal quantization step for binary
or ternary quantization yielding the minimum quantization error is not given by max(w maxxw))min((() 2--1
Shrinking 8 at low bit-width results in more clustered quantization grid lattice that fits the weights better, which leads smaller overall error. general, / positively correlated with the bit-width used. For per-channel quantization, the best / [0.8, 0.85) on INT2 quantization, whereas the empirically optimal around 0.9 for INT3 quantization. As for INT4, 3 is simply set to 1, that is, chose
used the standard quantization step. In addition, for per-group quantization, for both INT2 and INT3 quantization. The ablation study of B is in the Appendix [A.2] We observed that this refinement the quantization step 5 significantly improves the performance of the PTQ method. In addition, the iteration number K Algorithm was set to 150 across all the experiments.
0.95
Table Perplexity quantized LLaMA2 models on Wikitext2 and C4. We report WikiText2 and C4 perplexity in this table. LLaMAI resutls can be found in the Appendix.
Datasets
Wikitex12
C4
LLaMA PPL4
2-7B
2-13B
2-70B
2-7B
2-13B
2-70B
FP16
Baseline
5.47
4.88
3.31
6.97
6.46
5.52
OPTQ
7.7c3
2.1e3
77.95
NAN
323.12
48.82
OmniQuant
37.37
17.21
7.81
90.64
26.76
12.28
W2A16
QulP
27.13
10.09
6.33
31.33
13.13
8.94
MagR+OPTQ
16.73
11.14
5.95
23.73
14.45
8.53
OPTQ
36.77
28.14
33.70
20.97
W2A16
g128
OmniQuant
11.06
8.26
6.55
15.02
11.05
8.52
MagR+OPTQ
9.94
7.63
5.52
14.08
10.57
8.05
RTN
539.48
10.68
7.52
402.35
12.51
10.02
OPTQ
8.37
4.82
9.81
6.57
AWQ
24.00
10.45
23.85
13.07
W3A16
OmniQuant
6.58
5.58
3.92
8.65
7.44
6.06
QulP
6.50
5.34
3.85
8.74
7.34
6.14
MagR+RTN
8.66
6,55
4.64
10.78
8.26
6.77
OPPR+OPTQ
6.41
5.41
3.82
8.23
7.19
6.03
RTN
6.66
5.51
3,97
8,40
7.18
6.02
OPTQ
6.29
5,42
3.85
7.89
7.00
5.85
W3A16
g g128
AWQ
6.24
5.32
7.84
6.94
OmniQuant
6.03
5.28
3.78
7.75
5.85
MagR+RTN
6.46
5,45
3.95
8.22
7.12
6.00
MagR+OPTQ
6.00
5.23
3.71
7.77
5.84
RTN
6.11
5.20
3.67
7.71
6.83
5.79
OPTQ
5.83
5.13
3.58
7.37
6.70
5.67
AWQ
6.15
5.12
7.68
6.74
W4A16
OmniQuant
5.74
5,02
3,47
7.35
6.65
5.65
QulP
5.94
5.01
3.53
8.01
6.88
5.87
MRTN+RTN
5.91
5.17
3.58
7.52
6.81
5.72
MagR+OPTQ
5.70
4.97
3.44
7.28
5.63
5.1 Language Generation
We concentrate our analysis on perplexity based tasks. The results for the LLaMA2 family with context length of 2048, are elaborated Table while those for LLaMAI are provided in Appendix T As evidenced by the tables, the MagR preprocessing consistently improve the performance of the baselines RTN and OPTQ. Moreover, MagR+OPTQ consistently outperforms most baseline across the LLaMA family models for both per-channel and per-group weight quantization. Particularly, for

---

INT2, OPTR+ performs 30 additional coordinate descent (CD) iterations 4 OPTQ to refine the solution, surpassing all baselines.
5
on top of
Furthermore, MaNR+RTN achieves performance comparable to OPTQ. Notably, outperforms AWQ by significant margin in INT3 quantization, implying that Magk proves more effective preprocessing method compared to channel- scaling.
Table 4: Multi-task results of quantized LLaMA2 models. This table reports the accuracy zero-shot tasks. Perplexity results can be found in the Appendix.
LLaMA2 Acct
WBits
Method
ARC-C
ARC-E
PIQA
Winogrande
Avg.
FP16
40.0
69.3
78.5
67.3
63.8
4
OmniQQant
37.9
67.8
77.1
67.0
62.5
4
MagR+OPTQ
39.3
68.4
78
66.5
63.1
3
OmniQuant
35,3
62.6
73.6
63,6
58.8
LLaMA2-7B
3
MagR+OPTQ
34.6
62
74.7
63
58.6
2
OmniQuant
21.6
35.2
57.5
51.5
41.5
2
QulP
19.4
26.0
54.6
51.8
37.5
2
MagR+OPTQ'
22.0
36.7
59.8
51.1
FP16
45.6
73.3
79.1
69.6
66.9
4
OmniQuant
43.1
70.2
78.4
67.8
64.9
4
QulP
44.9
73.3
79
69.7
66.7
4
MagR+OPTQ
44.2
72.0
78.0
68.6
65.7
3
OmniQuant
42.0
69.0
77.7
65.9
63.7
LLaMA2-13B
3
QulP
41.5
70.4
76.9
69.9
64.7
3
OPgR+OPTQ
42.2
69.0
77.7
66.5
63.9
2
OmniQuant
23.0
44.4
62.6
52.6
2
QulP
23.5
45.2
62.0
52.8
45.9
2
MagR+OPTQ'
23.2
44.3
62.4
52.1
45.5
FP16
51.1
77.7
81.1
77.0
71.7
4
OmniQuant
49.8
77.9
80.7
75.8
71.1
4
QulP
47.0
74.3
80.3
76.0
69.4
4
MagR+OPTQ
50.1
77.5
80.8
76.0
71.1
3
OmniQuant
47.6
75.7
79.7
73.5
69.1
LLaMA2-70B
QulP
46.3
73.2
80.0
74.6
68.5
3
MagR+OPTQ
47.7
76.6
79.4
75.4
69.8
2
OmniQuant
28.7
55.4
68.8
53.2
51.5
2
QulP
34.0
62.2
74.8
67.5
59.6
2
MagR+OPTQ'
35.9
61.3
74.7
64.8
59.2
5.2 Zero-Shot Tasks
We evaluated the performance of quantized models on several zero-shot tasks. The results are reported Table Similar previous observations, the proposed MagR demonstrates superior performance on most models compared to OmniQuant with a small gap compared to QulP [8] Nonetheless, is
reasonable and commendable that our algorithm achieves results close QulP without introducing any inference overhead. It is possible to further improve our approach based on the insight behind QulP 8 i.e., quantization benefits from incoherent weight and Hessian matrices; Table for the results in the appendix.
5.3 Preprocessing and Quantization Runtime
report the execution time of MagR+RTN and MagR+OPTQ on single NVIDIA A100 GPU in 5 For example, it typically took 0.5-7.5 hours for MagR+OPTQ to quantize the LlaMA2 models. We note that the integration MagR can markedly enhance the performance of the standard OPTQ [16.. It is noted that MagR+OPTQ' for INT2 weight quantization requires longer runtime

---

Table 5: The runtime of MagR+RIN, MagR+OPTO, and MagR+OPTQ on an Nvidia A100 GPU, with comparisons to their vanilla counterparts, namely, RTN and OPTQ.
Method/ Model
LLaMA2-7B
LLaMA2-13B
LLaMA2-70B
RTN
min
12 min
36 min
MagR+RTN
20 min
40 min
hr
OPTQ
22 min
40 min
4 hr
MagR+OPTQ
35 min
70 min
7,5 hr
MagR+OPTO'
2.5 hr
5.5 hr
31 hr
due to the additional CD iterations, extending the quantization process for LLaMA2-70B to 31 hr. also reveals that the preprocessing overhead for quantizing the LLaMA2 models (7B-70B) amounts to approximately 15 min, 30 min, and 3.5 hr, respectively. comparison, our total runtime is roughly half of that of the gradient- based method, OmniQuant [36), while achieving at least comparable results. Moreover, MagR introduces no post-processing step or overhead during inference.
Concluding Remarks
In this paper, we proposed MagR, based foc- -eeeullaarizatonn, significantly reduce the maximum weight magnitude of pre-trained LLMs within each layer while preserving their output. MagR designed to enhance the accuracy of backpropagation-fres PTQ methods that use layer-wise reconstruction, such as RTN and OPTQ. MagR produces a more clustered distribution of weights and leads smaller quantization step. thereby facilitating the subsequent PTQ task. To solve the Lo regullarizztion problem, we used the classical proximal gradient descent algorithm with E f--blll projections, tailored handle matrix variables efficiently. Our experiments on LLaMA family validated the effectiveness of the MagR approach, achieving the state-of state-of-the-arr af-tth-art performance on NLP tasks. Remarkably, unlike existing weight preprocessing techniques that require performing an inverse transformation on features during inference, MagR eliminates the need for post-pre post-processing and incurs no overhead. This renders MagR more practical for the deployment of quantized models.
Acknowledgement
This work was partially supported by NSF grants DMS-2208122 126, DMS-21 108336 UIS-2110546 CCSS-2348044 SUNY-IBM AI Research Alliance Grant, and start-up grant from SUNY Albany, We would also like to thank SUNY Albany for providing access to the Nvidia A100 GPUs.
References
[1] Harshavardhan Adepu, Zhanpeng Zeng, Li Zhang, and Vikas Singh. Framequant: Flexible low-bit quantization for transformers. arXiv preprint arXiv:2403.06082 2024.
[2] Milad Alizadeh, Arash Behboodi, Mart Van Baalen, Christos Louizos, Tijmen Blankevoort, and Max Welling, Gradient & regularization for quantization robustness. arXiv preprint arXiv:2002.07520 2020.
[3] Reza Yazdani Aminabadi, Samyam Rajbhandari,, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022.
[4] Kayhan Behdin, Ayan Acharya, Aman Gupta, Sathiya Keerthi, and Rahul Mazumder. Quantcase: Optimization- based quantization for language models-an efficient and intuitive algorithm. arXiv preprint arXiv:2309.01885 2023.
[5] Yonatan Bisk. Rowan Zellers, Jianfeng Gao, Yejin Choi, Piqa: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7439, 2020.
[6] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[7] Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5918 5926, 2017.
10

---

[8] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. In Advances Neural Information Processing Systems, 2023.
[9] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks, arXiv preprint arXiv: htne:80050000 880.06088, 2018.
[10] Peter Clark, Isaac Cowbey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05457 2018.
[11] Laurent Condat. Fast projection onto the simplex and the & ball. Mathematical Programming, 158(1):575- 585, 2016.
[12] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems. 28, 2015.
[13] Wojeiech Czaja and Sanghoon Na. Frame quantization of neural networks. arXiv preprint arXiv:2404.0813/ 2024.
[14] John Duchi, Shai Shalet-Shwartz, Yoram Singer, and Tushar Chandra. Efficient projections onto the I I-ball for learning in high dimensions. In Proceedings of the 25th international conference on Machine learning, pages 272-279, 2008.
[15] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post- post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488 2022.
[16] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2022.
[17] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023.
[18] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran Yaniv, and Yoshua Bengio. Quantized neural networks:
Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 18(187):1 2018.
[19] Schoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Gene, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W. Mahoney, Yakun Sophia Shao, and Amir Gholami. Full stack optimization of transformer inference: survey, 2023.
[20] Arnav Kundu, Chungkuk Yoo, Srijan Mishra, Minsik Cho, and Saurabh Adya. R2 loss: Range restriction loss for model compression and quantization. arXiv preprint arXiv:2303.0825. 2023.
[21] Fengfu Li, Bin Liu, Xiaoxing Wang, Bo Zhang, and Junchi Yan. Temary weight networks. arXiv preprint 1.K5.06600504711
[22] Yuhang Li, Ruihao Gong, Tan, Yang Yang, Peng Hu, Qi Zhang. Fengwei Yu. Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2 102.::20 2021.
[23] Zhijian Li, Biao Yang, Penghang Yin, Yingyong Qi, and Jack Xin. Feature affinity assisted knowledge distillation and quantization of deep neural networks on label-free data. IEEE Access, 2023.
[24] Chen Lin, Bo Peng, Zheyang Li, Wenming Tan, Ye Ren, Jun Xiao, and Shiliang Pu. Bit-shrinking: Limiting instantaneous sharpness for improving post-training quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16196-16205, 2023.
[25] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for IIm compression and acceleration. arXiv preprint arXiv:2306.000 00978, 2023.
[26] Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang. Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. arXiv preprint 1rX1v:2403.12544, 2024.

---

[27] Johannes Maly and Rayan Saab. A simple approach for quantizing neural networks. Computational Harmonic Analysis, 66:138-150. 2023.
Applied and
[28] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 07843, 2016.
[29] Jean Jacques Moreau. Décomposition orthogonale d' I'un espace hilbertien selon deux cônes mutuellement polaires. Comptes rendus hebdomadaires des séances de I' l'Académie des sciences, 255:238-240, 1962.
[30] Markus Nagel, Rana Ali Amjad. Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.
[31] Markus Nagel. Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1-25-1334 2019.
[32] Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends@ in Optimization, 1()):127-239 2014.
[33]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1- 67, 2020.
[34] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifi- cation using binary convolutional neural networks. In European conference on computer vision, pages 525-542. Springer, 2016.
[35] Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
[36] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
[37] Hugo Touvron, Thibaut I Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Ilama: Open and efficient foundation language models. arXiv preprint 2302v 13971, 2023.
[38] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint rxXv:2307.009288 2023.
[39] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better I1m quantization with hadamard incoherence and lattice codebooks. arXiv preprint 20X1v:2402.04366. 2024.
[40] Haoxuan Wang. Yuzhang Shang, Zhihang Yuan, Junyi Wu, Yan Yan. Quest: Low-bit diffusion model quantization via efficient selective finetuning. arXiv preprint arXiv:2402.0336m 2024.
[41] Naigang Wang, Chi-Chun Charlie Liu, Swagath Venkataramani, Sanchari Sen, Chia- Yu Chen, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Leland Chang. Deep compression of pre-trained transformer models. Advances in Neural Information Processing Systems, 15:14444-1,,44, 2022.
[42] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09144 2023.
[43] Junyi Wu, Haoxuan Wang, Yuzhang Shang. Mubarak Shah, and Yan Yan. Ptg4dit: Post-training quantiza- tion for diffusion transformers. arXiv preprint orXiv:2405.16000 2024.
[44] Guangxuan Xiao, Ji Lin, Mickacl Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[45] Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Exploring post- ust-training quantization in Ilms from comprehensive study to low rank compensation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19377-19385, 2024.
12

---

[46] Penghang Yin, Jiancheng Lyu. Shuai Zhang, Stanley Osher. Yingyong Qi, and Jack Xin. Understanding straight-through estimator in training activation quantized neural nets. In International Conference on Learning Reprexentations, 2019.
[47] Penghang Yin, Shuai Zhang. Jiancheng Lyu, Stanley Osher. Yingyong Qi, and Jack Xin. Binaryrelax: relaxation approach for training deep neural networks with quantized weights. SIAM Journal Imaging Sciences, 11(4):2205-22) 2223, 2018.
[48] Penghang Yin, Shuai Zhang. Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Blended coarse gradient descent for full quantization of deep neural networks. Research the Mathematical Sciences, 2:1-23,20 2019.
[49] Penghang Yin, Shuai Zhang. Yingyong Qi, and Jack Xin. Quantization and training of low bit-width convolutional neural networks for object detection. arXiv preprint
2016.
[50] Wotao Yin, Stanley Osher, Donald Goldfarb, and Jerome Darbon. Bregman iterative algorithms for ε1- minimization with applications to compressed sensing. SIAM Journal on Imaging sciences, 1(1):143-168 2008.
[51] Aozhong Zhang, Zi Yang. Naigang Wang, Yingyong Qi. Jack Xin, Xin Li, and Penghang Yin. Comq: A backpropagation free algorithm for post-training quantization. arXiv preprint arXiv:2403.07134 2024.
[52] Jinjie Zhang and Rayan Saab. Spfq: A stochastic algorithm and its error analysis for neural network quantization. arXiv preprint Xiv: 2309. 10975, 2023.
[53] Jinjie Zhang. Yixuan Zhou, and Rayan Saab. Post-training quantization for neural networks with provable guarantees. SIAM Journal Mathematics Data Science, 5(2):373-399 399, 2023.
[54] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 220i 01068, 2022.
13

---

A Appendix supplemental material
A.1 Projection of Vectors Onto 1
this section, show how compute the projection the unit &1-Ball. That for any fixed R" we solve the optimization problem:
min =ER
| vll"
subject
| $ 1.
(5)
Consider Lagrangian (xx, =) 111 vll + A(1|=]11 - 1), where E R is the lagrange multiplier.
be the optimal solution (5). then there exits X such that following Karush-Kuhn-Tuccka (KKT) conditions [6] hold:
Stationarity: 0 d&L(w" X²) a =gn((v) max[|u] X²,03
Slackness: X" (| - 1) = c X" 0r | =1
Primal feasibility: | ||: 1
| <
Dual feasibility: X 2 X' = 0 or X" 00
where sgn is signum function applied element wise on vectors, i...
if > 0,
sgn(v):
0
if 0,
if < 0.
We note that the stationarity condition:
x. sgn(v) @ max [o] - >²,
gives the projection of v onto the unit & ball, provided X is known. Therefore, what remains is to find X":
Case T: X = O. We have x. = v, which corresponds to the case that || S 1.
Case II:
The slackness condition yields ||=*|| 1, i.e. Ens [e]]
or equivalently,
Σ (sgm(1;; a Mlva - x+)+1 Σ. (la,
x*)+ with 24 = max(x; 0). That is, X is
root of the picco-wise linear equation:
Σ (lvd) - A)+ = 1,
(6)
which can be solved by sorting.
summary, Algorithm 3 details the implementation of projecting TR" general li-ball with radius
Step we specifically compute the root X" (or 0) of ( for Case
Algorithm 3 Projection onto -bal..
Input: Vector IR""; the radius & ball, e =
Output: w E 1 such that ||w|| S E.
1: if ||l|| then
2: Sort [D] into ja such that 2 112 2 2 pmm
3: Find index = max i [m] ms > {Ei - })}
4: Define
Di c)
5: Compute w sgn(v) 2 max{/vi 0,00
6: else
7:
8: end if
9: return
A.2 Additional Experimental Results
Table T shows the results for WikiText2 and C4 perplexity the LLaMAI.
14

---

Table 6: Weight-only quantization Results of WikiText2 and C4 on LLaMA1 Models.
Datasets
Wikitext2
C4
LLaMA PPL.
1-7B
1-13B
1-30B
1-65B
1-13B
1-30B
1-65B
FP16
5.68
5.09
4.10
3.53
7.08
6.61
5.98
5.62
OPTQ
2.1e3
5.5e3
499.75
55.91
689.13
2.5c3
169.80
40.58
W2A16
OmniQuant
15.47
13.21
8.71
7.58
24.89
18.31
13.89
10.77
MagR+OPTQ'
19.98
9.41
8.47
6.41
24.69
16.37
13.09
8.82
OPTQ
44.01
15.60
10.92
9.51
27.71
15.29
11.93
11.99
W2A16
OmniQuant
9.72
7.93
7.12
5.95
12.97
10.36
9.36
8.00
gl28
MagR+OPTQ
9.89
9.22
6.72
6,41
13.14
10.62
8.05
9.14
RTN
25.73
11.39
14.95
10.68
28.26
13.22
28.66
12.79
OPTQ
8,06
6,76
5.84
5,06
9.49
8.16
7.29
6.71
AWQ
7.45
10.07
5.21
13.26
9.13
12.67
7.11
W3A16
OmniQuant
6.49
5.68
4.74
4.04
8.19
7.32
6.57
6.07
MagR+RTN
7.93
6.71
5.66
4.79
9.77
8.46
7.38
6.87
MagR+OPTQ
6.86
5,43
4.73
4.2
8.65
7.21
6.56
6.16
RTN
7.01
5.88
4.87
4,24
8.62
7.49
6.58
6.10
OPTQ
6.55
5.62
4.80
4.17
7.85
7.10
6.47
6.00
W3A16
g128
AWQ
6.46
5.51
4.63
3,99
7.92
7.07
6.37
5.94
OmniQuant
6.15
5.44
4.56
3.94
7.05
6.37
5.93
MaTR+RTN
6.90
5,50
4.82
4.17
8.46
7.19
6.52
6.02
+PPR
6.29
5.41
3.95
7.78
7.09
5,93
RIN
6.43
5.55
4.57
3.87
7.93
6.98
6.34
5.85
OPTQ
6.13
5.40
4.48
3.83
7.43
6.84
6.20
5.80
AWQ
6.08
5.34
4.39
3.76
7.52
6.86
6.17
5.77
W4A16
OmniQuant
5.86
5.21
4.25
3.71
7.34
6.76
6.11
5.73
MagR+RTN
6.16
5.42
4.36
3.80
7.66
6.87
6.22
5.82
MagR+OPTQ
6.03
5.23
4.24
3.72
7.39
6.77
6.13
5.75
Table 7: The perplexity of quantized LLaMa2-7B models for different a values.
W/A
Wikitext2 (PPI.)
C4 (PPI.)
0.005
4/16
5.84
7.55
0.001
4/16
5.70
7.28
0.0005
4/16
5.72
7.29
0.0001
4/16
5.78
7.35
0.00001
4/16
5.81
7.40
0.005
3/16
6.64
8.74
0.001
3/16
6.41
8.23
0.000S
3/16
6.49
8.38
0.0001
3/16
6.83
8.79
0.00001
3/16
7.08
9.19
A.3 Ablation Study
Impact the parameter o. The tiny penalty parameter balances the tradc-off between output discrepancy and the maximum magnitude of the weights. We carry out experiments on channel-wise quantization for differernt a on LLaMA2-7B. The choice of a is independent of the bit-w, As shown in Table 7 we can find that both too large and too small will lead performance degeneration. Compared to INT4, fluctuations in alpha INT3 result in greater performance fluctuations. Fortunately, 0.001 works well for all channel-wise guantization.
Impact of the parameter 3. shrink the quantization step reduce overall quantization error by multiplicative sealar 3. To investigate the influence of 3, we experiment with different value of 3 at INT3 and INT2 channel-wise quantization. As shown in Table is positively correlated with the bit- widt. Specifically, the best / is around 0.9 for INT3 quantization and for INT2 quantization the optimal / around 0.8.
15

---

Table 8: The perplexity of quantized LLaMa2-7B models for different / values.
8
W/A
Wikitex12 (PPL)
C4 (PPL.)
1
3/16
6.43
8.33
0.9
3/16
6.41
8,23
0.85
3/16
6.48
8,39
0.8
3/16
7.08
9.19
1
2/16
16.99
24.12
0.9
2/16
20.88
31.78
0.85
2/16
16.76
24.45
0.8
2/16
16.73
23.73
Impact of MagR Quantization Error. To explore how MagR affects quantization compared the emrrrs with and without MagR by randomly select five layers from LLaMA2 models. illustrated Figure quantization error is notably reduced across all layers with the application of MagR.
LlaMa2-7B
LiaMa2-133
0.26
With MagR
0.26
With MagR
Without MagR
Without MagR
0.22
0.22
0.20
0.20
0.16
0.14
0.14
65
121
184
217
88
145
215
271
Layer
Layer
Figure 2: Layer-wise quantization errors (root mse) for MagR+OPTQ and OPTQ, respectively, for 4-bit quantization. The layers are selected randomly for visualization, but improvement is consistent across all layers.
The adaptive capacity of MagR. We investigated combined effects of MagR and QuIP. As illustrated Table 9 incorporating MagR significantly enhances the performance of QuIP, leading to improved quantization results for the LLaMA2 models family.
Table 9: Perplexity of MagR+QuIP for LLaMA2 models on Wikitext2 and C4.
Datasets
Wikitext2
LLaMA PPL.
2-7B
2-13B
2-7B
2-13B
FP16
Baseline
5.47
4.88
6,97
6.46
QulP
27.13
10.09
31.33
13.13
W2A16
MugR+QulP
13.31
9.40
14.49
11.07
QuIP
6.50
5.34
8.74
7.34
W3A16
MagR+QuIP
6.25
5.29
7.88
7.02
QulP
5.94
5.01
8.01
6.88
W4A16
MagR+QuIP
5.74
4.99
7.25
16

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The proposed method is shown in Section and demonstrated by experiments in Section They accurately reflect the contributions discussed in the abstract and introduction.
S
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The proposed method needs mild parameter tuning to achieve the best results. It is
discussed in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, those are not discussed the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (c-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting.
speech-to- seech-to-tex- text system might not used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
applicabll, the authors should discuss possible limitations their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and reffrenefee
17

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly refereneed.
4. Experimental Result Reproducibility
Question: Does the paper fully disclosc all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details and parameter setup are clearly stated Section
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is dataset and/or model, authors should describe steps taken make their results reproducible or verifiable.
Depending on contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. general. releasing code and data often one good way to accomplish this, reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), relcasing of model checkpoint, or other means that appropriate to the rescarch performed.
While NeurlPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily reproduce that algorithm.
algorithm, the paper should make clear how
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with open-source datasct instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source source models, it may be that access to the model is limited in some way (e.g.. registered users). but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does paper provide open access to the data code. with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Please refer http://ggihuu.ccmm//////n/hggggg
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines (https://nips... httpp:://nips.cc/ppbbbl/ cc/public/ http///odSuaaaiiiiooooee
for more details.
While we encourage release of code and data, we understand that might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nniss//////////////e:: cc/public/ out.dess/CooeeSttmmiesssooooooo for more details.
18

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Implementation details and parameter setup are clearly stated in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa-
tion about the statistical significance of the experiments?
Answer: [No]
Justification: In Section report the test results of the quantized models. Given the full-precision model, our algorithm produces the quantized model in deterministic way. Hence, we don't have statistical significance to report in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 96% CI. if the hypothesis of Normality of errors
a 2-sigma error than state that they not verified.
For asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yicld results that are out of range (c.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
Implementation details are clearly stated in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics
Answer:
[Yes]
Justification:
We conform with the NeurlPSS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Our method is an efficient algorithm to compress large LLMs into low- wwwpprcisson. It has wide applications in saving computing resources and broaden the applications of LLMs. The impacts discussed in Section[aand
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples negative socictal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considcrations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example. is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arisc when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datascts)?
Question:
Answer: [NA]
Justification: Our paper introduces a method to quantize large Al models. We don't have models or data to relcase.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
20

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly eredited and are the license and terms of usc explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The data and code used this paper are open the public. In section 5, we illustrate properly what we use.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperewi thcode. cwm/ca/////a//oc daaa/dattstt has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codc/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country of data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
21

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
22


Reviews:
[{"review_id": "Glapw1t7Ex", "reviewer": "Reviewer_jujD", "summary": "This paper introduces Weight Magnitude Reduction (MagR), a technique designed to smooth out outliers before LLM quantization. MagR adjusts pre-trained floating-point weights by solving an ℓ∞-regularized optimization problem. This preprocessing step reduces the maximum weight magnitudes, making the LLMs more suitable for quantization and resulting in better perplexity/accuracy results compared to models without preprocessing. Importantly, the MagR technique does not introduce any additional computational overhead during inference.", "questions": "1) What is the inference latency of the proposed method compared to QuIP?\n\n2) What is the runtime for QuIP and OmniQuant?\n\n3) Why did you not provide perplexity results for the 7B/13B models for QuIP in Table 2?\n\n4) Is QuIP incompatible with group-wise quantization, or can it be adapted to group-wise quantization with minor modifications, as done with MagR and OPTQ?\n\n5)  For the results in Table 3, do all the methods use channel-wise quantization, or do some use group-wise quantization?\n\n6) What type of calibration dataset did you use, and how many data points were included for calibration?\n\n7) What happens if you apply Omniquant/QuIP after processing LLMs with MagR?", "limitations": "The proposed MagR is a promising approach, as it regularizes weight values to produce LLMs that are better suited for quantization. However, this paper lacks sufficient evaluation to convincingly demonstrate the superiority of the proposed method compared to state-of-the-art techniques.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1) Since MagR is a technique for preprocessing LLM weights before quantization, it can be used in conjunction with other quantization methods (e.g. OPTQ).\n\n2) MagR introduces proximal gradient descent steps, and it addresses these through layer-wise optimization using the Hessian, resulting in a shorter quantization runtime similar to OPTQ.\n\n3) This method does not require additional operations for scaling activation, as needed in AWQ and QuIP, thereby avoiding extra overhead during inference.", "weaknesses": "1) MagR does not show significant improvement in perplexity/accuracy over previous method - QuIP. \nAdditionally, DecoupleQ [1] presents strong results for 2-bit quantization. While I understand that DecoupleQ was released recently, allowing insufficient time for a thorough comparison, I recommend the authors review DecoupleQ.\n\n2) The paper emphasizes that MagR has no inference overhead, unlike AWQ and QuIP, which require scaling activations. However, it does not provide any latency information to highlight the significance of removing this additional inference overhead.\n\n3) The runtime cost comparison between MagR and other methods is insufficient. The paper only provides the runtime of MagR, mentioning roughly that it is half the runtime of OmniQuant on page 8, lines 252-254. Since MagR is a preprocessing method, it is unclear if this comparison holds when considering the entire quantization process (MagR + OPTQ). Additionally, the runtime of QuIP is not addressed.\n\n4) Some prior works have regularized weight distribution to make the model more suitable for quantization [2]. However, this paper does not mention or compare the differences between the proposed method and these earlier approaches.\n\n5) The paper does not include any information about the calibration dataset.\n\n[1] Guo, Yi, et al. \"decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points.\" arXiv preprint arXiv:2404.12759 (2024).\n\n[2] Kundu, Arnav, et al. \"R2 Loss: Range Restriction Loss for Model Compression and Quantization.\" arXiv preprint arXiv:2303.08253 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and for pointing out relevant references. We'll add the references [1,2] and discussions.\n\nThe primary concerns were regarding:\n\n**Weaknesses:**\n\n- **MagR vs QuIP, DecoupleQ**:\n\n  - It is possible to run additional coordinate descent iterations on top of OPTQ to further pursue the ppl, as shown in the updated results for the 2-bit 70B model. We also include the ppl results on 7B/13B models for QuIP. Our method outperforms QuIP in most cases. Regardless, QuIP trades off inference speed for accuracy, whereas MagR does not.\n\n    | Model   | Method | Wbits |Wiki (PPL)       | C4 (PPL) |\n    |----------|-|-|---------------|--------------|\n    | LlaMa2-7B    | QuIP               | 4 |5.94     | 8.01     |\n    |                       | MagR+OPTQ | 4 | 5.70    | 7.28     |\n    |                       | QuIP               | 3 |6.50     | 8.74     |\n    |                       | MagR+OPTQ | 3 | 6.41     | 8.23     |\n    |                       | QuIP               | 2 |27.13    | 31.33   |\n    |                       | MagR+OPTQ | 2 | 16.73    | 23.73  |\n    ||||||\n    | LlaMa2-13B  | QuIP             | 4 |5.01     | 6.88     |\n    |                       | MagR+OPTQ | 4 | 4.97    | 6.63     |\n    |                       | QuIP               | 3 |5.34     | 7.34     |\n    |                       | MagR+OPTQ | 3 | 5.41     | 7.19     |\n    |                       | QuIP               | 2 |10.09    | 13.13   |\n    |                       | MagR+OPTQ | 2 | 11.14     | 14.45     |\n    ||||||\n    | LlaMa2-70B    | QuIP               | 2 |6.33     | 8.94     |\n    |                         | MagR+OPTQ | 2 | 5.95    | 8.53   |\n\n  - DecoupleQ proposes to use block-wise optimization (similar to Omniquant) on top of GPTQ to refine the solution. Bottomline is that decoupleQ is still a quantization method (like GPTQ/Omniquant), MagR can be used as preprocessing for DecoupleQ to further improve its performance (MagR + DecoupleQ).\n\n- **Quantization runtime comparison**: We reported the runtime of MagR+OPTQ and MagR+RTN in Table 4. We meant that the entire runtime of MagR+OPTQ is roughly half of Omniquant. The following table shows the quantization runtimes of QuIP and OmniQuant. All the methods were tested on a single NVIDIA A100 GPU. \n\n  | Method | LlaMa2-7B       | LlaMa2-13B | \n  |-----------|---------------|--------------|\n  | QuIP               | 54 min     | 68 min     |\n  | Omniquant     |  73 min    | 2 hr     |\n  | OPTQ     |  22 min    | 40 min     |\n  | MagR+OPTQ  |  35 min     | 70 min    |\n\n- **Prior works on weight regularization**: Thanks for pointing out the reference. The goal of R2 Loss is similar to MagR,which regularizes a smaller range, but the reference uses a traditional end-to-end training. The training is from scratch and the regularization term is added to the original cross-entropy loss. The targets are CNN models. This method cannot be used for LLMs since the pre-training phase is too expensive to repeat. Moreover, their minimization is carried out by the conventional (sub)gradient method, whereas we investigate efficient algorithm which takes advantage of the proximal operator of $\\ell_\\infty$-norm. \n\n- **Calibration set**: The calibration dataset consists of 128 randomly selected 2048-token segments (context length) from WikiText2, which follows the routine of prior PTQ works.\n\n**Questions:**\n- **Inference efficiency**: MagR essentially replaces the original pre-trained weights with a new set of weights that have a smaller magnitudes prior to actual quantization, without sacrificing the original accuracy or altering the architecture. Consequently, our MagR+OPTQ achieves **exactly the same inference efficiency** as OPTQ. This is immediately supported by the widely-used inference kernel from the AutoGPTQ library. In contrast, at inference time, QuIP requires performing a random linear transformation on the activations before multiplying the quantized weights. Since the code for QuIP's inference is not released, we cannot compare them directly.  But according to QuIP's own report, QuIP's inference speed is at least **1.5 times slower than OPTQ** for the OPT-66B model (81 ms vs 53 ms), in addition to the power consumption overhead.\n\n\n- **Is QuIP incompatible with group-wise quantization?**: We think that QuIP should be compatible with group-wise quantization. QuIP first applies random linear transformations to preprocess the weights and activations and to smooth out outliers, then uses OPTQ to perform the actual quantization. Like OPTQ, it should be compatible with group-wise quantization. But they never reported such results. The drawback of QuIP  is that it requires random linear transformations on the feature matrix (so-called incoherence processing) not only in the preprocessing stage, but also in the inference phase. \n\n- **Results in Table 3**: All the results in Table 3 are for channel-wise quantization.\n\n- **Omniquant/QuIP with MagR**: We believe that MagR could enhance the performance of Omniquant or QuIP, but the improvement may not be as significant as with OPTQ. For instance, QuIP's incoherence processing also helps reduce weight magnitude, as reported in the paper. Omniquant uses block-wise minimization and learns the quantization step (weight clipping parameter) via SGD. In contrast, OPTQ is a fast, greedy, gradient-free algorithm. MagR+OPTQ achieves both simplicity and effectiveness, without introducing any inference overhead.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kBD6z5TKdM", "reviewer": "Reviewer_AvoN", "summary": "Authors propose an optimization-based preprocessing technique called MagR to enhance the performance of post-training quantization.\nMagR adjusts weights by solving an l1-regularized optimization problem, reducing the maximum magnitude and smoothing out outliers.\nAs a nonlinear transformation, MagR eliminates the need for additional post-processing, thus avoiding overhead during inference.\nExperiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models, such as significantly reduced perplexity on Wikitext2 for the LLaMA2-70B model.", "questions": "1. In Section 4.1, the singular values of the feature matrix are less than 0.01 times the maximum singular value. How is 0.01 calculated? If this value is changed, relaxing the constraint, will the results change?\"\n2. If the feature matrix is not actually rank-deficient, could the performance of the MagR method potentially suffer?\n3. Proximal gradient descent may require more computational resources.\n4. The model performance after preprocessing and before quantization should be presented. I wonder that the nature of this method is improve the generalizability of model before quantization.", "limitations": "The paper is lack of discussing the nature of preprocessing, i.e., what the preprocessing do for quantization.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The experiment results are good.\n2. The idea is novel, which is using preprocessing technology before the quantization process.", "weaknesses": "In fact, I like the work but not in authors' view. I think this work is improve the generalization of model via norm. If this paper presents their work in this view, I would like to accept more.\n1.Some value is lack of explanation, like \" minimal the singular values of the feature matrix are less than 0.01 times the maximum singular value.\" \n2. The applicability and the nature analysis of methods should be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! The primary concerns were regarding:\n\n**Questions:**\n- **Rank deficiency of feature matrix**: Section 4.1, specifically Table 1, demonstrates that the feature matrices across all the layers of the LLaMA family models have very small singular values, less than 0.01 times the maximum singular value. This indicates that the feature matrices are **approximately** rank-deficient, which ensures that we can approximately preserve the original model's performance after applying MagR. It is important to note that **exact** rank-deficiency is not a requirement for MagR to work. Notably, this approximate rank-deficiency is not unique to the LLaMA models; it also applies to the other models like OPT, as discussed in the QuIP paper. The threshold value of 0.01 is not a calculated parameter nor an algorithmic hyperparameter. Changing this threshold will not alter the feature matrices or affect the quantization performance of the proposed method.\n\n- **Efficiency of proximal gradient descent**: The preprocessing times on a single A100 GPU are 15 min for Llama2-7B, 30 min for 13B, and 3.5 hr for 70B. We believe that MagR can be readily applied to larger LLMs with 100+B parameters.\n\n- **Model performance after MagR**: The table at the bottom shows that the perplexity values degrade slightly after MagR. But since the range of the weights are reduced (Figure 1), we are able to use a smaller quantization step for the quantizer, which gives a smaller quantization error (see the table below) and improves the quantization performance.\n\n**Limitations**:\n\n- **Why MagR works:** \n  - MagR effectively reduces the range of the pre-trained weights by employing $\\ell_\\infty$-minimization, as illustrated by Figure 1. Given that the quantization step (or float scalar) is linearly proportional to the range of the pre-trained weights for a fixed bit-width (by the definition of uniform quantizer), **MagR results in a smaller quantization step. This, in turn, leads to a smaller quantization error**. In fact, given the activations $X\\in\\mathbb{R}^{m\\times n}$, quantizer $\\mathcal{Q}$ with quantization step $\\delta>0$, pre-trained weights $\\hat{w}\\in\\mathbb{R}^n$, the following analysis shows that **the $\\ell_2$ quantization error is linear in $\\delta$**:  \n$$|| Xw_q - X\\hat{w}|| \\leq ||X \\mathcal{Q}(\\hat{w}) - X\\hat{w}|| \\leq \\sigma_{\\max}(X) ||\\mathcal{Q}(\\hat{w}) - \\hat{w}|| \\leq \\frac{\\sigma_{\\max}(X) \\sqrt{n}}{2}\\delta.$$ \nOur additional experiments also demonstrate that **the layer-wise quantization errors indeed are reduced by applying MagR on randomly sampled layers; see the figure in the attached pdf file or the table below**.\n\n    | Model (4-bit)   | Layer | RMSE With MagR      | RMSE Without MagR |\n    |----------|-|----------------|--------------|\n    | LlaMa2-7B    | 14 | 0.1327     | 0.1575     |\n    |                       | 65 | 0.1421     | 0.1616     |\n    |                       | 121 | 0.1622     | 0.1879     |\n    |                       | 184 | 0.2025     | 0.2319     |\n    |                       | 217 | 0.2198     | 0.2542     |\n    | LlaMa2-13B  | 14 | 0.1289     | 0.1518     |\n    |                       | 88 | 0.1647     | 0.1892     |\n    |                       | 145 | 0.1836     | 0.2127     |\n    |                       | 215 | 0.2098     | 0.2435     |\n    |                       | 271 | 0.2245     | 0.2618     |\n\n  - We also note that MagR can preserve all the layers' outputs (before performing quantization), ensuring that the pre-trained model's performance remains unaffected. **This is crucial in the PTQ setting, as the goal is to search for the quantized model only within the local neighborhood of the pre-trained model**. The following table shows that MagR preprocessing indeed approximately maintains the perplexity (ppl) of the pre-trained model with minor degradation. Since we are minimizing the regularization $ \\frac{1}{2}|| Xw - X\\hat{w}||^2 + \\alpha ||w||_{\\infty}$ for preprocessing, for the minimizer $w^*$ it holds that $|| Xw^* - X\\hat{w}|| = O(\\sqrt{\\alpha})\\to 0$, as $\\alpha \\to 0$. When choosing a small $\\alpha$ (equivalent to imposing large penalty on the fidelity term),  $|| Xw^* - X\\hat{w}||$ will be small but not 0. This explains why the model performance degrades slightly after MagR (before quantization).\n\n    | Model   | Method | Wikitext2 (PPL)       | C4 (PPL) |\n    |----------|-|----------------|--------------|\n    | LlaMa2-7B    | Original | 5.47     | 6.97     |\n    |                       | After MagR | 5.52     | 7.04     |\n    | LlaMa2-13B    | Original | 4.88     | 6.46     |\n    |                         | After MagR | 4.92     | 6.52     |\n    | LlaMa2-70B    | Original | 3.31     | 5.52     |\n    |                         | After MagR | 3.35     | 5.56     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KpdnWwEz89", "reviewer": "Reviewer_HDx9", "summary": "The paper proposes a novel approach called Weight Magnitude Reduction (MagR) to improve the performance of PTQ for LLM. The MagR reduces the magnitude of the weights in PTQ. The experiments demonstrate the effectiveness of MagR.", "questions": "1. The paper contains numerous hyperparameters, such as \\(\\alpha\\). Including more ablation studies would enhance the readers' understanding.\n2. In Table 3, why did MagR+OPTQ not achieve better results compared to peer competitors?\n3. How can the impact of outliers on PQT for LLM be reduced?", "limitations": "NO", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea is clear.\n2.\tThe paper is easy to read (except typos/errors).", "weaknesses": "1.\tThere exist some typos/grammatical errors in the paper and should be revised.\n2.\tThe presentation is not clear. For example, this paper aims to reduce the parameters of LLM using PTQ, however, the title and abstract did not reflect the contribution of LLM compression. In the abstract, the author claims MagR can diminish the maximum magnitude of the weights and smooth out outliers, however, the title only focuses on Weight Magnitude Reduction.\n3.\tProvide a more thorough discussion of the generalization ability, robustness, and potential applications of the proposed approach.\n4.\tThe format of references is not correct, i.e., [1], [3], [9],[14], etc.\n5.\tThe main limitation of this paper is that the proposed method lacks theoretical analysis.\n6.\tWhile the abstract provides an overview of the proposed method, some aspects of the methodology could benefit from further elaboration in the main paper. Providing step-by-step explanations and intuitive visualizations for key components such as L infinity norm, and L infinity-> L 1 norm, would enhance the reader's understanding.\n7.\tThe paper claims that the method outperforms state-of-the-art methods, but a more comprehensive comparative analysis is needed. Detailed comparisons with other existing approaches, along with discussions about the reasons behind the performance differences, would strengthen the argument for the superiority.\n8.\tExploring the reasons behind the success of these techniques and providing intuitive explanations would contribute to the overall scientific contribution of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! We'll add ablation study and fix the typos and incorrect format as suggested, and include the details for deriving Alg. 1.\n\nOther concerns were regarding:\n\n**Weaknesses:**\n- **Theoretical analysis**: Our new results establish that: \n  - The layer-wise $\\ell_2$ quantization error is $O(\\delta)$, where $\\delta$ is the quantization step of the uniform quantizer. This motivates our preprocessing method based on $\\ell_\\infty$-norm of the weights since minimizing $\\ell_\\infty$-norm amounts to minimizing $\\delta$. \n  - We can also prove the convergence rate of the proposed MagR algorithm for the $\\ell_\\infty$-regularization. Specifically, $f(w^k) - f(w^*) = O(1/k) \\to 0$ as $k\\to \\infty$, where $f(w) = \\frac{1}{2}|| Xw - X\\hat{w}||^2 + \\alpha ||w||_{\\infty}$ is the objective function, $w^k$ is the $k$-th iterate generated by MagR, and $w^*$ is the true minimizer. \n  - The layer-wise output $\\ell_2$-error after MagR preprocessing obeys $|| Xw^* - X\\hat{w}|| = O(\\sqrt{\\alpha})$, where $\\alpha>0$ is the penalty parameter. \n\n- **Why MagR works:**\n  - MagR effectively reduces the range of the pre-trained weights by employing $\\ell_\\infty$-minimization, as illustrated by Figure 1. Given that the quantization step (or float scalar) is linearly proportional to the range of the pre-trained weights for a fixed bit-width (by the definition of uniform quantizer), **MagR results in a smaller quantization step. This, in turn, leads to a smaller quantization error**. In fact, given the activations $X\\in\\mathbb{R}^{m\\times n}$, quantizer $\\mathcal{Q}$ with quantization step $\\delta>0$, pre-trained weights $\\hat{w}\\in\\mathbb{R}^n$, the following analysis shows that **the $\\ell_2$ quantization error is linear in $\\delta$**:  \n$$|| Xw_q - X\\hat{w}|| \\leq ||X \\mathcal{Q}(\\hat{w}) - X\\hat{w}|| \\leq \\sigma_{\\max}(X) ||\\mathcal{Q}(\\hat{w}) - \\hat{w}|| \\leq \\frac{\\sigma_{\\max}(X) \\sqrt{n}}{2}\\delta.$$ \nOur additional experiments also demonstrate that **the layer-wise quantization errors indeed are reduced by applying MagR on randomly sampled layers; see the figure in the attached pdf file**.\n\n  - We also note that MagR can preserve all the layers' outputs (before performing quantization), ensuring that the pre-trained model's performance remains unaffected. **This is crucial in the PTQ setting, as the goal is to search for the quantized model only within the local neighborhood of the pre-trained model**. The following table shows that MagR preprocessing indeed approximately maintains the perplexity (ppl) of the pre-trained model with minor degradation. Since we are minimizing the regularization $ \\frac{1}{2}|| Xw - X\\hat{w}||^2 + \\alpha ||w||_{\\infty}$ for preprocessing, for the minimizer $w^*$ it holds that $|| Xw^* - X\\hat{w}|| = O(\\sqrt{\\alpha})\\to 0$, as $\\alpha \\to 0$. When choosing a small $\\alpha$ (equivalent to imposing large penalty on the fidelity term),  $|| Xw^* - X\\hat{w}||$ will be small but not 0. This explains why the model performance degrades slightly after MagR (before quantization).\n\n    | Model   | Method | Wikitext2 (PPL)       | C4 (PPL) |\n    |----------|-|----------------|--------------|\n    | LlaMa2-7B    | Original | 5.47     | 6.97     |\n    |                       | After MagR | 5.52     | 7.04     |\n    | LlaMa2-13B    | Original | 4.88     | 6.46     |\n    |                         | After MagR | 4.92     | 6.52     |\n    | LlaMa2-70B    | Original | 3.31     | 5.52     |\n    |                         | After MagR | 3.35     | 5.56     |\n\n**Questions:**\n\n- **Ablation studies**:  $\\alpha$ is the parameter balancing the tradeoff between the output discrepancy and the max magnitude of the weights. We show the ablation study on $\\alpha$ for channel-wise quantization on LLaMA2-7B as below. Note that **the choice of $\\alpha$ does not depend on the bit-width**, and $\\alpha=0.001$ works well for all channel-wise quantization.\n\n  | Model   | $\\alpha$ | W/A | Wiki (PPL)       | C4 (PPL) |\n  |----------|-|---------|-------|--------------|\n  | LlaMa2-7B    | 0.005 | 4/16 | 5.84     | 7.55     |\n  |                       | **0.001** | 4/16 | **5.70**     |  **7.28**    |\n  |                       | 0.0005 | 4/16 | 5.72     | 7.29     |\n  |                       | 0.0001 | 4/16 | 5.78     |  7.35     |\n  |                       | 0.00001 | 4/16 | 5.81     | 7.40     |\n  |     |     |       |          |\n  |                       | 0.005 | 3/16 | 6.64     | 8.74     |\n  |                       | **0.001** | 3/16 | **6.41**     |  **8.23**    |\n  |                       | 0.0005 | 3/16 | 6.49    | 8.38     |\n  |                       | 0.0001 | 3/16 | 6.83    |  8.79     |\n  |                       | 0.00001 | 3/16 | 7.08  | 9.19      |\n\n  | Model   | $\\beta$ | W/A | Wiki (PPL)       | C4 (PPL) |\n  |----------|-|---------|-------|--------------|\n  | LlaMa2-7B    | 1    | 3/16 | 6.43     | 8.33     |\n  |                       | **0.9** | 3/16 | 6.41     | **8.23**     |\n  |                       | 0.85 | 3/16 | 6.48     |  8.39     |\n  |                       | 0.8 | 3/16 | 7.12     | 9.46     |\n  | | | | | |\n  |                       | 1      | 2/16 | 16.99     | 24.12     |\n  |                       | 0.9   | 2/16 | 20.88     |  31.78    |\n  |                       | 0.85 | 2/16 | 16.76     | 24.45     |\n  |                       | **0.8** | 2/16 | 16.73     |  **23.73**     |\n\n- **Why MagR+OPTQ not achieve better results in Table 3**:  The reasoning tasks in Table 3 require diverse knowledge and skills. Quantization may impede the model's ability to excel across all tasks. Therefore, a quantized model with better perplexity does not necessarily imply higher multi-task accuracy, especially when the perplexity values are close.\n\n- **Impact of outliers on PTQ**: Outliers have much larger magnitude than other weights, which result in a large quantization step and a large quantization error. MagR addresses this issue by minimizing $\\ell_\\infty$-norm to reduce the quantization step.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TeF2unnPre", "reviewer": "Reviewer_5GSj", "summary": "- The authors propose an optimization-based preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post-training quantization (PTQ) for large language models (LLMs). They motivate their method from previous work, showing that linear transformation of weights can render more quantization-friendly, allowing for sub-4bit quantization. However, such methods normally require a post-processing step during inference to undo that linear transformation, introducing a run-time overhead.\n- Instead, they propose a non-linear transformation without needing an additional post-processing step. That transformation aims to minimise the $\\ell_{\\infty}$ norm, or the maximum value of the weights per channel or column while preserving the layer's output. By making the weight distribution more compact around zero and removing outliers, they are able to quantize weights to 2 or 3 bits with little accuracy drop compared to competing methods.\n- They formulate the optimization as an $\\ell_{\\infty}$ -regularisation and solve it using proximal gradient descent. In fact, the problem reduces to computing the projection of weight vectors onto the $\\ell_1$ ball, for which there are several efficient algorithms available.", "questions": "- Table 1: It is unclear to me how all these numbers are relevant in supporting the method. One can see from the maximum value that all Llama Models are rank-deficient based on the maximum value being below 100%. How are the rest of the statistics informative?\n- Line 124-125: From all the solutions available, why choose the lowest infinity norm? Why is this optimal? What is missing there is the optimality of this norm for quantization purposes. There, in fact, a discussion about quantization noise as a perturbation bounded by $\\ell_{\\infty}$ -norm in a paper by M. Alizadeh et. al called \"Gradient $\\ell_1$ regularization for quantization robustness\"\n- Table 3: activation quantization bitwidth missing\n- Please provide ablation studies about the penalty factor $\\alpha$. What is the effect of it not being tiny? How sensitive are the results?\n- To fully support the claims of the paper, it would be nice to see an ablation study on other norms as regularization. Otherwise, a theoretical justification is required about the optimality of the $\\ell_{\\infty}$ -norm.\n- Paragraph starting at line 210: it is unclear how you establish the optimality of the shrinking factor $\\beta$. In fact, I cannot see why you would not search for the optimal clipping threshold based on minimizing the MSE per column or group. This is a standard method for weight clipping in LLM quantization literature.\n- Section 5.3: the section's title is misleading. Maybe rename it to preprocessing runtime? Most commonly, runtime refers to the inference runtime\n- A section on inference runtime comparison to QuIP is required. The main claim of the paper is that the method is faster than QuIP due to the lack of transformation, but no experimental evidence is provided to support it.", "limitations": "The paper lacks a thorough discussion of each limitation. The only reference I found was in lines 245-6 about extending MagR into incoherence processing.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- MagR achieves state-of-art weight quantization without introducing any additional computation overhead. This is a very important contribution as all other competing methods with the linear transformation of weight, such as QuIP, require a linear transformation of the transformer block input/output. Having said this, the inference runtime comparison to QuIP is missing from the experiments section\n- The authors extensively evaluate the methods, evaluating a large corpus of datasets and benchmarks. They also prove that they outperform strong competing methods, such as OmniQuant, in almost all cases.\n- The idea of regularizing the weight's $\\ell_{\\infty}$ norm is quite novel and very effective. In fact, I am surprised this has not been established before in PTQ literature.", "weaknesses": "- The paper is very focused on ppl and accuracy results on a wide range of benchmarks and datasets at the cost of other important ablation studies. For example:\n\t- ablation study on the penalty factor $\\alpha$ and its impact on the final accuracy.\n\t- Inference runtime comparison is missing compared to competing methods, such as QuIP. In fact, the main claim of the paper is that it is faster than QuIP, but there are no studies supporting it.\n\t- Results with activation quantization beyond FP16 and runtime improvements vs baselines.\n- The connection between the rank-deficient feature matrix argument and $\\ell_{\\infty}$ regularisation requires further theoretical justification. There must be a theoretical guarantee for the optimality of that optimizaiton. In fact, the method would be stronger if it was formulated more generally as an $\\ell_p$ constrained, and the authors show that $\\ell_{\\infty}$ is better constrained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! We'll discuss the reference by Alizadeh et. al in the revised paper.\n\nThe primary concerns were regarding:\n\n**Weaknesses:** \n\n\n  - **The use of** $\\ell_\\infty$-**norm**: MagR aims to reduce the range of the pre-trained weights (not the quantization noise), and $\\ell_\\infty$-norm is naturally the best to serve this purpose. So the nature of MagR is different than $\\ell_1$-gradient regularization proposed by Alizadeh et. al. Given that the quantization step (or float scalar) is linearly proportional to the range of the pre-trained weights for a fixed bit-width (by the definition of uniform quantizer), **MagR results in a smaller quantization step. This, in turn, leads to a smaller quantization error**. In fact, we can show that the $\\ell_2$ quantization error is $O(\\delta)$.\nOur additional experiments also demonstrate that **the layer-wise quantization errors indeed are reduced by applying MagR**; see the figure in the attached pdf file. Another notable advantage of $\\ell_\\infty$-norm over the general $\\ell_p$-norm is its closed form of proximal operator, which is the core of the algorithm design for MagR. \n\n  - **Activation quantization**: By its nature, MagR is designed to improve the performance of weight quantization without introducing inference overhead. So we evaluated our method for only weight quantization. We intend to investigate activation quantization in our future work.\n\n- **Ablation study on $\\alpha$**:  $\\alpha$ is the parameter balancing the tradeoff between the output discrepancy and the max magnitude of the weights. We show the ablation study on $\\alpha$ for channel-wise quantization on LLaMA2-7B as below. Note that **the choice of $\\alpha$ does not depend on the bit-width** and $\\alpha=0.001$ is the best choice for channel-wise quantization.\n\n  | Model   | $\\alpha$ | W/A | Wiki (PPL)       | C4 (PPL) |\n  |----------|-|---------|-------|--------------|\n  | LlaMa2-7B    | 0.005 | 4/16 | 5.84     | 7.55     |\n  |                       | **0.001** | 4/16 | **5.70**   |  **7.28**    |\n  |                       | 0.0005 | 4/16 | 5.72     | 7.29     |\n  |                       | 0.0001 | 4/16 | 5.78     |  7.35     |\n  |                       | 0.00001 | 4/16 | 5.81     | 7.40     |\n  |     |     |       |          |\n  |                       | 0.005 | 3/16 | 6.64     | 8.74     |\n  |                       | **0.001** | 3/16 | **6.41**     |  **8.23**    |\n  |                       | 0.0005 | 3/16 | 6.49    | 8.38     |\n  |                       | 0.0001 | 3/16 | 6.83    |  8.79     |\n  |                       | 0.00001 | 3/16 | 7.08  | 9.19      |\n\n\n  \n**Questions:**\n\n  - **Fraction ranks in Table 1**: The statistics in Table 1 not only show that all feature matrices are nearly rank-deficient, but also gives us an idea of how low-rank they are and the overall distribution of their fraction ranks across the architectures. Intuitively, MagR works better for low-rank feature matrix because its kernel space would be larger (of a higher dimension), and MagR potentially produces weights with smaller $\\ell_\\infty$-norm.\n\n- **The choice of $\\beta$**:  Thanks for your suggestion. Since our main focus is on MagR preprocessing to reduce the quantization error, we prioritized simplicity and efficiency of the quantization algorithm by fixing the $\\beta$, which has proven to be effective.  We agree that optimizing the quantization error with respect to $\\beta$ could potentially further improve the performance, and we intend to investigate this in the revision.\n\n- **Preprocessing runtime**: Thanks for your suggestion, we will rename it.\n\n- **Inference time**: MagR essentially replaces the original pre-trained weights with a new set of weights that have a smaller magnitudes prior to actual quantization, without sacrificing the original accuracy or altering the architecture. Consequently, our MagR+OPTQ achieves **exactly the same inference efficiency** as OPTQ. This is immediately supported by the widely-used inference kernel from the AutoGPTQ library. In contrast, at inference time, QuIP requires performing a random linear transformation on the activations before multiplying the quantized weights. Since the code for QuIP's inference is not released, we cannot compare them directly.  But according to QuIP's own report, QuIP's inference speed is at least **1.5 times slower than OPTQ** for the OPT-66B model (81 ms vs 53 ms).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive feedback. We address the common concerns below:

**(1) Why MagR works:** 
- MagR effectively reduces the range of the pre-trained weights by employing $\ell_\infty$-minimization, as illustrated by Figure 1. Given that the quantization step (or float scalar) is linearly proportional to the range of the pre-trained weights for a fixed bit-width (by the definition of uniform quantizer), **MagR results in a smaller quantization step. This, in turn, leads to a smaller quantization error**. In fact, given the activations $X\in\mathbb{R}^{m\times n}$, quantizer $\mathcal{Q}$ with quantization step $\delta>0$, pre-trained weights $\hat{w}\in\mathbb{R}^n$, the following analysis shows that **the $\ell_2$ quantization error is linear in $\delta$**:  
$$|| Xw_q - X\hat{w}|| \leq ||X \mathcal{Q}(\hat{w}) - X\hat{w}|| \leq \sigma_{\max}(X) ||\mathcal{Q}(\hat{w}) - \hat{w}|| \leq \frac{\sigma_{\max}(X) \sqrt{n}}{2}\delta.$$ 
Our additional experiments also demonstrate that **the layer-wise quantization errors indeed are reduced by applying MagR on randomly sampled layers; see the figure in the pdf file**.

- We also note that MagR can preserve all the layers' outputs (before performing quantization), ensuring that the pre-trained model's performance remains unaffected. **This is crucial in the PTQ setting, as the goal is to search for the quantized model only within the local neighborhood of the pre-trained model**. The following table shows that MagR preprocessing indeed approximately maintains the perplexity (ppl) of the pre-trained model with minor degradation. Since we are minimizing the regularization $ \frac{1}{2}|| Xw - X\hat{w}||^2 + \alpha ||w||_{\infty}$ for preprocessing, for the minimizer $w^*$ it holds that $|| Xw^* - X\hat{w}|| = O(\sqrt{\alpha})\to 0$, as $\alpha \to 0$. When choosing a small $\alpha$ (equivalent to imposing large penalty on the fidelity term),  $|| Xw^* - X\hat{w}||$ will be small but not 0. This explains why the model performance degrades slightly after MagR (before quantization).

  | Model   | Method | Wikitext2 (PPL)       | C4 (PPL) |
  |----------|-|----------------|--------------|
  | LlaMa2-7B    | Original | 5.47     | 6.97     |
  |                       | After MagR | 5.52     | 7.04     |
  | LlaMa2-13B    | Original | 4.88     | 6.46     |
  |                         | After MagR | 4.92     | 6.52     |
  | LlaMa2-70B    | Original | 3.31     | 5.52     |
  |                         | After MagR | 3.35     | 5.56     |

**(2) Inference efficiency:** MagR essentially replaces the original pre-trained weights with a new set of weights that have a smaller magnitudes prior to actual quantization, without sacrificing the original accuracy or altering the architecture. Consequently, our MagR+OPTQ achieves **exactly the same inference efficiency** as OPTQ. This is immediately supported by the widely-used inference kernel from the AutoGPTQ library. In contrast, at inference time, QuIP requires performing a random linear transformation on the activations before multiplying the quantized weights. Since the code for QuIP's inference is not released, we cannot compare them directly.  But according to QuIP's own report, QuIP's inference speed is at least **1.5 times slower than OPTQ** for the OPT-66B model (81 ms vs 53 ms), in addition to the power consumption overhead.

**(3) Preprocessing time:**  We would like to highlight that a major contribution of our work is efficiently addressing the computational challenge of large-scale $\ell_\infty$-regularization with matrix variables. The preprocessing times for MagR (without quantization) on **a single A100 GPU** are modest: **15 min for Llama2-7B, 30 min for 13B, and 3.5 hr for 70B**. We believe that MagR is readily applicable to larger LLMs with 100+B parameters.

**(4) Ablation study on** $\alpha$:  $\alpha$ is the parameter balancing the tradeoff between the output discrepancy and the max magnitude of the weights. We show the ablation study on $\alpha$ for channel-wise quantization on LLaMA2-7B as below. Note that **the choice of $\alpha$ does not depend on the bit-width** and $\alpha=0.001$ is the best choice for channel-wise quantization.

  | Model   | $\alpha$ | W/A | Wiki (PPL)       | C4 (PPL) |
  |----------|-|---------|-------|--------------|
  | LlaMa2-7B    | 0.005 | 4/16 | 5.84     | 7.55     |
  |                       | **0.001** | 4/16 | **5.70**   |  **7.28**    |
  |                       | 0.0005 | 4/16 | 5.72     | 7.29     |
  |                       | 0.0001 | 4/16 | 5.78     |  7.35     |
  |                       | 0.00001 | 4/16 | 5.81     | 7.40     |
  |     |     |       |          |
  |                       | 0.005 | 3/16 | 6.64     | 8.74     |
  |                       | **0.001** | 3/16 | **6.41**     |  **8.23**    |
  |                       | 0.0005 | 3/16 | 6.49    | 8.38     |
  |                       | 0.0001 | 3/16 | 6.83    |  8.79     |
  |                       | 0.00001 | 3/16 | 7.08  | 9.19      |



In light of our responses to the reviewers' concerns, we would be very grateful if you would reconsider your opinion. We believe our work proposes a simplistic and effective method for quantizing LLMs without introducing any inference overhead.

Meta-review: This paper studies a post-training quantization (PTQ) method to compress LLMs, such as the LLAMA family of models. Recent PTQ approaches use linear transformations to make quantization easier but introduce computational overhead when making predictions. Therefore, this paper proposes a new PTQ method, MagR, that does not increase the inference computation. The main idea is to optimize the pretrained weights to have a smaller l-inf norm without changing the original layer-wise features, which is formulated in Eqn. (2). When the range of pre-trained weights is reduced, the quantization step (which is proportional to the range) is therefore reduced, which leads to reduced quantization error. The computation of the l-inf constrained optimization problem is solved efficiently using a proximal gradient descent algorithm. The paper shows promising results in the sub-4bit quantization regime (INT4/INT3/INT2) when compared to other PTQ methods.
Reasons for acceptance:
- Novel idea in reducing the weight range for PTQ.
- Fast calculation of the L-inf regularization problem (15 min for Llama2-7B on a single A100 GPU).
- No overhead in inference time compared to other linear-transform-based PTQ methods.
- Principled methods for weight reduction and a clear explanation of why the idea of reducing the weight range works.
- Potential to be combined with other PTQ methods.

This paper achieved unanimous support from the reviewers after lots of discussions. The paper can be further improved after incorporating some of the discussions with the reviewers, e.g., explicit discussions on inference time savings, prior work on regularization for better model quantization, and other suggested experiments and ablation studies by the reviewers.
Decision: Accept (poster)