MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization

Authors: aozhong  zhang
naigang  wang
yanxia  deng
xin  li
zi  yang
penghang  yin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:01:00

Abstract: In this paper, we present a simple optimization-based preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post-training quantization. For each linear layer, we adjust the pre-trained floating-point weights by solving an $\ell_\infty$-regularized optimization problem. This process greatly diminishes the maximum magnitude of the weights and smooths out outliers, while preserving the layer's output. The preprocessed weights are centered more towards zero, which facilitates the subsequent quantization process. To implement MagR, we address the $\ell_\infty$-regularization by employing an efficient proximal gradient descent algorithm. Unlike existing preprocessing methods that involve linear transformations and subsequent post-processing steps, which can introduce significant overhead at inference time, MagR functions as a non-linear transformation, eliminating the need for any additional post-processing. This ensures that MagR introduces no overhead whatsoever during inference. Our experiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models. For example, we achieve a Wikitext2 perplexity of 6.7 on the LLaMA2-70B model for per-channel INT2 weight quantization without incurring any inference overhead.

Main paper: 
 introduction: 
 Large language models (LLMs) have achieved outstanding performance across a broad range of applications, demonstrating remarkable success. However, their unprecedented model size has led to many computation operations and substantial memory footprints, becoming significant barriers to their practical deployment and adoption in production environments. Accordingly, it is highly desirable to develop efficient model compression techniques for LLMs so they can be more widely deployed in resource-limited scenarios. Among the various techniques to compress and accelerate deep neural networks (DNNs), low-precision quantization has proven to be highly effective across numerous application domains and is widely adopted for accelerating DNNs. For LLMs, the inference runtime is dominated by the token generation process, where output tokens are produced sequentially, one at a time. This process is known to be memory bandwidth bound [3, 19] . As a result, the quantization of LLMs has primarily focused on reducing the bit-width of model weights, with the dual goals of lowering the model's footprint to enable deployment on resource-constrained devices and decreasing the memory bandwidth requirements to improve computational efficiency and accelerate inference.The enormous computational demands for pre-training and fine-tuning Large Language Models (LLMs) have led to the emergence of Post-Training Quantization (PTQ) [4, 15, 22, 24, 27, 31, 41, 51, 52, 53, 43, 40] , as a promising solution for quantizing these models. Unlike Quantization Aware Training (QAT) [7, 9, 12, 18, 21, 23, 46, 47, 48, 49] , which is designed to minimize a global training loss for quantization parameters, PTQ directly applies low-precision calibration to a pre-trained full-precision model using a minimal set of calibration samples. By aiming to identify an optimal quantized model locally through the minimization of a simplified surrogate loss, PTQ offers computational savings and resource efficiency compared to QAT. However, PTQ often lags behind QAT in accuracy, particularly for ultra-low precision lower than 4-bit. Thus, it remains an open problem to achieve an improved balance between cost and performance for PTQ-based approaches.Motivation. To achieve state-of-the-art performance, the latest advances in PTQ [8, 25, 26, 36, 42] have proposed applying a linear transformation to process the pre-trained weights within a linear layer. This strategy of linear transformation aims to make the weights more suitable for the subsequent quantization procedure by reducing their magnitudes and suppressing outliers. In a nutshell, given the features X and weights W , one constructs linear transformation T such that T W is better conditioned than W in terms of being quantization-friendly. Such designs of T include diagonal matrices (so-called channel-wise scaling) [25, 36, 42] , random transformations [8, 39] , and finite frames [1, 13] . Then, quantization is performed on T W instead of the original weights W . To preserve the layer's output, however, the inverse transformation T -1 has to be in turn applied to the features X, namely, XW = (XT -1 )(T W ) ≈ (XT -1 )Q(T W ),with Q(T W ) being the quantized weights. PTQ done this way requires modifications on the original neural architecture, which involves additional computations of XT -1 and extra memory storage for T -1 at inference time. As a result, these steps introduce overhead that offsets the benefits provided by quantization. This raises a natural question:Can we effectively process the weights at the preprocessing stage to facilitate quantization without introducing inference overhead?To address this problem, we propose a simple optimization-based technique called Weight Magnitude Reduction (MagR). MagR functions as a non-linear transformation on weights without altering the original features/activations. The optimization program is designed to find new weights with minimal maximum magnitude, i.e., the ℓ ∞ norm, while preserving the layer's outputs.Contributions. We propose a non-linear approach, MagR, based on channel-wise ℓ ∞ -regularized least squares, to reduce the quantization scale without compromising the performance of pre-trained model, facilitating subsequent weight quantization while requiring no post-processing or inference overhead. See Figure 1 for comparing weight magnitudes before and after applying MagR. To address the ℓ ∞ -regularization problem, we develop an efficient and parallelizable proximal gradient descent algorithm that involves computing ℓ 1 -ball projections at each iteration. Specifically, MagR preprocessing on a single Nvidia A100 GPU takes merely 15 min for LLaMA2-7B and 3.5 hr for the 70B model. Our results on INT weight-quantization demonstrate that MagR can significantly boost the performance in the sub-4bit regime when combined with fast gradient-free methods for layer-wise PTQ, such as rounding-to-nearest (RTN) [30] and OPTQ [16] . This approach achieves performance for weight quantization at least comparable to state-of-the-art PTQ methods on natural language processing (NLP) tasks, including gradient-based methods using block-wise reconstruction. 
related work: 
 Recently, as the sizes of language models are exploding, there has been growing interest in developing post-training quantization (PTQ) methods [8, 16, 25, 26, 36, 44, 45] for large-scale AI models like large language models (LLMs) to reduce the model sizes and accelerate inference by representing weight matrices in low precision. PTQ methods directly find the low-precision representation of the model without re-training, thereby preferred by extreme large-scale AI models. The OPTQ [16] uses approximate second-order information to calibrate the quantization. The method successfully compresses LLMs into 3 or 4 bits and can achieve reasonable accuracy in 2 bits. Researchers have found that the extreme values and the distribution of the weight entries highly affect the quantization errors and the quantized model quality. The original weight can be converted into a more quantizationfriendly one by linear transformations. The approach can significantly reduce the quantization errors while bringing more time overhead during inference because of the linear transformation. OmniQuant [36] proposes learnable weight clippings and equivalent transformations to avoid the influence of extreme values. AWQ [25] searches for the most significant entries in the weight by looking at the activation and selects the scales that protect these entries. SmoothQuant [44] passes the difficulty Figure 1 : Motivation behind MagR: we can effectively reduce the magnitude of weights at the preprocessing stage. Each point denotes the maximum magnitude before (x-coordinate) and after (y-coordinate) applying MagR within a sampled channel (or column) of the weight matrix from three random layers of LLaMa2-7B [38] . These column-wise maximum magnitudes are typically more than halved through MagR.in activation quantization to weights by an equivalent linear transformation. QuIP [8] , AffineQuant [26] and FrameQuant [1] apply a linear transformation before quantization to make the transformed weight quantization-friendly. These approaches achieve high performance for extreme bits, like 2 bits, but introduce additional inference overhead though the transformation is carefully designed to be efficient. OmniQuant [36] and AffineQuant [26] can be adopted for weight-activation quantization by considering the activations in the proposed methods. The work [45] introduces a low-rank compensation method on top of other quantization methods, which employs low-rank matrices to reduce quantization errors with a minimal increase in model size. By modeling the quantization residual as an ℓ ∞ -bounded perturbation, [2] proposes applying an ℓ 1 penalty on the gradient of loss to enhance quantization robustness.The works most closely related to ours are [20] and [27] , both utilizing ℓ ∞ norm to regularize or constrain the weight range to a smaller scale. The Range Regularization (R 2 ) method [20] applies an ℓ ∞ penalty or its variants to the conventional network loss to regularize the weight range during end-to-end model pre-training, optimized via SGD. However, this approach becomes practically infeasible for large-scale models. In [27] , a layer-wise pre-processing technique is proposed, which involves solving an intractable ℓ 0 -minimization problem while constraining the ℓ ∞ -norm of weights.Table 6 shows the results for WikiText2 and C4 perplexity on the LLaMA1. Impact of the parameter α. The tiny penalty parameter α balances the trade-off between output discrepancy and the maximum magnitude of the weights. We carry out experiments on channel-wise quantization for differernt α on LLaMA2-7B. The choice of α is independent of the bit-width. As shown in Table 7 , we can find that both too large and too small α will lead performance degeneration. Compared to INT4, fluctuations in alpha at INT3 result in greater performance fluctuations. Fortunately, α = 0.001 works well for all channel-wise quantization.Impact of the parameter β. We shrink the quantization step to reduce the overall quantization error by a multiplicative scalar β. To investigate the influence of β, we experiment with different value of β at INT3 and INT2 channel-wise quantization. As shown in Table 8 , β is positively correlated with the bit-width. Specifically, the best β is around 0.9 for INT3 quantization and for INT2 quantization the optimal β is around 0.8. Impact of MagR on Quantization Error. To explore how MagR affects quantization error, we compared the errors with and without MagR by randomly select five layers from LLaMA2 models. As illustrated in Figure 2 , quantization error is notably reduced across all layers with the application of MagR.Figure 2 : Layer-wise quantization errors (root mse) for MagR+OPTQ and OPTQ, respectively, for 4-bit quantization. The layers are selected randomly for visualization, but improvement is consistent across all layers.The adaptive capacity of MagR. We investigated the combined effects of MagR and QuIP. As illustrated in Table 9 , incorporating MagR significantly enhances the performance of QuIP, leading to improved quantization results for the LLaMA2 models family. Justification: The proposed method is shown in Section 4 and demonstrated by experiments in Section 5. They accurately reflect the contributions discussed in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The proposed method needs mild parameter tuning to achieve the best results. It is discussed in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
background: 
 First, we clarify the mathematical notations that will be used throughout this paper:Notations. We denote vectors by bold small letters and matrices by bold capital ones. For a positive integer n, [n] := {1, 2, . . . , n} denotes the set containing all positive integers up to n. For any two vectors x, y ∈ R n , ⟨x, y⟩ := n i=1 x i y i is the inner product. We denote by ∥x∥:= ⟨x, x⟩ = n i=1 x 2 i the Euclidean norm; ∥x∥ 1 := n i=1 |x i | is the ℓ 1 -norm; ∥x∥ ∞ := max 1≤i≤n |x i | is the ℓ ∞ -norm. For any matrix X ∈ R m×n , X ⊤ ∈ R n×mis the transpose. We denote the spectrum norm of X by ∥X∥ = σ max (X), which equals its maximum singular value. Its Frobenius norm is given by ∥X∥ F = m i=1 n j=1 X 2 i,j . Moreover, for vectors x and y, x ⊙ y := (x 1 y 1 , . . . , x n y n ) ∈ R n denotes the Hadamard or element-wise product, and likewise for two matrices.Layerwise PTQ. Post-training quantization via layerwise reconstruction calls for solving a least squares problem with a discrete constraint. For the pre-trained weights W within a linear layer, we aim to find the quantized weights W q that minimize the following function minEQUATIONwhere X ∈ R (b•l)×m is the feature matrix associated with a batch of calibration data consisting of b samples stacked together, and each data sample is represented by an l × m sub-matrix. Q ⊂ R m×n is an appropriate set of all feasible quantized weights.The most straightforward PTQ technique, known as RTN, involves directly rounding the weight matrix W without utilizing any additional data. An improvement over RTN was introduced by AWQ [25] , which enhances the quantization process by incorporating channel-wise scaling on W .Thanks to the simplicity of the layer-wise formulation (1), several efficient gradient-free algorithms [4, 16, 51, 53] have been recently proposed to address layer-wise quantization, including OPTQ. Built on top of OPTQ, QuIP subjects X and W to random orthogonal transformations to produce "incoherent" weight and Hessian matrices, leading to superior accuracy with sub-4bit quantization. However, this advantage comes with a trade-off; during inference, QuIP requires random orthogonal transformations on the feature inputs of linear layers, rendering noticeably slower throughput compared to OPTQ.Uniform Quantizer. Given a set of points w ∈ R m , the commonly-used (asymmetric) uniform quantizer [9] defines the quantization step δ = max(w)-min(w) w q = δ • clamp w δ -z, 0, 2 b -1 + z .In per-channel (or per-group) PTQ, the quantization step δ is conventionally calculated based on the channel-wise (or group-wise, respectively) minimum and maximum values of the pre-trained weights W , as defined above, and remains constant throughout the quantization procedure.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: The paper does not include any theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
the proposed method: 
 In this section, we present the Weight Magnitude Reduction (MagR) method based on ℓ ∞ -norm regularization, which is applied just before the quantization step within each linear layer. The intuition behind MagR is based on the following simple estimate of the layer-wise quantization error. Given the feature/activation matrix X, the quantizer Q, and any pre-trained weights w ∈ R m , we have:min wq∈Q ∥Xw q -Xw∥ ≤ ∥X(Q(w) -w)∥ ≤ ∥X∥∥Q(w) -w∥ ≤ σ max (X) √ m 2 δ,where δ = max(w)-min(w)2 b -1is the quantization step size. This shows that reducing the range of weights helps to suppress the quantization error. With this in mind, MagR preprocessing is designed to achieve two key effects:• First, it effectively reduces the channel-wise (or column-wise) maximum magnitude of the weights, as illustrated by Figure 1 . • Second, it preserves the model's original performance with minimal accuracy loss after preprocessing. Table 1 demonstrates that MagR preprocessing maintains the perplexity of the pre-trained models, with only minor degradation. To illustrate the idea behind the proposed MagR method, let us consider a pre-trained weight vector ŵ ∈ R m of a linear layer and the associated feature input matrix X. MagR leverages the fact that the feature matrix X across all layers of LLMs is approximately rank-deficient. Specifically, if X is exactly rank-deficient, the linear system modeling the layer's output, Xw = X ŵ with variables w, generally has infinitely many solutions. That is, for any ν in the non-trivial kernel space of X, we have that w = ŵ + ν preserves the layer's output. Among all solutions, MagR aims to identify the weight vector w with the smallest extreme value in magnitude. In [8] , the authors empirically observed that the Hessian matrix X ⊤ X is approximately low-rank across all layers in open pre-trained (OPT) models [54] . Here we examined the feature matrix of LLaMA models [37, 38] . Our approximate fraction rank of the feature matrix X is defined as the fraction of singular values of X such that σ(X) > 0.01 • σ max (X). Table 2 illustrates that all feature matrices extracted from LLaMA models are indeed rank-deficient according to this definition.Let us consider the quantization of a weight vector for simplicity. Given pre-trained weight vector ŵ, we would like to find a new set of weights w with the smallest maximum magnitude, such that the layer output is preserved up to a small error ε > 0, i.e.,min w∈R m ∥w∥ ∞ subject to ∥Xw -X ŵ∥ ≤ ε.To efficiently implement MagR, we consider the following mathematically equivalent ℓ ∞regularization problem instead:EQUATIONwhere α > 0 serves as the regularization parameter, balancing fidelity against the ℓ ∞ regularizer. To maintain the output of the layer, α should typically be set to a small value. Indeed, let w * be the minimizer of (2), we have that the ℓ 2 error of the layer's output introduced by MagR is O( √ α):∥Xw * -X ŵ∥ ≤ ∥Xw * -X ŵ∥ 2 + 2α∥w * ∥ ∞ ≤ ∥X ŵ -X ŵ∥ 2 + 2α∥ ŵ∥ ∞ = 2α∥ ŵ∥ ∞ ,where ∥ ŵ∥ ∞ is a constant independent of α, and the second inequality uses that w * is the minimizer.Proximal Gradient Descent. Note that ℓ ∞ -norm is a convex but non-differentiable function. In theory, the optimization problem (2) can be simply solved by a subgradient algorithm, but it is significantly slower than the more sophisticated proximal gradient algorithm which matches the convergence rate of standard gradient descent.With the step size η > 0, proximal gradient descent [32] takes the following iteration:EQUATIONwhere prox t∥•∥∞ with the scalar t > 0 is the (scaled) proximal operator of ℓ ∞ -norm function, defined asprox t∥•∥∞ (v) := arg min x∈R m 1 2 ∥x -v∥ 2 + t∥x∥ ∞ .To ensure the convergence of (3), it is sufficient to choose the step sizeη ≤ 1 λ max (X ⊤ X),where λ max (X ⊤ X) is the maximum eigenvalue of X ⊤ X.Proximal Operator of ℓ ∞ -Norm. It remains to determine the proximal operator of ℓ ∞ -norm. It turns out we can compute it by leveraging the celebrated Moreau decomposition [29, 32] : for any t > 0,EQUATIONThat is, computing the proximal operator of ℓ ∞ norm amounts to evaluating the projection onto ℓ 1 ball, which is defined asproj ∥•∥1≤1 (v) := arg min x∈R m ∥x -v∥ 2 subject to ∥x∥ 1 ≤ 1.Fortunately, computing projection onto the ℓ 1 ball is an established task, and there are several efficient algorithms available. For example, see [11] and the references therein. Here we adopted a simple algorithm of O(m log m) time complexity as in [14] , which supports parallelizable or vectorized implementation for the projections of a batch of weight vectors, i.e., a weight matrix, as will be described in the next subsection. The implementation mainly involves sorting and soft-thresholding [50] ; see Algorithm 3 and its derivation in Appendix A.1 for the details.MagR for Weight Matrix. In practical implementation of MagR, we preprocess the entire weight matrix W = [w 1 , . . . , w n ] ∈ R m×n within each linear layer. For per-channel quantization (or per-column quantization in our setting), the ℓ ∞ penalty is imposed column-wise on the weight matrix to reduce the quantization scale of each channel. That is, MagR amounts to solvingmin W ∈R m×n 1 2 ∥XW -X Ŵ ∥ 2 F + α n j=1 ∥w j ∥ ∞In this case, we take the following iteration:W k+1 = prox ηα∥•∥∞ W k -η • X ⊤ X(W k -Ŵ ) ,where the proximal operator prox t∥•∥∞ and the corresponding projection proj ∥•∥1≤1 in (4) are applied column-wise to the matrix input. Hereby we summarize MagR for processing one linear layer in Algorithm 1 with the column-wise ℓ 1 -ball projection as detailed in Algorithm 2, which generalizes Algorithm 3 in Appendix A.1, by handling matrix inputs (or batches of vectors).Algorithm 1 Per-channel MagR for one linear layer.Input: Pre-trained weight matrix Ŵ ∈ R m×n ; Hessian matrix H = X ⊤ X ∈ R m×m ; max iteration number K; step size η = 1 λmax(H) ; penalty parameter α > 0. Output: Preprocessed weights W ∈ R m×n . 1: Initialize W 0 = Ŵ . 2: for k = 0, . . . , K -1 do 3: V k = W k -η • H(W k -Ŵ )gradient descent step 4:W k+1 = V k -ηα • proj ∥•∥1≤1 V k ηα proj ∥•∥1≤1 is described in Alg. 2 5: end for 6: return W = W KExtension to Per-Group Quantization. By using more float scaling factors, per-group quantization becomes a preferred strategy for mitigating accuracy loss at extremely low bit-widths. In this approach, a weight vector w ∈ R m is segmented into groups of weights, each containing d elements, with all weights within a group sharing a common scaling factor for quantization. Here, per-group MagR applies an ℓ ∞ penalty to each vector of grouped weights. Consequently, the ℓ 1 -ball projection is independently performed on these vectors, while maintaining the gradient descent step unchanged. We note that the group-wise ℓ 1 -ball projection can be easily done using Algorithm 2, with an additional reshaping of the inputV ∈ R m×n into R d×( m d •n) .Algorithm 2 Column-wise projection onto the unit ℓ 1 -ball.Input: Matrix V ∈ R m×n ; the radius of ℓ 1 ball, ϵ = 1. Output: W ∈ R m×n such that all columns ∥w j ∥ 1 ≤ ϵ, ∀ j ∈ [n].1: Create a binary mask M ∈ R m×n filtering out the columns of V with ∥v j ∥ 1 ≤ ϵ.2: Sort |V | column-wise in descending order into U .3: Find indexρ j = max i ∈ [m] : u i,j > 1 i i r=1 u r,j -ϵ , ∀ j ∈ [n] 4: Define θ j = 1 ρj ρj r=1 u r,j -ϵ , ∀ j ∈ [n] 5: Tile θ ∈ R n into Θ ∈ R m×n along the row. 6: Compute W = (1 -M ) ⊙ V + M ⊙ sgn(V ) ⊙ max{|V | -Θ, 0} 7: return WQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Implementation details and parameter setup are clearly stated in Section 5.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 Overview. We tested the proposed MagR for INT4, INT3, and INT2 weight quantization. In our notations, the weight and activation bits are denoted by 'W' and 'A', respectively. Additionally, we implemented group-wise weight quantization with the group size denoted by 'g'. For example, W2A16g128 signifies INT2 weight and FP16 activation (i.e., INT2 weight-only quantization) with a group size of 128.We employed our MagR processing approach on top of the two gradient-free PTQ methods in main text, RTN and OPTQ [16] , to quantize the LLaMA1 (7B-65B) [37] and LLaMA2 (7B-70B) [38] model families. In the Appendix A.2, we extend MagR with QuIP [8] (MagR+QuIP) to quantize LLaMA2 (7B-70B) model families. By applying MagR on top of RTN (MagR+RTN), we achieved better results than AWQ [25] for per-channel INT3 and INT4 weight quantization. Additionally, MagR combined with OPTQ (MagR+OPTQ) achieved state-of-the-art performance for INT3 and INT4 qunatization. To enhance the per-channel INT2 quantization, we ran 30 additional iterations of coordinate descent algorithm [4, 51] on top of OPTQ, which we denote by MagR+OPTQ † . It turns out MagR+OPTQ † is superior to both Omniquant [36] and QuIP [8] in terms of perplexity (Table 9 ), and falls just short of QuIP in zero-shot tasks for 13B and 70B models (Table 4 ). Note that QuIP uses random orthogonal transformations (so-called Incoherence Processing) to process both the weights and features, resulting in 1.5× slower throughput than OPTQ. In contrast, MagR-based method does not introduce any overhead whatsoever compared with OPTQ.In conclusion, our MagR-based PTQ method is intuitive yet effective in compressing models into extreme bit-widths, while maintaining performance without introducing any inference overhead.Datasets and Evaluation. Following the previous work [16, 25, 36] , we evaluate the quantized model on language generation tasks on WikiText2 [28] and C4 [33] . Additionally, we test its performance on zero-shot tasks, including PIQA [5] , ARC (Easy and Challenge) [10] , and Winogrande [35] . For the language generation experiments, our implement is based on the OPTQ's [16] repository, which is built using PyTorch. For executing all zero-shot tasks, we adhere to the lm-eval-harness [17] .Baseline: For the language generation task, we compare our method with RTN, OPTQ [16] , AWQ [25] and OmniQuant [36] on LLaMA1 and LLaMA2 models. In addition to the aforementioned methods, we also conduct a comparison with QuIP [8] on the LLaMA2-70B model. In the zero-shot task, we focus on four individual tasks and compare the average accuracy across all four tasks with Omniquant [36] .Implementation details. We utilized the HuggingFace implementations of the LLaMA1 and LLaMA2 models and perform quantization on a single NVIDIA A100 GPU with 80GB of memory. Following the OPTQ method, we load one block consisting of 7 linear layers into GPU memory at a time. In line with previous work [8, 16] , the input matrix X is obtained by propagating the calibration data through the quantized layers.The choice of parameters. To ensure that the MagR-processed layer output XW is faithful to the original X Ŵ , we need to use a tiny penalty parameter α in (2) . For per-channel quantization, α was fixed to be 10 -3 in our experiments, but we did find that setting it to a smaller value of 5 × 10 -4 or 10 -4 can sometimes slightly improve the perplexity (with a relative change of < 1% in ppl). Similarly for per-group quantization, we set α to 10 -4 , while reducing it to 5 × 10 -5 or 10 -5 could sometimes also slightly improve the perplexity. An ablation study on α is provided in the Appendix A.2. Furthermore, we used a multiplicative scalar β < 1 to decay the standard quantization step δ = max(w)-min(w) 2 b -1(or equivalently, the quantization scale) of the quantizer. In other words, our δ = β • max(w)-min(w) 2 b -1. It has been shown in existing works [21, 34] that, optimal quantization step for binary or ternary quantization yielding the minimum quantization error is not given by max(w)-min(w) 2 b -1. Shrinking δ at low bit-width results in a more clustered quantization grid lattice that fits the weights better, which leads to a smaller overall error. In general, β is positively correlated with the bit-width used. For per-channel quantization, the best β ∈ [0.8, 0.85] on INT2 quantization, whereas the empirically optimal β is around 0.9 for INT3 quantization. As for INT4, β is simply set to 1, that is, we used the standard quantization step. In addition, for per-group quantization, we chose β = 0.95 for both INT2 and INT3 quantization. The ablation study of β is in the Appendix A.2. We observed that this refinement on the quantization step δ significantly improves the performance of the PTQ method. In addition, the iteration number K in Algorithm 1 was set to 150 across all the experiments. We evaluated the performance of quantized models on several zero-shot tasks. The results are reported in Table 4 . Similar to previous observations, the proposed MagR demonstrates superior performance on most models compared to OmniQuant, with a small gap compared to QuIP [8] . Nonetheless, it is reasonable and commendable that our algorithm achieves results close to QuIP without introducing any inference overhead. It is possible to further improve our approach based on the insight behind QuIP [8] -i.e., quantization benefits from incoherent weight and Hessian matrices; see Table 9 for the results in the appendix.We report the execution time of MagR+RTN and MagR+OPTQ on a single NVIDIA A100 GPU in Table 5 . For example, it typically took 0.5-7.5 hours for MagR+OPTQ to quantize the LlaMA2 models. We note that the integration of MagR can markedly enhance the performance of the standard OPTQ [16] . It is noted that MagR+OPTQ † for INT2 weight quantization requires a longer runtime due to the additional CD iterations, extending the quantization process for LLaMA2-70B to 31 hr. It also reveals that the preprocessing overhead for quantizing the LLaMA2 models (7B-70B) amounts to approximately 15 min, 30 min, and 3.5 hr, respectively. In comparison, our total runtime is roughly half of that of the gradient-based method, OmniQuant [36] , while achieving at least comparable results. Moreover, MagR introduces no post-processing step or overhead during inference.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Please refer https://github.com/AozhongZhang/MagR Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
concluding remarks: 
 In this paper, we proposed MagR, based on ℓ ∞ -regularization, to significantly reduce the maximum weight magnitude of pre-trained LLMs within each layer while preserving their output. MagR is designed to enhance the accuracy of backpropagation-free PTQ methods that use layer-wise reconstruction, such as RTN and OPTQ. MagR produces a more clustered distribution of weights and leads to a smaller quantization step, thereby facilitating the subsequent PTQ task. To solve the ℓ ∞ -regularization problem, we used the classical proximal gradient descent algorithm with ℓ 1 -ball projections, tailored to handle matrix variables efficiently. Our experiments on LLaMA family validated the effectiveness of the MagR approach, achieving the state-of-the-art performance on NLP tasks. Remarkably, unlike existing weight preprocessing techniques that require performing an inverse transformation on features during inference, MagR eliminates the need for post-processing and incurs no overhead. This renders MagR more practical for the deployment of quantized models.A.1 Projection of Vectors Onto ℓ 1 -BallIn this section, we show how to compute the projection onto the unit ℓ1-Ball. That is, for any fixed v ∈ R m , we solve the optimization problem:EQUATIONConsider the Lagrangian L(x, λ) = 1 2 ∥x -v∥ 2 + λ(∥x∥1 -1), where λ ∈ R is lagrange multiplier. Let x * be the optimal solution of (5), then there exits λ * such that the following Karush-Kuhn-Tucker (KKT) conditions [6] hold:• Stationarity: 0 ∈ ∂xL(x * , λ * ) ⇔ x * = sgn(v) ⊙ max{|v| -λ * , 0} • Slackness: λ * (∥x * ∥1 -1) = 0 ⇔ λ * = 0 or ∥x * ∥1 = 1. • Primal feasibility: ∥x * ∥1 -1 ≤ 0 ⇔ ∥x * ∥1 ≤ 1 • Dual feasibility: λ * ≥ 0 ⇔ λ * = 0 or λ * > 0where sgn is the signum function applied element-wise on vectors, i.e.,sgn(v)i =      1 if vi > 0, 0 if vi = 0, -1 if vi < 0.We note that the stationarity condition:x * = sgn(v) ⊙ max{|v| -λ * , 0}gives the projection of v onto the unit ℓ1 ball, provided λ * is known. Therefore, what remains is to find λ * :• Case I: λ * = 0. We have x * = v, which corresponds to the case that ∥v∥1 ≤ 1. which can be solved by sorting.In summary, Algorithm 3 details the implementation of projecting v ∈ R m onto a general ℓ1-ball with radius ϵ.In Step 4, we specifically compute the root λ * (or θ) of ( 6) for Case II.Algorithm 3 Projection onto ℓ 1 -ball.Input: Vector v ∈ R m ; the radius of ℓ 1 ball, ϵ = 1. Output: w ∈ R m such that ∥w∥ 1 ≤ ϵ. 1: if ∥v∥ 1 > ϵ then 2: Sort |v| into µ such that µ 1 ≥ µ 2 ≥ . . . ≥ µ m . 3: Find index ρ = max i ∈ [m] : µ i > 1 i i r=1 µ r -ϵ 4: Define θ = 1 ρ ( ρ r=1 µ r -ϵ) 5: Compute w = sgn(v) ⊙ max{|v| -θ, 0} 6: else 7: w = v 8: end if 9: return w A.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Implementation details and parameter setup are clearly stated in Section 5.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: In Section 5, we report the test results of the quantized models. Given the full-precision model, our algorithm produces the quantized model in a deterministic way. Hence, we don't have statistical significance to report in this paper.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Implementation details are clearly stated in Section 5.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
 
 Appendix:  


Decision: This paper studies a post-training quantization (PTQ) method to compress LLMs, such as the LLAMA family of models. Recent PTQ approaches use linear transformations to make quantization easier but introduce computational overhead when making predictions. Therefore, this paper proposes a new PTQ method, MagR, that does not increase the inference computation. The main idea is to optimize the pretrained weights to have a smaller l-inf norm without changing the original layer-wise features, which is formulated in Eqn. (2). When the range of pre-trained weights is reduced, the quantization step (which is proportional to the range) is therefore reduced, which leads to reduced quantization error. The computation of the l-inf constrained optimization problem is solved efficiently using a proximal gradient descent algorithm. The paper shows promising results in the sub-4bit quantization regime (INT4/INT3/INT2) when compared to other PTQ methods.
Reasons for acceptance:
- Novel idea in reducing the weight range for PTQ.
- Fast calculation of the L-inf regularization problem (15 min for Llama2-7B on a single A100 GPU).
- No overhead in inference time compared to other linear-transform-based PTQ methods.
- Principled methods for weight reduction and a clear explanation of why the idea of reducing the weight range works.
- Potential to be combined with other PTQ methods.

This paper achieved unanimous support from the reviewers after lots of discussions. The paper can be further improved after incorporating some of the discussions with the reviewers, e.g., explicit discussions on inference time savings, prior work on regularization for better model quantization, and other suggested experiments and ablation studies by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper introduces Weight Magnitude Reduction (MagR), a technique designed to smooth out outliers before LLM quantization. MagR adjusts pre-trained floating-point weights by solving an ℓ∞-regularized optimization problem. This preprocessing step reduces the maximum weight magnitudes, making the LLMs more suitable for quantization and resulting in better perplexity/accuracy results compared to models without preprocessing. Importantly, the MagR technique does not introduce any additional computational overhead during inference.
strength_weakness: strength_weakness: 1) MagR does not show significant improvement in perplexity/accuracy over previous method - QuIP. 
Additionally, DecoupleQ [1] presents strong results for 2-bit quantization. While I understand that DecoupleQ was released recently, allowing insufficient time for a thorough comparison, I recommend the authors review DecoupleQ.

2) The paper emphasizes that MagR has no inference overhead, unlike AWQ and QuIP, which require scaling activations. However, it does not provide any latency information to highlight the significance of removing this additional inference overhead.

3) The runtime cost comparison between MagR and other methods is insufficient. The paper only provides the runtime of MagR, mentioning roughly that it is half the runtime of OmniQuant on page 8, lines 252-254. Since MagR is a preprocessing method, it is unclear if this comparison holds when considering the entire quantization process (MagR + OPTQ). Additionally, the runtime of QuIP is not addressed.

4) Some prior works have regularized weight distribution to make the model more suitable for quantization [2]. However, this paper does not mention or compare the differences between the proposed method and these earlier approaches.

5) The paper does not include any information about the calibration dataset.

[1] Guo, Yi, et al. "decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points." arXiv preprint arXiv:2404.12759 (2024).

[2] Kundu, Arnav, et al. "R2 Loss: Range Restriction Loss for Model Compression and Quantization." arXiv preprint arXiv:2303.08253 (2023).
questions: questions: 1) What is the inference latency of the proposed method compared to QuIP?

2) What is the runtime for QuIP and OmniQuant?

3) Why did you not provide perplexity results for the 7B/13B models for QuIP in Table 2?

4) Is QuIP incompatible with group-wise quantization, or can it be adapted to group-wise quantization with minor modifications, as done with MagR and OPTQ?

5)  For the results in Table 3, do all the methods use channel-wise quantization, or do some use group-wise quantization?

6) What type of calibration dataset did you use, and how many data points were included for calibration?

7) What happens if you apply Omniquant/QuIP after processing LLMs with MagR?
limitations: limitations: The proposed MagR is a promising approach, as it regularizes weight values to produce LLMs that are better suited for quantization. However, this paper lacks sufficient evaluation to convincingly demonstrate the superiority of the proposed method compared to state-of-the-art techniques.
paper_summary: paper_summary: Authors propose an optimization-based preprocessing technique called MagR to enhance the performance of post-training quantization.
MagR adjusts weights by solving an l1-regularized optimization problem, reducing the maximum magnitude and smoothing out outliers.
As a nonlinear transformation, MagR eliminates the need for additional post-processing, thus avoiding overhead during inference.
Experiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models, such as significantly reduced perplexity on Wikitext2 for the LLaMA2-70B model.
strength_weakness: strength_weakness: In fact, I like the work but not in authors' view. I think this work is improve the generalization of model via norm. If this paper presents their work in this view, I would like to accept more.
1.Some value is lack of explanation, like " minimal the singular values of the feature matrix are less than 0.01 times the maximum singular value." 
2. The applicability and the nature analysis of methods should be discussed.
questions: questions: 1. In Section 4.1, the singular values of the feature matrix are less than 0.01 times the maximum singular value. How is 0.01 calculated? If this value is changed, relaxing the constraint, will the results change?"
2. If the feature matrix is not actually rank-deficient, could the performance of the MagR method potentially suffer?
3. Proximal gradient descent may require more computational resources.
4. The model performance after preprocessing and before quantization should be presented. I wonder that the nature of this method is improve the generalizability of model before quantization.
limitations: limitations: The paper is lack of discussing the nature of preprocessing, i.e., what the preprocessing do for quantization.
paper_summary: paper_summary: The paper proposes a novel approach called Weight Magnitude Reduction (MagR) to improve the performance of PTQ for LLM. The MagR reduces the magnitude of the weights in PTQ. The experiments demonstrate the effectiveness of MagR.
strength_weakness: strength_weakness: 1.	There exist some typos/grammatical errors in the paper and should be revised.
2.	The presentation is not clear. For example, this paper aims to reduce the parameters of LLM using PTQ, however, the title and abstract did not reflect the contribution of LLM compression. In the abstract, the author claims MagR can diminish the maximum magnitude of the weights and smooth out outliers, however, the title only focuses on Weight Magnitude Reduction.
3.	Provide a more thorough discussion of the generalization ability, robustness, and potential applications of the proposed approach.
4.	The format of references is not correct, i.e., [1], [3], [9],[14], etc.
5.	The main limitation of this paper is that the proposed method lacks theoretical analysis.
6.	While the abstract provides an overview of the proposed method, some aspects of the methodology could benefit from further elaboration in the main paper. Providing step-by-step explanations and intuitive visualizations for key components such as L infinity norm, and L infinity-> L 1 norm, would enhance the reader's understanding.
7.	The paper claims that the method outperforms state-of-the-art methods, but a more comprehensive comparative analysis is needed. Detailed comparisons with other existing approaches, along with discussions about the reasons behind the performance differences, would strengthen the argument for the superiority.
8.	Exploring the reasons behind the success of these techniques and providing intuitive explanations would contribute to the overall scientific contribution of the work.
questions: questions: 1. The paper contains numerous hyperparameters, such as \(\alpha\). Including more ablation studies would enhance the readers' understanding.
2. In Table 3, why did MagR+OPTQ not achieve better results compared to peer competitors?
3. How can the impact of outliers on PQT for LLM be reduced?
limitations: limitations: NO
paper_summary: paper_summary: - The authors propose an optimization-based preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post-training quantization (PTQ) for large language models (LLMs). They motivate their method from previous work, showing that linear transformation of weights can render more quantization-friendly, allowing for sub-4bit quantization. However, such methods normally require a post-processing step during inference to undo that linear transformation, introducing a run-time overhead.
- Instead, they propose a non-linear transformation without needing an additional post-processing step. That transformation aims to minimise the $\ell_{\infty}$ norm, or the maximum value of the weights per channel or column while preserving the layer's output. By making the weight distribution more compact around zero and removing outliers, they are able to quantize weights to 2 or 3 bits with little accuracy drop compared to competing methods.
- They formulate the optimization as an $\ell_{\infty}$ -regularisation and solve it using proximal gradient descent. In fact, the problem reduces to computing the projection of weight vectors onto the $\ell_1$ ball, for which there are several efficient algorithms available.
strength_weakness: strength_weakness: - The paper is very focused on ppl and accuracy results on a wide range of benchmarks and datasets at the cost of other important ablation studies. For example:
	- ablation study on the penalty factor $\alpha$ and its impact on the final accuracy.
	- Inference runtime comparison is missing compared to competing methods, such as QuIP. In fact, the main claim of the paper is that it is faster than QuIP, but there are no studies supporting it.
	- Results with activation quantization beyond FP16 and runtime improvements vs baselines.
- The connection between the rank-deficient feature matrix argument and $\ell_{\infty}$ regularisation requires further theoretical justification. There must be a theoretical guarantee for the optimality of that optimizaiton. In fact, the method would be stronger if it was formulated more generally as an $\ell_p$ constrained, and the authors show that $\ell_{\infty}$ is better constrained.
questions: questions: - Table 1: It is unclear to me how all these numbers are relevant in supporting the method. One can see from the maximum value that all Llama Models are rank-deficient based on the maximum value being below 100%. How are the rest of the statistics informative?
- Line 124-125: From all the solutions available, why choose the lowest infinity norm? Why is this optimal? What is missing there is the optimality of this norm for quantization purposes. There, in fact, a discussion about quantization noise as a perturbation bounded by $\ell_{\infty}$ -norm in a paper by M. Alizadeh et. al called "Gradient $\ell_1$ regularization for quantization robustness"
- Table 3: activation quantization bitwidth missing
- Please provide ablation studies about the penalty factor $\alpha$. What is the effect of it not being tiny? How sensitive are the results?
- To fully support the claims of the paper, it would be nice to see an ablation study on other norms as regularization. Otherwise, a theoretical justification is required about the optimality of the $\ell_{\infty}$ -norm.
- Paragraph starting at line 210: it is unclear how you establish the optimality of the shrinking factor $\beta$. In fact, I cannot see why you would not search for the optimal clipping threshold based on minimizing the MSE per column or group. This is a standard method for weight clipping in LLM quantization literature.
- Section 5.3: the section's title is misleading. Maybe rename it to preprocessing runtime? Most commonly, runtime refers to the inference runtime
- A section on inference runtime comparison to QuIP is required. The main claim of the paper is that the method is faster than QuIP due to the lack of transformation, but no experimental evidence is provided to support it.
limitations: limitations: The paper lacks a thorough discussion of each limitation. The only reference I found was in lines 245-6 about extending MagR into incoherence processing.

Comments: title: Thanks for your reply
Thanks for the reply
Reviewed score
QuIP Minimizes a Different Quantization Error
QuIP quantization error
comment: We sincerely appreciate your valuable feedback and thank you for raising your score!
Thank you for the response. After reading it and other reviewer's comments, I will raise my score.
We sincerely appreciate your valuable feedback and your recognition of the novelty of our work!

Thanks again!
Having reviewed the responses, ablation studies and latest results in the conversation with reviewer jujD, I have decided to increase my score to strong accept, provided the additional results and analyses are included in the camera-ready version.

From my initial review, I found the method itself innovative but concerns about the presentation and certain claims. The authors have been diligent in addressing the points raised by me and other reviewers.
Thank you for your feedback and for raising your score. We appreciate your recognition of the contributions of our work. We will thoroughly review and cite previous works related to weight regularization that are pertinent to our approach.
Thank you for your response.

After reading your reply, I have decided to raise my score to 7, as the additional experimental results on MagR combined with other quantization methods have convinced me of the effectiveness of the proposed work.

However, I would like to request that the authors properly cite previous works related to weight regularization. While I understand that this paper is not identical to earlier approaches, applying weight regularization to achieve quantization-friendly models is a well-studied technique, and this paper currently lacks citations of these relevant works.
We hope our rebuttal have addressed the reviewers' concerns, and we would greatly appreciate it if you could reconsider your rating.
Thank you for your feedback!
---
1. ... it is crucial to thoroughly clarify the differences between R2 Loss and MagR, particularly in terms of loss term design or training efficiency ...
---
As we previously noted, R2 regularization applies $\ell_\infty$ penalty to the **traditional network loss** during end-to-end model pre-training, optimized using **standard SGD**, which is **practically infeasible for LLMs**. In contrast, MagR, as a concurrent approach, operates directly on the pre-trained model in a layer-by-layer manner, utilizing **linear least squares loss** to preserve each layer's output. MagR employs **proximal gradient descent** algorithm specifically tailored for this objective, **enabling efficient processing of LLMs**. This mathematically elegant algorithm represents a main contribution of our work.

---
2. ... MagR+OPTQ does not consistently achieve the best ppl/accuracy results ...
---

While we don’t claim that MagR+OPTQ always achieves the highest accuracy, it does perform at or near the top in most cases. Moreover, its accuracy can be further enhanced by applying techniques like additional coordinate descent iterations or learnable weight clipping as suggested by Reviewer 5GSj. Accuracy is just one of many critical performance metrics. More importantly, we introduce a technique that **incurs no additional overhead at inference time while achieving both training efficiency and accuracy comparable to the state-of-the-art, which typically sacrifices inference speed**. Inference speed is crucial for real-world applications, particularly in resource-limited settings such as edge computing.

---
3. ... the paper only discusses MagR+OPTQ without exploring the results of MagR combined with other quantization methods ...
---

 We would like to remind the reviewer that **we also reported the substantial performance gain of MagR over the nearest round method (MagR+RTN vs RTN) in Table 2.**

|Method | Wbits |Wiki (PPL)| | || C4 (PPL)|||
|-|-|-|-|-|-|-|-|-|
|| |7B|13B|70B|-|7B|13B|70B|
|Baseline| FP16|5.47|4.88|3.31||6.97|6.46|5.52|
||||||
|RTN|4/16|6.11|5.20|3.67|-|7.71|6.83|5.79|
|MagR+RTN|4/16|5.91|5.17|3.58|-|7.52|6.81|5.72|
||||||
|RTN|3/16|539.48|10.68|7.52|-|402.35|12.51|10.02|
|MagR+RTN|3/16|8.66|6.55|4.64|-|10.78|8.26|6.77|

Here we also demonstrate that MagR can enhance the performance of QuIP:

|Model|Method|Wbits|Wiki (PPL)| C4 (PPL)|
|-|-|-|-|-|
|LlaMa2-7B|QuIP|4|5.94|8.01|
| |MagR+QuIP|4|5.74|7.25|
| |QuIP|3|6.50|8.74|
| |MagR+QuIP|3|6.25|7.88|
| |QuIP|2|27.13|31.33|
| |MagR+QuIP|2|13.31|14.49|
||||||
|LlaMa2-13B|QuIP|4|5.01|6.88|
| |MagR+QuIP|4|4.99|6.63|
| |QuIP|3|5.34|7.34|
| |MagR+QuIP|3|5.29|7.02|
| |QuIP|2|10.09|13.13|
| |MagR+QuIP|2|9.40|11.07|


In summary, MagR preserves the behavior of the pre-trained model, as demonstrated by the new data, while allowing for small quantization steps in the subsequent PTQ process, leading to reduced quantization error. So theoretically, MagR can be combined with other PTQ methods.
Thank you for your careful and detailed response.

While I find the concept of MagR in regularizing weight values before quantization interesting, the paper seems to lack essential information needed to verify the significance of the proposed work:

1.	As you correctly noted, R2 Loss [2] was evaluated on CNNs using a full training approach. Since the loss term designed to regularize the weight distribution can be applied across various model architectures, it is crucial to thoroughly clarify the differences between R2 Loss and MagR, particularly in terms of loss term design or training efficiency. Although the difference between R2 Loss and MagR is briefly discussed in the rebuttal, the explanation provided is not detailed enough to fully clarify the distinction.

2.	The proposed MagR+OPTQ does not consistently achieve the best ppl/accuracy results compared to previous works.

3.	If the key novelty of MagR lies in pre-processing LLMs before quantization, it should be compatible with various quantization methods to enhance ppl/accuracy. However, the paper only discusses MagR+OPTQ without exploring the results of MagR combined with other quantization methods. Consequently, it is unclear whether MagR should be regarded as a general pre-processing solution or if ‘MagR+OPTQ’ is intended as a new quantization scheme. If MagR is indeed a broadly applicable pre-processing method, it would be highly valuable. If not, I have reservations about the significance of ‘MagR+OPTQ,’ given the limited ppl/accuracy improvement as discussed earlier.

Therefore, I still perceive MagR as a limited contribution and will maintain my stance.
Thank you for your questions. 

In response to the second question, we evaluated the quantization errors for INT4 quantization, and we did not apply weight clipping at 4-bit (i.e., $\beta=1$). This means that the reduction in quantization error is entirely due to the MagR preprocessing.

Regarding the quantization error with QuIP, it targets a different quantization error in a distinct parameter space compared to our method or OPTQ. Specifically, we solve $\min_{W_q\in\mathbb{Q}} ||X(W_q - W)||^2$, where the focus is directly on minimizing the error in the original weight space. In contrast, QuIP first applies orthogonal transformations $U$ and $V$ to the weights and activations before quantization: $\min_{W_q} ||XV V^{\top} (W_q - W)U||^2 := \min_{\tilde{W_q}\in\mathbb{Q}} ||\tilde{X} (\tilde{W_q} - \tilde{W})||^2$. Here, $\tilde{X} = XV$, $\tilde{W_q} = V^{\top} W_q U$, and $\tilde{W} = V^{\top} W U$ represent the transformed activations and weights. QuIP only quantizes the transformed weights $\tilde{W_q}$, and their corresponding $W_q$ is actually float. In contrast, we directly quantize $W_q$ as is done in standard OPTQ. As a result, the quantization errors between our method and QuIP are not directly comparable.

We hope we have addressed the reviewer’s questions.
Dear authors,

Thank you for your thorough response to our comments and questions. I have one more question.

In the attached figure, you compare MagR's quantization error with OPTQ. Do you have time to add the quantization error with the transformed weights of QuIP? This would be quite useful in understanding whether your optimization is theoretically superior to the random linear transformation of QuIP (beyond QuIP's additional compute overhead). 

Also, is the quantization error calculated before or clipping, namely, is $\beta$<1 for these results? 

Thanks