Title: MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization
Venue: NeurIPS 2024
Accepted: yes
Keywords: post-training quantization

model compression

$\ell_\infty$-regularization
Abstract: In this paper, we present a simple optimization-based preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post-training quantization. For each linear layer, we adjust the pre-trained floating-point weights by solving an $\ell_\infty$-regularized optimization problem. This process greatly diminishes the maximum magnitude of the weights and smooths out outliers, while preserving the layer's output. The preprocessed weights are centered more towards zero, which facilitates the subsequent quantization process. To implement MagR, we address the $\ell_\infty$-regularization by employing an efficient proximal gradient descent algorithm. Unlike existing preprocessing methods that involve linear transformations and subsequent post-processing steps, which can introduce significant overhead at inference time, MagR functions as a non-linear transformation, eliminating the need for any additional post-processing. This ensures that MagR introduces no overhead whatsoever during inference. Our experiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models. For example, we achieve a Wikitext2 perplexity of 6.7 on the LLaMA2-70B model for per-channel INT2 weight quantization without incurring any inference overhead.

MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization

Abstract

In this paper, we present a simple optimization-based preprocessing technique called Weight Mag nitude R eduction (MagR) to improve the performance of posttraining quantization. For each linear layer, we adjust the pre-trained floating-point weights by solving a channel-wise ℓ ∞ -regularized optimization problem. This process greatly diminishes the maximum magnitude of the weights and smooths out outliers, while preserving the layer's output. The preprocessed weights exhibit reduced range, which facilitates the subsequent quantization process. To implement MagR, we address the ℓ ∞ -regularization by employing an efficient proximal gradient descent algorithm. Unlike existing preprocessing methods that involve linear transformations and subsequent post-processing steps, which can introduce significant overhead at inference time, MagR functions as a non-linear transformation, eliminating the need for any additional post-processing. This ensures that MagR introduces no overhead whatsoever during inference. Our experiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models. For example, we achieve a Wikitext2 perplexity of 5.95 on the LLaMA2-70B model for per-channel INT2 weight quantization without incurring any inference overhead. The code is available at https://github.com/AozhongZhang/MagR

1 Introduction

Large language models (LLMs) have achieved outstanding performance across a broad range of applications, demonstrating remarkable success. However, their unprecedented model size has led to many computation operations and substantial memory footprints, becoming significant barriers to their practical deployment and adoption in production environments. Accordingly, it is highly desirable to develop efficient model compression techniques for LLMs so they can be more widely deployed in resource-limited scenarios. Among the various techniques to compress and accelerate deep neural networks (DNNs), low-precision quantization has proven to be highly effective across numerous application domains and is widely adopted for accelerating DNNs. For LLMs, the inference runtime is dominated by the token generation process, where output tokens are produced sequentially, one at a time. This process is known to be memory bandwidth bound [3, 19]. As a result, the quantization of LLMs has primarily focused on reducing the bit-width of model weights, with the dual goals of lowering the model's footprint to enable deployment on resource-constrained devices and decreasing the memory bandwidth requirements to improve computational efficiency and accelerate inference.

The enormous computational demands for pre-training and fine-tuning Large Language Models (LLMs) have led to the emergence of Post-Training Quantization (PTQ) [4, 15, 22, 24, 27, 31, 41, 51, 52, 53, 43, 40], as a promising solution for quantizing these models. Unlike Quantization Aware Training (QAT) [7, 9, 12, 18, 21, 23, 46, 47, 48, 49], which is designed to minimize a global training loss for quantization parameters, PTQ directly applies low-precision calibration to a pre-

trained full-precision model using a minimal set of calibration samples. By aiming to identify an optimal quantized model locally through the minimization of a simplified surrogate loss, PTQ offers computational savings and resource efficiency compared to QAT. However, PTQ often lags behind QAT in accuracy, particularly for ultra-low precision lower than 4-bit. Thus, it remains an open problem to achieve an improved balance between cost and performance for PTQ-based approaches.

Motivation. To achieve state-of-the-art performance, the latest advances in PTQ [8, 25, 26, 36, 42] have proposed applying a linear transformation to process the pre-trained weights within a linear layer. This strategy of linear transformation aims to make the weights more suitable for the subsequent quantization procedure by reducing their magnitudes and suppressing outliers. In a nutshell, given the features X and weights W , one constructs linear transformation T such that TW is better conditioned than W in terms of being quantization-friendly. Such designs of T include diagonal matrices (so-called channel-wise scaling) [25, 36, 42], random transformations [8, 39], and finite frames [1, 13]. Then, quantization is performed on TW instead of the original weights W . To preserve the layer's output, however, the inverse transformation T -1 has to be in turn applied to the features X , namely,

with Q ( TW ) being the quantized weights. PTQ done this way requires modifications on the original neural architecture, which involves additional computations of XT -1 and extra memory storage for T -1 at inference time. As a result, these steps introduce overhead that offsets the benefits provided by quantization. This raises a natural question:

Can we effectively process the weights at the preprocessing stage to facilitate quantization without introducing inference overhead?

To address this problem, we propose a simple optimization-based technique called Weight Magnitude Reduction (MagR). MagR functions as a non-linear transformation on weights without altering the original features/activations. The optimization program is designed to find new weights with minimal maximum magnitude, i.e., the ℓ ∞ norm, while preserving the layer's outputs.

Contributions. We propose a non-linear approach, MagR, based on channel-wise ℓ ∞ -regularized least squares, to reduce the quantization scale without compromising the performance of pre-trained model, facilitating subsequent weight quantization while requiring no post-processing or inference overhead. See Figure 1 for comparing weight magnitudes before and after applying MagR. To address the ℓ ∞ -regularization problem, we develop an efficient and parallelizable proximal gradient descent algorithm that involves computing ℓ 1 -ball projections at each iteration. Specifically, MagR preprocessing on a single Nvidia A100 GPU takes merely 15 min for LLaMA2-7B and 3.5 hr for the 70B model. Our results on INT weight-quantization demonstrate that MagR can significantly boost the performance in the sub-4bit regime when combined with fast gradient-free methods for layer-wise PTQ, such as rounding-to-nearest (RTN) [30] and OPTQ [16]. This approach achieves performance for weight quantization at least comparable to state-of-the-art PTQ methods on natural language processing (NLP) tasks, including gradient-based methods using block-wise reconstruction.

2 Related Work

Recently, as the sizes of language models are exploding, there has been growing interest in developing post-training quantization (PTQ) methods [8, 16, 25, 26, 36, 44, 45] for large-scale AI models like large language models (LLMs) to reduce the model sizes and accelerate inference by representing weight matrices in low precision. PTQ methods directly find the low-precision representation of the model without re-training, thereby preferred by extreme large-scale AI models. The OPTQ [16] uses approximate second-order information to calibrate the quantization. The method successfully compresses LLMs into 3 or 4 bits and can achieve reasonable accuracy in 2 bits. Researchers have found that the extreme values and the distribution of the weight entries highly affect the quantization errors and the quantized model quality. The original weight can be converted into a more quantizationfriendly one by linear transformations. The approach can significantly reduce the quantization errors while bringing more time overhead during inference because of the linear transformation. OmniQuant [36] proposes learnable weight clippings and equivalent transformations to avoid the influence of extreme values. AWQ [25] searches for the most significant entries in the weight by looking at the activation and selects the scales that protect these entries. SmoothQuant [44] passes the difficulty

AfterMagR

0.20

0.15

0.10

Magnitude lumn-wise

0.05

8

AfterMagR

Layer 119

0.25

MagnitudeA

0.20

0.15

0.10

umn-wise

0.05

0.00

0.00

0.05

0.10

0.15

0.20

0.25

【

Column-wiseMaxMagnitudeBeforeMagR

AfterMagR

Column-wise

0.00

0.00

0.05

0.10

0.15

0.20

Column-wiseMaxMagnitudeBeforeMagR

an xn

0.20

0.15

0.10

0.05

0.00

0.00

0.05

0.10

0.15

0.20

Column-wiseMaxMagnitudeBeforeMagR

in activation quantization to weights by an equivalent linear transformation. QuIP [8], AffineQuant [26] and FrameQuant [1] apply a linear transformation before quantization to make the transformed weight quantization-friendly. These approaches achieve high performance for extreme bits, like 2 bits, but introduce additional inference overhead though the transformation is carefully designed to be efficient. OmniQuant [36] and AffineQuant [26] can be adopted for weight-activation quantization by considering the activations in the proposed methods. The work [45] introduces a low-rank compensation method on top of other quantization methods, which employs low-rank matrices to reduce quantization errors with a minimal increase in model size. By modeling the quantization residual as an ℓ ∞ -bounded perturbation, [2] proposes applying an ℓ 1 penalty on the gradient of loss to enhance quantization robustness.

The works most closely related to ours are [20] and [27], both utilizing ℓ ∞ norm to regularize or constrain the weight range to a smaller scale. The Range Regularization (R 2 ) method [20] applies an ℓ ∞ penalty or its variants to the conventional network loss to regularize the weight range during end-to-end model pre-training, optimized via SGD. However, this approach becomes practically infeasible for large-scale models. In [27], a layer-wise pre-processing technique is proposed, which involves solving an intractable ℓ 0 -minimization problem while constraining the ℓ ∞ -norm of weights.

3 Background

First, we clarify the mathematical notations that will be used throughout this paper:

Notations. We denote vectors by bold small letters and matrices by bold capital ones. For a positive integer n , [ n ] := { 1 , 2 , . . . , n } denotes the set containing all positive integers up to n . For any two vectors x , y ∈ R n , ⟨ x , y ⟩ := ∑ n i =1 x i y i is the inner product. We denote by ∥ x ∥ := √ ⟨ x , x ⟩ = √∑ n i =1 x 2 i the Euclidean norm; ∥ x ∥ 1 := ∑ n i =1 | x i | is the ℓ 1 -norm; ∥ x ∥ ∞ := max 1 ≤ i ≤ n | x i | is the ℓ ∞ -norm. For any matrix X ∈ R m × n , X ⊤ ∈ R n × m is the transpose. We denote the spectrum norm of X by ∥ X ∥ = σ max ( X ) , which equals its maximum singular value. Its Frobenius norm is given by ∥ X ∥ F = √ ∑ m i =1 ∑ n j =1 X 2 i,j . Moreover, for vectors x and y , x ⊙ y := ( x 1 y 1 , . . . , x n y n ) ∈ R n denotes the Hadamard or element-wise product, and likewise for two matrices.

Layerwise PTQ. Post-training quantization via layerwise reconstruction calls for solving a least squares problem with a discrete constraint. For the pre-trained weights W within a linear layer, we aim to find the quantized weights W q that minimize the following function

where X ∈ R ( b · l ) × m is the feature matrix associated with a batch of calibration data consisting of b samples stacked together, and each data sample is represented by an l × m sub-matrix. Q ⊂ R m × n is an appropriate set of all feasible quantized weights.

Layer 42

Layer 210

The most straightforward PTQ technique, known as RTN, involves directly rounding the weight matrix W without utilizing any additional data. An improvement over RTN was introduced by AWQ [25], which enhances the quantization process by incorporating channel-wise scaling on W .

Thanks to the simplicity of the layer-wise formulation (1), several efficient gradient-free algorithms [4, 16, 51, 53] have been recently proposed to address layer-wise quantization, including OPTQ. Built on top of OPTQ, QuIP subjects X and W to random orthogonal transformations to produce 'incoherent" weight and Hessian matrices, leading to superior accuracy with sub-4bit quantization. However, this advantage comes with a trade-off; during inference, QuIP requires random orthogonal transformations on the feature inputs of linear layers, rendering noticeably slower throughput compared to OPTQ.

Uniform Quantizer. Given a set of points w ∈ R m , the commonly-used (asymmetric) uniform quantizer [9] defines the quantization step δ = max( w ) -min( w ) 2 b -1 and zero-point z = ⌊ min( w ) δ ⌉ , and it quantizes w onto the scaled integer grids Q = { z · δ, ( z +1) · δ, . . . , ( z +(2 b -1) ) · δ } m as follows:

In per-channel (or per-group) PTQ, the quantization step δ is conventionally calculated based on the channel-wise (or group-wise, respectively) minimum and maximum values of the pre-trained weights W , as defined above, and remains constant throughout the quantization procedure.

4 The Proposed Method

In this section, we present the Weight Magnitude Reduction (MagR) method based on ℓ ∞ -norm regularization, which is applied just before the quantization step within each linear layer. The intuition behind MagR is based on the following simple estimate of the layer-wise quantization error. Given the feature/activation matrix X , the quantizer Q , and any pre-trained weights w ∈ R m , we have:

where δ = max( w ) -min( w ) 2 b -1 is the quantization step size. This shows that reducing the range of weights helps to suppress the quantization error. With this in mind, MagR preprocessing is designed to achieve two key effects:

First, it effectively reduces the channel-wise (or column-wise) maximum magnitude of the weights, as illustrated by Figure 1.

Second, it preserves the model's original performance with minimal accuracy loss after preprocessing. Table 1 demonstrates that MagR preprocessing maintains the perplexity of the pre-trained models, with only minor degradation.

4.1 Approximately Rank-Deficient Feature Matrix

To illustrate the idea behind the proposed MagR method, let us consider a pre-trained weight vector ˆ w ∈ R m of a linear layer and the associated feature input matrix X . MagR leverages the fact that the feature matrix X across all layers of LLMs is approximately rank-deficient. Specifically, if X is exactly rank-deficient, the linear system modeling the layer's output, Xw = X ˆ w with variables w , generally has infinitely many solutions. That is, for any ν in the non-trivial kernel space of X , we have that w = ˆ w + ν preserves the layer's output. Among all solutions, MagR aims to identify the weight vector w with the smallest extreme value in magnitude.

In [8], the authors empirically observed that the Hessian matrix X ⊤ X is approximately low-rank across all layers in open pre-trained (OPT) models [54]. Here we examined the feature matrix of LLaMA models [37, 38]. Our approximate fraction rank of the feature matrix X is defined as the fraction of singular values of X such that σ ( X ) > 0 . 01 · σ max ( X ) . Table 2 illustrates that all feature matrices extracted from LLaMA models are indeed rank-deficient according to this definition.

4.2 MagR via ℓ ∞ -Regularization

Let us consider the quantization of a weight vector for simplicity. Given pre-trained weight vector ˆ w , we would like to find a new set of weights w with the smallest maximum magnitude, such that the layer output is preserved up to a small error ε > 0 , i.e.,

To efficiently implement MagR, we consider the following mathematically equivalent ℓ ∞ -regularization problem instead:

where α > 0 serves as the regularization parameter, balancing fidelity against the ℓ ∞ regularizer. To maintain the output of the layer, α should typically be set to a small value. Indeed, let w ∗ be the minimizer of (2), we have that the ℓ 2 error of the layer's output introduced by MagR is O ( √ α ) :

where ∥ ˆ w ∥ ∞ is a constant independent of α , and the second inequality uses that w ∗ is the minimizer.

Proximal Gradient Descent. Note that ℓ ∞ -norm is a convex but non-differentiable function. In theory, the optimization problem (2) can be simply solved by a subgradient algorithm, but it is significantly slower than the more sophisticated proximal gradient algorithm which matches the convergence rate of standard gradient descent.

With the step size η > 0 , proximal gradient descent [32] takes the following iteration:

where prox t ∥·∥ ∞ with the scalar t > 0 is the (scaled) proximal operator of ℓ ∞ -norm function, defined as

To ensure the convergence of (3), it is sufficient to choose the step size

where λ max ( X ⊤ X ) is the maximum eigenvalue of X ⊤ X .

Proximal Operator of ℓ ∞ -Norm. It remains to determine the proximal operator of ℓ ∞ -norm. It turns out we can compute it by leveraging the celebrated Moreau decomposition [29, 32]: for any t > 0 ,

That is, computing the proximal operator of ℓ ∞ norm amounts to evaluating the projection onto ℓ 1 ball, which is defined as

Fortunately, computing projection onto the ℓ 1 ball is an established task, and there are several efficient algorithms available. For example, see [11] and the references therein. Here we adopted a simple algorithm of O ( m log m ) time complexity as in [14], which supports parallelizable or vectorized implementation for the projections of a batch of weight vectors, i.e., a weight matrix, as will be described in the next subsection. The implementation mainly involves sorting and soft-thresholding [50]; see Algorithm 3 and its derivation in Appendix A.1 for the details.

MagR for Weight Matrix. In practical implementation of MagR, we preprocess the entire weight matrix W = [ w 1 , . . . , w n ] ∈ R m × n within each linear layer. For per-channel quantization (or per-column quantization in our setting), the ℓ ∞ penalty is imposed column-wise on the weight matrix to reduce the quantization scale of each channel. That is, MagR amounts to solving

In this case, we take the following iteration:

where the proximal operator prox t ∥·∥ ∞ and the corresponding projection proj ∥·∥ 1 ≤ 1 in (4) are applied column-wise to the matrix input. Hereby we summarize MagR for processing one linear layer in Algorithm 1 with the column-wise ℓ 1 -ball projection as detailed in Algorithm 2, which generalizes Algorithm 3 in Appendix A.1, by handling matrix inputs (or batches of vectors).

Algorithm 1 Per-channel MagR for one linear layer.

Input: Pre-trained weight matrix ˆ W ∈ R m × n ; Hessian matrix H = X ⊤ X ∈ R m × m ; max iteration number K ; step size η = 1 λ max ( H ) ; penalty parameter α > 0 .

Output: Preprocessed weights W ∈ R m × n .

1: Initialize W 0 = ˆ W .

2: for k = 0 , . . . , K -1 do

3: V k = W k -η · H ( W k -ˆ W )

4: W k +1 = V k -ηα · proj ∥·∥ 1 ≤ 1 ( V k ηα )

5: end for

6: return W = W K

Extension to Per-Group Quantization. By using more float scaling factors, per-group quantization becomes a preferred strategy for mitigating accuracy loss at extremely low bit-widths. In this approach, a weight vector w ∈ R m is segmented into groups of weights, each containing d elements, with all weights within a group sharing a common scaling factor for quantization. Here, per-group MagR applies an ℓ ∞ penalty to each vector of grouped weights. Consequently, the ℓ 1 -ball projection is independently performed on these vectors, while maintaining the gradient descent step unchanged. We note that the group-wise ℓ 1 -ball projection can be easily done using Algorithm 2, with an additional reshaping of the input V ∈ R m × n into R d × ( m d · n ) .

gradient descent step proj ∥·∥ 1 ≤ 1 is described in Alg. 2

Algorithm 2 Column-wise projection onto the unit ℓ 1 -ball. Input: Matrix V ∈ R m × n ; the radius of ℓ 1 ball, ϵ = 1 . Output: W ∈ R m × n such that all columns ∥ w j ∥ 1 ≤ ϵ , ∀ j ∈ [ n ] . 1: Create a binary mask M ∈ R m × n filtering out the columns of V with ∥ v j ∥ 1 ≤ ϵ . 2: Sort | V | column-wise in descending order into U . 3: Find index ρ j = max { i ∈ [ m ] : u i,j > 1 i ( ∑ i r =1 u r,j -ϵ )} , ∀ j ∈ [ n ] 4: Define θ j = 1 ρ j (∑ ρ j r =1 u r,j -ϵ ) , ∀ j ∈ [ n ] 5: Tile θ ∈ R n into Θ ∈ R m × n along the row. 6: Compute W = (1 -M ) ⊙ V + M ⊙ sgn( V ) ⊙ max {| V | -Θ , 0 } 7: return W

5 Experiments

Overview. We tested the proposed MagR for INT4 , INT3 , and INT2 weight quantization. In our notations, the weight and activation bits are denoted by 'W' and 'A' , respectively. Additionally, we implemented group-wise weight quantization with the group size denoted by 'g' . For example, W2A16g128 signifies INT2 weight and FP16 activation (i.e., INT2 weight-only quantization) with a group size of 128.

We employed our MagR processing approach on top of the two gradient-free PTQ methods in main text, RTN and OPTQ [16], to quantize the LLaMA1 (7B-65B) [37] and LLaMA2 (7B-70B) [38] model families. In the Appendix A.2, we extend MagR with QuIP [8] (MagR+QuIP) to quantize LLaMA2 (7B-70B) model families. By applying MagR on top of RTN (MagR+RTN), we achieved better results than AWQ [25] for per-channel INT3 and INT4 weight quantization. Additionally, MagR combined with OPTQ (MagR+OPTQ) achieved state-of-the-art performance for INT3 and INT4 qunatization. To enhance the per-channel INT2 quantization, we ran 30 additional iterations of coordinate descent algorithm [4, 51] on top of OPTQ, which we denote by MagR+OPTQ † . It turns out MagR+OPTQ † is superior to both Omniquant [36] and QuIP [8] in terms of perplexity (Table 9), and falls just short of QuIP in zero-shot tasks for 13B and 70B models (Table 4). Note that QuIP uses random orthogonal transformations (so-called Incoherence Processing) to process both the weights and features, resulting in 1 . 5 × slower throughput than OPTQ. In contrast, MagR-based method does not introduce any overhead whatsoever compared with OPTQ.

In conclusion, our MagR-based PTQ method is intuitive yet effective in compressing models into extreme bit-widths, while maintaining performance without introducing any inference overhead.

Datasets and Evaluation. Following the previous work [16, 25, 36], we evaluate the quantized model on language generation tasks on WikiText2 [28] and C4 [33]. Additionally, we test its performance on zero-shot tasks, including PIQA [5], ARC (Easy and Challenge) [10], and Winogrande [35]. For the language generation experiments, our implement is based on the OPTQ's [16] repository, which is built using PyTorch. For executing all zero-shot tasks, we adhere to the lm-eval-harness [17].

Baseline : For the language generation task, we compare our method with RTN, OPTQ [16], AWQ [25] and OmniQuant [36] on LLaMA1 and LLaMA2 models. In addition to the aforementioned methods, we also conduct a comparison with QuIP [8] on the LLaMA2-70B model. In the zero-shot task, we focus on four individual tasks and compare the average accuracy across all four tasks with Omniquant [36].

Implementation details. We utilized the HuggingFace implementations of the LLaMA1 and LLaMA2 models and perform quantization on a single NVIDIA A100 GPU with 80GB of memory. Following the OPTQ method, we load one block consisting of 7 linear layers into GPU memory at a time. In line with previous work [8, 16], the input matrix X is obtained by propagating the calibration data through the quantized layers.

The choice of parameters. To ensure that the MagR-processed layer output XW is faithful to the original X ˆ W , we need to use a tiny penalty parameter α in (2). For per-channel quantization, α was fixed to be 10 -3 in our experiments, but we did find that setting it to a smaller value of 5 × 10 -4 or 10 -4 can sometimes slightly improve the perplexity (with a relative change of < 1% in ppl). Similarly for per-group quantization, we set α to 10 -4 , while reducing it to 5 × 10 -5 or 10 -5 could

sometimes also slightly improve the perplexity. An ablation study on α is provided in the Appendix A.2.

Furthermore, we used a multiplicative scalar β < 1 to decay the standard quantization step δ = max( w ) -min( w ) 2 b -1 (or equivalently, the quantization scale) of the quantizer. In other words, our δ = β · max( w ) -min( w ) 2 b -1 . It has been shown in existing works [21, 34] that, optimal quantization step for binary or ternary quantization yielding the minimum quantization error is not given by max( w ) -min( w ) 2 b -1 . Shrinking δ at low bit-width results in a more clustered quantization grid lattice that fits the weights better, which leads to a smaller overall error. In general, β is positively correlated with the bit-width used. For per-channel quantization, the best β ∈ [0 . 8 , 0 . 85] on INT2 quantization, whereas the empirically optimal β is around 0 . 9 for INT3 quantization. As for INT4 , β is simply set to 1, that is, we used the standard quantization step. In addition, for per-group quantization, we chose β = 0 . 95 for both INT2 and INT3 quantization. The ablation study of β is in the Appendix A.2. We observed that this refinement on the quantization step δ significantly improves the performance of the PTQ method. In addition, the iteration number K in Algorithm 1 was set to 150 across all the experiments.

5.1 Language Generation

We concentrate our analysis on perplexity-based tasks. The results for the LLaMA2 family with context length of 2048, are elaborated in Table 9, while those for LLaMA1 are provided in Appendix Table 6. As evidenced by the tables, the MagR preprocessing consistently improve the performance of the baselines RTN and OPTQ. Moreover, MagR+OPTQ consistently outperforms most baseline across the LLaMA family models for both per-channel and per-group weight quantization. Particularly, for

INT2 , MagR+OPTQ † performs 30 additional coordinate descent (CD) iterations [4, 51] on top of OPTQ to refine the solution, surpassing all baselines.

Furthermore, MagR+RTN achieves performance comparable to OPTQ. Notably, it outperforms AWQby a significant margin in INT3 quantization, implying that MagR proves more effective as a preprocessing method compared to channel-wise scaling.

5.2 Zero-Shot Tasks

We evaluated the performance of quantized models on several zero-shot tasks. The results are reported in Table 4. Similar to previous observations, the proposed MagR demonstrates superior performance on most models compared to OmniQuant, with a small gap compared to QuIP [8]. Nonetheless, it is reasonable and commendable that our algorithm achieves results close to QuIP without introducing any inference overhead. It is possible to further improve our approach based on the insight behind QuIP [8] - i.e., quantization benefits from incoherent weight and Hessian matrices; see Table 9 for the results in the appendix.

5.3 Preprocessing and Quantization Runtime

We report the execution time of MagR+RTN and MagR+OPTQ on a single NVIDIA A100 GPU in Table 5. For example, it typically took 0.5-7.5 hours for MagR+OPTQ to quantize the LlaMA2 models. We note that the integration of MagR can markedly enhance the performance of the standard OPTQ [16]. It is noted that MagR+OPTQ † for INT2 weight quantization requires a longer runtime

due to the additional CD iterations, extending the quantization process for LLaMA2-70B to 31 hr. It also reveals that the preprocessing overhead for quantizing the LLaMA2 models (7B-70B) amounts to approximately 15 min, 30 min, and 3.5 hr, respectively. In comparison, our total runtime is roughly half of that of the gradient-based method, OmniQuant [36], while achieving at least comparable results. Moreover, MagR introduces no post-processing step or overhead during inference.

6 Concluding Remarks

In this paper, we proposed MagR, based on ℓ ∞ -regularization, to significantly reduce the maximum weight magnitude of pre-trained LLMs within each layer while preserving their output. MagR is designed to enhance the accuracy of backpropagation-free PTQ methods that use layer-wise reconstruction, such as RTN and OPTQ. MagR produces a more clustered distribution of weights and leads to a smaller quantization step, thereby facilitating the subsequent PTQ task. To solve the ℓ ∞ -regularization problem, we used the classical proximal gradient descent algorithm with ℓ 1 -ball projections, tailored to handle matrix variables efficiently. Our experiments on LLaMA family validated the effectiveness of the MagR approach, achieving the state-of-the-art performance on NLP tasks. Remarkably, unlike existing weight preprocessing techniques that require performing an inverse transformation on features during inference, MagR eliminates the need for post-processing and incurs no overhead. This renders MagR more practical for the deployment of quantized models.

References

Harshavardhan Adepu, Zhanpeng Zeng, Li Zhang, and Vikas Singh. Framequant: Flexible low-bit quantization for transformers. arXiv preprint arXiv:2403.06082 , 2024.

Milad Alizadeh, Arash Behboodi, Mart Van Baalen, Christos Louizos, Tijmen Blankevoort, and Max Welling. Gradient ℓ 1 regularization for quantization robustness. arXiv preprint arXiv:2002.07520 , 2020.

Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022.

Kayhan Behdin, Ayan Acharya, Aman Gupta, Sathiya Keerthi, and Rahul Mazumder. Quantease: Optimization-based quantization for language models-an efficient and intuitive algorithm. arXiv preprint arXiv:2309.01885 , 2023.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020.

Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5918-5926, 2017.

Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems , 2023.

Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 , 2018.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Laurent Condat. Fast projection onto the simplex and the ℓ 1 ball. Mathematical Programming , 158(1):575585, 2016.

Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems , 28, 2015.

Wojciech Czaja and Sanghoon Na. Frame quantization of neural networks. arXiv preprint arXiv:2404.08131 , 2024.

John Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. Efficient projections onto the l 1-ball for learning in high dimensions. In Proceedings of the 25th international conference on Machine learning , pages 272-279, 2008.

Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems , 35:4475-4488, 2022.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations , 2022.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023.

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research , 18(187):1-30, 2018.

Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W. Mahoney, Yakun Sophia Shao, and Amir Gholami. Full stack optimization of transformer inference: a survey, 2023.

Arnav Kundu, Chungkuk Yoo, Srijan Mishra, Minsik Cho, and Saurabh Adya. R2 loss: Range restriction loss for model compression and quantization. arXiv preprint arXiv:2303.08253 , 2023.

Fengfu Li, Bin Liu, Xiaoxing Wang, Bo Zhang, and Junchi Yan. Ternary weight networks. arXiv preprint arXiv:1605.04711 , 2016.

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426 , 2021.

Zhijian Li, Biao Yang, Penghang Yin, Yingyong Qi, and Jack Xin. Feature affinity assisted knowledge distillation and quantization of deep neural networks on label-free data. IEEE Access , 2023.

Chen Lin, Bo Peng, Zheyang Li, Wenming Tan, Ye Ren, Jun Xiao, and Shiliang Pu. Bit-shrinking: Limiting instantaneous sharpness for improving post-training quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16196-16205, 2023.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978 , 2023.

Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403.12544 , 2024.

Johannes Maly and Rayan Saab. A simple approach for quantizing neural networks. Applied and Computational Harmonic Analysis , 66:138-150, 2023.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 , 2016.

Jean Jacques Moreau. Décomposition orthogonale d'un espace hilbertien selon deux cônes mutuellement polaires. Comptes rendus hebdomadaires des séances de l'Académie des sciences , 255:238-240, 1962.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning , pages 7197-7206. PMLR, 2020.

Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1325-1334, 2019.

Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends® in Optimization , 1(3):127-239, 2014.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision , pages 525-542. Springer, 2016.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137 , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396 , 2024.

Haoxuan Wang, Yuzhang Shang, Zhihang Yuan, Junyi Wu, and Yan Yan. Quest: Low-bit diffusion model quantization via efficient selective finetuning. arXiv preprint arXiv:2402.03666 , 2024.

Naigang Wang, Chi-Chun Charlie Liu, Swagath Venkataramani, Sanchari Sen, Chia-Yu Chen, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Leland Chang. Deep compression of pre-trained transformer models. Advances in Neural Information Processing Systems , 35:14140-14154, 2022.

Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145 , 2023.

Junyi Wu, Haoxuan Wang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Ptq4dit: Post-training quantization for diffusion transformers. arXiv preprint arXiv:2405.16005 , 2024.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087-38099. PMLR, 2023.

Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Exploring post-training quantization in llms from comprehensive study to low rank compensation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19377-19385, 2024.

Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding straight-through estimator in training activation quantized neural nets. In International Conference on Learning Representations , 2019.

Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Binaryrelax: A relaxation approach for training deep neural networks with quantized weights. SIAM Journal on Imaging Sciences , 11(4):2205-2223, 2018.

Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Blended coarse gradient descent for full quantization of deep neural networks. Research in the Mathematical Sciences , 6:1-23, 2019.

Penghang Yin, Shuai Zhang, Yingyong Qi, and Jack Xin. Quantization and training of low bit-width convolutional neural networks for object detection. arXiv preprint arXiv:1612.06052 , 2016.

Wotao Yin, Stanley Osher, Donald Goldfarb, and Jerome Darbon. Bregman iterative algorithms for ℓ 1 -minimization with applications to compressed sensing. SIAM Journal on Imaging sciences , 1(1):143-168, 2008.

Aozhong Zhang, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin Li, and Penghang Yin. Comq: A backpropagation-free algorithm for post-training quantization. arXiv preprint arXiv:2403.07134 , 2024.

Jinjie Zhang and Rayan Saab. Spfq: A stochastic algorithm and its error analysis for neural network quantization. arXiv preprint arXiv:2309.10975 , 2023.

Jinjie Zhang, Yixuan Zhou, and Rayan Saab. Post-training quantization for neural networks with provable guarantees. SIAM Journal on Mathematics of Data Science , 5(2):373-399, 2023.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

A Appendix / supplemental material

A.1 Projection of Vectors Onto ℓ 1 -Ball

In this section, we show how to compute the projection onto the unit ℓ 1 -Ball. That is, for any fixed v ∈ R m , we solve the optimization problem:

Consider the Lagrangian L ( x , λ ) = 1 2 ∥ x -v ∥ 2 + λ ( ∥ x ∥ 1 -1) , where λ ∈ R is the lagrange multiplier. Let x ∗ be the optimal solution of (5), then there exits λ ∗ such that the following Karush-Kuhn-Tucker (KKT) conditions [6] hold:

Stationarity: 0 ∈ ∂ x L ( x ∗ , λ ∗ ) ⇔ x ∗ = sgn( v ) ⊙ max {| v | -λ ∗ , 0 }

Slackness: λ ∗ ( ∥ x ∗ ∥ 1 -1) = 0 ⇔ λ ∗ = 0 or ∥ x ∗ ∥ 1 = 1 .

Primal feasibility: ∥ x ∗ ∥ 1 -1 ≤ 0 ⇔∥ x ∗ ∥ 1 ≤ 1

Dual feasibility: λ ∗ ≥ 0 ⇔ λ ∗ = 0 or λ ∗ > 0

where sgn is the signum function applied element-wise on vectors, i.e.,

We note that the stationarity condition:

gives the projection of v onto the unit ℓ 1 ball, provided λ ∗ is known. Therefore, what remains is to find λ ∗ :

Case I: λ ∗ = 0 . We have x ∗ = v , which corresponds to the case that ∥ v ∥ 1 ≤ 1 .

Case II: λ ∗ > 0 . The slackness condition yields ∥ x ∗ ∥ 1 = 1 , i.e. , ∑ n i =1 | x ∗ i | = 1 , or equivalently, ∑ n i =1 | sgn( v i )( | v i | -λ ∗ ) + | = ∑ n i =1 ( | v i | -λ ∗ ) + = 1 with x + := max { x, 0 } . That is, λ ∗ is the root of the piece-wise linear equation:

which can be solved by sorting.

In summary, Algorithm 3 details the implementation of projecting v ∈ R m onto a general ℓ 1 -ball with radius ϵ . In Step 4, we specifically compute the root λ ∗ (or θ ) of (6) for Case II.

Algorithm 3 Projection onto ℓ 1 -ball.

Input: Vector v ∈ R m ; the radius of ℓ ball, ϵ = 1 .

1 Output: w ∈ R m such that ∥ w ∥ 1 ≤ ϵ .

1: if ∥ v ∥ 1 > ϵ then

2: Sort | v | into µ such that µ 1 ≥ µ 2 ≥ . . . ≥ µ m .

3: Find index ρ = max { i ∈ [ m ] : µ i > 1 i ( ∑ i r =1 µ r -ϵ )}

4: Define θ = 1 ρ ( ∑ ρ r =1 µ r -ϵ )

5: Compute w = sgn( v ) ⊙ max {| v | -θ, 0 }

6: else

7: w = v

8: end if

9: return w

A.2 Additional Experimental Results

Table 6 shows the results for WikiText2 and C4 perplexity on the LLaMA1.

A.3 Ablation Study

Impact of the parameter α . The tiny penalty parameter α balances the trade-off between output discrepancy and the maximum magnitude of the weights. We carry out experiments on channel-wise quantization for differernt α on LLaMA2-7B. The choice of α is independent of the bit-width. As shown in Table 7, we can find that both too large and too small α will lead performance degeneration. Compared to INT4 , fluctuations in alpha at INT3 result in greater performance fluctuations. Fortunately, α = 0 . 001 works well for all channel-wise quantization.

Impact of the parameter β . We shrink the quantization step to reduce the overall quantization error by a multiplicative scalar β . To investigate the influence of β , we experiment with different value of β at INT3 and INT2 channel-wise quantization. As shown in Table 8, β is positively correlated with the bit-width. Specifically, the best β is around 0.9 for INT3 quantization and for INT2 quantization the optimal β is around 0.8.

Impact of MagR on Quantization Error. To explore how MagR affects quantization error, we compared the errors with and without MagR by randomly select five layers from LLaMA2 models. As illustrated in Figure 2, quantization error is notably reduced across all layers with the application of MagR.

LlaMa2-7B

LlaMa2-13B

0.26

With MagR

0.26

WithMagR

WithoutMagR

WithoutMagR

0.24

0.24

0.22

Quantization error

Quantization error

0.22

0.20

0.20

0.18

0.18

0.16 -

0.16

0.14

0.14

14

65

121

184

217

14

88

145

215

271

Layer index

Layer index

The adaptive capacity of MagR. We investigated the combined effects of MagR and QuIP. As illustrated in Table 9, incorporating MagR significantly enhances the performance of QuIP, leading to improved quantization results for the LLaMA2 models family.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The proposed method is shown in Section 4 and demonstrated by experiments in Section 5. They accurately reflect the contributions discussed in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The proposed method needs mild parameter tuning to achieve the best results. It is discussed in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include any theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Implementation details and parameter setup are clearly stated in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer https://github.com/AozhongZhang/MagR

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Implementation details and parameter setup are clearly stated in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: In Section 5, we report the test results of the quantized models. Given the full-precision model, our algorithm produces the quantized model in a deterministic way. Hence, we don't have statistical significance to report in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Implementation details are clearly stated in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our method is an efficient algorithm to compress large LLMs into low-precision. It has wide applications in saving computing resources and broaden the applications of LLMs. The impacts are discussed in Section 1 and 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper introduces a method to quantize large AI models. We don't have models or data to release.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The data and code used in this paper are open to the public. In section 5, we illustrate properly what we use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors propose an optimization-based preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post-training quantization (PTQ) for large language models (LLMs). They motivate their method from previous work, showing that linear transformation of weights can render more quantization-friendly, allowing for sub-4bit quantization. However, such methods normally require a post-processing step during inference to undo that linear transformation, introducing a run-time overhead.\n- Instead, they propose a non-linear transformation without needing an additional post-processing step. That transformation aims to minimise the $\\ell_{\\infty}$ norm, or the maximum value of the weights per channel or column while preserving the layer's output. By making the weight distribution more compact around zero and removing outliers, they are able to quantize weights to 2 or 3 bits with little accuracy drop compared to competing methods.\n- They formulate the optimization as an $\\ell_{\\infty}$ -regularisation and solve it using proximal gradient descent. In fact, the problem reduces to computing the projection of weight vectors onto the $\\ell_1$ ball, for which there are several efficient algorithms available.\n\nSTRENGTHS:\n- MagR achieves state-of-art weight quantization without introducing any additional computation overhead. This is a very important contribution as all other competing methods with the linear transformation of weight, such as QuIP, require a linear transformation of the transformer block input/output. Having said this, the inference runtime comparison to QuIP is missing from the experiments section\n- The authors extensively evaluate the methods, evaluating a large corpus of datasets and benchmarks. They also prove that they outperform strong competing methods, such as OmniQuant, in almost all cases.\n- The idea of regularizing the weight's $\\ell_{\\infty}$ norm is quite novel and very effective. In fact, I am surprised this has not been established before in PTQ literature.\n\nWEAKNESSES:\n- The paper is very focused on ppl and accuracy results on a wide range of benchmarks and datasets at the cost of other important ablation studies. For example:\n\t- ablation study on the penalty factor $\\alpha$ and its impact on the final accuracy.\n\t- Inference runtime comparison is missing compared to competing methods, such as QuIP. In fact, the main claim of the paper is that it is faster than QuIP, but there are no studies supporting it.\n\t- Results with activation quantization beyond FP16 and runtime improvements vs baselines.\n- The connection between the rank-deficient feature matrix argument and $\\ell_{\\infty}$ regularisation requires further theoretical justification. There must be a theoretical guarantee for the optimality of that optimizaiton. In fact, the method would be stronger if it was formulated more generally as an $\\ell_p$ constrained, and the authors show that $\\ell_{\\infty}$ is better constrained.\n\nQUESTIONS:\n- Table 1: It is unclear to me how all these numbers are relevant in supporting the method. One can see from the maximum value that all Llama Models are rank-deficient based on the maximum value being below 100%. How are the rest of the statistics informative?\n- Line 124-125: From all the solutions available, why choose the lowest infinity norm? Why is this optimal? What is missing there is the optimality of this norm for quantization purposes. There, in fact, a discussion about quantization noise as a perturbation bounded by $\\ell_{\\infty}$ -norm in a paper by M. Alizadeh et. al called \"Gradient $\\ell_1$ regularization for quantization robustness\"\n- Table 3: activation quantization bitwidth missing\n- Please provide ablation studies about the penalty factor $\\alpha$. What is the effect of it not being tiny? How sensitive are the results?\n- To fully support the claims of the paper, it would be nice to see an ablation study on other norms as regularization. Otherwise, a theoretical justification is required about the optimality of the $\\ell_{\\infty}$ -norm.\n- Paragraph starting at line 210: it is unclear how you establish the optimality of the shrinking factor $\\beta$. In fact, I cannot see why you would not search for the optimal clipping threshold based on minimizing the MSE per column or group. This is a standard method for weight clipping in LLM quantization literature.\n- Section 5.3: the section's title is misleading. Maybe rename it to preprocessing runtime? Most commonly, runtime refers to the inference runtime\n- A section on inference runtime comparison to QuIP is required. The main claim of the paper is that the method is faster than QuIP due to the lack of transformation, but no experimental evidence is provided to support it.\n\nLIMITATIONS:\nThe paper lacks a thorough discussion of each limitation. The only reference I found was in lines 245-6 about extending MagR into incoherence processing.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach called Weight Magnitude Reduction (MagR) to improve the performance of PTQ for LLM. The MagR reduces the magnitude of the weights in PTQ. The experiments demonstrate the effectiveness of MagR.\n\nSTRENGTHS:\n1.\tThe idea is clear.\n2.\tThe paper is easy to read (except typos/errors).\n\nWEAKNESSES:\n1.\tThere exist some typos/grammatical errors in the paper and should be revised.\n2.\tThe presentation is not clear. For example, this paper aims to reduce the parameters of LLM using PTQ, however, the title and abstract did not reflect the contribution of LLM compression. In the abstract, the author claims MagR can diminish the maximum magnitude of the weights and smooth out outliers, however, the title only focuses on Weight Magnitude Reduction.\n3.\tProvide a more thorough discussion of the generalization ability, robustness, and potential applications of the proposed approach.\n4.\tThe format of references is not correct, i.e., [1], [3], [9],[14], etc.\n5.\tThe main limitation of this paper is that the proposed method lacks theoretical analysis.\n6.\tWhile the abstract provides an overview of the proposed method, some aspects of the methodology could benefit from further elaboration in the main paper. Providing step-by-step explanations and intuitive visualizations for key components such as L infinity norm, and L infinity-> L 1 norm, would enhance the reader's understanding.\n7.\tThe paper claims that the method outperforms state-of-the-art methods, but a more comprehensive comparative analysis is needed. Detailed comparisons with other existing approaches, along with discussions about the reasons behind the performance differences, would strengthen the argument for the superiority.\n8.\tExploring the reasons behind the success of these techniques and providing intuitive explanations would contribute to the overall scientific contribution of the work.\n\nQUESTIONS:\n1. The paper contains numerous hyperparameters, such as \\(\\alpha\\). Including more ablation studies would enhance the readers' understanding.\n2. In Table 3, why did MagR+OPTQ not achieve better results compared to peer competitors?\n3. How can the impact of outliers on PQT for LLM be reduced?\n\nLIMITATIONS:\nNO", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose an optimization-based preprocessing technique called MagR to enhance the performance of post-training quantization.\nMagR adjusts weights by solving an l1-regularized optimization problem, reducing the maximum magnitude and smoothing out outliers.\nAs a nonlinear transformation, MagR eliminates the need for additional post-processing, thus avoiding overhead during inference.\nExperiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models, such as significantly reduced perplexity on Wikitext2 for the LLaMA2-70B model.\n\nSTRENGTHS:\n1.The experiment results are good.\n2. The idea is novel, which is using preprocessing technology before the quantization process.\n\nWEAKNESSES:\nIn fact, I like the work but not in authors' view. I think this work is improve the generalization of model via norm. If this paper presents their work in this view, I would like to accept more.\n1.Some value is lack of explanation, like \" minimal the singular values of the feature matrix are less than 0.01 times the maximum singular value.\" \n2. The applicability and the nature analysis of methods should be discussed.\n\nQUESTIONS:\n1. In Section 4.1, the singular values of the feature matrix are less than 0.01 times the maximum singular value. How is 0.01 calculated? If this value is changed, relaxing the constraint, will the results change?\"\n2. If the feature matrix is not actually rank-deficient, could the performance of the MagR method potentially suffer?\n3. Proximal gradient descent may require more computational resources.\n4. The model performance after preprocessing and before quantization should be presented. I wonder that the nature of this method is improve the generalizability of model before quantization.\n\nLIMITATIONS:\nThe paper is lack of discussing the nature of preprocessing, i.e., what the preprocessing do for quantization.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Weight Magnitude Reduction (MagR), a technique designed to smooth out outliers before LLM quantization. MagR adjusts pre-trained floating-point weights by solving an ℓ∞-regularized optimization problem. This preprocessing step reduces the maximum weight magnitudes, making the LLMs more suitable for quantization and resulting in better perplexity/accuracy results compared to models without preprocessing. Importantly, the MagR technique does not introduce any additional computational overhead during inference.\n\nSTRENGTHS:\n1) Since MagR is a technique for preprocessing LLM weights before quantization, it can be used in conjunction with other quantization methods (e.g. OPTQ).\n\n2) MagR introduces proximal gradient descent steps, and it addresses these through layer-wise optimization using the Hessian, resulting in a shorter quantization runtime similar to OPTQ.\n\n3) This method does not require additional operations for scaling activation, as needed in AWQ and QuIP, thereby avoiding extra overhead during inference.\n\nWEAKNESSES:\n1) MagR does not show significant improvement in perplexity/accuracy over previous method - QuIP. \nAdditionally, DecoupleQ [1] presents strong results for 2-bit quantization. While I understand that DecoupleQ was released recently, allowing insufficient time for a thorough comparison, I recommend the authors review DecoupleQ.\n\n2) The paper emphasizes that MagR has no inference overhead, unlike AWQ and QuIP, which require scaling activations. However, it does not provide any latency information to highlight the significance of removing this additional inference overhead.\n\n3) The runtime cost comparison between MagR and other methods is insufficient. The paper only provides the runtime of MagR, mentioning roughly that it is half the runtime of OmniQuant on page 8, lines 252-254. Since MagR is a preprocessing method, it is unclear if this comparison holds when considering the entire quantization process (MagR + OPTQ). Additionally, the runtime of QuIP is not addressed.\n\n4) Some prior works have regularized weight distribution to make the model more suitable for quantization [2]. However, this paper does not mention or compare the differences between the proposed method and these earlier approaches.\n\n5) The paper does not include any information about the calibration dataset.\n\n[1] Guo, Yi, et al. \"decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points.\" arXiv preprint arXiv:2404.12759 (2024).\n\n[2] Kundu, Arnav, et al. \"R2 Loss: Range Restriction Loss for Model Compression and Quantization.\" arXiv preprint arXiv:2303.08253 (2023).\n\nQUESTIONS:\n1) What is the inference latency of the proposed method compared to QuIP?\n\n2) What is the runtime for QuIP and OmniQuant?\n\n3) Why did you not provide perplexity results for the 7B/13B models for QuIP in Table 2?\n\n4) Is QuIP incompatible with group-wise quantization, or can it be adapted to group-wise quantization with minor modifications, as done with MagR and OPTQ?\n\n5)  For the results in Table 3, do all the methods use channel-wise quantization, or do some use group-wise quantization?\n\n6) What type of calibration dataset did you use, and how many data points were included for calibration?\n\n7) What happens if you apply Omniquant/QuIP after processing LLMs with MagR?\n\nLIMITATIONS:\nThe proposed MagR is a promising approach, as it regularizes weight values to produce LLMs that are better suited for quantization. However, this paper lacks sufficient evaluation to convincingly demonstrate the superiority of the proposed method compared to state-of-the-art techniques.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}