I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models

## Abstract

Abstract Post-training quantization (PTQ) serves as a potent technique to accelerate the inference of large language models (LLMs). Nonetheless, existing works still necessitate a considerable number of floating-point (FP) operations during inference, including additional quantization and de-quantization, as well as non-linear operators such as RMSNorm and Softmax. This limitation hinders the deployment of LLMs on the edge and cloud devices.
In this paper, we identify the primary obstacle to integer-only quantization for LLMs lies in the large fluctuation of activations across channels and tokens in both linear and non-linear operations. To address this issue, we propose I-LLM, a novel integer-only fully-quantized PTQ framework tailored for LLMs. Specifically, (1) we develop Fully-Smooth Block-Reconstruction (FSBR) to aggressively smooth inter-channel variations of all activations and weights. (2) to alleviate degradation caused by inter-token variations, we introduce a novel approach called Dynamic Integer-only MatMul (DI-MatMul). This method enables dynamic quantization in full-integer matrix multiplication by dynamically quantizing the input and outputs with integer-only operations. (3) we design DI-ClippedSoftmax, DI-Exp, and DI-Normalization, which utilize bit shift to execute non-linear operators efficiently while maintaining accuracy.
The experiment shows that our I-LLM achieves comparable accuracy to the FP baseline and outperforms non-integer quantization methods. For example, I-LLM can operate at W4A4 with negligible loss of accuracy. To our knowledge, we are the first to bridge the gap between integer-only quantization and LLMs. We’ve published our code on anonymous.4open.science , aiming to contribute to the advancement of this field.

## 1 Introduction

Large Language Models (LLMs) have paved the way for general artificial intelligence with their remarkable performance across a wide range of tasks. However, the rising number of parameters and computing power requirements of LLMs pose significant challenges when it comes to deployment.

Post-training quantization (PTQ) is a powerful technique employed to accelerate the inference process of LLMs. Previous PTQ methods for LLMs have primarily relied on simulated quantization (aka. fake quantization) [, , , ], where integer values are typically used for inputs/outputs and compute-intensive operations are performed using dequantized floating-point (FP) values (as shown in Fig4). Although this scheme offers benefits in scenarios where data transmission bandwidth is limited, it does not effectively reduce computational costs and thus has little effect on compute-bound situations. Besides, non-linear operations (e.g., Softmax and RMSNorm) often involve complex operations, including transcendental functions such as exponential functions and square root functions. These functions are typically performed on dedicated FP units and may require multiple iterations for accurate computation.

Figure: Figure 1: Differences in activations of the non-linear operator in LLaMA2-7b across the channel/token dimensions.
Refer to caption: extracted/5646459/figures/activation.png

In contrast, integer-only quantization [, , , , ] utilizes low-precision integer arithmetic for all operations, including linear operations (e.g., matrix multiplication) and non-linear operations. It enables quantized models to take full advantage of fast and efficient integer arithmetic units, resulting in promising speedup effects and reduction of latency and power consumption. Additionally, integer-only quantization facilitates deployment on popular edge processors specifically designed for embedded, mobile, or IoT devices, which often lack dedicated floating point units. Examples of such edge processors include ARM Cortex-M [], GreenWaves GAP-9 [], and Google’s Edge TPU []. Note that Turing Tensor Cores in GPU server-class devices also have introduced support for integer logical units, offering notably lower latency compared to FP arithmetic.

However, the aforementioned integer-only methods are designed specifically for CNNs or small Transformer networks (e.g., Bert [] and ViT []), which renders them inadequate for LLMs.
First, they are incapable of straightforwardly supporting the non-linear operators inherent in LLMs, such as SwiGLU and RMSNorm.
Furthermore, their accuracy on LLMs deteriorates significantly (as depicted in Table 4), even when those non-supported operators are executed in full-precision mode. This is because as the model size increasing, the presence of activation outliers in both linear and non-linear layers becomes prominent. As illustrated in Fig 2, Llama2-7B exhibits substantial variations in activation magnitude at both the token and channel levels, making previous approaches ineffective.
Last but not least, these methods are limited to 8-bit quantization, whereas LLMs would benefit from lower bit-width quantization (e.g., 4-bit) to address the extensive computational requirements and storage demands.
Consequently, how to accurately perform LLMs with efficient integer-only arithmetic remains an unresolved issue that requires further investigation.

In this paper, we identify the primary obstacle to integer-only quantization for LLMs lies in the large fluctuation of activations across channels and tokens in both linear and non-linear operators. To address this issue, we introduce I-LLM, a novel integer-only PTQ framework tailored for LLMs: (1) We propose Fully-Smooth Block-Reconstruction (FSBR) to harmonize the variance in activation across channels. While Omniquant [] and Smoothquant [] share some similarities, they primarily focus on smoothing the activation in serial norm-linear and parallel linear-linear operations. We argue that mitigating the disparities of all suitable activation-activation and activation-weight pairs of LLMs (see Fig.5) significantly enhances accuracy. For instance, the input of SwiGLU encounters numerous disparities on both token-wise and channel-wise dimensions, as depicted in Fig.2-a. To achieve smoothing on such a non-linear operator, we decompose SiLU into $x\cdot\sigma(x)$ to apply FSBR, and as a result the input becomes more consistent and amenable to quantization, as shown in Fig.2-b. (2) To alleviate the degradation resulting from inter-token variations, we present a novel approach called Dynamic Integer-only MatMul (DI-MatMul). DI-MatMul facilitates quantization on full-integer matrix multiplication by employing integer-only operations to quantize the input and outputs dynamically. Traditional static quantization methods, which are characterized by their lack of robustness and adaptability, often falter when encountering input beyond the calibration set. In contrast, DI-MatMul is designed to proactively recognize and adapt to the diverse range of input data, thereby reducing quantization errors and enhancing overall model performance. (3) For non-linear operators, we design DI-ClippedSoftmax, DI-Exp, and DI-Norm, which leverage the efficiency of bit shifting to replace complex math calculations while maintaining accuracy.
Our contributions are summarized as:

- 1.
We identify the primary obstacle to integer-only quantization for LLMs lies in the large fluctuation of activations across channels and tokens in both linear and non-linear operators. To address inter-channel variations, we propose FSBR to effectively reduces disparities among all suitable activation-activation and activation-weight pairs, thereby markedly improving accuracy.
- 2.
We attribute the failure of previous integer-only quantization methods on LLMs to various range in activation tokens. To tackle this problem, we introduce DI-MatMul, which enables dynamic quantization on input and output through full-integer matrix multiplication. DI-MatMul proactively adapts to the range of input, minimizing quantization errors and improving overall performance.
- 3.
We introduce DI-Exp, DI-ClippedSoftmax, and DI-Norm, innovative integer-only operators that harness the power of bit shifting to replace complex mathematical computations within the non-linear functions of LLMs, without compromising on accuracy.
- 4.
To the best of our knowledge, this work represents the first attempt to utilize integer-only quantization for LLMs, enabling their deployment on edge devices that lack floating-point capabilities. Experiments demonstrate remarkable accuracy when compared to SOTA non-integer quantization techniques, e.g., I-LLM on LLAMA-13b achieves an approximate 20% reduction in perplexity.

## 2 Related Work

LLMs Quantization. LLMs quantization can be broadly categorized into weight-only and weight-activation quantization. To alleviate the computational burdens, some works [, , , , , , , ] make efforts in weight-only quantization. GPTQ [] and QuIP [] achieve high compression rates by optimizing matrix multiplications operation. AWQ [] and OWQ [] demonstrate performance improvements by accounting for the impact of activation outliers on weight quantization. Moreover, works such as QLORA [], QA-lora [] and LoftQ [] leverage Parameter Efficient Fine-Tuning (PEFT) techniques to achieve weight compression with fine-tuning tasks. Different with weight-only quantization methods, towards to accelerate the LLMs inference, the weight-activation quantization methods [, , , , , ] quantize both the weights and activations, including the KV cache. The primary challenge in quantizing activations lies in outliers, leading to significant quantization errors. To tackle this issue, ZeroQuant [] proposes a fine-grained hardware-friendly quantization scheme for both weight and activations. SmoothQuant [] migrates the quantization difficulty from activations to weights with a mathematically equivalent transformation (i.e., per-channel scaling). OmniQuant [] further enhances performance by training the quantization parameters. While these methods have mitigated the quantization error, their inference pipelines still involve partially FP operations on non-linear operators such as Softmax, Normalization, and SiLU. In this study, our focus on achieving Integer-only inference for LLMs model using advanced PTQ [, , , ] techniques.

Integer-only Quantization. Current quantization methods for LLMs often involve dequantized FP operations during inference, limiting the utilization of efficient low-precision arithmetic units. Integer-only quantization, eliminating dequantization, enables complete inference using Integer-only arithmetic, promising enhanced model acceleration. Previous approaches [, ] leverage dyadic arithmetic for Integer-only pipeline on CNNs, but these are tailored for linear operations and are unsuitable for non-linear operations in ViTs. Applying INT arithmetic solely to linear operations while retaining FP arithmetic for non-linear ones maybe a straightforward solution, but this demands custom hardware and introduces computational overheads. Advancements include Fully-8bit [] and I-BERT [], which address non-linear operations through employs L1 LayerNorm and INT polynomial approximations for the non-linear operations. However, these methods face inefficiencies or fail to fully exploit hardware advantages. Based on I-BERT[], FQ-ViT[] extends INT arithmetic to part of the operations but overlooks significant non-linear operations like GELU. While some methods[, ] retain FP arithmetic during approximation, they cannot meet the requirement for Integer-only arithmetic. Recently, I-ViT[] introduces Integer-only quantization for ViTs, yet its suitability for LLMs is questionable due to differing data distributions, and its computational graph includes partially INT32 precision operations. In this paper, we focus on Integer-only inference for LLMs, maintaining the entire computational graph at INT8 precision or lower bit (e.g., 6/4-bit), enhancing LLMs’ inference efficiency on edge processors.

## 3 Method

Figure: Figure 3: Typical LLM quantization vs. I-LLM. The former requires dequantization and involves FP arithmetic, while the latter performs the entire inference using integer-only arithmetic.
Refer to caption: extracted/5646459/figures/i-llm-Smooth_vs_I-LLM.png

Challenges of Integer-Only Quantization for Large Language Models.
Presently, integer-only LLMs encounter two primary hurdles: (1) quantizing the activation of LLMs, especially those originating from non-linear operators, poses a formidable challenge. As evidenced in Fig 2, the divergence in these non-linear activations often surpasses that of linear operators, particularly pronounced in models such as LLaMA []. Previous methods have failed to address these non-linear activations, and straightforwardly quantizing non-linear layers may lead to substantial accuracy degradation. (2) prior integer-only quantization techniques have overlooked the distinctive traits of LLMs, including divergent activation scales and the substantial overhead of loading large-scale weights. Even the W8A8 method introduced in I-BERT can lead to catastrophic outcomes, as shown in Fig 4, let alone more aggressive quantization methods like W6A6 or W4A4.

In this section, we introduce a novel integer-only quantization framework termed I-LLM. As illustrated in Fig 5, this framework incorporates a differentiable approach within the Post-Training Quantization (PTQ) paradigm, termed Fully-Smooth Block Reconstruction. This method is designed to effectively balance all feasible parameter pairs, as elaborated in Section 3.2.
Furthermore, we advance the develop of dynamic quantization within the integer-only context by introducing DI-MatMul, a dynamic integer-only matrix multiplication method, which is elucidated in Section 3.3. Additionally, we detail integer-only non-linear approximations, including DI-ClippedSoftmax, DI-exp, and DI-Norm, that are built upon DI-MatMul and are presented in Section 3.4. These operators facilitate 8-bit input activations while minimizing accuracy loss.

### 3.1 Basic Mathematical Notations

Matrices are denoted by bold uppercase letters such as $\bm{X}$, vectors by bold lowercase letters such as $\bm{x}$. Floating-point and integer numbers are distinguished using the superscript $I$, for example, $x^{I}$ for integers and $x$ for floating-point numbers. The notation $\mathcal{Q}$ means the quant function, $\lfloor\cdot\rfloor$ represents the floor function, and $\lfloor\cdot\rceil$ denotes rounding to the nearest integer. The symbol $\otimes$ indicates element-wise multiplication, while $\oslash$ denotes element-wise division. Other mathematical symbols follow their standard definitions.

### 3.2 Fully-Smooth Block-Reconstruction (FSBR)

As mention above, to mitigate the issue of non-linear layer activations in LLMs being affected by channel and token differences (Fig 2and 2), we propose Fully-Smooth Block-Reconstruction (FSBR). An intuitive method is to train a smoothing coefficient for all activations and weights to aid in restoring the model’s quantization accuracy, especially the quantization accuracy of non-linear operators. Therefore, we consider the activations of all non-linear layers and learn the smoothing coefficients for all possible equivalent smoothing transformations at the channel level. On the left side of Fig 5, four paradigms for achieving inter-channel smoothing during block reconstruction are shown: Parallel Linear-Linear, Serial Linear-Norm, Serial Linear-Linear, and NonLinear Act-Smooth. The first three smoothing methods are indeed linear, making them easier to implement.

However, when addressing non-linear operators (NonLinear Act-Smooth) in LLMs, such as the gated activation function (SwiGLU), it becomes challenging to apply any linear transformations to them. For convenience, we can represent SwiGLU using the following formula:

$$ $\displaystyle\operatorname{SwiGLU}(\bm{x},\bm{W},\bm{V},\bm{b},\bm{c})$ $\displaystyle=\operatorname{SiLU}(\bm{x}\bm{W}+\bm{b})\otimes(\bm{x}\bm{V}+\bm {c})$ (1) $\displaystyle=\bm{x1}\otimes\bm{x2}\otimes\sigma(\bm{x1})$ $$

Where $x\in\mathbb{R}^{ic}$ is a vector, $\bm{W},\bm{V}\in{\mathbb{R}^{ic\times oc}}$ are weight matrices, and $\bm{b},\bm{c}\in{\mathbb{R}^{oc}}$ represent biases. $\bm{x1}=\bm{x}\bm{W}+\bm{b}$, $\bm{x2}=\bm{x}\bm{V}+\bm{c}$. $\operatorname{SiLU}$ stands for Sigmoid-Weighted Linear Unit. $\sigma$ represents the sigmoid activation function. In order to jointly balance the activation and weight quantization difficulties of $\bm{x1}$ and $\bm{x2}$, we introduce a smoothing factor $\bm{s}$, expressed in the formula as follows:

$$ $\displaystyle\operatorname{SwiGLU}(\bm{x},\bm{W},\bm{V},\bm{b},\bm{c})$ $\displaystyle=(\bm{x1}\otimes\bm{s})\otimes(\bm{x2}\oslash\bm{s})\otimes\sigma ^{\prime}(\bm{x1}\otimes\bm{s})$ $\displaystyle=\bm{x1^{\prime}}\otimes\bm{x2^{\prime}}\otimes{\sigma^{\prime}}( \bm{x1^{\prime}})\$ $$

Where $\bm{W}^{\prime}=\bm{W}\otimes\bm{s},\quad\bm{b}^{\prime}=\bm{b}\otimes\bm{s},
\quad\bm{V}^{\prime}=\bm{V}\oslash\bm{s},\quad\bm{c}^{\prime}=\bm{c}\oslash\bm
{s}$, $\bm{x1^{\prime}}=\bm{x}\bm{W}^{\prime}+\bm{b^{\prime}},\quad\bm{x2^{\prime}}=
\bm{x}\bm{V}^{\prime}+\bm{c^{\prime}},\quad\sigma^{\prime}(\bm{x1^{\prime}})=
\sigma(\bm{x1^{\prime}}\oslash\bm{s})$.
As shown by the red dashed box in Fig 5, during reconstruction, we first incorporate $\bm{s}$ into the weights. After calculating $\bm{x1^{\prime}}$ and $\bm{x2^{\prime}}$, we quantize them before proceeding with the computation. The forward inference process of $\operatorname{SwiGLU}$ can be expressed as: $\operatorname{SwiGLU}(\bm{x})=\mathcal{Q}(\mathcal{Q}(\bm{x})\mathcal{Q}(\bm{W
^{\prime}})+\bm{b^{\prime}})\otimes\mathcal{Q}(\mathcal{Q}(\bm{x})\mathcal{Q}(
\bm{V^{\prime}})+\bm{c^{\prime}})\otimes\sigma^{\prime}(\mathcal{Q}(\mathcal{Q
}(\bm{x})\mathcal{Q}(\bm{W^{\prime}})+\bm{b^{\prime}}))$. After block reconstruction, it is natural to incorporate the smoothing factor $\bm{s}$ into the weights. The only difference lies in replacing the original activation function $\sigma$ with $\sigma^{\prime}$, which incurs negligible overhead. Fig 2 presents the output activation distribution of the gated unit in SwiGLU before and after the FSBR. It can be observed that the significant imbalance between channels and tokens, as depicted in Fig 2-a, is effectively alleviated after FSBR (Fig 2-b).

Figure: Figure 5: Details of I-LLM in a transformer block. The left side of the figure illustrates various paradigms for channel-wise smoothing during the FSBR process. The right side depicts the integer-only execution pipeline for both linear operators, such as matrix multiplication (MatMul), and non-linear operators.
Refer to caption: extracted/5646459/figures/i-llm-main_method_pipeline.png

It is worth noting that SmoothQuant[] and OmniQuant[] are subsets of FSBR. FSBR encompasses various previous equivalent quantization algorithms, providing more possibilities for optimizing the distribution of weights and activations. Through mutual optimization across channels, the network demonstrates improved robustness to quantization, as shown in Table 5.

### 3.3 Dynamic Interger-only MatMul (DI-MatMul)

The dynadic arithmetic pipeline is an efficient approach that implements floating-point operations using integer bit-shifts, allowing linear operations to be performed solely through integer arithmetic. Initially developed for convolutional neural networks [], this technique has been extended to Vision Transformers by I-ViT []. However, these methods primarily focus on static quantization, where the quantization parameters of the model’s activation are fixed during runtime. In the context of LLMs, even after applying inter-channel smoothing, there still exists considerable variation in activation on a token-wise basis, as shown in Fig. 6 in Appendix A. Employing static quantization can result in significant quantization errors and subsequent degradation in accuracy, as depicted in the Fig. 4. Therefore, implementing dynamic quantization for inputs and outputs while adhering to Integer-only constraints poses a significant challenge.

We propose a novel DI-MatMul approach, where the matrix multiplication is formulated as $\bm{Y^{I}},s_{y},zp^{I}=\mathcal{M}(s_{1},zp_{1}^{I},\bm{X^{I}_{1}},s_{2},zp_{
2}^{I},\bm{X^{I}_{2}})$. Herein, $s_{1}$, $s_{2}$, and $s_{y}$ are floating-point scalars representing the quantization steps for the inputs and outputs respectively; while $zp_{1}^{I}$ and $zp_{2}^{I}$ denote zero-points. To avoid floating-point operations in our method, we represent quantization step $s$ using a dyadic number (DN), i.e., $\frac{m^{I}}{2^{k^{I}}}$ where both $m^{I}$ and $k^{I}$ are 8-bit integers. Consequently, the entire matrix multiplication can be expressed as:

$$ $\displaystyle\bm{Y^{I}},m^{I}_{y},k^{I}_{y},zp^{I}=\mathcal{M}(m_{1}^{I},k_{1} ^{I},zp_{1}^{I},\bm{X^{I}_{1}},m_{2}^{I},k_{2}^{I},zp_{2}^{I},\bm{X^{I}_{2}})$ (2) $$

For a single matrix multiplication operation:

$$ $\displaystyle\bm{P^{I}}=(\bm{X_{1}^{I}}-zp^{I}_{1})(\bm{X_{2}^{I}}-zp^{I}_{2}) ,\quad\bm{Y}=\bm{P^{I}}\cfrac{m^{I}_{1}m^{I}_{2}}{2^{k^{I}_{1}+k^{I}_{2}}}$ (3) $$

The intermediate result of the matrix multiplication is denoted as $\bm{P^{I}}$. By applying Eq 15 and preliminary Eq 2, we obtain the following approximation:

$$ $\displaystyle\frac{m^{I}_{y}}{2^{k^{I}_{y}}}\approx s_{y}=\frac{(p^{I}_{\max}- p^{I}_{\min})\cdot m^{I}_{1}\cdot m^{I}_{2}}{(2^{n^{I}}-1)\cdot 2^{k^{I}_{1}+k ^{I}_{2}}}$ (4) $$

where $p^{I}_{\max}$ and $p^{I}_{\min}$ denote the maximum and minimum values in $\bm{P^{I}}$. To obtain the quantization scale of the output:

$$ $\displaystyle\mathop{\arg\min}\limits_{m^{I}_{y},k^{I}_{y}}\left|\left|\frac{( p^{I}_{\max}-p^{I}_{\min})\cdot m^{I}_{1}\cdot m^{I}_{2}}{(2^{n^{I}}-1)\cdot 2 ^{k^{I}_{1}+k^{I}_{2}}}-\frac{m^{I}_{y}}{2^{k^{I}_{y}}}\right|\right|_{1}\quad \text{s.t.}\quad m^{I}_{y},k^{I}_{y}\in{0,1,2,\ldots,255}$ (5) $$

Obtaining the optimal values for $m^{I}_{y}$ and $k^{I}_{y}$ may require multiple iterations. However, by induction, it is known that if an optimal value of $k^{I}_{0}$ is achieved for $k^{I}_{y}$, then increasing it to $k^{I}_{0}+1$ will not result in a worse outcome. Therefore, we set $m^{I}_{y}=256$ to determine the value of $k^{I}_{y}$, and subsequently solve for the remaining variables as follows:

$$ $\displaystyle k^{I}_{y}=\left\lfloor\log_{2}\left(\frac{256\cdot(2^{n^{I}}-1) \cdot 2^{k^{I}_{1}+k^{I}_{2}}}{p^{I}_{\max}-p^{I}_{\min}}\right)\right\rfloor= \left\lfloor\log_{2}\left(\frac{(2^{n^{I}}-1)\cdot 2^{k^{I}_{1}+k^{I}_{2}+8}}{ p^{I}_{\max}-p^{I}_{\min}}\right)\right\rfloor$ (6) $\displaystyle m^{I}_{y}=\left\lfloor\frac{(p^{I}_{\max}-p^{I}_{\min})\cdot m^{ I}_{x1}\cdot m^{I}_{x2}}{n^{I}}\gg(k^{I}_{1}+k^{I}_{2}-k^{I}_{y})\right\rceil$ (7) $\displaystyle z^{I}_{y}=\left\lfloor\frac{-p^{I}_{\min}\cdot(2^{n^{I}}-1)}{p^{ I}_{\max}-p^{I}_{\min}}\right\rceil,\quad\bm{Y^{I}}=\left\lfloor\frac{(\bm{P^{ I}}-p^{I}_{\min})\cdot(2^{n^{I}}-1)}{p^{I}_{\max}-p^{I}_{\min}}\right\rceil$ (8) $$

The implementation of $\left\lfloor\operatorname{log_{2}}(\cdot)\right\rfloor$ can be achieved using either the Most Significant Bit (MSB) method or by performing iterative right shifts. As shown in Eq (6), (7), and (8), our approach introduces only a few additional integer-only scalar computations during runtime, making it more efficient than previous methods.
For the Dense layers in Large Language Models (LLMs), one can simply replace an input with its corresponding weights. During inference, since the input usually consists of a single token, this allows for per-token dynamic quantization naturally. In the prefill phase, we adjust the required dimensions accordingly; for instance, we define $\bm{m^{I}}$ and $\bm{k^{I}}$ as vectors in $\mathbb{R}^{\text{token}}$.

### 3.4 Dynamic Non-Linear Integer-only Operations

#### 3.4.1 Dynamic Interger-only Clipped Softmax & Dynamic Integer-only Exponent function

The Softmax function converts attention scores into a probability distribution. It can be represented as follows:

$$ $\operatorname{Softmax}\left(x_{i}\right)=\frac{e^{x_{i}}}{\sum_{j=0}^{i}e^{x_{ j}}}=\frac{e^{x_{i}-x_{\text{max}}}}{\sum_{j=0}^{i}e^{x_{j}-x_{\text{max}}}}, \quad\text{where }i=1,2,\cdots,d$ (9) $$

We introduce DI-ClippedSoftmax to avoid quantizing the entire input range. As shown in Fig 2, the activation inputs to the Softmax function in LLMs can exhibit significant outliers, with their magnitudes being proportional to the number of tokens. Fortunately, the DI-MatMul approach proposed in Section 3.3 allows for the differences between tokens to be handled individually. However, simply applying 8-bit quantization to the Softmax inputs would result in substantial precision loss, as demonstrated by the ablation studies in Section 5.
Starting from the observation that the value of the exponential function at $-20$ can be considered to be zero, this implies that, for each token, only the values within the range $(x_{\text{max}}-20,x_{\text{max}})$ contribute to the result. For a single token, we can constrain the quantization range by limiting $p^{I}_{\text{min}}$ as defined in Eq 4. Assuming we want to confine the range within $(p_{\text{max}}-c,c)$, we first represent $c$ as a dyadic number, i.e., $c=\frac{m^{I}_{c}}{2^{k^{I}_{c}}}$. Combining this with Eq 4, the entire truncation process can be expressed as follows:

$$ $\displaystyle p^{I}_{min}=\operatorname{max}(p^{I}_{min},p^{I}_{max}-c^{I}) \quad\text{, where }c^{I}=m_{c}^{I}\cdot 2^{k_{1}^{I}+k_{2}^{I}-k_{c}^{I}}$ (10) $$

Therefore, regardless of the dynamic range, the length of our quantization range will never exceed $c$. In our hyper-parameters tuning experiments (Table 5), we select the optimal value of $c=15$, ensuring that the maximum quantization error does not exceed 0.047.

We also propose Dynamic Interger-only Exponent function(DI-Exp), an exponential computation method that performs non-linear calculations of the exponential function using only shift operations. For a dynamically quantized input composed of $\bm{x^{I}},m^{I}_{x},k_{x}^{I}$, where $\bm{x^{I}}$ is already the result after subtracting the maximum value, the computation of the exponential can be expressed as:

$$ $\displaystyle\operatorname{e}^{\bm{x^{I}}\cdot\left(m^{I}_{x}\gg k^{I}_{x} \right)}=\operatorname{2}^{\bm{x^{I}}\cdot\left(m^{I}_{x}\gg k^{I}_{x}\right) \cdot\log_{2}e}=2^{\bm{x^{I}}\cdot m^{I}_{x}\cdot\left(\log_{2}e\gg k_{x}^{I} \right)}$ (11) $$

We let $s=\frac{\log_{2}e}{2^{k_{x}^{I}}}$ and $s^{I}=\left\lfloor\frac{2^{k_{x}^{I}}}{\log_{2}e}\right\rceil$, then:
$\operatorname{e}^{\bm{x^{I}}\cdot\left(m^{I}_{x}\gg k^{I}_{x}\right)}\approx 2
^{\left(\left\lfloor\frac{\bm{x^{I}}}{s^{I}}\right\rfloor+\left(\bm{x^{I}}\%s^
{I}\right)s\right)}\approx 2^{-q^{I}+r^{I}\cdot s}$

where $q^{I}=-\left\lfloor\frac{\bm{x^{I}}}{s^{I}}\right\rfloor,\quad r^{I}=\bm{x^{I}
}\%s^{I}$
It is known that $r^{I}\cdot s\in\left(-1,0\right]$, therefore, we can simply perform linear interpolation within this interval. That is,
$2^{r^{I}\cdot s}\approx 1-\frac{r^{I}}{2\cdot s^{I}}$
Finally, we obtain:

$$ $\displaystyle\operatorname{e}^{\bm{x^{I}}\cdot\left(m^{I}_{x}\gg k^{I}_{x} \right)}\approx\left(1-\frac{r^{I}}{2\cdot s^{I}}\right)\gg q^{I}$ (12) $$

In DI-Exp, nonlinearity is achieved using only shift operations, which improves computational efficiency compared to complex methods such as quadratic fitting and Taylor series expansion. Only the calculation of $s^{I}$ and the linear interpolation within a small interval introduce errors, while other computations remain equivalent to the original. Detailed implementation of DI-Exp is in Algorithm1.

#### 3.4.2 Dynamic Interger-only Normalization & Dynamic Interger-only SwiGLU

DI-Norm. LayerNorm and RMSNorm are commonly used normalization methods in LLMs. RMSNorm represents a lightweight improvement over LayerNorm, as it does not necessitate the computation of means and biases, thereby reducing computational complexity. Both RMSNorm and LayerNorm exhibit similar characteristics, with significant fluctuations at the channel level. As depicted in the Fig 2, RMSNorm often exhibits more pronounced differences between channels due to the absence of a centering step.
We propose Dynamic Integer-only Normalization (DI-Norm) to adapt to fluctuations between channels. When conducting FSBR, we perform per-channel quantization on the inputs of LayerNorm and RMSNorm. During inference, while computing the mean is straightforward, calculating the variance requires an appropriate root mean square function. Unlike the Newton’s method used in I-BERT, to ensure consistency between inference and training, we employ a bit-wise check method as mentioned in the Algorithm 4 to achieve higher precision.

DI-SwiGLU. During the block reconstruction stage, we decompose the SiLU activation function of SwiGLU into $\operatorname{SiLU}(\bm{x})=\bm{x}\cdot\sigma(\bm{x})$ to achieve non-linear smoothing. As described in Algorithm 3, we implement the Dynamic Integer-only SwiGLU (DI-SwiGLU), consisting of a single sigmoid nonlinearity and two multiplication operations, where the sigmoid implementation involves multiple invocations of our proposed DI-Exp operator.

## 4 Experiments

Implementation Details. I-LLM ensures that the activation of all non-linear operators remains at 8 bits, while the weights and activations of linear operators are determined by the current quantization configuration, such as W4A8. Consistent with other methods, we employ 128 samples for reconstruction. During the reconstruction phase, we maintain that the input to Softmax is not quantized and ensure that all smoothing coefficients maintain a common learning rate of $5\times 10^{-3}$. After the reconstruction, all operators will be replaced with respective versions supporting dynamic integer-only inference. All experiments are conducted on Nvidia A6000 GPU.

Models & Evaluation Metric.
We conduct experiments on several commonly used open-source LLMs, including OPT [], LLaMA [], and LLaMA2 []. Additionally, we also evaluated the recently impressive LLaMA3-8B model.
For the sake of comparison, we tested the impact of quantization on Perperxity on two of the most commonly used datasets WikiText2[] and C4[]. Moreover, accuracy is evaluated in zero-short tasks including PIQA [], ARC[], BoolQ [], HellaSwag [], Winogrande[].

### 4.1 Quantitative Results

Notably, our work is the first to address integer-only quantization for LLMs, whereas all the methods we compare against in our experiments are not integer-only except I-Bert. Although this may seem somewhat unfair for us, I-LLM still demonstrates the superiority through its remarkable performance.

Fig 4 illustrates the efficacy of our 8-bit quantization technique on widely adopted large-scale language models. Despite SmoothQuant’s prior success with 8-bit quantization, our approach demonstrates performance for each model that is closer to floating-point precision. This suggests that even with low-bit quantization using integer-only conditions (e.g., 8-bit), we can achieve performance comparable to floating-point representation. These findings strongly validate the effectiveness of our proposed I-LLM, as it enables integer-only operators to yield results highly akin to those obtained through floating-point operations, using simple arithmetic and integer bit-shifting.

**Table 1: Quantitative weight-activation quantization PPL($\downarrow$) results of I-LLM. We report WikiText2 and C4 perplexity of LLaMA Family in this table.**
| #Bits | Method | LLaMA-7B | LLaMA-13B | LLaMA-30B | LLaMA2-7B | LLaMA2-13B | LLaMA3-8b |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WikiText2 | C4 | WikiText2 | C4 | WikiText2 | C4 | WikiText2 | C4 | WikiText2 | C4 | WikiText2 | C4 |  |  |
| FP16 | - | 5.68 | 7.08 | 5.09 | 6.61 | 4.10 | 5.98 | 5.47 | 6.97 | 4.88 | 6.46 | 6.14 | 8.88 |
| W6A6 | SmoothQuant | 6.03 | 7.47 | 5.42 | 6.97 | 4.55 | 6.34 | 6.2 | 7.76 | 5.18 | 6.76 | 7.08 | 10.16 |
| OmniQuant | 5.96 | 7.43 | 5.28 | 6.84 | 4.38 | 6.22 | 5.87 | 7.48 | 5.14 | 6.74 | 6.97 | 10.08 |  |
| I-LLM | 5.84 | 7.32 | 5.23 | 6.79 | 4.32 | 6.25 | 5.68 | 7.27 | 5.10 | 6.74 | 6.61 | 9.77 |  |
| W4A4 | SmoothQuant | 22.25 | 32.32 | 40.05 | 47.18 | 192.40 | 122.38 | 83.12 | 77.27 | 35.88 | 43.19 | 418.88 | 312.86 |
| OmniQuant | 11.26 | 14.51 | 10.87 | 13.78 | 10.33 | 12.49 | 14.26 | 18.02 | 12.30 | 14.55 | 437.88 | 315.69 |  |
| AffineQuant | 10.28 | 13.64 | 10.32 | 13.44 | 9.35 | 11.58 | 12.69 | 15.76 | 11.45 | 13.97 | - | - |  |
| I-LLM | 9.10 | 12.33 | 7.99 | 10.96 | 7.24 | 9.85 | 10.44 | 12.92 | 9.76 | 12.57 | 21.19 | 30.9 |  |

Table: Table 2: Quantitative weight-activation quantization PPL($\downarrow$) results on OPT Family of I-LLM.

As shown in Table 1 and Table 2, we report the perplexity performance of I-LLM on the C4 and WikiText2 datasets. As depicted in these tables, I-LLM consistently surpasses previous methods across a diverse array of LLM families (LLaMA-1, LLaMA-2, LLaMA-3, OPT) and varying levels of precision.
Particularly noteworthy is its performance under the W4A4 setting, where our proposed method achieves perplexity values that are consistently 10% to 30% lower than those attained by state-of-the-art methods.

**Table 3: The performance of various methods for 4-bit and 6-bit quantization on the LLaMA family models across six zero-shot datasets.**
| LLaMA / Acc($\uparrow$) | #Bits | Method | PIQA($\uparrow$) | ARC-e($\uparrow$) | ARC-c($\uparrow$) | BoolQ($\uparrow$) | HellaSwag($\uparrow$) | Winogrande($\uparrow$) | Avg.($\uparrow$) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-7B | FP16 | - | 77.47 | 52.48 | 41.46 | 73.08 | 73.00 | 67.07 | 64.09 |
| W6A6 | SmoothQuant | 76.75 | 51.64 | 39.88 | 71.75 | 71.67 | 65.03 | 62.81 |  |
| W6A6 | OmniQuant | 77.09 | 51.89 | 40.87 | 72.53 | 71.61 | 65.03 | 63.17 |  |
| W6A6 | I-LLM | 76.99 | 52.66 | 40.78 | 72.94 | 71.31 | 65.67 | 63.39 |  |
| W4A4 | SmoothQuant | 49.80 | 30.40 | 25.80 | 49.10 | 27.40 | 48.00 | 38.41 |  |
| W4A4 | LLM-QAT | 51.50 | 27.90 | 23.90 | 61.30 | 31.10 | 51.90 | 41.27 |  |
| W4A4 | LLM-QAT+SQ | 55.90 | 35.50 | 26.40 | 62.40 | 47.80 | 50.60 | 46.43 |  |
| W4A4 | OS+ | 62.73 | 39.98 | 30.29 | 60.21 | 44.39 | 52.96 | 48.43 |  |
| W4A4 | OmniQuant | 66.15 | 45.20 | 31.14 | 63.51 | 56.44 | 53.43 | 52.65 |  |
| W4A4 | AffineQuant | 69.37 | 42.55 | 31.91 | 63.73 | 57.65 | 55.33 | 53.42 |  |
| W4A4 | I-LLM | 67.25 | 45.58 | 32.59 | 63.88 | 58.89 | 57.06 | 54.21 |  |
| LLaMA-13B | FP16 | - | 79.10 | 59.89 | 44.45 | 68.01 | 76.21 | 70.31 | 66.33 |
| W6A6 | SmoothQuant | 77.91 | 56.60 | 42.40 | 64.95 | 75.36 | 69.36 | 64.43 |  |
| W6A6 | OmniQuant | 78.40 | 57.28 | 42.91 | 67.00 | 75.82 | 68.27 | 64.95 |  |
| W6A6 | I-LLM | 77.48 | 56.94 | 44.03 | 64.92 | 75.24 | 69.14 | 64.63 |  |
| W4A4 | SmoothQuant | 61.04 | 39.18 | 30.80 | 61.80 | 52.29 | 51.06 | 49.36 |  |
| W4A4 | OS+ | 63.00 | 40.32 | 30.38 | 60.34 | 53.61 | 51.54 | 49.86 |  |
| W4A4 | OmniQuant | 69.69 | 47.39 | 33.10 | 62.84 | 58.96 | 55.80 | 54.37 |  |
| W4A4 | AffineQuant | 66.32 | 43.90 | 29.61 | 64.10 | 56.88 | 54.70 | 52.58 |  |
| W4A4 | I-LLM | 67.95 | 48.15 | 34.47 | 62.29 | 63.13 | 59.98 | 56.00 |  |
| LLaMA-30B | FP16 | - | 80.08 | 58.92 | 45.47 | 68.44 | 79.21 | 72.53 | 67.44 |
| W6A6 | SmoothQuant | 77.14 | 57.61 | 42.91 | 65.56 | 78.07 | 69.92 | 65.20 |  |
| W6A6 | OmniQuant | 78.40 | 57.28 | 42.91 | 67.00 | 75.82 | 68.27 | 64.95 |  |
| W6A6 | I-LLM | 79.43 | 58.88 | 45.14 | 73.36 | 78.51 | 72.61 | 67.99 |  |
| W4A4 | SmoothQuant | 58.65 | 35.53 | 27.73 | 60.42 | 35.56 | 48.06 | 44.83 |  |
| W4A4 | OS+ | 67.63 | 46.17 | 34.40 | 60.70 | 54.32 | 52.64 | 52.62 |  |
| W4A4 | OmniQuant | 71.21 | 49.45 | 34.47 | 65.33 | 64.65 | 59.19 | 56.63 |  |
| W4A4 | AffineQuant | 66.32 | 43.90 | 29.61 | 64.10 | 56.88 | 54.70 | 52.58 |  |
| W4A4 | I-LLM | 71.38 | 51.81 | 37.12 | 65.69 | 67.79 | 61.40 | 59.20 |  |

Table 3 presents our performance on six zero-shot tasks, employing both W4A4 and W6A6 settings. Particularly striking is the performance of the LLaMA-30b model, configured with full quantization at W6A6 precision, which achieves an average accuracy on these tasks surpassing even that of the floating-point model. This achievement underscores the potential of fully quantized integer-only methods in maintaining the generalization capabilities of LLMs to a considerable degree.

### 4.2 Ablation Study

Contribution of Fully-Smooth Block Reconstruction
In Table 5, we meticulously assess the impact of various PTQ methods on model accuracy. To maintain impartiality, this experiment abstains from utilizing integer-only operators; instead, all quantization procedures are substituted with pseudo-quantization. The nodes necessitating quantization align with those delineated in Fig 4. Notably, conventional non-integer-only methodologies often overlook the influence of activation in non-linear layers, leading to compromised model accuracy during integer inference. However, with the integration of FSBR, these activations are thoughtfully considered during PTQ, effectively reinstating the model’s accuracy under full quantization.

**Table 4: Impact of different PTQ methods and integer-only operators on LLaMA-7B PPL($\downarrow$) across WikiText2 and C4 datasets.**
| LLaMA-7B | W4A4 | W6A6 |  |  |
| --- | --- | --- | --- | --- |
| Method | WikiText2 | C4 | WikiText2 | C4 |
| SmoothQuant | 256.58 | 218.47 | 6.09 | 7.6 |
| OmniquantQuant | 122.18 | 183.2 | 5.99 | 7.57 |
| FSBR | 9.44 | 12.72 | 5.83 | 7.02 |
| +DI-CLippedSoftamx | 9.44 | 12.72 | 5.83 | 7.02 |
| +DI-Swiglu | 9.12 | 12.38 | 5.83 | 7.04 |
| +DI-Norm | 9.52 | 12.63 | 5.85 | 7.35 |

Impact of Integer-Only Operators.
As shown in Table 5, we intricately outline the precise impact of each integer-only operator on the holistic accuracy of the model. Additionally, we ablate the influence of the clipping coefficient $c$ in Eq 10 within DI-ClippedSoftmax on model accuracy, as depicted in Table 5. Notably, owing to residual connections, the quantization of DI-Norm engenders a discernible decrement in performance, a phenomenon meticulously anticipated in our comprehensive analysis. In Table 5, we elaborate on the influence of each integer-only operator on the overall model accuracy. Specifically, Table 5 illustrates the impact of the clipping coefficient $c$ in the Eq 10 from DI-ClippedSoftmax on model accuracy. It’s worth noting that the quantization of DI-Norm results in a performance decline, as expected due to residual connections, a phenomenon that aligns with our expectations.

## 5 Conclusion

In this paper, we present I-LLM, a fully-quantized integer-only PTQ framework for LLMs. We address the challenge of fluctuating activations in both linear and non-linear operations by proposing Fully-Smooth Block-Reconstruction (FSBR) to harmonize inter-channel variations and Dynamic Integer-only MatMul (DI-MatMul) to handle inter-token variations. Additionally, we design DI-ClippedSoftmax, DI-Exp, and DI-Norm as lightweight integer-only operators to replace complex math caculations. Experiments demonstrate that I-LLM outperforms simulated quantization methods and achieves comparable accuracy to the floating-point baseline.
Notably, I-LLM can operate at W4A4 quantization setting with negligible loss of accuracy, bridging the gap between low-bit integer-only quantization and LLMs. This work opens up avenues for the efficient deployment of LLMs on edge devices without floating-point capabilities. Looking ahead, we are committed to deploying I-LLM on specialized hardware and cloud platforms, with the aim of achieving even greater acceleration performance and further optimizing the operational efficiency of LLMs in various computational environments.

## Appendix A Appendix

### A.1 Quantization Preliminaries

Quantization & Dequantization. Quantization typically refers to mapping a floating-point number to a discrete interval with integer number. Here, we only consider uniform quantization. The quantization process can be expressed as follows:

$$ $\displaystyle\bm{X}^{I}=\text{clamp}\left(\left\lfloor\frac{\bm{X}}{s}\right \rceil+zp^{I},0,2^{n^{I}}-1\right)$ (13) $$

$$ $\displaystyle s=\frac{x_{\max}-x_{\min}}{2^{n^{I}}-1}$ (14) $$

$$ $\displaystyle zp^{I}=\left\lfloor\frac{-x_{\min}}{s}\right\rceil$ (15) $\displaystyle X^{\prime}=(X^{I}-zp^{I})\cdot s$ (16) $$

where $\bm{X}$ is the floating-point tensor, $\bm{X^{I}}$ is its quantized counterpart and $\bm{X^{\prime}}$ is the dequantized resuls of $\bm{X^{I}}$. Here, $n^{I}$ represents the number of bits (e.g., 8). $s$ is the quantization step size, determined by $x_{\min}$, $x_{\max}$, and $n^{I}$.clamp represents truncation function. The choice of $s$ greatly affects the accuracy of the quantized model. We can obtain $s$ from the activations of some samples, which is called static quantization. Alternatively, we can derive it from runtime statistics, known as dynamic quantization. Quantization can also be distinguished by its granularity into per-channel quantization and per-token quantization [].

### A.2 Dynamic Interger-only Algorithms

Figure: Algorithm 1 Dynamic Integer-only Exp: DI-Exp

Figure: Algorithm 2 Dynamic Integer-only Softmax : DI-Softmax

Figure: Algorithm 3 Dynamic Integer-only SwiGLU : DI-SwiGLU

Figure: Algorithm 4 Dynamic Integer-only RMSnorm : DI-RMSnorm

### A.3 Fully Results

Figure: Figure 6: The output activation distribution of qkv of LLMs before and after the FSBR.
Refer to caption: extracted/5646459/figures/act_qkv.png

### A.4 Limitation and Discussion

We have shown evidence that our I-LLM can replicate float-point performance with an integer-only module with 8-bit or lower bits. However, due to time constraints, we are currently evaluating the model’s latency on real hardware devices, but have not obtained quantitative results. Another limitation is that we only focused on natural language models, however, it would be interesting to explore how I-LLM performs in computer vision tasks. We leave this for future work.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: I have few questions and comments as below: 

1. In Table 1, the results for SmoothQuant under the W4A4 setting (e.g., 1.8e4 for the OPT family) are unusually high, especially compared to LLaMA models. This discrepancy should be explained.

2. The experimental setup is unclear, especially regarding Table 4, where latency and speedup for traditional W4A4 are reported. What framework was used, and was Tensor Core applied?

3. To better understand the efficiency of integer-only quantization, comparisons with other quantization works like QServe [1] and Atom [2] should be included.

4. In the quantization algorithms, how are the scale factor and zero-point stored? If they are stored as integers, does this significantly impact accuracy? A discussion on this trade-off is needed.


[1].QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
[2].Atom: Low-bit quantization for efficient and accurate llm serving
soundness: 2
strengths: 1. This paper addresses the challenge of integer-only quantization, which is often overlooked by existing work as typical LLM quantization methods usually store intermediate results as floating-point values.

2. The paper introduces innovative integer-only operators, such as DI-Exp, DI-ClippedSoftmax, and DI-Norm, to replace computationally intensive floating-point operations.

3. The experimental section includes a thorough comparison across multiple model types and configurations, as well as an ablation study, demonstrating the framework’s efficiency.
summary: This paper proposes an integer-only post-training quantization (PTQ) framework to accelerate the inference of large language models, called I-LLM. The authors introduce three main techniques: Fully-Smooth Block Reconstruction (FSBR), which reduces inter-channel activation disparities; Dynamic Integer-only MatMul, enabling dynamic quantization and integer-only matrix multiplication; and integer-only non-linear operators such as DI-ClippedSoftmax and DI-Exp, which use bit-shifting for efficiency. Experimental results show that I-LLM achieves accuracy on par with floating-point baselines while delivering significant improvements in computational performance and memory efficiency.
weaknesses: 1. The motivation for this work is not clearly explained, especially regarding why integer-only quantization is necessary. The trade-off between accuracy and inference performance needs more discussion. Additionally, the configuration of different quantization types for weights and activations (e.g., W8A8, W4A8, W4A4) is not discussed.

2. The experimental setup lacks clarity, and more results on inference performance are needed. See detailed comments 1-3.

3. The innovation of the Fully-Smooth Block Reconstruction method is limited, as it closely resembles SmoothQuant. Additionally, the overhead of dynamic quantization should be demonstrated in the experimental results.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - What framework and baseline setup were used for W4A4 quantization, and can more detail be provided about the experimental environment?
- The description of FSBR is unclear. Could the authors provide a more explicit breakdown of its application across different computing units beyond MatMul?
- The authors also emphasize the outliers across the token (which sounds new), but Fig. 3 shows that the token-wise distribution looks much flatter than the channel-wise distribution, which is well known.
soundness: 2
strengths: - The integer-only quantization of non-linear operations offers potential efficiency improvements for hardware without FP support.
- Demonstrates notable performance in W4A4 quantization settings, showing significant reductions in latency and memory usage on specific LLMs.
summary: The paper proposes I-LLM, an integer-only post-training quantization (PTQ) framework for large language models (LLMs), aiming to improve inference efficiency on hardware lacking floating-point support. The key contributions include the Fully-Smooth Block-Reconstruction (FSBR) to smooth inter-channel variations in activation and dynamic quantization methods (Dynamic Integer-only MatMul and others). Experiments on several LLMs, such as LLaMA and OPT, demonstrate improved modest speed and memory usage under W4A4 settings, with minimal accuracy loss compared to floating-point (FP) models. Despite these claims, the methodology largely extends known techniques, showing limited novelty, and lacks sufficient evaluation against state-of-the-art (SOTA) approaches on challenging quantization settings.
weaknesses: - The proposed techniques, such as FSBR, heavily build on prior works like OmniQuant for quantization-parameter tuning and I-VIT for dynamic quantization, limiting originality.
- The absence of comprehensive SOTA comparison limits the rigor of performance claims, particularly missing comparisons with rotation-based methods (e.g., SpinQuant (Liu et al., 2024)) and LUT-based approximations (e.g., NN-LUT(Yu et al., 2022)).
- FSBR and DI-MatMul introduce computational overhead with on-the-fly operations like 8-bit scalar multiplication/division, yet no detailed ablation study quantifies the latency impact per Transformer component.
- The evaluation datasets and tasks are limited, and broader testing across more diverse and challenging benchmarks is required to substantiate the generalization of results.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Detail more in depth the steps needed in training.
soundness: 4
strengths: First method that covers all the steps of integer inference.
Presents implementation results, showing significant speedup even in non-adhoc designed hardware
end-to-end results in inference are presented, not only sub-steps
summary: The paper present an end-to-end quantization method that is applied to all the inference sub-steps in LLMs, including non-linear operations and all attention calculations. The technique presented leverages block and dynamic quantization concepts. Specific integer approximation for non-linearities are presented to avoid moving o fp when computing them, End-to-end results are presented, and an implementation of the method on GPU is also presented, showing significant speedups.
weaknesses: It seems the method is not post-training (the smoothing coefficients are learned), hence it is not applicable for the largest LLMs for which training is not really easy to repeat. 
Notation is hard to follow and could be simplified to get better intuition on the methods applied. 
Approximation proposed for non-linear functions are not really well justified
Similar ideas are present in I-BERT, I-ViT, BRECQ. Thus, the contribution of this paper seems to be incremental (see BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction)

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: NA
soundness: 3
strengths: The authors provide valuable insights into the quantization of large language models (LLMs), demonstrating how such models can maintain high accuracy despite the reduced precision.
summary: This study introduces a novel quantization approach for large language models (LLMs), featuring custom-designed modules: DI-MatMul, DI-ClippedSoftmax, DI-Exp, and DI-Normalization. These modules collectively outperform state-of-the-art (SOTA) methods, offering enhanced performance in LLM applications.
weaknesses: Consider combining quantization with pruning to achieve enhanced model efficiency and reduced computational overhead.
Consider a hybrid quantization approach, where different layers utilize varied precision levels, such as W4A4 for certain layers and W8A8 for others, to balance efficiency and performance.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: Please refer to Weaknesses
soundness: 3
strengths: 1. The proposed framework enables integer-only quantization for LLMs: by completely avoiding floating-point operations, I-LLM takes an important step toward making LLMs deployable on edge devices, achieving faster and more efficient inference on hardware without the need for floating-point support.

2. The authors conducted extensive experiments across various LLM architectures, model sizes, and datasets, with the table data from the manuscript demonstrating overall outstanding performance.

3. This paper introduces techniques like FSBR and DI-MatMul that optimize LLM quantization accuracy by addressing variations across channels and tokens. These techniques help maintain high precision during the inference process.
summary: This paper introduces I-LLM, a novel integer-based post-training quantization (PTQ) framework for large language models (LLMs). Traditional PTQ techniques involve a mix of integer and floating-point operations, which limits deployment on edge devices that lack floating-point capabilities. The authors propose three techniques: Full-Smooth Block Reconstruction (FSBR) to smooth activation variations across channels, Dynamic Integer Matrix Multiplication (DI-MatMul) to manage variations between tokens, and dynamic integer implementations for nonlinear operations. Experiments show that I-LLM achieves comparable post-compression performance with existing methods while significantly reducing computational overhead.
weaknesses: 1. Section 3.2 of the paper proposes 'training a smoothing coefficient for all activations and weights to aid in restoring the model’s quantization accuracy.' However, the training and solving process of this coefficient is not discussed, which could lead to confusion and misunderstandings. If the reason for not detailing this part is that the method aligns with SmoothQuant or OmniQuant, it should be explicitly cited and clearly explained.

2. Equations (1) and (2) extend SmoothQuant by applying smoothing to 'NonLinear Act-Smooth.' However, the motivation of SmoothQuant is to reduce the difficulty of quantization by lowering the smoothness of activations through a smoothing coefficient. In Equations (1) and (2), the smoothing coefficient is counterbalanced between the Gate-layer and Up-layer using '*s' and '/s', respectively. The paper does not discuss the rationale behind this operation or why 'W' scale is 'times' while the 'V' is 'division'. 

3. The definition of $\sigma$ in Equation (2) is confusing. Based on Equation (2) and line 262, it follows that $\sigma'(x1) = \sigma(x1 / s)$, and $\sigma'(x1') = \sigma(x1' / s)$. So we can get $\sigma(x1' / s) = \sigma(x1 / s)$, i do believe this is not a right equation or hope the author can make a clarification on this.

4. The authors state in line 270 that 'SmoothQuant and OmniQuant are subsets of FSBR.' However, based on the description in this section, it appears that FSBR actually adopts the techniques of SmoothQuant and OmniQuant and extends them within 'NonLinear Act-Smooth.' Referring to them as subsets is inaccurate and could lead to misunderstandings.

5. I have carefully reviewed and evaluated the anonymous code repository provided by the authors. Regarding the DI-MatMul computation process mentioned in Section 3.3 (specifically in *quantize/quant_modules/QuantMatMul, quantize/quantizer), its implementation and definition of 'dynamic' is consistent with the OmniQuant codebase. If there are any omissions or misunderstandings, I would appreciate further clarification from the authors.

6. Since the quantizer used in the paper results in the same post-quantization weight bit-width and additional quantizer parameters (scaling factor, zero factor) as methods like OmniQuant, the specific factors behind the reduction in **Weight Memory** listed in Table 4 for I-LLM are not clearly discussed. It would be helpful if the authors could clarify which specific parameter compression or operation contributes to this memory efficiency advantage.

7. In table 5, 'OmniquantQuant'->'Omniquant', 'CLippedSoftamx'->'CLippedSoftmax'

Meta-review: The paper proposes I-LLM, an integer-only PTQ framework for LLMs. The goal is to eliminate floating-point operations, enabling efficient inference on hardware without floating-point support. This is particularly useful for targets like mobile devices, where resources are scarce. The main contributions of this work are:

(a) A method to smooth activation variations across channels and layers, enabling efficient integer-only quantization. This helps to reduce the negative impact on accuracy.

(b) A matrix multiplication approach that adjusts quantization dynamically to mitigate inter-token variations in activation distributions.

(c) Custom implementations of non-linear operations, which use integer arithmetic to approximate these operations.

(d) Experiments across multiple LLM architectures (e.g., LLaMA, OPT) and configurations (e.g., W4A4, W6A6). Results indicate that I-LLM achieves comparable accuracy to floating-point baselines while offering significant improvements in computational speed and memory efficiency.

## Strengths

(a) Presenting an integer-only inference framework for LLMs, including non-linear operations.

(b) The introduction of integer-only operations (DI-MatMul, DI-Softmax, and other non-linear functions).

(c) Experimental results show that I-LLM achieves reductions in latency and memory usage (e.g., 2.49$\times$ speedup for LLaMA-7B) under W4A4 configurations, with minimal accuracy loss.


## Weaknesses

(a) The proposed techniques, particularly FSBR, build heavily on prior methods like SmoothQuant and OmniQuant. The contributions largely involve extending existing techniques, with narrow contributions.

(b) While the paper includes comparisons with OmniQuant, it omits evaluations against other state-of-the-art methods like SpinQuant and LUT-based approaches (e.g., NN-LUT) -- also mentioned by reviewers.

(c) The rationale for smoothing operations and the computational overhead introduced by DI-MatMul, remain unclear. Reviewers noted inconsistencies in the mathematical explanations and an insufficient breakdown of latency impacts.

(d) Some concerns about the reproducibility of the codebase (integer-only operators were not released).

## Summary

After reading the reviews and the rebuttal, I recommend **Reject** for this paper. I believe this a good direction to explore, however, the paper in its current form has very narrow contributions, performs limited comparison with SOTA methods, and lacks clear explanations of key aspects of the proposed method.

Decision: Reject