Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
Authors: Janghwan Lee
Minsoo Kim
Seungcheol Baek
Seok Joong Hwang
Wonyong Sung
Jungwook Choi
Venue: EMNLP 2023 Main (2023)
Keywords: Post-training quantization
LLM
Transformers
Numerical format

Abstract: Large Language Models (LLMs) are proficient in natural language processing tasks, but their deployment is often restricted by extensive parameter sizes and computational demands. This paper focuses on post-training quantization (PTQ) in LLMs, specifically 4-bit weight and 8-bit activation (W4A8) quantization, to enhance computational efficiency—a topic less explored compared to weight-only quantization. We present two innovative techniques: activation-quantization-aware scaling (AQAS) and sequence-length-aware calibration (SLAC) to enhance PTQ by considering the combined effects on weights and activations and aligning calibration sequence lengths to target tasks. Moreover, we introduce dINT, a hybrid data format combining integer and denormal representations, to address the underflow issue in W4A8 quantization, where small values are rounded to zero. Through rigorous evaluations of LLMs, including OPT and LLaMA, we demonstrate that our techniques significantly boost task accuracies to levels comparable with full-precision models. By developing arithmetic units compatible with dINT, we further confirm that our methods yield a 2$\times$ hardware efficiency improvement compared to 8-bit integer MAC unit.

Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
Janghwan Lee'; Minsoo Kim'; Seungcheol Back", Seok Joong Hwang', Wonyong Sung" and Jungwook Choi
(twanii0288) minsoo253330hanyyae ac. kr, {bsc11235, nzthing)ossppon..com
wysungesnu.com ac.kr, kr, choij@hanyang.ce
Hanyang University, PSAPOON Korea Inc., SSouul National University Republic of Korea
Abstract
rameters, significantly reducing memory require- ments and enhancing GPU performance by alle- viating memory bandwidth bottlenecks (Frantar et al., 2023; Lin et al., 2023; Lee et al., 2023a). Nevertheless, LLMs' computational complexity re- mains concern. For example, GPT-3 (Brown et al., 2020) requires at least 350 GFLOPs of com- putation for single token, but PTQ methods of- revert compressed weights higher precisions like 16-bit floating- poittn (FP16) for computation, which inefficient given the resource demands of multipy--ccumullte (MAC) operations. With com- puting platforms evolving through high-bandwidth memory (Gurumurthi et al., 2021) and processing- in-memory (Kim et al., 2021; He et al., 2020) to resolve the memory bandwidth bottleneck, address- ing LLMs computational needs becomes more imperative.
Large Language Models (LLMs) are proficient in natural language processing tasks, but their deployment is often restricted by extensive parameter sizes and computational demands. This paper focuses on post-training quantiza- tion (PTQ) I.MMs, specifically 4-hit weight and 8-bit activation (W4A8) quantization, to enhance computational efficiency-a topic less explored compared to weight-only quantiza- tion. We present two innovative techniques: drtivation-qaaniizatinnnoo
scaling (AQAS) and sequence-lengtthawaar calibration (SLAC) to enhance PTQ by considering the combined effects on weights and activations and align- ing calibration sequence lengths target tasks. Moreover, we introduce dINT, hybrid data format combining integer and denormal repre- sentations, to address the underflow issue in W4A8 quantization, where small values are rounded to zero. Through rigorous evalua- tions of LLMs, including OPT and LLaMA, we demonstrate that our techniques significantly boost task accuracies to levels comparable with full-precision models. By developing arith- metic units compatible with dINT, we further confirm that methods yield a 2x hardware efficiency improvement compared to 8-bit inte- ger MAC unit.
A PTQ strategy that effectively quantizes both weights and activations is thus appealing as it reduces the hardware complexity of MAC units, enhancing computational throughput (Sun et al., 2019; Dettmers et al., 2022; Xiao et al., 2022). PTQ research specific to LLM's computation efficiency is growing, focusing utilizing INT8-INT8 MAC units, common in GPUs (Andersch al., 2022). ILM. (Dettmers et al., 2022), for instance, used INT8 quantization for weights and activations, but directed activation outliers through an FP16 dat- apath, isolating them. SmoothQuant (Xiao et al., 2022) extended this by employing activation chan- nel scaling to target outliers and adjusting corre- sponding weights for balanced quantization. How- ever, these studies do not address challenges faced when weights reduced 4 bits, revealing unexplored
1 Introduction
Large language models (LLMs) have achieved breakthroughs in many natural language processing tasks such as translation, summarization, reason- ing, and conversation, often matching or exceeding human performance (Zhang et al., 2022; Touvron et al., 2023; Chowdhery et al., 2022; Brown et al., 2020; OpenAL, 2023). However, the extensive pa- rameters of LLMs present deployment challenges due to the high memory bandwidth needed for high throughput inference. Post-training quantization (PTQ) addresses this by "compressing" weight pa-
for combined effects on weight and activation quantization.
This paper delves into the challenges of post- training quantization (PTQ) for both weights and activations in large language models (LLMs). We pinpoint two primary hurdles in achieving efficient
"qqul contribution corresponding author

---

4-bit weight and 8-bit activation (W4A8) quantiza- tion. First, LLMs like OPT (Zhang et al., 2022) and LLaMA (Touvron et al., 2023) have distinct weight and activation range characteristics, making exist- ing PTQ methods unsuitable for universal use. For example, AWQ's s (Lin et al., 2023) uctivation-awan aware scaling makes activations prone to quantization er- rors, while OPTQ's (Frantar et al., 2023) weight calibration struggles with varying activation ranges. We propose two novel solutions for this first hurdle: activation-quann quantization-ann aware scaling (AQAS) and wawueence-lenth-wwarrr calibration (SLAC). AQAS optimizes quantization scales by jointly consid- ering weights and activations, yielding balanced quantization. SLAC aligns the sequence length of the application task with that of the PTQ calibra- tion dataset, mitigating the impact of variations in activation diversity, which significantly affects the PTQ calibration process.
2 Background
2.1 Weight-only PTQ for LLMs
Various weight-only PTQ techniques have emerged to alleviate memory-bandwidth constraints in LLM inference by compressing weights to 4 bits while maintaining accuracy (Park al., 2023; Kwon et al., 2022; Frantar al., 2023; Lin al., 2023; Lee et al., 2023a). For example, OPTQ (Fran- tar et al., 2023) reduces output distortion from column-wise weight quantization by sequentially updating unquantized weights using activation Hes- sians. AWQ (Lin et al., 2023) scales weights according to activation magnitudes for improved quantization, while OWQ (Lee et al., 2023a) and SPQR (Dettmers et al., 2023) isolate sensitive weights, retaining them at higher precision. How- ever, these approaches entail high-precision com- putations and complex arithmetic units. We demon- strate that these weight compression methods are sub-optimal for activation quantization in common LLMs, often exacerbating challenges by ignoring activation dynamics. Consequently, we introduce advanced techniques specifically designed to ad- dress these intricacies, enhancing weight quantiza- tion accuracy when the activation is also quantized.
Second, we observe that underflow, where small- magnitude values round to zero, severely impacts W4A8 quantization LLMs because the quan- tization error associated with values rounding to zero constitutes a significant portion of the out- put error. While underflow is a well known issue in reduced- -precssio formats for deep neural net- works (DNNs) (Sun et al., 2019, 2020; Chmiel al., 2022: Jin al., 2022). previous PTQ research in LLMs mainly focuses on outliers, neglecting underflow. We discover that standard INT4 repre- sentation discards crucial small maglitmg weights when multiplied with activations. As existing data formats like integer, floating-point or logarithmic formats are inadequate for this underflow issue, we introduce dINT, new integer format with de- normal representation. dINT merges the uniform coverage of integers with the denormal of floating- points, effectively mitigating underflow and im- proving accuracy. We also propose a MAC unit supporting dINT to ensure hardware efficiency.
2.2 Weight and Activation PTQ for LLMs
Quantizing both weights and activations enables the use of lower-precision MAC units, signif- icantly saving logic area and power consump- tion (Horowitz, 2014). As such, many studics aim to reduce DNN's computational burden (Sun et al., 2019; Lee et al., 2023b), especially in LLMs (Dettmers et al., 2022; Xiao et al., 2022; Liu et al., 2023; Bondarenko al.. 2021). For instance, LLM.Int8 (Dettmers et al., 2022) and SmoothQuant (Xiao et al., 2022) employ GPU- supported INTS-INT8 MAC operations for effi- ciency, with ILM. processing outliers sepa- rately and SmoothQuant adjusting activations and weights. Additionally, (Liu et al., 2023; Bon- darenko et al., 2021) employ quantization-awan aware fine-tuning for further reductions to W4A8 or W4A4, but face noticeable accuracy losses despite expensive fine-tuning. This paper proposes novel solutions that address the accuracy drop in com- bined weight and activation quantization with bit- precision down to W4A8, achieving superior re- sults compared to prior works without fine-tuning.
We evaluate AQAS, SLAC, and dINT on OPT and LLaMA, focusing on language modeling, zero- shot reasoning, and 5-shot in co-contett learning. The results show that integrating these methods for W4A8 PTQ significantly improves task accuracies for both OPT and LLaMA across diverse set of benchmarks (Wikitext, Common Sense Ques- tion Answering (CSQA), and Massive Multitask Language Understanding (MMLU)) and the model sizes ranging from 125M to 65B parameters.

---

X
60-
20
X Norm
Martuul
Min Range
Q
Q(Wwaa)
Fused LN
yirc
10
X
Norm
X-
KotMol
- - r) @(W)e))
0-
Sesien
Seggan 2048
0-
Secken-256
Seqjen-121
Sa2len-312
-20
-10
204t
LLaMA
Wyc1
30
20
30
(a) Compulation pattern OPT (top) LLaMA (below)
Layer Number
Number
(c) Activation Min-Max range per layer sequence length
Activation Min
Weight Min Max Max-Max Range
(Left: OPT, Right: LLaMA)
MrrMMx Range
40-
60
097.6.70
UaMA-70
0.5
Seglen-32
Seqlen-2048
Seqlen-32
Seqien 204g
0.0
Max
Nax
10
0.5
-20
-1.0
Layer Number
0
10
Layer Number
20
30
channel number
channel number
channel number
channel number
(b) Min-Max range of input activation and weight per layer
(d) Max values of per-ehannel input activation (Left: OPT, Right: LLaMA)
Figure (a) Illustaaioon fused-layernorm (fused-LN) OPT (top) and layernorm (LN) in LLaMA (bottom) computation patterns within Transformer layer. Note that two computation patterns yield ths same output if computed in full-precision, but they deviate when activation and weight are quantized. (b) Min-Max range of input activations (left) and weight (right) as operands of matrix multiplication. (c) Min-Max range of input activation varying sequence length from 128 to 2048 (Orange: OPT-6.7B, Blue: LLaMA-7B). (d) Max values of per-channel input activation for OPT-6.7B (left) and LLaMA-7B (right) for different input sequence lengths (32 and 2048).
2.3 Underflow for Reduced-Precisine LLMs
to varied weight distributions across models, OPT- 66B experiences substantial perplexity increase in the wikitext benchmark with INT4 weights, soar- ing from 9.34 110 (Frantar al., 2023), whereas GLMM-30D shows no performance degradation on the MMLU benchmark when INT4 weights are applicd (Zeng et al., 2023). We posit that these discrepancies arise from variances in pre-training configurations such as datasets, learning rates, layer structures, and self-attention directionality, as well as options designed for efficient inference, such as operation fusion techniques like layernorm fu- sion. Significantly, existing PTQ research has over- looked these unique traits intrinsic each model that are pivotal for the combined optimization of activation and weight quantization. Therefore, we delve into the weight and activation distributions of widely-used OPT and LLaMA models during quan- tization to understand PTQ limitations and develop novel methods to address them.
Underflow, the numerical error from small values rounding zero due limited bit-precision, has been actively studied as a critical issue in reduced- precision DNN training. For instance, (Sun al., 2019) counters underflow in 8-bit floating-pon point by adjusting the exponent bias, (Sun et al., 2020) uti- lizes radix-4 format to represent wider magnitude ranges in 4-bit floating-por et al., 2022) uses stochastic underflow to address biased quantization in FP4 gradients. In fixed-point representation, (Jin et al., 2022) explores optimal formats by analyzing underflow and overflow trade- offs based on fractional length. Contrary to these studies focusing on the training phase, our paper in- vestigates underflow's 's impact on PTQ of LLMs for the first time and introduces an enhanced integer format to combat it.
(FP4), and (Chmiel
3 Improving PTQ for Weight and Activation Quantization
3.1 Model Analysis: OPT vs. LLaMA
We aim to advance L.M quantization beyond the realms of 4-bit weight-only PTQ or W8A8 PTQ by investigating the combined effects of weight and ac- tivation quantization. When quantizing both weight and activation, it is important to note that LLMs dis- play distinct weight and activation characteristics. For example, OPT has been found have 0.1% ac- tivation outliers by (Detterrs et al., 2022), whereas GLM-130B (Zeng et al., 2023) reported 30% of outliers in its model. In the context of weight, due
To understand the adverse effects of quantization on restricting dynamic range, we examine the minimum and maximum values (Min-Max range) across the layers of LLMs. Fig. I(a) illustrates the computation patterns within layer of LLMs and Fig. (bb) displays Min-Max range of activa- tions (left) and weights (right) as operands of ma- trix multiplication for each FC layer in OPT and LLaMA. Notably, there are contrasting trends

---

Min-Max ranges; OPT has broad activation range but a narrow weight range, while LLaMA exhibits the opposite. This distinction stems from the way these LLMs process activations at layernorm. As depicted in Fig. ((a), in OPT, the layernorm pa- rameters are fused to the subsequent FC layer's weights (Fig. ((a) top), allowing only normalized activation enter the FC layer. Conversely, lay- ernorm is not fused in LLaMA (Fig. ((a below), resulting in scaled activation as imput to FC layers. Although layernorm fusion preserves functionality in full-precision computation, this presence or ab- sence of layernorm fusion activation processing contributes to significantly distinct behaviors under quantization, will be discussed in the following sections.
Raseline
SmoothQuant
AWO
AQAS
60
AbsimaxXX
40-
20-
Key projection layers
Key projection lsyers
(a) Inpul activation
(b) Weight
Absolute max value of (a) input activation and (b) weight after scaling by each method (OPT-6.7H 7B). We observed that these trends were significantly pro- nounced in OPT models due to large outliers. (See Fig. 6 for the same plot for LLaMA.)
Figure 2:
and quantization-sensiiive range for weight. On the other hand, AWQ (Lin et al., 2023) scales weights for 4-bit quantization but significantly increases activation diversity, making activation quantization problematic. In other words, the existing scaling- based PTQ techniques such as SQ and AWQ cannot resolve the issue of conflicting trends in activation and weight outliers. To address this, we introduce activation-quant tir---annizzaionavvaaa scaling (AQAS), a hybrid of SQ and AWQ. AQAS aims to find scal- ing values that minimize the output error caused by quantized weights and activations. We mean squared error (MSE) loss as the objective function, aligning with previous studies on layer- wise opti- mization (Nagel et al., 2020; Frantar et al., 2022). Our objective function is as follows:
Another insightful finding from our model anal- ysis is the variation in activation diversity based on sequence lengths. Fig. I(c) displays the Min-Max range sequence length varies from 128 to 2048 (Orange: OPT-6.7B, Blue: ILL.MA-BB.. Notably, OPT's activation range remains stable across se- quence lengths, while LLaMA's activation range expands, suggesting challenges in range calibration for quantization. Fig. (d) contrasts maximum val- ues per channel for OPT and LLaMA at varying sequence lengths. OPT displays consistent out- liers at the same channels, dominating its activa- tion dynamic ranges. In contrast, LLaMA's outliers increase in magnitude and shift across channels, indicating varied activation dynamic ranges. This distinction in activation diversity is significant for quantization. While PTQ generally presumes con- sistent dynamic ranges for calibrating quantization ranges "offline", these findings emphasize the ne- cessity of considering distinct activation dynamic range and incorporating sequence length into cali- bration. The following sections discuss methods to optimize weight and activation quantization, build- ing on these model-specifie insights.
argmin ||Q(W diag(s)) -xx) -X- WX|
(1) We define the weight W E RM scale factor RCC. and activation X E RCXT where M rep- resents the output feature dimension, C represents the input feature dimension, and T denotes the number of tokens. Fig. demonstrates that AQAS considers activation quantization's impact to adjust activation magnitudes, easing activation quantiza- tion. Additionally, as compared to SQ, AQAS ad- justs weight magnitudes more moderately, making 4-bit weight quantization feasible.
3.2 Acdivatinn-Quaatizaaionn-- Scaling
The distinct properties of outliers in weights and ac- tivations illustrated in Fig. 1(b) pose challenges of applying prior scaling techniques. Fig. 2 illustrates the absolute maximum of (a) input activations and (b) weights at the "Key" layers (for self-attention) in OPT-6.7B when different scaling methods are applied. Specifically, SmoothQuant (Xiao al., 2022) (SQ) scales activation for 8-bit quantization, but descales weights, resulting in a more diverse
3.3 Sequence-Lengtt-AAaar Calibration
As shown in Fig. 1(c), variation in activation di- versity depending the sequence length affects the quantization performance. Specifically, weight- update-based quantization like OPTQ (Frantar et al., 2023) struggles with models like LLAMA that have increasing activation diversity during cal- ibration. To delve deeper into this phenomenon,

---

OPT-6.78
Soquence Length
64
128
512
2048
std
Weight Update Rabio
Least:
Max-126
FP (LLaMA-7B)
70.92
INT4
69.37
2
Least 2048
Leest-1024
Max-1024
Max-
INT4 OPTQ
68. 14
69.89
65.96
65.19
2.54
INT4 AQAS+OPTQ
69.99
70.48
68.92
70.20
0.68
Table 1: W4A8 Quantization zero-shot evaluation of CommonSenseQA (average score of PIQA, Winogrande and Arc_ eas_e default calibration sequence length is 2048)
0
10
15
20
25
30
LLaMA-7B
Weight Update Ratio
4
Least-12
Max-128
Least-1024
Mox-1024
2
Leasz 2048
Max-2048
we analyze the approach adopted by OPTQ, which employs weight adjustments in response to quan- tization error using activation Hessian, formulated follows (Frantar et al., 2023):
0
10
15
20
25
30
LLaMA-78 w/ AQAS
Weight Update Ratio
4
128
Least-512
Max-512
Wg - quant(w.)
Least-1024
Max-1024
8F
H )m
(2)
2
49
0
15
30
BºE H, OW? 2XX",
Layer Number
(3)
(a)
Input Sequence Length- 128
where X denotes the layer input activation, W is weights of linear layer, wy is weight element to quantize, and denotes optimal weight update recovering quantization error. We examine the represent- ing the second derivative of quantization error (E), to assess changes in weights due to OPTQ. Fig. 3(a) shows the weight update ratio for OPT and LLaMA with varying calibration sequence lengths. OPT remains relatively consistent, while LLaMA dis- plays varying weight update ratios for varying se- quence length, suggesting activation diversity af- fects OPTQ's weight updates.
FP
calib-128
callb 2048
Min Range
-6
-8
weight update ratio, (H.²)..q/ [HF')eg (HHF')))H)))
-10
0
5
10
15
20
25
30
Layer Number
(b)
Figure 3: (a) Comparison of weight update ratio in Eq. 2 in OPT-6.7B, LLaMA-7B, and LLaMA-7B with AQAS scaling. (b) Minimum input activation range for the query layer three models: W4A8 (calibrated with 128 and 2048 sequence lengths) and full- pell-precision (FP), all evaluated under an input sequence length 128.
This sensitivity of OPTQ updates prompts us to further explore its implications for performance. We evaluate the zero-shot performance of OPTQ for W4A8 quantization by varying the calibra- tion sequence length on PIQA, Winogrande, and Arc_ Arc_easy tasks from CSQA (Bisk et al., 2019; Sak- aguchi et al., 2019; Clark et al., 2018), which have sequence lengths ranging from tens to hundreds (note that the type of calibration dataset was kept consistent). Table reveals that when the cali- bration sequence length (e.g., 512 or 2048) sig- nificantly deviates from task's sequence lengths, OPTQ's performance suffers (up 4% degrada- tion), even falling below basic nearest-rounding quantization. However, when the sequence lengths are aligned (c.g., 64 or 128), OPTQ performs ex- ceptionally well.
The large standard deviation accuracies for matching versus non-matching sequence lengths suggests that LLaMA's activation diversity substan- tially impacts OPTQ's 's accuracy. To mitigate this, propose the sequence-length-avaal calibration (SLAC) method. This approach involves determin- ing the expected sequence length during the target task's inference phase and aligning the sequence length of the calibration dataset accordingly. Such a taec-fpe PTQ calibration process enhances the robustness and accuracy of the model's infer- ence. The efficacy of SLAC, particularly in the CSQA benchmark. is substantiated by experiments detailed in Sec. 5.3.
The effectiveness of the SLAC method is evi-

---

dent when comparing the dynamic range of quan- tized models with their full-pree full-precision counterparts. Fig. 3 (b) demonstrates that using calibration data aligned with the input sequence length (calib-122) results in dynamic range more consistent with that of the full-precision model (FP), unlike mod- els calibrated with mismatched sequence lengths (calib-2048).
follows:
[[(WX (W + A. A,)
(4)
-[((AX))) = +E,(AXXA,X).
Fig. 4(a) exemplifies the underflow issues, illustrat- ing the distinct impacts of quantization errors on final model accuracy, measured perplexity. The figure highlights that setting small values near zero to exactly zero, while leaving other values unquan- tized, impairs performance. In contrast, quantizing larger values and precisely representing those near zero significantly improve accuracy. Fig. 4(b) pro- vides breakdown of error terms across layers in OPT W4A8, indicating correlation between high total error and substantial underflow error. This un- derlines the necessity for a method that effectively addresses underflow errors.
Integrating SLAC with AQAS effectively en- hances weight and activation quantization. As il- lustrated in Fig. 3(a), AQAS efficiently mitigates the sensitivity to input sequence length regarding weight updates. Morcover, Table shows that the standard deviation related to the calibration dataset's ss length is significantly reduced from 2.54 0.68 through AQAS. Consequently, combining AQAS with OPTQ proves advantageous for infer- ences across diverse sequence lengths, and employ- ing the SLAC method for calibration according to the target dataset's sequence length further bolsters performance.
4.2 Integer with Denormal Representation
Inspired by our observations and the denormal num- bers in floating-poin representation, we introduce a new integer format called integer with denormal representation (dINT). As illustrated by Fig. 4(c), dINT uses two bins around zero to ensure lower magnitudes are effectively represented. In b-bit quantization, two values are reserved for special cases, the quantization range represents integers from to 2 3. These special values in dINT have magnitudes equal to half of the chosen step size, which a power of two to enable computation by simple bit-shift operations. Our experimental find- ings have confirmed that this choice of half-step size consistently delivers the most robust perfor- mance, surpassing other special values designed for bit shifting, as elaborated in Appendix A.5. The quantization and dequantization procedures for dINT are detailed below:
4 Overcoming PTQ Underflow for LLMs
By employing AQAS to address activation quanti- zation errors weight scaling, and utilizing SLAC to align the sequence length of the calibration dataset with that of the target inference, we achieve a substantial improvement in the performance of our W4A8 models. However, encounter persis- tent performance degradation issues. In this sec- tion, we unveil "underflow" issues as a previously overlooked cause of accuracy degradation in PTQ applied to LLMs and propose new numerical for- mat to mitigate this problem.
4.1 Observations
ei, < 3s 4
We identify underflow as a main contributor to per- formance degradation. To dissect the causes of degradation when converting the weights of the scaled model to 4-bit, we split the quantization eтror into two parts: rounding error (A,) and un- derflow error (Aw). The rounding error accounts for the error when the quantized value is non-zero, whereas the underflow error represents the error occurring when the quantized value rounds zero. By considering the total error (A) induced by quan- tization as a combination of A and A,, we can express the expected output quantization error as
Xint
C2, for
-3s $ n < -8
(5)
4
4
clamp ( + 2, 0,
clse
2 for Xint = ej
X, = =s 2 for Ximi = C2
(6)
(Xii - 2) s, else
where p represents the number of uniform steps, calculated p 25 for a given bit number b. The step size obtained by dividing the quanti- zation range by p. and a is the zero-point for asym- metric quantization. Ci and e2 denote the positive

---

INT4
INT4 w/o rounding
INT4 w/o underflow
dI
10"
10
10
10a
(o) Eliminating Rounding and Underflow error in INT4
(c) Histogram of dINT4
Ouiput Enor
Total error
Rounding
INT4
dINT4
16
Layer (b) Breakdown of ouput errors in INT4
Layar (d) Breakdown ouput erros dINT4
Figure 4: (a) INT4 without rounding sets small values near zero to zero, preserving the rest and causing performance degradation. INT4 without underflow preserves only values zero, improving performance. (b) Impact of underflow error and rounding error on the output crror. Significant impact of underflow error on the output error in INT4. (c) Proposed dINT4 preserves two small values near zero, preventing performance degradation. (d) Using the proposed dINT4 to reduce underflow error leads significant reduction in output error.
Value
INT4
FP4 (1-3-0)
diNT4
HW Performance
MAC Input Formats
Precision
INT8 x INT8
dINT4 x INT8 (Savings)
Area (um²)
86.33
44.57 (1.93x)
Power (mW)
0,1595
0.0624 (2.56x)
Table 2: Evaluation hardware performance of MAC units (7nm, IGHz).
-5
C2
C1
10
15
Figure 5: (Blue) Values to be quantized. (Orange) INT4 quantized values, evenly spaced. (Green) FP4 quantized values, dense resolution for small values but coarse resolution for large magnitudes. (Red) Proposed dINT4 format, balanced quantization range with separate special value for small values.
dINT is effective most scenarios. Furthermore, we design and synthesize a MAC unit using dINT and compare to a traditional 8-bit integer MAC unit using Synopsys Design Compiler and a com- mercial 7nm technology (1GHz) for area efficiency evaluation. As shown in Table 2, dINT achieves 1.93 and 2.56X savings in and power sumption, respectively. This underscores dINT's effectiveness in tackling underflow issues with min- imal output errors and hardware implementation efficiency.
and negative special values in dINT that represent small magnitudes. These values are encoded with distinct bits, analogous to encoding inf or NaN. During dequantization, if the value corresponds to special value; other-
C1 C2, it is represented wise, dequantization proceeds in standard integer formats. Fig. 5 shows that dINT4 strikes a balance between INT4, which has uniform dynamic range coverage but underflow issues, and FP4, which densely represents small values to avoid underflow but coarsely covers the dynamic range.
Experimental Results
5.1 Experimental Settings
In our experimental settings, we implement a com- prehensive evaluation to assess the effectiveness of our AQAS, SLAC, and dINT4 techniques in LLMs. This involves conducting quantized in- ference with 8-bit activations and 4-bit weights across spectrum of tasks, encompassing language modeling, reasoning, and the MMLU benchmark. To enhance both computational and memory ef- ficiency in activation quantization, we broaden our approach to incorporate the quantization "Value (for attention map calculation)", which are
4.3 Advantages
Fig. 4(d) showcases the benefits of dINT in reduc- ing output quantization error. By plotting each term of Eq. 4, we observe that dINT primarily mitigates underflow error, which substantially lowers the out- put error. Although in instances like layer 9, the output error slightly increases due to a widened step size causing a rise in rounding error, the mag- nitude this increment minimal. On the whole,

---

Weight Scaling
OPT Family
LLaaM Family
OPTQ
A-bits
W/V-hits
12SM
1.3B
2.7B
6.7B
13B
30B
7B
13B
30B
Baseline
31.95
16.41
14.32
12.29
11.50
10.67
5.68
5.09
4.10
42.78
35.17
24.03
15.52
36.43
150.94
6.87
6.14
5.52
V
36.04
19.35
15.61
15.16
15.82
25.23
7.57
5.85
5.13
sQ
/
41.31
23.48
33.86
1,596.83
897.25
19.43
7.08
6.27
6.17
INT4
36.22
17.70
15.22
12.82
11.93
11.06
8.26
5.91
5.11
INTS
44.17
25.04
15.88
322.85
670.00
4,246.25
6.75
5.84
5.26
AWQ
/
39.10
19.35
16.00
432.74
183.83
4,848.16
6.52
5.78
5.02
/
36.57
17.68
15.34
13.42
12.19
11.08
6.69
5.81
5.14
AQAS
35.62
17.48
15.08
12.97
12.08
11.04
6.60
5.71
5.07
V
dINT4
34.92
17.28
15.03
12.89
11.98
11.04
6.48
5.67
4.72
Table 3: PPL results of W4A8V4 (Weight-4bit, Activation-Sbit, Value-4bit) at standard language modeling evaluation with OPT and LLaMA family models, applying OPTQ (Frantar et al., 2023) with various weight scaling techniques and two numerical formats.
specifically cached to expedite the inference stage during generation (Kwon et al., 2023). We com- pare our methods against baseline techniques, in- cluding weight scaling of SQ (Xiao et al., 2022), AWQ (Lin al., 2023), and weight update based method, OPTQ (Frantar et al., 2023). Task details, models, calibration methods, and quantization tech- niques used in the experiments are outlined in Ap- pendix 1, and an ablation study exploring aspects such as reducing precision to 3-bit, weight-only quantization with dINT, and other 4-bit formats is detailed in Appendix A.6.
Weight Scaling
OPT
LLaMA
OPTQ
W/V-bits
6.7B
13B
7B
13B
Baseline
69.11
69.38
70.92
74.48
63.23
48.67
69.37
72.22
4
64.42
$6.62
64.94
71.69
INT4
SQ
67.80
68.96
68.23
72.63
67.79
68.80
69.04
73.50
AQAS
V
dINT4
68.19
68.96
68.74
73.31
V
INT4
67.48
68.24
70.48
73.50
AQAS*
dINT4
67.92
68.80
71.01
73.64
Table 4: Average accuracy for CommonSense QA (CSQA) tasks including PIQA, Winogrande, and ARC easy. AQAS* denotes the AQAS method with the SLAC approach (calibrating the dataset's sequence length to 128).
5.2 Evaluation on Language Modeling Task
We first evaluate perplexity (PPL) as the language modeling performance for various PTQ methods. Table 3 presents W4A8 quantization results with different PTQ combinations. For OPT models, OPTQ generally reduces perplexity. However, when combined with weight scaling methods like SQ and AWQ for 4-bit weight quantization, there's s significant accuracy drop (i.e., spikes in PPL., which OPTQ cannot fully mitigate in most OPT models (except 6. 6.7B and 13B). AQAS effectively curtails the accuracy drop of 4-bit weight quan- tization, and combining it with OPTQ enhances accuracy. Utilizing dINT4 for 4-bit quantization further lowers perplexity, maintaining a gap of less than 1.0 compared to the full-precision baseline, with the exception of OPT- 125M, which is sensi- tive quantization. For LLaMA models, OPTQ with 4-bit weight quantization raises perplexity due increased activation diversity, as discussed in Sec. 3.3. Weight scaling methods like AWQ and AQAS aid in performance recovery, and dINT4
further minimizes perplexity, keeping it within 1.0 of the baseline. We detail the results of applying our proposed AQAS and dINT4 strategies to mod- els with over 60 billion parameters, specifically OPT-66B and LLaMA-65B, in Appendix A.2.
5.3 Evaluation on Zero-shot Reasoning Tasks We carry out experiments for the zero-shot Com- monSense QA (CSQA) (Bisk et al., 2019; Sak- aguchi et al., 2019; Clark et al., 201 (018) benchmark by comparing different quantization options, akin to prior tests. As noted in Sec. 3.3, LLaMA models undergo performance decline with OPTQ without weight scaling, whereas OPT models, less affected by activation diversity, show performance gains ing OPTQ even without scaling. Among weight scaling techniques, AQAS exhibits superior perfor- mance, and employing the dINT4 format further enhances results.

---

Weight
OPTQ
A-bits
W/V-bits
LLaMA Family
Conclusion
Scaling
7B
13B
30B
65B
Baseline
35.20
47.15
58.50
63.60
We address Post-training Quantization (PTQ) in
28.05
40.82
48.40
57.20
Large Language Models (LLMs), specifically tar- geting 4-bit weight and 8-bit actívation (W4A8) quantization boost computational efficiency. We present CedivationnQuuaniizatin--- ing (AQAS) and Sequeence-Lentth tion (SLAC), refining PTQ by taking into account weights and activations, and aligning sequence lengths. To combat the underflow issue in W4A8 quantization, where small magnitudes are rounded down to zero, we introduce dINT, hybrid for- mat blending integer and denormal representations. Through extensive evaluations on LLMs such OPT and LLaMA, we demonstrate marked im- provements task accuracy and adaptability, Addi- tionally, with the development of MAC units com- patible with dINT, we achieve a twofold increase in hardware efficiency.
27.05
53.30
INT4
INT8
SQ AQAS
29.32
52.83
30.81
44.23
53.67
59.60
Scal- Calibra-
dINT4
31,00
44.73
55,50
61.40
Table 5: Average MMLU accuracy. The detailed accu- racy for each item can he found in Table 7.
Due to the shorter input sentences zero-shot CSQA compared to the default OPTQ calibra- tion dataset, employing SLAC, which considers the LLaMA models' activation diversity based on sequence length, improves performance for both INT4 and dINT4 formats. However, aligning the calibration length with the target task's sequence length for the OPT models does not result signif- icant improvements. This can be attributed to the OPT models' lower sensitivity to weight updates due to activation diversity during the calibration process, as discussed in Section 3.3, which differs from the behavior of the LLaMA models.
Limitation
We conducted thorough analysis of model- specific characteristics in LLMs and identified lim- itations in current PTQ methods. However, fur- ther investigation is needed to understand the spe- cific phenomena observed in certain LLM mod- els during the pre-training process. Additionally, exploring more advanced collaborations of PTQ techniques at lower bit precision for weights and activations holds promise for future research.
As a result, we attain performance within 1% of full precision for both OPT and I.LaMA mod- els using 8-bit activation and 4-bit weight, notably achieving full LLaMA-7B 7B by comprehensively accounting for the model's activation characteristics.
performance in
5.4 Evaluation on In-Context Learning Tasks
We evaluate the MMLU benchmark on several op- tions that cxhibited strong performance in language modeling. To assess the efficacy of our proposed method in in-context learning, we conduct 5-shot inference. Given that OPT models are deemed unsuitable for the MMLU benchmark (Lin al., 2023), restrict the experiments LLaMA mod- cls. Consistent with language modeling results, AQAS, accounting for both weight and activation quantization errors, delivers the best performance. Moreover, effectively managing underflow error bolsters performance across all models, with a no- table 2% performance enhancement observed in the LLaMA-30B model. To evaluate the efficacy of our upproach large-scale models, further expand the experiment LLaMA-65B. The results demon- strate that dINT4 significantly enhances MMLU accuracy by conserving small-magnitude values. Detailed results for each category within MMLU are provided in the Appendix A.3.
A Acknowledgement
This work was supported by Institute of Infor- mation & communications Technology Planning & Evaluation(IITP)) grants funded by the Korea govemmnen((MSTT))22200000000 of AI Deep-Learning Processor and Module for 2,000 TFLOPS Server, 2020-0-01297, Develop- ment of Ultra-Low Power Deep Learning Pro- cessor Technology using Advanced Data Reuse for Edge Applications), the Technology Innova- tion Program (1415178807, Development of In- dustrial Intelligent Technology for Manufacturing, Process, and Logistics) funded By the Ministry of Trade, Industry & Energy(MOTIE, Korea), and the National Research Foundation of Korea (NRF) grant funded by the Korea govemmen((MSTT)(N.. 202 ARII1C11133113)
Development

---

References
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023, OPTQ: Accurate quantization for generative pre-trained transformers. In The Fleventh International Conference on Learning Reprexenta- lions.
Michael Andersch, Greg Palmer, Ronny Krashinsky, Nick Stam, Vishal Mehta, Gonzalo Brito, and Sridhar Ramaswamy. 2022. Nvidia hopper architecture in- depth. https:///evell how.eeeeeeeee..iiiiieeeeeee nvidii. com/blog/ indppt/
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing. Travis Hoppe. Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.0000
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2021. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 202/ Conference on Empiri- cal Methods in Natural Language Processing, pages 7947-7969, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Sudhanva Gurumurthi, Kijun Lee, Munseon Jang, Vi- las Sridharan, Aaron Nygren, Yesin Ryu, Kyomin Sohn. Taekyun Kim, and Hoeju Chung. 2021. Hbm3 ras: Enhancing resilience at scale. JEEE Computer Architecture Letters, 20(2):158-161.
Mingxuan He, Choungki Song, Ilkon Kim, Chunseok Jeong, Scho Kim, I1 Park, Mithuna Thottethodi, and T. N. Vijaykumar. 2020. Newton: A dram-maker's ss accelerator-in-memndd
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages Inc.
(aim) architecture for ma- chine learning. In 2020 53rd Annual IEEF/ACM International Symposium on Microarchitecture (MI- CRO), pages 372-385.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing.
Mark Horowitz. 2014. Energy table for 45mm process.
Curran Associates,
Qing Jin, Jian Ren, Richard Zhuang, Sumant Hanu- mante, Zhengang Li. Zhiyu Chen. Yanzhi Wang. Kaiyuan Yang. and Sergey Tulyakov. 2022. F8net: Fixed-point 8-bit only multiplication for network quantization. In International Conference on Learn- ing Representations.
Brian Chmiel, Ron Banner, Flad Hoffer, Hilla Ben Yaa- cov, and Daniel Soudry. 2022. Logarithmic unbiased quantization: Simple bit training in deep learning.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Jin Hyun Kim, Shin-haeng Kang, Sukhan Lee, Hyeonsu Kim, Woongjae Song. Yuhwan Ro, Seungwon Lee. David Wang, Hyunsung Shin, Bengseng Phuah, Ji- hyun Choi. Jinin So, YeonGon Cho. JoonHo Song, Jangseok Choi, Jeonghyeon Cho, Kyomin Sohn, Youngsoo Sohn, Kwangil Park, and Nam Sung Kim. 2021. Aquabolt-xl: Samsung hbm2-pim with in- memory processing for ml accelerators and beyond. In 2021 IEEE Hot Chips 33 Symposium (HCS), pages 1-26.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arX1vv1803.05457VL.
Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung- Woo Ha, Nako Sung, and Dongsoo Lee. 2022. AlphaTTuning: Quantization-aware parameter- efficient adaptation of large-scale pre-trained lan- guage models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3288 3305, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zeulemoyer. 2022. Llm.int8(:: 8-bit matrix multi- plication for transformers at scale. arXiv preprint arXiv:2208.07339
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alis- tarh. 2023. Spqr: A sparse-quantized representation for near-lossless Ilm weight compression.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management large language model serv- ing with pagedattention. In Proceedings of the 29th
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprini arXiv:2210.1777 133.3
17323.

---

Symposiun on Operating Systems Principles. SOSP '23. page 611 626,, New York, NY, USA. Association for Computing Machinery.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1 13971.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Funhhyok Park. 2023a. Owq: Lessons learned from activation outliers for weight quantization in large language models.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien De- mouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large lan- guage models. arXiv preprint 1xXi::2211100436
Janghwan Lee, Youngdeok Hwang, and Jungwook Choi. 2023b. Finding optimal numerical format for sub-8- bit post-training quantization of vision transformers. ICASSP 2023 2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.
Ji Lin, Jiamiig Tang, Haotian Tang, Shang Yang. Xingyu Dang, and Song Han. 2023, Awq: Activation- aware weight quantization for IIm compression and acceleration. arXiv.
Aohan Zeng. Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding. Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia. Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang. Yuxiao Dong, and Jie Tang. 2023, GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference Learning Representations.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Emie Chang, Pierre Stock, Yashar Mchdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023. Llm-qat: Data-free quantization aware training for large language models.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
Stephen Merity, Caiming Xiong. James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quanti- zation. In Proceedings of the 37th International Con- ference Machine Learning, ICML JMLR R.o..
OpenAI. 2023, Gpt-4 technical report. arXiv preprint 0xX7::2303.08774
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim. Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lec. 2023. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models.
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2019. Winogrande: An adver- sarial winograd schema challenge at scale.
Xiao Sun, Jungwook Choi, Chia-Yu Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi (Viji) Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. 2019. Hybrid 8-bit floating point (hfp8) training and inference for deep neural net- works. In Advances Neural Information Process- ing Systems, volume 32. Curran Associates, Inc.
Xiao Sun, Naigang Wang. Chia-Ya Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkatara- mani, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, and Kailash Gopalakrishnan. 2020. Ultra- low precision 4-bit training of deep neural networks. In Advances in Neural Information Processing Sys- tems, volume 33, pages 1796-1801 Curran Asso- ciates, Inc.

---

A Appendix
Wikitext PPL
OPT--6B
LLaMA-65B
FP16 baseline
10.09
3.53
A.1 Experimental Details
INT4
3,222.02
4,80
Baseline Setup. As comparative baselines for weight scaling, we employ SQ (Xiao al.,
AQAS OPTQ
Table 6: Wikitext perplexity for W4A8V4 inference OPT-66B and LLaMA-65B.
2022) as the scale method for 8-bit activation quantization and AWQ (Lin et al., 2023) as the scaling method for 4-bit weight quantization. In terms of weight rounding, we evaluate both options provided by OPTQ (Frantar et al., 2023), which offers additional optimization, and the standard nearest rounding method. As for numerical format, we compare dINT with existing INT4 methods in terms of performance.
bration process by modifying the objective Eq. to minimize the distortion the attention output. We use a randomly extracted dataset from Pile (Gao et al., 2020) for AWQ (Lin et al., 2023), SmoothQuant (Xiao et al., 2022), and AQAS methods. When calibrating weights with OPTQ, follow the baseline calibration setting provided in the OPTQ repository². We use a subset of the C4 dataset, randomly selecting 128 samples with a sequence length of 2048.
Task and Models. We evaluate our proposed approach on various tasks, including language mod- eling using Wikitext (Merity et al., 2016)), CSQA (PIQA (Bisk al., 2019). WinoGrande (Sakaguchi al., 2019), and ARC easy (Clark et al., 2018)), well MMLU (Hendrycks benchmarks. For benchmark models, we assess range of options across OPT and LLaMA models, ranging from 125M to 66B, which are widely used LLMs.
A.2 Language modeling in >60B Models
al., 2021)
Table 6 To assess the effectiveness of our approach large models, conduct language modeling experiments on OPT-66B and LLaMA-65B, with the objective of determining whether our method performs well even on models with over 60 billion parameters. demonstrates that proposed scaling method and numerical format can significantly re- duce perplextiy in language modeling task.
Quantization Settings. We apply quantiza- tion to both the weights and activations of all matrix multiplications in the Decoder layer. We conduct our experiments by implementing the quantizer within the PyTorch framework. For activations, except for Value, we apply 8-bit quantization, while for memory-intensive compo- nents such as weights and Value, we utilize 4-bit quantization. Similar to commonly used methods for LLM quantization (Dettmers et al., 2022; Yao al., 2022), apply loken-wise quantization for activations, and output channel-wise quantization for weights. For Value, we upply channel-wise quantization, taking into account the dimensions where partial-sum accumulation occurs when multiplied by the self-attention map. We apply Min-Max asymmetric quantization determine the step size and the zero-point for both activation and weight.
A.3 Few-shot MMLU Benchmarks
The results for the each category in the 5-shot MMLU benchmark for LLaMA models are dis- played in Table As demonstrated in Table 7, AQAS exhibits higher accuracy compared to other scaling methods, emphasizing the importance of considering both weight and activation quantiza- tion. Furthermore, it's noteworthy that the of dINT, which effectively mitigates underflow. achieves the highest accuracy.
A.4 Finding Scales for AQAS
To automatically determine the channel-wise scale factor in AQAS, it is necessary to select representa- tive values for both activation and weight channels. In SQ (Xiao et al., 2022), the maximum magni- tude was used as the criterion, while in AWQ (Lin et al., 2023), the absolute mean value was used to explore the scale factor. As shown in Table 8, we explore both cases and found that selecting the
Calibration Settings. During the calibra- tion process to find the weight scale, we follow the calibration setting from the AWQ r For the attention operation, we adjust the cali-
Shurp://iituuh.oom///STDAASA

---

LLaMA 7B
Wikitext
Scaling
OPTQ A-bits
W/V-bits
Hums
STEM
Social
Avg.
OPT
LLaMA
AQAS
OPTQ
Baseline
33,90
30.50
38.20
38.20
35.20
125M
1.38
6.78
28.65
25.75
31.15
28.05
Buseline
31.95
16.41
14.32
12:29
11.50
10.67
5,68
sQ
V
/
INT8
INT4
27.80
26,99
24.88
29.60 06
25.71
27.40
33.04
30.41
27.05
29.32
Mean
37.46
36.07
18.16
17.50
15.43
15.19
13.08
12.99
12,11
12,07
11.06
11,02
6.72
6.67
AQAS
29.61
29.85
30.65
33.62
30.81
Max
35.62
36.57
17.48
17.68
15.08
15.34
12.97
13:42
12.19
12.08
11.08
11.04
6.60
6.69
dINT4
29.67
29.26
32.24
33.37
31,00
PIQA
LLaMA 13B
Scaling OPTQ A-biss
W/V/bis
Hums
STEM
Social
Other
Avg.
AQAS
OPTQ
OPT
1.aaMA
125M
1:38
2.78
6.78
138
30B
7B
Baseline
36.40
47.15
Baseline
63.00
71.71
73.78
76.28
75.90
77.58
78.35
36.96
35.16
45.56
47.19
40.82
61,86
70.78
72.91
75.57
74.86
76.66
77.42
34.10
Mean
SQ
INTS
INT4
40.96
34.26
48.72
49.20
43.12
V
61.81
71.22
72,85
75.14
74.97
77.20
77.37
70.51
72.69
74.27
75.14
77.15
77.26
AQAS
35.02
Max
V
62.30
71.11
73.56
75.08
75.84
77.48
76.28
4NT4
42.64
35.79
50.83
50.31
44.73
WineGrande
LA
308
OPT
LLaMA
AQAS
OPTQ
Scaling
OPTQ
b-tits
W/V-buis
Humm
STEM
Social
Other
Avg.
125M
1.38
2,78
6.78
138
308
7B
Baseline
56,40
46.70
67.30
63.60
58.50
Baseline
59.51
73.01
67.09
46.00
54,50
54.50
48.40
Mean
51,54
58.88
60.14
65.59
65.51
67.56
64.96
50.20
42 60
61.90
39.70
53.30
49.96
57.62
61.64
65,82
64.88
68.03
64.80
INT4
SQ
43.31
39.65
52.83
52,41
60.62
61.64
65:27
64.88
67.40
63.77
51.75
43:21
60.68
59.53
53.67
V
49.72
58.17
60.14
63.77
65.27
68.11
65.59
AQAS
dINT4
$2.90
44.00
61.10
35.50
Table 8: Comparing the performance of AQAS when exploring channel-wise quantization using the criteria of absolute mean and max values.
L 65B
Scaling OPTQ A-hits|
W/V-bits
Hums
STEM
Social
Other
Aug.
Baseline
61.90
52,10
73,40
67.60
63.60
$4,10
46.40
66.90
62.50
57.20
SQ
/
INT8
INT4
56.00
57,20
47.80
47.
67.20
67.90
63.90
64.30
59.30
58,50
/
125M
1.3B
OPT
2.7B
6.7B
LLaMA
7B
Avg.
57.20
47.80
69.50
64.80
59.60
Baseline
31.95
16.41
14.32
12.29
5.68
16.13
AQAS
dINT4
59,50
50.40
70.70
65.80
61.40
0.50
34.92
17.28
15.03
12.89
6.48
17.32
0.25
35.84
17.22
14.96
12.83
6.28
17.43
Table 7: MMLU accuracy on LLaMA models.
0.125
35.20
17.23
14.97
12.84
6.32
17,32
maximum magnitude as the representative value of- ten yielded better performance. Similar to previous research (Lin et al., 2023), we use a grid search to find the appropriate scale, and after determining the scale factor, we make adjustments by additionally clipping the weights.
Table 9: dINT4's special value sweep, W4A8V4 in- ference with AQAS+OPTQ Where ei is the positive special value, and is the step size.
Weight scaling
OPT
LLaMA
OPTQ
W/V-bits
125M
2.7B
6.7B
7B
Baseline
31.95
14.32
12.29
5.68
A.5
Sweep of the Special Value in dINT
INT3
1.7c3
4.3c4
1.204
94.97
dINT3
127.94
8.7e3
55.24
10.99
The dINT format defines the special value as half of the step size s. If we change this value, the magnitude of the state representing small values will differ. We conduct additional sweeps with different power-of-two values (e.g., 0.25, 0.125) to observe the impact in Table 9. In most cases, setting to 0.25 times the proves to be good choice, but in the case of the OPT- 125M model, it shows significant increase in perplexity. To select a value that generally works well, we set e to be half of s.
INT3
54.84
36.38
69.45
24.85
AQAS
dINT3
46.34
20.67
17.42
10.04
Table 10: Perplexity is assessed using standard language modeling on the Wikitext datasct, with activations quan- tized 8 bits and weights and values to bits. We em- ploy AQAS and OPTQ and compare the performance of INT3 and dINT3. Notably, dINT3 considerably reduces performance degradation.
in which we retain 8-bit activation while reducing weight and Value precision to bits. As shown in Table 10, as bit precision decreases and the impact of underflow becomes more significant, the effectiveness of dINT becomes more pronounced.
A.6 Ablation Study
Reducing Precision to 3 Bits. To achieve addi- tional memory savings, we conduct experiments

---

Precision
Format
Method
125M
OPT
2.78
138
Table 13: Experiments on various configurations of 4- bit floating-poine FP4 (1-e-m) represents floating-point format with 1-bit sign bit, e-bit exponent, and m-bit mantissa. We conduct experiments on Wikitext PPL. PIQA accuracy, and MMLUU average accuracy. Among W4 format
Model
Precision
FP16
baseline
Wikitext
5.68
78.29
PIQA
MMLU
35.20
FP16 baseline
31,95
16.41
14.32
12.29
11.50
FP4
163582.55
51.69
26.88
LLaMA-7B
FP4 (1-2-1)
26.52
62.84
27.31
RIN
58.37
195.10
499.39
39.18
29.57
W4A16
FP4 (1-3-0)
6.30
76.77
31.46
INT3
OPTQ
41.93
18.53
15.79
13.13
12.01
dINT4
6.07
77.91
32.53
AWQ
41.45
18.56
15.63
12.99
12.03
W3A16
KIN
54,53
21.70
24.15
15,80
13.14
FP16 baseline
5.09
78.78
47.15
JINT3
OPTQ
39.60
18.36
15.52
12.99
11.91
(1-11-1-2)
74763.98
52.29
24.72
AWQ
39,46
18.29
15:47
12.97
12.03
LLaMA-13B
W4A16
(1-2-1)
7.95
31.51
FP4 (1-3-0)
5.56
78.62
40.76
JINT4
5.38
79.05
44.35
Table 11: Performance comparison of W3A16 infer- ence results with various statc-of-the-art methods when applying group-wist quantization (group size: 128).
FPI6 baseline
4.10
80.96
58.50
34027.07
51.52
25.32
LLaMA-308
FP4 (1-2-1)
9.10
71.22
32.05
W4A16
FP4 (1-3-0)
4.57
79.71
53.50
dINT4
4.36
80.41
55.87
OPT
Procision
Formst
Method
125M
1.38
2.7B
6.7B
13B
baseline
12.29
11.50
RTN
35.52
17.69
15.12
13:02
11.89
INT4
OPTQ
34.23
16.92
14.69
12.51
11.60
AWQ
33.96
1688
14.61
12.44
11.60
RTN
19.13
18:25
15.54
13.35
12.14
W4416 g=128
FP4
OPTQ
11.91
FP4 configurations, 3-bit exponent exhibits the best performance, while dINT surpassing it.
RTN
34.40
17.06
14.83
12.75
11.79
OPTQ
34.04
14.64
12.47
11.59
AWQ
33.66
16.78
14,56
12.44
11.61
Table 12: Performance comparison of W4A16 infer- ence results with various tate-offthheart methods when applying group-wind wise quantization (group size: 128).
Precision
Format
OPTQ
125M
2.7B
13B
FP16 baseline
31.95
16.41
14.32
12.29
11.50
43.15
29.90
19.70
14.18
12.89
INT4
36.29
17.68
15.15
12.88
11.73
42.17
18.52
16.01
13.38
12.33
W4A16
FP4
37.52
18.31
15.39
13.18
11.89
By solely changing the numerical format without applying weight scaling, we are able to signifi- cantly reduce the perplexity of the ILaMA-7B model from 94.97 to 10.99, This underscores the influence of underflow model performance.
15.53
12.06
dINT4
35.10
17.30
14.94
12.65
11.68
Table 14: Comparing the performance of various for- mats in welght-only quantization for the language mod- eling task, using both dINT and OPTQ together shows the best performance.
Weight-Only Quantization Method with dINT. dINT, as numerical format, can be integrated with existing PTQ methods. We combine the dINT format with otate-of-the-art PTQ methods for LLMs, namely OPTQ and AWQ, and compare their performance with the integer format. As shown Table 11 and Table 12, dINT outperforms the traditional integer format both 3-bit and 4-bit quantization. This indicates that underflow significantly affects the performance of weight quantization in LLMs.
Daseline Smeothquant
Beseline
Absmmaxx
20
AWO AQAS
Nosmmattii
AVO
SmoothQuant
20-
10
20
30
10
20
30
Key projection layers
Key projection layers (b) Weight
(a) Input activation
Figure 6: The variation in the absolute max values of weights and activations when applying weight scaling LLaMA-7B.
details can be found in Table 13. As shown in Table 14, FP4 achieves some performance improve- ment compared to uniform quantization due to its wider dynamic range. However, dINT4 outper- forms the other two formats by effectively repre- senting a wide range of values with uniform inter- vals while accurately representing small values. It demonstrates better performance and good compat- ibility with existing optimization techniques such as OPTQ.
Other 4-Bit Formats. To compare the perfor- mance of 4-bit quantization formats, we evaluate performance by applying integer, floating-poin and dINT4 to the weights, without considering ac- tivation quantization. We employ 4-bit floating- point (FP4), consisting of single sign bit and three exponent bits. While alternative configura- tions with different exponent and mantissa bits are available, we experimentally determine the neces- sity of a 3-bit exponent for the FP4. Additional


Reviews:
[{"review_id": "X78pk7MjFr", "reviewer": "Reviewer_WZ3Z", "summary": "", "questions": "Please see the *Reasons To Reject section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback. Responses to your comments are provided below.   \n\n> **Reasons To Reject 1.** The method is only verified on the W4A8E4 setting. However, in previous work SmoothQuant(SQ), AWQ can be applied with various precision.\n\nThank you for the valuable comment on the necessity of experiments across various precisions. Following the reviewer’s advice, we extended our experiments to include a range of precisions (W8A8, W3A16, and W3A8) studied in the previous work (SQ, AWQ). As shown in Table R1, the proposed AQAS and dINT achieve consistent accuracy gain for aggressive 3-bit weight AND activation quantization. \n\n| Wikitext-2 | OPT/LLaMA | W8A8 | W8A8 | W3A16 | W3A16 | W3A8 | W3A8\n-- | -- | -- | -- | -- | -- | -- | --\nScaling method | Weight format | OPT-6.7B | LLaMa-13B | OPT-6.7B | LLaMa-13B | OPT-6.7B | LLaMa-13B\n| - | FP Baseline | 12.29 | 5.09 | 12.29 | 5.09 | 12.29 | 5.09\nSQ | INT | 12.31 | 5.13 | 50053.79 | 142.54 | 48777.33 | 123.71\nAWQ | INT | 99.40 | 5.15 | 17.60 | 7.43 | 3013.52 | 7.53\nAQAS | INT | **12.29** | **5.11** | 16.63 | 6.34 | 17.96 | 6.47\nAQAS | dINT | **12.29** | **5.11** | **14.39** | **5.99** | **15.10**| **6.08**\n\nTable R1. Comparison of SQ, AWQ, and AQAS scaling methods for various PTQ precisions (Wikitext 2 PPL, the lower the better).\n\n- In the case of W8A8, AWQ's lack of consideration for activation quantization results in significant performance degradation, particularly in models with notable activation outliers like OPT. Although SQ outperforms AWQ by scaling activation outliers, it still falls short of AQAS, which delivers performance nearly on par with full-precision models.  \n\n- In the case of W3A16 (weight-only quantization), AWQ specialized for low-precision weight-only quantization significantly recovers accuracy compared to SQ, which suffers from a severe accuracy drop. However, AWQ still exhibits noticeable accuracy loss at 3-bit weight quantization. Conversely, AQAS achieves performance that is much closer to full-precision models. Note that the dINT format significantly improves weight quantization performance in sub-4-bit precision scenarios.  \n\n- In more aggressive bit precision settings for both weight AND activation quantization (i.e., W3A8), both SQ and AWQ exhibit a substantial accuracy drop in both OPT and LLaMA. Conversely, AQAS and dINT effectively mitigate this accuracy degradation, particularly in the OPT model, bringing results nearly in line with full-precision baselines. This observation highlights the combined efficacy of AQAS in adjusting quantization scales and dINT in enhancing the representation of small values for aggressively precision-scaled PTQ.  \n\n&nbsp;\n\n> **Reasons To Reject 2-1.** In Table 2, comparing the area and power of INT8-based MAC units and dINT4-based MAC units is unfair.\n\nThank you for allowing us to elucidate the comparison between INT8-INT8 and dINT4xINT8 units in Table 2. Our study primarily explored whether the dINT4-based MAC unit could offer performance gains proportionate to the bit-precision reduction over the INT8-based unit. As noted in previous research [R1], lowering bit-precision can proportionally enhance both area and power efficiency, leading to increased throughput (e.g., transitioning from FP8-FP8 to INT4-INT4 boosts throughput 4x, as per [R2]). Table 2 shows that by reducing weight precision from 8-bit to 4-bit, our dINT format yields 2x area and 2.5x power efficiency improvements, underscoring dINT's ability to offer scalable performance despite its representation (Eq. 5) distinct from INT.\n\n[R1] M. Horowitz. \"Energy table for 45nm process\", Stanford VLSI wiki. [Online]. Available: https://sites.google.com/site/seecproject  \n[R2] Agrawal et al., “9.1 A 7nm 4-Core AI Chip with 25.6TFLOPS Hybrid FP8 Training, 102.4TOPS INT4 Inference and Workload-Aware Throttling”, ISSCC 2021\n\n&nbsp;\n\n> **Reasons To Reject 2-2.** In Figure 5 the FP4 is shown with mantissa=1, exponent=3, and no bias bit (correct me if I am wrong), which is not an optimal format for FP4. Also, I guess the error will be lower than dINT4 if the FP4 quantization error is visualized in Figure 4.\n\nFollowing the reviewer's suggestion, we conducted an ablation study on various FP4 bit configurations (sign-exponent-mantissa: 1-3-0, 1-2-1, 1-1-2). Note that instead of utilizing a distinct exponent bias, we have employed a scale factor similar to integer quantization for representing the maximum data magnitude. Benchmark results (Table R2) and quantization error analysis (Table R3) demonstrate that our chosen FP4 1-3-0 outperforms the other FP4 configurations, yet dINT further excels over FP4 1-3-0.\n\nModel | Precision | W4 format | Wikitext (PPL ↓) | PIQA (ACC ↑) | MMLU (avg ↑)\n-- | -- | -- | -- | -- | --\n|LLaMA-7B |   | FP16 baseline | 5.68 | 78.29 | 35.20 \n|  | W4A16 | FP4 (1-1-2) | 165582.55 | 51.69 | 26.88\n|  |   | FP4 (1-2-1) | 26.52 | 62.84 | 27.31\n|  |   | FP4 (1-3-0) | 6.30 | 76.77 | 31.46\n|  |   | dINT4 | **6.07** | **77.91** | **32.53**\nLLaMA-13B |   | FP16 baseline | 5.09 | 78.78 | 47.15\n|  | W4A16 | FP4 (1-1-2) | 74763.98 | 52.29 | 24.72\n|  |   | FP4 (1-2-1) | 7.95 | 74.65 | 31.54\n|  |   | FP4 (1-3-0) | 5.56 | 78.62 | 40.76\n|  |   | dINT4 | **5.38** | **79.05** | **44.35**\nLLaMA-30B |   | FP16 baseline | 4.10 | 80.96 | 58.50\n|  | W4A16 | FP4 (1-1-2) | 34027.07 | 51.52 | 25.32\n|  |   | FP4 (1-2-1) | 9.10 | 71.22 | 32.05\n|  |   | FP4 (1-3-0) | 4.57 | 79.71 | 53.50\n|  |   | dINT4 | **4.36** | **80.41** | **55.87**\n\nTable R2. Experimental results of weight-only quantization (W4A16) utilizing FP4 variants and dINT4\n\n|  | Total error | by Rounding error | by Underflow error | by Cross-term error\n|-- | -- | -- | -- | --\n|FP4 (1-2-1) | 39.4017 | **4.2784** | 34.6510 | 0.4723\n|FP4 (1-3-0) | 17.6443 | 17.6315 | **0.0163** | **0.0035**\n|dINT4 | **7.9299** | 7.4314 | 0.4594 | 0.0390\n\nTable R3. Analysis of output quantization error (MSE) in LLaMa single layer with breakdown by weight quantization error using Eq.4. (LLaMa-7B Query projection Layer 15, unit: 1e-3) \n\nHere are key insights yielded from this ablation study: \n\n**Performance in benchmarks**: As evident from Table R2, employing a 1-bit exponent severely compromises performance, and even in the case of FP4 1-2-1, accuracy loss persists. While FP4 1-3-0 demonstrates robust performance compared to other FP4 options, our proposed dINT4 exhibits the best accuracy across all tasks.\n\n**Quantization error analysis**: Table R3 shows the breakdown of the mean-squared error (MSE) at the output of input-weight multiplication (cf. Eq. 4). While FP4 1-2-1 has the least rounding error, it exhibits significant underflow error. Conversely, FP4 1-3-0 reduces underflow error but has elevated rounding error. dINT strikes a balance between these errors, resulting in enhanced inference accuracy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy-to-follow. The visualization of the activation range (Fig.1) and quantization error (Fig.4) are helpful in understanding the advantage of the proposed method and data format.\n2. Insights on the dynamic range of different layers and variations with different sequence lengths are provided. These findings are helpful for future research on LLM quantization (both PTQ and QAT).\n3. Comprehensive experimental design across various LLMs and tasks. These results clearly demonstrate the effectiveness of the proposed method.\n4. The evaluation of the hardware performance of MAC units on dINT4 and INT8 is informative.", "reasons_to_reject": "1. The method is only verified on the W4A8E4 setting. However, previous work (SmoothQuant, AWQ) can be applied with various precision.\n2. Some comparisons may be unfair:\n    - In Table 2, comparing the area and power of INT8-based MAC units and dINT4-based MAC units is unfair.\n    - In Figure 5 the FP4 is shown with mantissa=1, exponent=3, and no bias bit (correct me if I am wrong), which is not an optimal format for FP4. Also, I guess the error will be lower than dINT4 if the FP4 quantization error is visualized in Figure 4.", "questions_to_authors": "Please see the *Reasons To Reject section.", "paper_topic_and_main_contributions": "This work focuses on LLMs' post-training quantization (PTQ), especially W4A8 quantization, to improve the efficiency of LLMs. The two core techniques proposed in this work are activation-quantization-aware scaling (AQAS) and sequence-length-aware calibration (SLAC). In addition, a novel data format called dINT is introduced to solve the underflow challenges effectively. The method is evaluated across different LLMs (OPT, LLaMas) and various tasks and metrics (PPL on language modeling, zero-shot reasoning tasks, few-shot MMLU, etc.) to demonstrate effectiveness and efficiency.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5KmVwSvAR5", "reviewer": "Reviewer_MJRd", "summary": "", "questions": "What is the unique contribution of this paper compared with similar investigations around W4A8 PTQ for language models: https://openreview.net/pdf?id=tvDRmAxGIjw, and clamping denormal strategy: https://arxiv.org/pdf/1909.13271.pdf?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the insightful feedback provided by the reviewer. In response to the raised questions, our answers are as follows.\n\n> **Reasons To Reject 1.** The experiment results cannot support the authors' argument well. The INT4 AQAS shows similar results as INT4 SQ in all experiments carried out in the paper. There is no good justification for using AQAS rather than SQ.\n\nThank you for letting us clarify the results of AQAS compared to SQ. We claim that AQAS performs robustly across various LLMs with distinct characteristics (OPT vs LLaMA) for W4A8 quantization, while SQ barely manages to retain comparable accuracy for OPT (only with the help of OPTQ), it still suffers noticeable accuracy degradation on LLaMA and LLaMA-2. \n\nTo elucidate the superior performance of AQAS over SQ, consider the following points:\n\n- As shown in Table 3, SQ alone falls short for W4A8, exhibiting PPL scores of 1596.83 and 897.25 on OPT-6.7B/13B, respectively.\n- While OPTQ assists SQ in achieving accuracy comparable to AQAS on OPT, SQ still suffers a noticeable performance drop on LLaMA-7B. Specifically, SQ shows a 2.58 PPL degradation compared to AQAS in wikitext as shown in Table 3.\n- AQAS's balanced scaling strategy ensures robust accuracy for both OPT and LLaMA. For instance, it exhibits PPL scores of 12.19 and 12.08 with and without OPTQ on OPT-13B and demonstrates a 1.03 higher PPL than SQ on LLaMA. Similar trends are also observed across the CSQA and MMLU tasks in Tables 4 and 5.\n- We newly added experimental results of the recently released LLaMA-2 model, as observed in Table R1, AQAS outperforms SQ in all tasks, regardless of whether OPTQ is applied. Importantly, the significant accuracy degradation of SQ on MMLU and CSQA tasks seen with LLaMA is consistently evident with LLaMA-2 as well.\n\nThe observed unstable performance of SQ on LLaMA models is because of the instability of SQ, **which unilaterally shifts the quantization difficulty by moving activation outliers to the weights**, showing varying performance depending on the models and the assistance of OPTQ. In contrast, AQAS, which explores **scaling while considering the quantization results of both activation and weight**, delivers more consistent and reliable performance across different models, underscoring its superiority in the W4A8 quantization.\n\nLLaMA-2-7B (W4A8) | WikiText-2 | MMLU | CSQA\n-- | -- | -- | --\nWeight Scaling | PPL | Acc | Acc\nFP | 5.42 | 41.6 | 71.76\nSQ | 6.82 | 38.6 | 68.55\nAQAS | 6.46 | 40.7 | 70.40\nSQ+OPTQ | 6.65 | 38.9 | 69.51\nAQAS+OPTQ | **6.45** | **41.1** | **70.94**\n\nTable R1. Performance comparison between SQ and AQAS across wikitext, MMLU, and CSQA (PIQA, winogrande, ARC_easy) tasks with LLaMA-2-7B.\n\nTo gain a deeper understanding of the differences between SQ and AQAS, we conducted an analysis focusing on the scale values and their corresponding layer output quantization errors (via SQNR)  in both methods, as presented in Table R2. Notably, in Layers 5 and 10, AQAS employs smaller scale values than SQ. These **contrasting scale values lead to substantially higher output quantization errors** in the initial layers for SQ compared to AQAS. Even though similar scale values emerge in later layers, these variations result in markedly different text generation performance, as evidenced by the PPL comparisons between SQ and AQAS in Table R2.\n\nOPT-6.7B | Layer | 5 | 10 | 20 | 30 | W4A8 PPL\n-- | -- | -- | -- | -- | -- | --\nSQ | Scale | 22.55 | 22.39 | 25.92 | 31.53 | 1586.83\nSQ | SQNR | 12.51 | 4.29 | 11.47 | 9.71 | -\nAQAS | Scale | 13.44 | 14.91 | 21.69 | 28.45 | **13.42**\nAQAS | SQNR | **16.76** | **5.91** | **11.90** | **9.93** | -\n\nTable R2. Comparison of scale values for outlier channel and per-layer output SQNR between SQ and AQAS in OPT-6.7B and wikitext-2 PPL results of W4A8 PTQ OPT-6.7B \n(SQNR: signal-to-quantization-noise ratio, higher the better)\n\n&nbsp;\n\n> **Reasons To Reject 2.** This paper lacks of novelty. I understand that this paper focuses on LLMs, but there are similar investigations around W4A8 PTQ for language models: https://openreview.net/pdf?id=tvDRmAxGIjw, and clamping denormal strategy: https://arxiv.org/pdf/1909.13271.pdf.  \n**Questions.** What is the unique contribution of this paper compared with similar investigations around W4A8 PTQ for language models: https://openreview.net/pdf?id=tvDRmAxGIjw, and clamping denormal strategy: https://arxiv.org/pdf/1909.13271.pdf?\n\nWe thank the reviewer for letting us clarify the novelty of our work over the suggested related works.\n\nWe would like to emphasize that our work proposes the **first comprehensive study to tackle unprecedented W4A8 quantization for large (Transformer-decoder-based) language models with tens of billions of parameters**. To overcome unique quantization challenges of systematic outliers (Dettmers et al., 2022) in LLMs, we comprehensively improve three key aspects of quantization, 1) scaling for quantization dynamic ranges, 2)  task-specific sequence-length calibration, and 3) number representation format for avoiding underflow, thereby extending the boundaries of weight and activation quantization of LLMs. Specifically, our contributions are as follows:\n\n- Analyzing the characteristics of activations and weights across different LLMs to address the challenges of activation/weight PTQ, thereby proposing a new quantization scaling method (AQAS) \n- Proposing a target task adaptive calibration methodology to offer a task-specific PTQ approach (SLAC).\n- Introducing the dINT format aimed at solving the previously overlooked issue of underflow in low-precision computations (dINT).\n\nHere's how each of these contributions contrasts with the papers you suggested:\n\n**[Comparison with W4A8 PTQ Paper [R1] https://openreview.net/pdf?id=tvDRmAxGIjw ]**\n\n- Unlike [R1], which focuses on encoder-only models, we perform a deep-dive into the challenges of low-precision PTQ for decoder-only LLMs, which exhibits unique characteristics of systematic outliers (Dettmers et al., 2022).\n- Our scaling methods AQAS, designed to be computationally viable for billion-scale LLMs, contrast with [R1]'s back-propagation-dependent approach, which is revealed to be unsuitable (Frantar et al., 2023). \n- Diverging from [R1]'s task-agnostic methods, we introduce a task-specific PTQ calibration technique called SLAC, which we showed critical for supporting duverse tasks.\n\n**[Comparison with AdaptivFloat [R2] https://arxiv.org/pdf/1909.13271.pdf ]**\n\n- [R2] did not investigate the trade-off between underflow and rounding errors in terms of quantization error, nor did it explore the necessity of denormal. As revealed in our study, this trade-off constitutes a critical determinant, especially in contexts of low precision, such as 4-bit precision.\n- To highlight the novelty of our approach, we conducted a brief ablation study on various formats of FP4 to demonstrate the significance of exploring the trade-off between underflow and rounding errors. \n   - Table R3 shows the breakdown of the mean-squared error (MSE) at the output of input-weight multiplication (cf. Eq. 4). While FP4 1-2-1 has the least rounding error, it exhibits significant underflow error. Conversely, FP4 1-3-0 reduces underflow error but has elevated rounding error. \n   - dINT strikes a balance between these errors, resulting in enhanced inference accuracy across wikitext, PIQA, and MMLU tasks as shown in Table R4.\n\n| W4 Format | Total error | by Rounding error | by Underflow error | by Cross-term error\n|-- | -- | -- | -- | --\n|FP4 (1-2-1) | 39.4017 | **4.2784** | 34.6510 | 0.4723\n|FP4 (1-3-0) | 17.6443 | 17.6315 | **0.0163** | **0.0035**\n|dINT4 | **7.9299** | 7.4314 | 0.4594 | 0.0390\n\nTable R3. Analysis of output quantization error (MSE) in W4A16 with LLaMA-7B single layer with breakdown by weight quantization error using Eq.4. (Layer 15 Query projection, unit: 1e-3) \n\nModel | Precision | W4 Format | Wikitext (PPL ↓) | PIQA (ACC ↑) | MMLU (avg ↑)\n-- | -- | -- | -- | -- | --\nLLaMA-7B | FP16 | - | 5.68 | 78.29 | 35.20\n|  | W4A16 | FP4 (1-2-1) | 26.52 | 62.84 | 27.31\n|  |   | FP4 (1-3-0) | 6.30 | 76.77 | 31.46\n|  |   | dINT4 | **6.07** | **77.91** | **32.53**\n\nTable R4. Experimental results of weight-only quantization (W4A16) utilizing FP4 variants and dINT4\n\nWe are grateful for the reviewer’s feedback which has significantly enhanced the clarity of the comparison between related work and our contributions.\n\n[R1] Bai et al, “Towards Efficient Post-training Quantization of Pre-trained Language Models” NeurIPS 2022  \n[R2] Tambe et al, “Algorithm-Hardware Co-Design of Adaptive Floating-Point Encodings for Resilient Deep Learning Inference” DAC 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors have conducted a large number of tests on LLMs, and show that the W4A8 PTQ works well. The results are solid.\n\n2. The authors have done model analysis and found interesting relations between distribution of outliers of activation and weight, variation in activation diversity and input sequence lengths in OPT and LLaMa.", "reasons_to_reject": "1. The experiment results cannot support the authors' argument well. The INT4 AQAS shows similar results as INT4 SQ in all experiments carried out in the paper. There is no good justification for using AQAS rather than SQ.\n\n2. This paper lacks of novelty. I understand that this paper focuses on LLMs, but there are similar investigations around W4A8 PTQ for language models: https://openreview.net/pdf?id=tvDRmAxGIjw, and clamping denormal strategy: https://arxiv.org/pdf/1909.13271.pdf.", "questions_to_authors": "What is the unique contribution of this paper compared with similar investigations around W4A8 PTQ for language models: https://openreview.net/pdf?id=tvDRmAxGIjw, and clamping denormal strategy: https://arxiv.org/pdf/1909.13271.pdf?", "paper_topic_and_main_contributions": "This paper focuses on using post-training quantization (PTQ) in LLMs to enhance computational efficiency. . The authors conduct study on applying W4A8 PTQ in specific LLM models (OPT and LLaMa) inference.\n\nThis main contribution of this paper is that it pushes the limit of LLMs inference compute efficiency. Other contributions include:\n1. The authors combine exitsing methods SQ and AWQ to control the distribution of outliers of activation and weight.\n2. The authors find there is relation between variation in activation diversity and input sequence lengths in OPT and LLaMa, and propose Sequence-Length-Aware Calibration (SLAC) to mitigate accuracy losses.\n3. The authors propose a new data format dINT to reduce underflow in low precision.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {}}, {"review_id": "PbIfvvF2HJ", "reviewer": "Reviewer_XAcF", "summary": "", "questions": "1.\tWhy wasn't the AWQ method compared in Table 4 and 5? \n2.\tSLAC is conspicuously absent from most experiments without being thoroughly analyzed. However, it does provide some assistance in improving results, as evidenced in table 4 during the LLaMA experiments. The reason behind excluding SLAC from other experiment sets remains unclear. Does this imply that SLAC generally does not contribute to accuracy improvement in most cases? \n3.\tAlthough the paper highlights a twofold increase in hardware efficiency, the provided experiments primarily focus on accuracy. Where can we find evidence or information demonstrating the improved hardware efficiency?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the insightful feedback provided by the reviewers. In response to the raised questions, our answers are as follows.\n\n> **Reasons To Reject 1.** In Section 3.1, there are inconsistencies in the description and arrangement of Figure 1(b). The description states that the min-max range of activation should be at the top and weight at the bottom, but the figure is presented in a left-right format. Additionally, the legend in Figure 1(b) only provides two descriptions, despite there being four lines in the graph. It can be inferred that the lower part represents the min value, and the upper part represents the max value, but it would be advisable to visually distinguish these parts using different colors or markers for improved clarity.\n\nThank you for your detailed comments. We apologize for any confusion caused by the incorrect placement and description of Figure 1 (b). In line with the reviewer’s suggestions, we will align the description and arrangement of Figure 1 (b) for clarity. Additionally, we will update in the final manuscript the legend to accurately represent all four lines, improving the visualization of four lines in the graph.\n\n&nbsp;\n\n> **Reasons To Reject 2**. Upon analyzing Figure 2, I perceive a limited distinction between SmoothQuant and AQAS. This similarity becomes more pronounced as the numbers increase, ultimately resulting in a diminishing difference and even intersecting towards the end.\n\nThank you for your detailed question regarding the distinctive differences between SmoothQuant (SQ) and AQAS. We aim to elucidate the differences between SQ and AQAS by highlighting that they utilize different **scale values**, leading to significant discrepancies in the **quantization layer output error**, and these differences consequently impact **text generation performance.**\n\nTo illustrate the impact of different scaling methods, we measured the scale values and the subsequent output quantization errors (via SQNR) for each layer in both SQ and AQAS, as presented in Table R1. Notably, in Layers 5 and 10, AQAS employs smaller scale values than SQ, corroborating the trends in absmax of weight seen in Figure 2. **These contrasting scale values lead to substantially higher output quantization errors** in the initial layers for SQ compared to AQAS. Even though similar scale values emerge in later layers, these variations result in markedly different text generation performance, as evidenced by the PPL comparisons between SQ and AQAS.\n\nOPT-6.7B | Layer | 5 | 10 | 20 | 30 | W4A8 PPL\n-- | -- | -- | -- | -- | -- | --\nSQ | Scale | 22.55 | 22.39 | 25.92 | 31.53 | 1586.83\nSQ | SQNR | 12.51 | 4.29 | 11.47 | 9.71 | -\nAQAS | Scale | 13.44 | 14.91 | 21.69 | 28.45 | **13.42**\nAQAS | SQNR | **16.76** | **5.91** | **11.90** | **9.93** | -\n\nTable R1. Comparison of scale for outlier channel and per-layer SQNR between SQ and AQAS in OPT-6.7B and wikitext-2 PPL results of W4A8 PTQ OPT-6.7B (SQNR: signal-to-quantization-noise ratio, higher the better)\n\nTo assess the impact of the analys on model accuracy, we conducted a performance comparison between SQ and AQAS on the recently released LLaMA-2 model. As seen in Table R2, AQAS consistently outperforms SQ across various tasks, including wikitext, MMLU, and CSQA, when applied to the LLaMA-2 model. This confirms that AQAS achieves significantly higher performance compared to SQ, reinforcing its robustness and efficacy.\n\nLLaMA-2-7B (W4A8) | WikiText-2 | MMLU | CSQA\n-- | -- | -- | --\nWeight Scaling | PPL | Acc | Acc\nFP | 5.42 | 41.6 | 71.76\nSQ | 6.82 | 38.6 | 68.55\nAQAS | 6.46 | 40.7 | 70.40\nSQ+OPTQ | 6.65 | 38.9 | 69.51\nAQAS+OPTQ | **6.45** | **41.1** | **70.94**\n\nTable R2. Performance comparison between SQ and AQAS across wikitext, MMLU, and CSQA (PIQA, winogrande, ARC_easy) tasks with LLaMA-2-7B.\n\n&nbsp;\n\n>**Reasons To Reject 3.**  The paper lacks detailed information and fails to provide a comprehensive description of the SLAC method. For instance, it does not present the formula or describe the process involved in implementing SLAC.  \n**Reasons To Reject 6.** SLAC is noticeably absent from the majority of experiments and lacks thorough analysis. Its presence is only observed in Table 4 during the LLaMA experiments, where SLAC shows some advantages. The reasoning behind excluding SLAC from other sets of experiments is unclear. It is recommended to include more analysis and comparisons of SLAC to provide a comprehensive evaluation.  \n**Question 2.** SLAC is conspicuously absent from most experiments without being thoroughly analyzed. However, it does provide some assistance in improving results, as evidenced in Table 4 during the LLaMA experiments. The reason behind excluding SLAC from other experiment sets remains unclear. Does this imply that SLAC generally does not contribute to accuracy improvement in most cases?\n\nThank you for asking for further clarification on SLAC. SLAC is a method emerged from observation that the mismatch in the sequence length between the calibration and the target tasks results in significant accuracy degradation, which has been neglected for the careful study in previous work. To address the reviewer's questions comprehensively, we have prepared detailed responses with two sections.\n\n**[Motivation and detailed description of SLAC]**\n\nSLAC aligns the sequence length of the application task with that of the PTQ calibration dataset, thereby enhancing the PTQ performance of the target task.\n\n- This approach is motivated by our discovery that variations in activation diversity, depending on the input sequence length, have a significant impact on the PTQ calibration process, as shown in Table 1.\n- To mitigate such impacts, SLAC aligns the expected sequence length in the target task inference phase with the sequence length of the calibration dataset.\n- The specific process of our proposed SLAC method is as follows: 1) Measure the expected sequence length for the target application task during the inference phase. 2) In the OPTQ calibration process, align the sequence length of the calibration dataset with the previously measured sequence length.\n- By following this approach, we can apply task-specific PTQ calibration, thereby enhancing the accuracy of the quantized model's inference under the target task in a robust manner.\n\n**[Experimental results and analysis of SLAC]**\n\nWe conducted experiments applying the SLAC method to the MMLU task. Through this experiment, we not only observed performance improvement but also conducted an analysis of SLAC with individual sub-tasks within MMLU. Below is a summary of our experimental findings and analysis.\n\n- In Table R3, **SLAC**: adjusting the sequence length of OPTQ calibration dataset based on the averaged sequence length (698) across every subject's dataset in MMLU, **default**: using a default sequence length (2048) employed in OPTQ. The evaluation was conducted both with and without the AQAS scaling method.\n\n   - In both LLaMA-7B and LLaMA-13B, we observed that the SLAC method improves the performance of MMLU. Notably, the performance enhancement was seen in both OPTQ and AQAS-scaled OPTQ, confirming that the SLAC method can be orthogonally applied to both calibration-based PTQ approaches (OPTQ) and the scaling-based methods combined with OPTQ (AQAS+OPTQ).\n\n- To investigate the effectiveness of the SLAC methodology, we selected a subset of subjects from the MMLU dataset, each characterized by different average sequence lengths. Our findings, summarized in Table R4, compared the accuracy of two PTQ models: SLAC-1978 and SLAC-587. These models were calibrated to the average sequence lengths specific to the History and Chemistry subjects, respectively, both of which are sub-tasks within MMLU.\n\n   - The results revealed a consistent trend: models calibrated to match the average sequence length of each subject yielded higher accuracy. This finding underscores the efficacy of the SLAC methodology, emphasizing that **task-specific customization of calibration sequence length** can lead to improved performance in W4A8 PTQ.\n\nMMLU | Calibration (Seq Len) | LLaMA-7B | LLaMA-13B\n-- | -- | -- | --\n|OPTQ | default (2048) | 27.4 | 43.0\n|  | SLAC (698.1) | **28.8** | **43.7**\n|AQAS+OPTQ | default (2048) | 31.0 | 44.2\n|  | SLAC (698.1) | **31.6** | **44.6**\n\nTable R3. W4A8 MMLU accuracy comparison between default calibration sequence length (2048) and SLAC (698.1) method for total MMLU dataset using OPTQ (+AQAS) PTQ method.\n\nMMLU single subject | Calibration (Seq Len) | History (seq len avg. 1978) | Chemistry (seq len avg. 587)\n-- | -- | -- | --\n| OPTQ | SLAC (1978) | **51.9** | 27.1\n| OPTQ | SLAC (587) | 51.2 | **30.4**\n\nTable R4. Results of varying calibration sequence length across each subject (History, Chemistry) of MMLU with LLaMA-7B (W4A8)\n\n&nbsp;\n\n> **Reasons To Reject 4.** The paper highlights an achievement of a twofold increase in hardware efficiency. However, it fails to include any experiments specifically demonstrating this improvement in hardware efficiency. Rather, the paper solely focuses on presenting accuracy results without a corresponding set of experiments showcasing the enhancement in hardware efficiency.  \n**Question 3.** Although the paper highlights a twofold increase in hardware efficiency, the provided experiments primarily focus on accuracy. Where can we find evidence or information demonstrating the improved hardware efficiency?\n\nAs highlighted by the reviewer, we have quantified the proposed dINT4xINT8 through a comparative analysis of area and power consumption against the state-of-the-art configuration, INT8xINT8 in Table 2. Our study primarily explored whether the dINT4-based MAC unit could offer performance gains proportionate to the bit-precision reduction over the INT8-based unit. As noted in previous research [R1], lowering bit-precision can proportionally enhance both area and power efficiency, leading to increased throughput (e.g., transitioning from FP8-FP8 to INT4-INT4 boosts throughput 4x, as per [R2]). Table 2 shows that by reducing weight precision from 8-bit to 4-bit, our dINT format yields 2x area and 2.5x power efficiency improvements, underscoring dINT's hardware efficiency to offer scalable performance despite its representation (Eq. 5) distinct to INT.\n\n[R1] M. Horowitz. \"Energy table for 45nm process\", Stanford VLSI wiki. [Online]. Available: https://sites.google.com/site/seecproject  \n[R2] Agrawal et al., “9.1 A 7nm 4-Core AI Chip with 25.6TFLOPS Hybrid FP8 Training, 102.4TOPS INT4 Inference and Workload-Aware Throttling”, ISSCC 2021\n\n&nbsp;\n\n> **Reasons To Reject 5.** Why wasn't the AWQ method compared in Tables 4 and 5?  \n**Question 1.** Why wasn't the AWQ method compared in Table 4 and 5?\n\nWe apologize for the confusion caused by not reporting the results of the AWQ method in Tables 4 and 5. We compared the performance of the AWQ method with SQ and AQAS methods in W4A8 performance for CSQA (PIQA, ARC_easy) and MMLU evaluation.\n\n- In the case of OPT-13B in Table R5, significant performance degradation is observed with AWQ+OPTQ. This result can be attributed to the AWQ scaling approach, which only considers weight quantization and exacerbates the difficulty of activation quantization in OPT, which has a wide dynamic range compared to LLaMA.\n\n- In the LLaMA model, which has opposite activation-weight dynamic ranges compared to OPT, the challenge of activation quantization is less pronounced, resulting in less performance degradation. However, its performance still falls short compared to AQAS in both CSQA and MMLU benchmarks.\n\nCSQA/MMLU | W/V-format | PIQA | ARC-easy | PIQA | ARC-easy | MMLU-avg\n-- | -- | -- | -- | -- | -- | -- \n| scale method |  model  | OPT-13B | OPT-13B | LLaMA-13B | LLaMA-13B | LLaMA-13B\n| - | FP | 75.90 | 67.13 | 78.78 | 74.58 | 47.15\n|- | INT4 | 66.16 | 50.34 | 77.80 | 70.41 | 42.95\n|SQ + OPTQ | INT4 | 75.52 | 66.33 | 78.56 | 71.21 | 43.12\n|AWQ + OPTQ | INT4 | 72.09 | 63.08 | 78.56 | 72.64 | 44.55\n|AQAS + OPTQ | INT4 | **75.84** | 65.28 | **78.84** | **73.23** | 44.23\n|AQAS + OPTQ | dINT4 | 75.41 | **66.12** | 78.02 | 72.77 | **44.73**\n\nTable R5. CSQA (PIQA, ARC_easy) and MMLU evaluation comparison between SQ, AWQ, and AQAS methods in W4A8 precision\n\nThrough comparison with AWQ, we can confirm that AQAS consistently delivers **high performance across both OPT and LLaMA models,** demonstrating its robustness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is mostly well-written and easy to follow. The frame is clear and easy to understand. The methods are described with enough detail, and the results are well presented and discussed. The paper analyzes in great detail the changes of weight and activation in each layer of the LLMs during the quantization process and gives solutions for each of the problem. ", "reasons_to_reject": "1.\tIn Section 3.1, there are inconsistencies in the description and arrangement of Figure 1(b). The description states that the min-max range of activation should be at the top and weight at the bottom, but the figure is presented in a left-right format. Additionally, the legend in Figure 1(b) only provides two descriptions, despite there being four lines in the graph. It can be inferred that the lower part represents the min value, and the upper part represents the max value, but it would be advisable to visually distinguish these parts using different colors or markers for improved clarity. \n2.\tUpon analyzing Figure 2, I perceive a limited distinction between SmoothQuant and AQAS. This similarity becomes more pronounced as the numbers increase, ultimately resulting in a diminishing difference and even intersecting towards the end. \n3.\tThe paper lacks detailed information and fails to provide a comprehensive description of the SLAC method. For instance, it does not present the formula or describe the process involved in implementing SLAC. \n4.\tThe paper highlights an achievement of a twofold increase in hardware efficiency. However, it fails to include any experiments specifically demonstrating this improvement in hardware efficiency. Rather, the paper solely focuses on presenting accuracy results without a corresponding set of experiments showcasing the enhancement in hardware efficiency. \n5.\tWhy wasn't the AWQ method compared in Table 4 and 5? \n6.\tSLAC is noticeably absent from the majority of experiments and lacks thorough analysis. Its presence is only observed in table 4 during the LLaMA experiments, where SLAC shows some advantages. The reasoning behind excluding SLAC from other sets of experiments is unclear. It is recommended to include more analysis and comparisons of SLAC to provide a comprehensive evaluation.\n", "questions_to_authors": "1.\tWhy wasn't the AWQ method compared in Table 4 and 5? \n2.\tSLAC is conspicuously absent from most experiments without being thoroughly analyzed. However, it does provide some assistance in improving results, as evidenced in table 4 during the LLaMA experiments. The reason behind excluding SLAC from other experiment sets remains unclear. Does this imply that SLAC generally does not contribute to accuracy improvement in most cases? \n3.\tAlthough the paper highlights a twofold increase in hardware efficiency, the provided experiments primarily focus on accuracy. Where can we find evidence or information demonstrating the improved hardware efficiency?\n", "paper_topic_and_main_contributions": "The paper addresses the quantization problem in Large Language Models by targeting the 4-bit weight and 8-bit activation (W4A8) quantization. It proposes activation-quantization-aware scaling (AQAS) to jointly quantize the weight and activation. Additionally, sequence-length-aware calibration (SLAC) is introduced to match sequence lengths and align calibration data ranges with target tasks. The dINT method is utilized to denormalize the representation.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to optimize the PTQ of LLMs. Initially, reviewers had several concerns regarding the clarity, less novelty, and unclear results comparison. But the authors have addressed them, therefore, I recommend acceptance of this paper.
Decision: Accept-Main