Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models
Authors: Jung Hwan Heo
Jeonghoon Kim
Beomseok Kwon
Byeongwook Kim
Se Jung Kwon
Dongsoo Lee
Venue: ICLR 2024 poster (2024)
Keywords: large language models
quantization
model compression
Primary area: generative models
TL;DR: Augmenting low-bit weight quantization methods with adaptive per-channel quantization

Abstract: Large Language Models (LLMs) have recently demonstrated a remarkable success across various tasks. However, efficiently serving LLMs has been a challenge due to its large memory bottleneck, specifically in small batch inference settings (e.g. mobile devices). Weight-only quantization can be a promising approach, but sub-4 bit quantization remains a challenge due to large-magnitude activation outliers. To mitigate the undesirable outlier effect, we first propose per-IC quantization, a simple yet effective method that creates quantization groups within each input channel (IC) rather than the conventional per-output channel (OC). Our method is motivated by the observation that activation outliers affect the input dimension of the weight matrix, so similarly grouping the weights in the IC direction can $\textit{isolate outliers to be within a group}$. We also find that activation outliers do not dictate quantization difficulty, and inherent weight sensitivities also exist. With per-IC quantization as a new outlier-friendly scheme, we then propose Adaptive Dimensions ($\textbf{AdaDim}$), a versatile quantization framework that can adapt to various weight sensitivity patterns. We demonstrate the effectiveness of AdaDim by augmenting prior methods such as Round-To-Nearest and GPTQ, showing significant improvements across various language modeling benchmarks for both base (up to $+4.7\%$ on MMLU) and instruction-tuned (up to $+10\%$ on HumanEval) LLMs.

Published as a conference paper at ICLR 2024
RETHINKING CHANNEL DIMENSIONS TO ISOLATE OUTLIERS FOR LOW-BIT WEIGHT QUANTIZATION OF LARGE LANGUAGE MODELS
Jung Hwan Heo x11 Jeonghoon Kim $2 Beomseok Kwon"
Byeongwook Kim² Se Jung Kwon Dongsoo Lee²
1 University of Southern California
2 NAVER Cloud
ABSTRACT
Large Language Models (LLMs) have recently demonstrated remarkable success across various tasks. However, efficiently serving LLMs has been challenge due to the large memory bottleneck, specifically in small batch inference settings (e.g. mobile devices). Weight-only quantization can promising approach, but sub-4 bit quantization remains a challenge due to large-magnitde activation outliers. To mitigate the undesirable outlier effect, we first propose per-IC quantization, simple yet effective method that creates quantization groups within cach input channel (IC) rather than the conventional per-output-chanell (per-OC). Our method is motivated by the observation that activation outliers affect the input dimension of the weight matrix, so similarly grouping the weights in the IC direction can isolate outliers within a group. We also find that activation outliers do not dictate quantization difficulty, and inherent weight sensitivities also exist. With per-IC quantization as a new outlier-friendly scheme. we propose Adaptive Dimensions (A (AdaDim), versatile quantization framework that can adapt to various weight sensitivity patterns. We demonstrate the effectiveness of AdaDim by augmenting prior methods such as Round-To- To-Neerest and GPTQ, showing significant improve- ments across various language modeling benchmarks for both base (up to +4.7% MMLU) and instruction-tuned (up to +10% on HvmanEval) LLMs. Code is available at: https: //gi thub. com/ johnheo/ addadm-mlmml
INTRODUCTION
The rise of Transformers (Vaswani et al., 2017) has led a remarkable success of Large Language Models (LLMs) (Brown et al., 2020; Touvron et al., 2023), achieving on par excelling human-level performance on various tasks (Bubeck et al., 2023). However, with the rapid scaling in model size, efficiently serving LLMS has become significant challenge. Specifically, the autoregressive decoding of an LLM is limited by memory bandwidth rather than compute (Kim et al., 2023b).
Low-bit weight quantization is a promising approach reduce storage and accelerate inference latency (Park et al., 2022). By reducing weight precision. one can pack multiple weights under equal bit width to increase memory I/O (e.g., 4x for FP16 + INT4). However, sub-4 bit quantization remains challenge due to the presence of activation outliers in billion parameter scale modern LLMs (Dettmers et al., 2022; Bondarenko et al., 2023). Prior works have sought to mitigate large activations amplifying the rounding errors in the corresponding weights, where a small subset of weights is much more sensitive than others (Lin et al., 2023; Dettmers et al., 2023; Yuan et al., 2023).
In this paper, we first propose per-input-channel (per-IC) quantization, simple yet effective scheme to address the activation outlier problem. The motivation behind per-IC quantization lies in how activation outliers impact specific input channels of the weight matrix. Thus, similarly grouping the weights in the IC direction can effectively isolate the effect of these outliers (ref. Figure 1). Such an approach is particularly feasible in the weight-only quantization context because it does not rely specialized INT8 GEMM kernels impose the Per-OC grouping constraint.
"Equal contribution
I Work done during an internship at NAVER Cloud 1 Corresponning authorjconghoon.som samuel @ aaammel@ggall.cm

---

Published as a conference paper at ICLR 2024
Activation outliers cause sensitive Weight outliers
Outliers in all groups
Outlier is isolated
Cout
Cin
X
W
Per Output Channel Quantization (Standard)
Per Input Channel Quantization (Ours)
Figure 1: Per-input-channel quantization. Activation outliers that affect certain imput channels (ICs) amplify quantization errors. Such sensitive ICs exist all groups in the conventional per- output-channel (per-OC) quantization. With our per-IC scheme, the outlier effect isolated.
By unlocking input dimension as new design parameter that helps to sidestep the activation outlier problem, we propose AdaDim, versatile quantization framework that can adapt to different weight sensitivity scenarios. We now organize our contributions below:
From analyzing the structural relationship between activation outliers and sensitive weights, we propose per-IC quantization. Unlike traditional per-OC quantization where the outlier effect is pervasive, per-IC quantization isolates the outlier effect.
Activation outliers emerge only in subset of the network, prompting selective application of per-IC quantization. We present Adaptive Dimensions (AdaDim), framework that adapts to various weight sensitivity patterns by using both per-IC and per-OC quantization.
Augmenting weight-only quantization methods such as Sound-To-Nearest (RTN) and GPTQ with AdaDim results in significant boost various language modeling abilities and special- ized tasks (math, coding) for both base and instruction-tuned LLMs.
2 RELATED WORK
Generative inference of an LLM is heavily memory bound (Sheng et al., 2023). In single-batch inference setting which is usually common for mobile devices, billions of floating point weights need to be read in from VRAM into on-chip cache to generate single next token. Thus, smaller weight precision can make memory I/O more efficient by packing multiple values under equal bit width.
2.1 LLM QUANTIZATION
Quantization is an effective method to reduce weight precision, accelerating inference and reducing storage. INT8 quantization maps both activations and weights to lower precision, so that specialized GEMM engines can effectively accelerate arithmetic computation for large matrix mulplications (Xiao al., 2022). Thus in autoregressive decoding workloads as in modern LLMs (Touvron et al., 2023), INT8 quantization is helpful for large batch sizes where compute the bottleneck, but not for small batches (let alone single batch) where memory is the bottleneck.
An alternative to address the memory bottleneck is weight-only quantization (Park et al., 2022), which leaves activations in high precision (c.g.. FP16) while pushing the weights to even lower precision (S 4 bits). We focus on weight-only quantization to accelerate memory I/O rather than compute, small-batch inputs do not saturate the powerful compute capacity of modern GPUs(Kim et al., 2023b). In order to preserve accuracy while minimizing the number of bits, group-wise per-channel quantization is commonly used (Shen et al., 2020; Kim et al., 2023a). It is fine-grained quantization scheme where group of consecutive weights inside each channel share the same quantization parameters.
2.2 THE ACTIVATION OUTLIER PROBLEM
INT8 quantization. Low-bit transformer quantization is complicated by the presence of activation outliers (Bondarenko et al., 2023). First characterized OPT models (Zhang et al., 2022) by Dettmers al. (2022), activation outliers emerge in small subset of hidden dimensions and have up 20x
2

---

Published as a conference paper at ICLR 2024
larger magnitude than other channels. Such outliers increase quantization range. which results in most of the activation values being mapped to the same quantization bucket, making quantization ineffective. To address this, recent works have attempted to suppress smooth such outliers (Wei et al., 2022b; Xiao et al., 2022) or use training (Liu et al., 2023b; Lee et al., 2023) to achieve acceptable INT8 accuracy.
Weight-only quantization. Although activation outliers can be seemingly unrelated, it has been discovered that they also make weight quantization difficult (Dettmers et al., 2023; Lin et al., 2023). When large activations are multiplied by dequantized weights that inherently possess rounding errors, even small errois can be amplified by the large inputs, causing non con-trivial perturbations to the layer output. To address the undesirable outlier effect, existing per-channel quantization methods keep outliers in higher precision via mixed- preecdsioniiionn formats (Dettmers et al., 2023; Kim et al., 2023b) or use scaling to give higher importance during quantization (Lin et al., 2023).
Reviewing renewed GPTQ. A pioneering work in LLM weight-only quantization is GPTQ (Frantar et al., 2022) which does iterative per-channel quantization while compensating the rounding errors with Hessian-based approximation. Contrary to what is explained in the original GPTQ paper as Arbitrary Order Insight, it has been found that the order in which channels are quantized are actually crucial (i.e.. --act --rdrr option in their repo'). ). Because salient (sensitive) weight channels compensating the error of non-salient weight channels is undesirable, the activation reordering technique sorts the channels in the descending order of activation magnitude to quantize salient channels first (Figure 8). Although this provides better accuracy, reordering incurs non-trivial hardware overhead due to quantization scales becoming non-contiguous in memory (Lin et al., 2023).
To avoid inefficient random memory access, GPTQ added another option (i.e., -it.----tttcgrouuppe that pre-computes the quantization parameters before reordering. This enables activation-aware weight updates while quantization parameters are still contiguous in memory. However, we find that such static quantization constraint fails to capture the changing weight dynamics that occur during iterative weight updates, leading to suboptimal accuracy (Table 2b).
METHODOLOGY
Overview. In this section, we first structurally analyze the relationship between activation outliers and weight sensitivity patterns (Figure 2). We then introduce per-IC quantization isolate the outlier effect, empirically validating its effectiveness in a preliminary study (Table 1). Finally, we propose Adaptive Dimensions (AdaDim), a versatile framework that can automatically choose whether to use per-IC or Per-OC quantization for each layer of the network.
Blko attn. v_proj
B1k15. wtw.mmppuuu. __pp_proj
B1k31. wwp.mppppp up_prro
dmoddl
Cin
Cout
Mild activations > Less sensitive rows; dominant sensitivity dimension (IC or OC) can change across network depth
Outlier activations Row (IC) sensitivity)
Figure 2: Weight sensitivity patterns of LLaMA-V2-7B, Darker colors indicate larger activation magnitude (red), and higher weight sensitivity (blue). Sensitivity is computed with fisher information approximated by the squared of the gradient. We downsample the grids with 16x16 maxpool kernel and take the log scale for clarity. Left: The presence of activation outliers lead to sensitive rows. Right: Mild activations lead to less sensitive rows, while the dominant sensitivity dimension (row or column) can change across network depth even for the same module.
3

---

Published as a conference paper at ICLR 2024
Table 1: Isolating outliers with per-IC quantization. We observe the largest activations before attn. qkv and mlp. down projections for LLaMA-V2 base models. Selectively applying per-IC quantization RTN only where outliers are present gives the most improvement in perplexity and multi-task in-context learning performance (MMLU) on INT4 with group size 128.
Model
Baseline
Module to apply Per-IC quant.
Metric
FPI6
Size
(RTN, Per-OC) (1)attn.qqy qkv (2)mpp. down (T)&(2
All
Wiki-2 ppl. (L)
8.79
9.22
9.17
9.11
9.09
9.11
7B
MMLU 5-shot (f)
45.98
44,54
44.77
44.7
45.21
44.38
Wiki-2 ppl. (L)
7.89
8.13
8.12
8.11
8.10
8.13
13B
MMLU 5-shot (f)
55.61
54,43
54.76
54.90
54.97
54.67
3.1
ANALYZING WEIGHT SENSITIVITY PATTERNS
Although activation outliers are prevalent in modern LLMs (Wei al., 2022c; 2023), they do not occur in every layer of the network. We investigate the structural relationship between activation outliers and sensitive weight outliers in the LLaMA-VE V2 base model family. We define weight sensitivity by using fisher information following Kim al. (2023b), which can be approximated by the squared of the gradient obtained by using a calibration set. Here, sensitivity is defined per weight instead of per channel, as we infer channel sensitivity from the sensitivity of its constituent individual weights.
Our preliminary study shows that the largest activations occur before QKV attention projection and DOWN feedforward projection (Figure 9). When visualizing the sensitivity of the corresponding weight matrices, we find that the hidden dimensions where activation outliers emerge have high correlation sensitive weight channels (left of Figure 2). However, even when outliers exist before Q and K projections, they can exhibit dominant OC sensitivity (Figure 10a). Thus, activation outliers cause sensitive rows but does not necessarily dictate the overall sensitivity.
Besides, when activation outliers do not exist, we observe that the weight matrix can have a mixture of sensitive IC and OC channels. Moreover, the dominant sensitivity dimension can actually switch across network depth, even if it is the same module (right of Figure 2). These observations motivate a weight quantization method that can adapt to different sensitivity scenarios, which is largely conditioned by the existence of activation outliers.
3.2 PER-IC QUANTIZATION
One common thread of existing per-channel quantization methods is their usage of per-OC channel quantization. When activation outliers emerge in certain hidden dimensions, the amplification effect is permeated across all quantization groups for per-OC quantization (Figure 1). In contrast, grouping within each IC yields a 1:1 mapping between hidden dimension to a quantization group which isolates the outlier effect to be within group. Thus, per-IC quantization can be more effective method that mitigates the outlier problem.
Motivation.
Selective outlier isolation. To first verify intuition that per-IC quantization isolate the undesirable activation outlier effect, we augment the standard RTN method. As Table 1, utilizing Per-IC quantization for modules influenced by activation outliers can effectively improve both perplexity and multi-task in-context learning ability of an LLM. We also observe that selective utilization per-IC quantization is important; naively applying it to all modules can actually hurt baseline RTN performance from 44.54 to 44.38, while selectively applying it to QKV and DOWN modules improve up 0.67 on average MMLU score. Thus, it is desirable to search for a selective scheme to apply Per-IC quantization, motivating our adaptive approach in Section 3.3.
3.3 ADAPTIVE PER-CH CHR-CCANNELL QUANTIZATION
Optimization objective. Beyond heuristically determining the channel quantization dimension by looking at the sensitivity patterns offline, we propose an adaptive method that can achieve this on the fly during quantization. For each linear layer WN (linear projection W at layer E) in neural network, we formulate channel quantization as a simple binary selection problem that chooses the optimization parameter dim either one of the two (OC IC) dimensions. To measure which dimension is more effective, we adopt the widely used reconstruction error metric (Nagel et al., 2020; Li et al., 2021), yielding the optimization objective as
4

---

Published as a conference paper at ICLR 2024
dim
arg min L(dim), C(dim) = 1[ (W.) WeXells (omçio)
(I)
where the reconstruction error L(dim) is defined as the L2 distance between the outputs of a full precision layer W X; and that of the quantized layer calibration set by randomly sampling from the pretraining corpus (e.g. The Pile). Here, the per- channel quantization function Qaim can either create quantization groups per-OC (standard) or per-IC (proposed), as illustrated in Figure 1. Since the search space of the dimension parameter is only two, AdaDim requires a very small number of forward passes to determine the optimal dimension.
To obtain X, we curate small
Augmenting RTN and GPTQ. Applying our AdaDim to RTN is straightforward: we independently quantize the full precision weights two different times (per-IC and per-OC), then choose the dimension that yields a lower reconstruction ertor. Hence, we always search for the optimal dimension with RTN and optionally apply GPTQ in the selected dimension. As the GPTQ (Frantar et al., 2022) algorithm consists of iteratively 1) computing the quantization error of a weight channel and 2) applying hessian-bart based weight updates, using our per-IC variant simply requires executing the quantization step 1) with per-IC RTN. We also considered applying AdaDim to AWQ but found it incompatible (see Appendix A,2 for further discussion). Furthermore, per-IC GPTQ can obtain additional benefits from the reordering scheme, which we detail further in Appendix A.I.
4 EXPERIMENTS
4.1 EVALUATION SETUP
Quantization sefting. In this work, we focus on weight-only per-channel (w/ uniform asymmetric setting) quantization with group size of 128, which is shown to be good accuracy/latency trade-off point (Dettmers& & Zettlemoyer, 2022). We also focus on INT3 quantization since shows the biggest relative improvements while INT4 quantization yields comparable performance across methods (see Appendix C). Following the settings of GPTQ and AWQ, we use a small calibration set from the Pile (Gao et al., 2020) dataset. For Instruction-tunce models, we also experiment with aask-specific calibration sets, which is randomly sampled from the training split of the respective tasks.
Models. For base model evaluation, we version 2 (V2) instead V1 of the LLaMA (Touvron et al., 2023) family with the exception of 33B since it is not yet publicly available. We further benchmark instruction-tuned models from the WizardIMM (Xu al., 2023) family, which is series of performant LLaMA-V2 models fine-tuned with specialized instruction dataset curation. We use WizardMath and WizardCoder-Python (Luo et al., 2023a;b) to test mathematical reasoning and code generation ability.
Tasks. Following previous literature (Dettmers et al., 2022: Yao et al., 2022), we evaluate the quantized models on zero-shot commonsense reasoning (CSR) ability, including PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2019), and ARC-easy (Clark al., 2018). Besides common sense abilities, we also evaluate multi-task generalization ability with five-shot setting (in-context learning) on MMLU (Hendrycks et al., 2020), which consists of 57 tasks including STEM, humanities, social science, and others (business, health, etc.). We used in--val-harnese (Gao et al., 2021) for CSR and reproducible repo² for MMLU. We report the average score of the four affremmntioned CSR tasks and the total average of MMLU.
Following (Luo et al., 2023a;b), we evaluate instruction-tuned models on mathematical reasoning with Chain-of-Thought (CoT) prompting (Wei et al., 2022a) on GSM8K (Cobbe et al., 2021) dataset, set of grade school math questions. We also test code generation with greedy decoding the HvmanEval (Chen et al., 2021) dataset, which includes hand-crafted programming problems written in Python. To curate domain-specific calibration sets, from MBPP (Austin et al., 2021), I), a crowd-sourced entry level Python problem set.
the training splits from GSM8k and
Baselines. We benchmark against vanilla round-to-neared quantization (RTN), GPTQ (Frantar et al., 2022), and AWQ (Lin et al., 2023) for LLM weight quantization. We thorougly ablate GPTQ's most up-to-date design parameters as in Table 2 to find a strong baseline for modern LLMs such as LLaMA-V2 (Touvron et al., 2023). We first observe that heuristically fixing which module to apply Ie quantization offline is inferior to the optimization-bai based (adaptive) setting where
5

---

Published as a conference paper at ICLR 2024
Table 2: GPTQ ablation using LLaMA-V2-13E with MMLU 5-shot average score. We that adaptive method is superior to the heuristic- based approach. Although reordering by itself gives the best results for OC cfg., it hardware inefficient, Thus, our final design is marked in green.
(a) Heuristic-based. Results when the decision use per-IC quantization with reorder fixed accord- ing to heuristic from an offline observation as Table 1.
(b) Optimization-base
Results when AdaDim
adaptively chooses either per-OC/per- quantiza- tion for each layer by minimizing the reconstruction error.
per-IC module
Accuracy (%)
ctg.
Accuracy (%)
none
50.56
IC
default
reorder
attn.qkv
51.47
OC
mip.down
50.4
default
51.07
51.54
qkv & down
48.93
reorder
52.18
52.68
all
51.4
reorder static
51.32
52.27
dimensions are chosen on-the-fly. Once the optimal dimension is selected with RTN, there are various combinations to apply the GPTQ algorithm: for OC dimension, test default (baseline), reorder (activation reordering). and the reorder + static (hardware-effickh test static groups for IC since it is already hardware-efficient. We empirically determine that static reordering for per-OC and reordering for per-IC yields the best accuracy-efficiency trade-off.
reordering) option. We do not
4.2 BASE MODEL QUANTIZATION
Base models serve as the fundamental backbone for modern L.Mss, which demonstrated remarkable capabilities in general knowledge understanding (Bubeck et al., 2023). To test the effect of quantiza- tion on such models, we evaluate the LLaMA family that widely used today (Taori al., 2023; Liu al., 2023a). We evaluate INT3 per-channel quantization with group size 128 (w3g128) sense reasoning with O-shot and MMLU with 5-shot (in-context learning). As in Figure 3, AdaDim enables notable improvements in existing methods such RTN and GPTQ. Remarkably, augmenting RTN with per-IC quantization yields a 4.7 % MMLU accuracy boost on the 7B model, surpassing both AWQ and GPTQ.
common
LLaMA
AdaDim Incompatible
w/ AdaDim
w/o AdaDim
(w3g128)
42
MMLU
40
52
54
66
5-shot
AWD
RTN
GPTQ
AWQ
RTN
GPTQ
AWQ
GPTQ
AWQ
RTN
GPTQ
69.5
72.0
66.5
69.0
74.0
71.5
CSR
66.0
68.5
735
D-shot Avg.
71.0
65.5
73.0
70.5
AWO
RTN
GPTQ
AWD
RTN
GPTQ
V2-7B
V2-13B
V1-33B
V2-708
Figure 3: Base model results. Evaluating the effectiveness of our AdaDim framework for LLaMA base models on Massive Multi-task Language Understanding (MMLU) and commonsense reasoning (CSR) tasks. Performance boosts (in green) indicate additional gains from adaptively switching to per-IC quantization. We observe notable gains over the original per-OC versions of RTN and GPTQ (Frantar al., 2022), often matching or even surpassing AWQ (Lin et al., 2023).

---

Published as a conference paper at ICLR 2024
Table We evaluate Vicuna, family of instruction-tunct LLaMA models with improved language modeling. AdaDim applied to RTN and GPTQ are trailed by postfix '-ada' and highlighted in green.
Vicuna-V1.5-7B
Vicuna-Vis V1.5-13B
Vicuna Reuna-VV1.3333 V1.3-33B
w3g128
MMLU Avg.
CSR Avg.
MMLU Avg.
CSR Avg.
MMLU Avg.
CSR Avg.
FP16
50.27
67,45
56.02
69.81
59.22
71.12
AWQ
45.79
65.4
53.27
68.06
55.3
69.24
RTN
45.18
65.61
51.61
67.90
53,39
68.75
RTN-ada
46.25
66,42
51.8
67.92
55,93
69.42
GPTQ
47.32
64.88
53.11
67.55
55.68
68.08
GPTQ-ada 47.45
65.7
53.29
67.72
57.08
69.37
Table Task-specific quantization. We evaluate instruction-tun WizardLM models on math and coding. Intuitively, we find that using calibration set from the target domain (e.g., code-code) rather than generic text corpus (c.g., pile-code) improves performance. Applying AdaDim (labeled green) consistently improves both RTN and GPTQ with up 10% on HumanEval, surpassing AWQ.
GSMBK pass@ (t)
HumanEval pass@1 I (t)
w3g128
WizMath-7B
WizMath- 13B
NizCoder-Py-7B WizCoder-Py-11BB
FP16
55.35
63.38
55.49
64.02
RTN
32.52
49.13
35.37
50.61
calib. set
hase
target
hase
target
base
target
base
target
AWQ
39.42
40.49
55.19
54.97
43.29
44.51
57.32
60.36
RTN-ada
37.38
39.12
50.95
53.15
42.68
45.12
60.37
60.98
GPTQ
38.29
41.09
54.21
57.16
31.71
45.12
53.05
56.71
GPTQ-ada
41.77
42.15
56.78
57.47
46.34
46.95
53.69
62.2
4.3 INSTDD MODEL QUANTIZATION
Instruction tuning has become the method of choice to boost the performance and user interaction experience of LLMSs (Wei et al., 2021; Sanh al., 2021; Chung et al., 2022). A well-known instruction-tuned model that also publicly available is Vicuna (Chiang et al., 2023) which is fine-tuned from LLaMA models. To benchmark how quantization affects their improved language modeling capabilities, we conduct MMLU and CSR evaluations in Table 3. AdaDim brings consistent improvements across various model scales; similar the base model results, strong performances displayed for RTN-ada on CSR and GPTQ-ada for MMLU, Performance boost is most noticeable in the 33B scale, where our adaptive solutions can bridge the degradation down ~2% points.
Task-specific quantization. Beyond general abilities like commonsense reasoning, fine-tuning to specific task has shown to be effective creating specialized LLMs (Luo et al., 2023a;b). WizardLM family is a set of LLaMA fine-tuned models that show very strong performance on open benchmarks such as GSMSk and Hvla Different tokens produce different activations; to better simulate test-time activation distribution of task-specific LLM, it may be desirable to also use task-relevant calibration set. To test this intuition, we use both the generic text corpus (WikiText-2, denoted as base) and task-relevant corpora (GSM8k for math and MBPP for coding, denoted target). As in Table 4, we confirm the proposition that using a target calibration set can bring improvements, sometimes significantly up to 8.51 % the 13B coding model for GPTQ-ada. Notably, RT'N-aa brings up to a 10.3% boost over vanilla RTN, with GPTQ-ada outperforming all other methods when using the target calibration set which may be due the task-specific weight updates that can serve as further fine-tuning.
4.4 ANALYSIS
Sweeping precision ranges. To test the generality of AdaDim across various quantization settings, use L-AMA-22777 to sweep INT3/INT4 precision with 256, 128, and 64 group sizes. As in Figure 4, AdaDim strictly improves perplexity scores when applied to RTN and GPTQ Prominently, RTN-ada shows significant perplexity improvements from vanilla RTN up to 1.06 on w3g256, even
7

---

Published as a conference paper at ICLR 2024
RTN
RTN-ada --- GPTQ
GPTQ-ada ----
FP16
8.5
46
9
44
9.5
10.5
11
38
11.5
36
w3g258 wGg128 w3g64 w4g258
w4g128 wdg84
w3g256
w3g128
w3g64 w4g256
w4g128 w4g64
WikiText-v2 Perplexity Quant. Confiy
MMLU Sashot (%) Quant. Confiy
Figure 4: Sweeping various quantization configurations for LLaMA-V2-7B. Average bits per weight increases from left to right (x-axis). AdaDim can further close the gap between INT3/4 and FP16.
First 10 Blocks of 1lame-2-7b
First 10 Blocks of ame-2-700
Aist 10 B ocks WiaadMaath-7
Recon Error Redlccion Ratio
AdDDm
Emoor Ratto
AdaDim
Rocon Emor Reductin Ratio
AdeDim
Peelic
PeNiC used
60
200
40
60
20
40
60
Module Index
Module Index
Module Index
Figure 5: Adaptive dimension selection. By adaptively switching to per-IC quantization, AdaDim can reduce the reconstruction error up to 6x in the RTN setting. The decisions vary across model size (7B vs. 70B) and task (language modeling math), showcasing the versatility of AdaDim.
surpassing GPTQ on various ranges, On MMLU, AdaDim provides accuracy lifts that are relatively non-uniform, which we speculate due to the nature of in-context learning.
Reduced reconstruction error. As in Figure 5, adaptively switching to per-IC quantization for RTN-ada can save up to 6x in reconstruction error, aligning with our objective in Eq. We notice that more savings come from carlier layers, which suggest that carlier part of the network is especially sensitive to activation outliers. Among the different modules, attn. v projection layer yields the most етror reduction, followed by the mlp. down projection layer. The switch to per-IC decisions occur at different modules for different model sizes and tasks, demonstrating the versatility of AdaDim.
Localized GPTQ updates. Activation outlier effect is pervasive in the standard Per-OC quantization, while it is isolated in the per-IC quantization (Figure 1). We study the impact of outlier isolation on GPTQ weight updates in Figure 6. GPTQ computes the weight update (analogous to gradient) by first capturing the rounding error then multiplying it by the inverse hessian (formed by activations). Thus, quantizing the sensitive weights altogether with IC grouping will lead to larger quantization errors, but it will be localized to few channels where the outliers emerge. Indeed, per-IC quantization uses larger, localized updates that target only few input channels. In contrast, standard per-OC quantization uses smaller updates to much more channels, caused by the pervasively spread outlier effect. Hence, per-IC quantization minimally perturbs the weight distribution by focusing on only few sensitive channels that contain densely populated outliers.
4.5 PER-IC KERNEL IMPLEMENTATION
We utilize the implementation of LUT-GEMM (Park al., 2022), Lookup Table-based (LU'T-based) matrix multiplication method. LUT-GEMM first involves the precomputation stage, where possible multiplication outcomes between quantized weights and activations are stored in a Lookup Table. During actual computation in forward pass, the matmul operations take the form of indexing the LUT (instead of multiplying) using the weight values as keys, enabling efficient GEMM operations. For our per-IC quantization, similar implementation is applied. During LUT generation. each per-IC quantization scaling factor can be multiplied in advance with activations to create the Lookup Table. In cases of group quantization, adjusting tile sizes to fit the groups and generating LUTTs
8

---

Published as a conference paper at ICLR 2024
Smaller, pervasive updates
Cout
Larger, localized updates
10
w/ AdaDim
10
w/o Ada
10
Cin
10
10a
10a
0.000
Weight Delta Magnitudc from GPTQ
0.025
0.050
c.075
0.100
0.125 0.55
AdaDim
AdaDli
Figure 6: GPTQ update behavior. Thanks to per-IC quantization's outlier isolation, GPTQ's error compensation (weight update) is localized to small subset of input channels instead of many input channels. By grouping sensitive weights together, AdaDim can use larger, localized updates that minimally perturb the original weight distribution. Left: A typical distribution of GPTQ's 's weight updates (magnitude) in LLaMA-V2 -2-7B''---'s attn. v layer. Middle/Right: Weight updates visualized in the 2D matrix form Cout)) ) zoomed in the top right quadrant.
INT4
INT3
4.00
5.00
Latency Speedup Factor
3.55
4.34
4.00
2.95
3.00
3.38
3.00
2.20
2.42
2.00
2.00
1.44
1.53
1.00
1.00
256
512
2048
128
256
2048
Group Size
Group Size
Figure 7: Latency speedup of our Per-IC kernel over cuBLAS across various group sizes. We measure the latency of the first FFN layer on OPT-175B model with 3-bit and 4-bit precision and corresponding kernel selections with hidden dimension size set to 12288 on an A100-80GB (000-800B-GPU GPU.
tile-by-tile basis allows for similar application of this method. Due to time and resource limitations, we do not perform experiments with a fully optimized kernel that may further optimize the latency of approach. Nevertheless, our per-IC kernel exhibits faster latency than the cuBLAS baseline in Figure 7. This indicates that our Per-IC quantization not only achieves accuracy improvements but also leads measurable speedups in inference latency. Further investigation into well-optimized per-IC kernel is a critical direction for our future research. For further discussion, please refer to Appendix D.
5 CONCLUSION
Per-IC quantization method offers simple yet effective resolution to the activation outlier challenge by strategically isolating the sensitive weights the IC direction. This methodology is further advanced by our Adaptive Dimensions (AdaDim) framework, which showcases adaptability to varying guantization sensitivities. Experimental results underline AdaDim's 's effecttivnesss, evinced by notable performance gains in both base and instruction-tuna LLMs across diverse language modeling benchmarks. Through this work, we hope to make a step forward in the practicality and accessibility of L.LMs in real-world applications.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank our colleagues from the AI Efficiency team at NAVER Cloud for constructive feedback, especially Gunho Park, the author of LUT-GEMM (Park et al., 2022), for helping with the kernel implementation. We also thank Lin and Elias Frantar for fruitful discussions.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint Sidi:2208.007320 2021.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing, arXiv preprini arXiv:2306. 0..6.2220 /2929, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Laaguagee models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, Curran Associates, Inc., 2020. URL https: / https//////ccce.....//o// neur cc/ ppaper/200/fflll.. flelleee147c066ffooo cbw.6741664449441111 (wt.6/77486bbbao.oo Bcw.aaccbbcccacoccc
pdd.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2303. 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023, URL https: // ///.mmss.org/bllgg...o 2023-
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling inrtuction-finttnne language models. arXiv preprint arXiv:2210. 11416, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint a 080300555, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv.2110.14163 2021.
10

---

Published as a conference paper at ICLR 2024
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. arXiv preprint arXiv:2212.09770, 2022.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int80: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.073333 2022.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized represen- tation for near-lossless Ilm weight compression, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101 100.00077 2020.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL httpp:://do org/ 0 5281 5371628.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. CoRR, abs/2009.03300, 2020.
URL httpaaaiii...oraaaa./
03300.
Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv preprint wwXiv:2305.14152 2023a.
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv, 2023b.
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. FlexRound: Learnable rounding based on element- wise division for post-training quantization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 18913-18939. PMLR, 23-29 Jul 2023. URL https: htttps://proceeeeee http://doocceenn..om/mm. mlr. press/v202/lee2333
Yuhang Li, Ruíhao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426. 2021.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023a.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Emne Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint ar55::200..77888 2023b.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint aXXiv22388.09583 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.085.. 08568, 2023b.

---

Published as a conference paper at ICLR 2024
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Gunho Park, Baeseong Park. Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmull for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.0995. 09557, 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodci, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:190..10641 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv: 1xx.v:2110008200 2021.
Sheng Shen, Zhen Dong. Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8815-8821, 2020.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E Gonzalez, et al. High-throughput generative inference of large language models with single gpu. arXiv preprint arXiv:2303, 06666, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing Ilama model. 2023.
https://gi thub. com/ wwwssuaaaasttaorr....
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971,13999.. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. urXiv preprint arXiv:2109.0163: 019.0165,, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837. 2022a.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models, 2022b. URL httpps://at //axxvv..//iiv///////// orgo htrg//bs/2225 13325.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint MXXX::2209.13322 2022c.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruíhao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppressiont: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint urXiv:2300.09145. 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10433 2022.
12

---

Published as a conference paper at ICLR 2024
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304..222444 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers, 2022. 01861.
URL httt:::/arxiv..rrg/bbb///
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang. Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? CoRR, abs/1905.07830, 2019. URL http://aaxxvv..rg//bbb////////// 07830.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
13

---

Published as a conference paper at ICLR 2024
A FURTHER DISCUSSIONS
Cin Shuffled w/ act. reorder
Per-OC GPTQ w/ reorder
Per-IC GPTQ w/ reorder
Memory access
Quant. order
B1k0. attn. v_ v_proj
Cin
Cin
Shuffle
Shuffle
Cin
Reordering allows sensitive channels to be quantized first
contiguous
contiguous
Figure 8: Activation reordering allows important (sensitive) channels to quantized first. This prevents salient channels from being updated to compensate for the quantization error of non-salient ones. However standard per-OC scheme, activation reordering causes quantization scales to be not contiguous, causing inefficient memory access. With per-IC, quantization scales placed in row-major order, which allows scales to be contiguous even after reordering.
A.1 ACTIVATION REORDERING FOR FREE
crucial nature of the GPTQ algorithm is that it prioritizes the weights that are quantized first, since the eиror of earlier quantized weights is compensated by later quantized weights. Activation reordering trick can help sensitive weights be quantized first, but it is only hardware-efficient with the static groups constraint. As the static groups setting pre-determines all quantization parameters independent of the weight update step, per-OC GPTQ with hardware-efficien static reordering leads suboptimal accuracy (Table 2b). In contrast, per-IC GPTQ can decouple quantization parameter's memory contiguity from the usage of activation reordering, thanks to the row-major layout (Figure 8). With per-IC quantization, we can maximize the performance of the GPTQ algorithm by using reordering for free without the static groups constraint.
A.2 INCOMPATIBILIT WITH AWQ
We found that AdaDim incompatible to AWQ (Lin et al., 2023), which is another competitive weight-only quantization approach alongside GPTQ (Frantar et al., 2022). The premise of AWQ is that when conventionally grouping the weights within each OC (per-OC quant.), each weight is multiplied by unique hidden dimension of activations. When activation outliers emerge in small subset of those channels, the corresponding weights (more sensitive ones) can be identified and scaled up. This effectively transforms the weight distribution such that sensitive weights now lie near the maximum of the distribution, so the min-max quantizer will near-losslessly quantize the scaled up outlier weights. In contrast, Per-IC quantization groups weights within each IC, so all the weights end up sharing the same hidden dimension; this results in all weights being scaled up equally, which essentially nullifies AWQ's s activation-based weight transformation. Nonetheless, AWQ can indeed be applied to layers where AdaDim uses Per-OC quantization, which may further boost performance.
14

---

Published as a conference paper at ICLR 2024
A.3 RUNTIME AND COMPUTE
Additional runtime and compute cost incurred from adopting AdaDim is minimal. One point of comparison is AWQ (Lin et al., 2023) which searches over the scale and clip parameters with grid size of 20 each, totaling 40 forward passes layer. In contrast, AdaDim requires two forward passes per layer, thanks the small search space of the dimension parameter.
B IMPLEMENTATION
def rtn_ada (x: torch. tensor, layer: torch.nn, tayer:torch.my
nn, w_bit:int, group_size: int str:
Inputs: calib. data x, layer to be quantized, and quant. parameters org_out layer (x)
org_sd [k: v. cpu for k, v in layer. state_ state_dict() items ())
best_err = float('in inf'); best_dim None; quant_sd = ()
for dim in ['oc', 'ic'l:
quantize the weight matrix place and save it temporarily quant_sd[dim] = per_channel (weight, dim, w_bit,
group_size)
quant_out layer recon_err (org_out quant_out). float () pow (2). mean () item if recon_err < best_err:
best_dim = dim
layer soad_state__e dictte_dicttee sorg_sd) recover full precision weight layer. load_ (quant_ [dbsst___mmm] del quant_sd return best_dim
def gptq_ada (x: torch tensor, layer: torch. w_bit:int, group_size:int)
weight bayer.weighte data. () best_dim rtn_ada layer, w_bit, group_size)
# run gptq on the searched dimension layer weight. data run_gptq (weigght,, best_dim, w_bit, group_size)
C ADDITIONAL RESULTS
Ablating GPTQ-ada. As in Table GPTQ-ada shows the best performance when using 256 samples.
Table 5: Calibration set size ablation on w3g128. We fix the max sequence length be 512 per sample, thus # of calibration tokens (# of samples x 512). We use 256 samples for our experiments.
LLAMA-V2-BB
LLAMA-V2-13B
# samples
MMLU Avg.
CSR Avg.
MMLU Avg.
CSR Avg.
35.15
64.16
47.56
$4.09
39.63
65.48
50.46
66.65
128
39.15
65.42
51.96
67.94
256
41.32
66.41
52.27
68.73
512
39.55
66.51
51.76
68.80
INT4 quantization (at least when used with the widely suggested group size of 128 (Dettmers &
Zettlemoyer, 2022)) does not have a "winning methodology" that consistently outperforms others (Table 6 & Table 7).
Additional experiments with SpQR. To benchmark AdaDim with more modern quantization schemes, we have conducted additional experiments with SpQR (Dettmers et al., 2023), novel weight-only quantization method featuring small grsup-wise double quantization and an FP16 outlier sparse representation. Table illustrates that Nound-To-Neante with AdaDim (RTN-ada) achieves comparable performance to SpQR. This similarity arises partly because RTN-ada has higher average bit-precision than SpQR. For more balanced comparison regarding average bit-precision with SpQR,
15

---

Published as a conference paper at ICLR 2024
Table 6: Vicuna results across all model scales.
Vicuna-V1.5-7B
Dicuna-V1.3-11
Vicuna- V1.3-33B
w4g128
MMLU Avg.
CSR Avg.
MMLU Avg.
CSR Avg.
MMLU Avg.
CSR Avg.
FP16
50.27
67.45
56.02
69.81
59.22
71.12
AWQ
23.06
38.39
55.68
69.36
29.73
62.49
RTN
49.18
67.36
54.88
69.00
58.32
70.82
RTN-ada
49.57
67.55
55.17
69.40
59.04
70.63
GPTQ
49.63
66.84
55.27
69.37
58.82
70.58
GPTQ-ada
49.56
66.94
55.07
69.59
59.06
70.61
Table LLaMA results across all model scales.
LLAMA-V2-7B
LLLMA-V2-13B
LJaMA-VI-33B
ILLMA-V2-70B
w4g128
MMLU Ave.
CSR Avg.
MMLU Avg.
CSR Avg.
MMLU Avg.
CSR Avg.
MMLU Avg.
CSR Avg.
FP16
45.98
67.93
55.61
70.33
58.46
72.96
69.29
74.96
AWQ
45.62
67.78
54.59
69.77
57.86
72,85
68.72
74.68
RTN
44.54
67.84
54,43
69.82
57.59
73.16
68.34
74.34
RTN-ada
45.04
67.65
54.57
70.04
57.63
72.75
68.97
74.99
GPTQ
44.93
68.18
54.20
70.00
58.00
72.51
68.99
74.66
GPTQ-ada 45.26
67.69
54.59
70.10
57.86
72.82
68.75
74.60
we conduct further experiments at lower bit-precisions. From Table 9, it is evident that SpQR outperforms RTN-ada within similar range of average bit bit-precision. However, it is crucial to note that SpQR requires an FP16 representation for outliers, necessitating an additional sparse inference kernel, which not requirement for AdaDim. We emphasize that by shifting the quantization channel dimensions from output-channe to input-channel, AdaDim can effectively enhance the performance of existing weight-only group-wise quantization methods.
Table 8: Perplexity (PPL) comparison between SpQR (Dettmers al., 2023) and Round-To-Nearest with AdaDim (RTN-ada) on the Wikitext2 and C4 datasets, employing 4-bit channel -wine configu- ration with grouping size of 128. The lower PPL, the better.
LLaMA-VI-7B Wikitext2 C4
LLaMA-VI-33B Wikitext2 C4
LLaMA-VI-65D Wikitext2 / C4
Method
Group Size
Avg Bits
FP16
16
5.68 / 7.08
4.10/5.98
3.53 / 5.62
RTN-ada
128
4.125
6.20/7.22
4.19/ 6.05
3.61 5.68
SpQR
3.9
5.87 7.28
4.25 6.08
3.68 5.70
Table 9: Perplexity (PPL) comparison between SpQR (Dettmers et al., 2023) and with AdaDim (RTN-ada) the Wikitext2 and C4 datasets, employing 3-bit channel-wit configu- ration with various grouping sizes. The lower PPL, the better.
LLMMA
Group Size
Avg Bits
Wikitext2 [PPL]
C4 [PPL]
FP16
16
3.53
5.62
RTN-ada
16
4.00
3.74
5.75
RTN-ada
32
3.50
3.82
5.80
RTN-ada
64
3.25
3.91
5.87
SpQR
16
3.63
3.71
5.73
D PER-IC KERNEL
Table 10 compares the latency of our per-IC kernel against the cuBLAS baseline and other per-OC kernels like OPTQ (Frantar et al., 2022), AWQ (Lin al., 2023), and LUT-GEMM (Park al., 2022). Although it shows slower latency compared to per-OC kernels due to its sub-optimal status, it's noteworthy that our Ier-IC approach is selectively applied to layer-wise manner and enhances the accuracy performance of per-OC baselines, detailed in the paper.
16

---

Published as a conference paper at ICLR 2024
Table 10: Latency comparison of the first FFN layer on OPT-175B model with 3-bit and 4-bit
precision and corresponding kernel selections with hidden dimension size (matrix size, m) is set to
12288 and various group sizes on A100-80GB-
Method
Group Size
Weight (1m ml
Input (m
Output (4m
Latency [mss]
cuBLAS
INT4
FP16
FP16
0.7258
OPTQ
128
INT3
FP16
FP16
0.3599
AWQ
128
INT4
FP16
FP16
0.3238
LUT-GEMM
128
INT4
FP16
FP16
0.2688
LUT-GEMM
128
INT3
FP16
FP16
0.225
128
INT4
FP16
FP16
0.50477
per-IC kernel
256
INT4
FP16
FP16
0.32926
(Ours)
512
INT4
FP16
FP16
0.24570
2048
INT4
FP16
FP16
0.20425
128
INT3
FP16
FP16
0.47336
per-IC kernel
256
INT3
FP16
FP16
0.29957
(Ours)
512
INT3
FP16
FP16
0.21457
2048
INT3
FP16
FP16
0.16735
E VISUALIZATIONS
BIk31.attn.gqv
Bik31.attn.out
11k31.mlppuu
B1k31.m mtw....mm...d....
0.4
0.3
0.4
0.1
0.2
Figure 9: Activation magnitude acToss different modules of LLaMA-V2-7B Block 31. We observe the largest activations before the QKV projection and DOWN projection, which we then selectively apply per-IC quantization to heuristically validate the outlier isolation effect.
17

---

Published as a conference paper at ICLR 2024
3
BANS JI
88:15 mbp.ppp..pp0o 2r0
Mc:5mrllggte pio
mloowwo proj
If tkti.mpp down. pro
14:31 www.pppp..mm groj
2k11 mlp gate proj
(a) LLAMA-V2-7B
BikO.attn gkv proj
BIkO.mlp down_ pro)
3K0 sttn.o.orol
Blkll.attn.qkv..rr
Biko m p.up gate proj
BK11 attn.o_pm)
3lk11.mlp.dow
Bikl1.mlp.up gate proj
(b) GPT-2. QKV projections are horizontally concatenated.
Figure 10: Weight sensitivity patterns shown in (Cins Cout) shape. Both sensitive rows and
columns exist across different modules and network depth. This hints at the potential effectiveness of AdaDim's 's versatile quantization scheme to other backbones such as GPT-2 (Radford et al., 2019).
18

---

Published as a conference paper at ICLR 2024
Al 32 Blocks of llama 2 7b
AdaDim Per-IC
Recon, Error Reduction Ratio
50
100
150
200
Module ndex
All 40 Blocks of llama-2-130
AdaDim Per CC used
Recon, Error Reduction Ratio
21
50
Module Index
All 80 Blocks of 1lama-2-70b
AdaDim
Recon, Error Reduction Rabio
100
200
300
400
500
Module
All 32 Blocks of WizardMath-77
AdaDim
Recon, Error Reduction Rabio
2
50
100
150
200
Module Index
Figure 11: Full visualization of adaptively selected per-IC quantization decisions and the subsequent reconstruction error savings for INT3 with group size 128.
19


Reviews:
[{"review_id": "1EIB0qowI9", "reviewer": "Reviewer_ubBB", "summary": "The paper focuses on the weight-only quantization of Large Language Models (LLMs) and claims the case of less than 4 bits has challenges because of the presence of large-magnitude activation outliers. Based on the observation, the authors propose a method called \"per-IC quantization\" within AdaDim framework, which creates quantization groups within each input channel (IC) rather than the conventional per-output-channel (per-OC). The experiment results show the proposed superiority.", "questions": "1. Typo error: The MMLU Avg. (49.56) of GPTQ-ada is bold, but GPTQ (49.63) is higher. Similar problem in CSR Avg.\n\n2. The improvement of GPTQ with ada is lower than the RTN in Tab. 6 and Tab. 7. Have compared with more baseline methods, and are there situations where there is no improvement?\n\n3. I agree the issue that accelerating the memory I/O is important in some scenes, but I didn’t find related memory evaluation/analysis result. I'd be glad if you could point this to me.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The high-level idea is good. The idea of \"per-IC quantization\" is clear and effective solution.\n\n2. This paper is well-written and organized.\n\n3. I agree the issue that accelerating the memory I/O is important in some scenes, and solving this challenge is very valuable in the industry.", "weaknesses": "1. The experiment results with activation quantization is expected, and more compared methods such [1] are necessary. \n\n2. The result of A.3 is important, more detailed analysis with diagram is expected.\n\n[1] Smoothquant: Accurate and efficient post-training quantization for large language models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c2ZJq23jIs", "reviewer": "Reviewer_JAtq", "summary": "This work introduces input-channel quantization and adaptive dimension (AdaDim) for input or output channel selection. The authors analyzes the outlier distributions and observed the traditional output-channel quantization is sub-optimal on some layers. Neither input nor output channel quantization is universally optimal, so the authors proposed AdaDim for channel selection. The performance improvement over SOTA 3-bit quantization is noticeable, but the improvement over 4-bit quantization is very minor.", "questions": "1. Why did the sensitivity is done in channel-wise, but the experiments were done with sub-channel? \n\n2. If you perform the sensitivity analysis based on the sub-channels, do you still see a large gap between input/output channel quantization?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The analysis on input/output channel sensitivity is very interesting (figure 2 and 9). \n\n2. Performance improvement on WizMath and WizCoder is pretty impressive (Table 4).\n\n3. While input channel quantization isn't new (weakness 1), AdaDim is novel.", "weaknesses": "1. Exploring non-output channel quantization isn't new. See [1] that explores input-channel and arbitrary combination of row and column groups for quantization.\n\n2. All evaluations in this paper use sub-channel quantization with group size of 128. While this is the SOTA configuration, I suggest authors to compare per-channel quantization (without groups) which might better demonstrate the advantage of input channel quantization over output channel quantization for certain layers. This is because the analysis was compare channel-wise sensitivity instead of sub-channel sensitivity.\n\n[1] Yuan, Z., Chen, Y., Xue, C., Zhang, C., Wang, Q. and Sun, G., 2021. Ptq-sl: Exploring the sub-layerwise post-training quantization. arXiv preprint arXiv:2110.07809.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PQdb6eWCrb", "reviewer": "Reviewer_horP", "summary": "This paper proposes a new quantization scheme for LLMs. Particularly, they propose to choose quantizers based on input channels rather than output channels, as it is usually done. They show that this can to some degree reduce the effect of weight outliers on the quality of the quantized network.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The numerical results suggest that this quantization framework can help reduce the loss of accuracy from quantization.", "weaknesses": "- First, I think the presentation of the paper is very hard to follow. The authors choose to write a paragraph rather using mathematical notations that can make the paper easier to follow. At core, given the activation as $WX$ with $W\\in R^{q\\times p}$, they propose to have $p$ quantizers where quantizer $i$ corresponds to column $i$ of $W$, rather than having $q$ of them each corresponding to a row of $W$. Other examples include using the term \"Activation\". Is activation referring to $WX$ or $X$? Or $W$ which are the weights? Isn't this work trying to circumvent outliers in $W$? Then why they talk about activation outliers in page 3? The method does not even seem to detect outliers, then why talk so much about outliers?  Overall, I find the paper confusing and hard to follow.\n\n- It seems transposing the direction of quantization helps with accuracy. But then what happens in terms of implementing this method in practice? Note that this per-IC quantization method is not a drop-in replacement for per-OC, or at least if it is, that requires to be substantiated with experiments. As far as I could tell there are no experiments on hardware implementation and inference time performance of the proposal. The hardware implementation is very briefly mentioned in Section 3.2, but the explanation actually creates more questions than answers, as it contain a bunch of new notations that are never properly defined. The authors say \"With per-IC quantization, we can maximize the performance of the GPTQ algorithm by using reordering for free without the static groups constraint.\" which again I don't think is numerically supported. \n\nOverall, I think the paper needs some rewriting to make it easier to understand, and several experiments to show the benefits of per-IC in inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)