LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
Authors: Gunho Park
Baeseong park
Minsub Kim
Sungjae Lee
Jeonghoon Kim
Beomseok Kwon
Se Jung Kwon
Byeongwook Kim
Youngjoo Lee
Dongsoo Lee
Venue: ICLR 2024 poster (2024)
Keywords: Neural Network Quantization
Model Compression
Generative Language Model
Transformer
Deep Learning
Primary area: infrastructure, software libraries, hardware, etc.

Abstract: Recent advances in self-supervised learning and the Transformer architecture have significantly improved natural language processing (NLP), achieving remarkably low perplexity.
However, the growing size of NLP models introduces a memory wall problem during the generation phase.
To mitigate this issue, recent efforts have focused on quantizing model weights to sub-4-bit precision while preserving full precision for activations, resulting in practical speed-ups during inference on a single GPU.
However, these improvements primarily stem from reduced memory movement, which necessitates a resource-intensive dequantization process rather than actual computational reduction.
In this paper, we introduce LUT-GEMM, an efficient kernel for quantized matrix multiplication, which not only eliminates the resource-intensive dequantization process but also reduces computational costs compared to previous kernels for weight-only quantization.
Furthermore, we proposed group-wise quantization to offer a flexible trade-off between compression ratio and accuracy.
The impact of LUT-GEMM is facilitated by implementing high compression ratios through low-bit quantization and efficient LUT-based operations.
We show experimentally that when applied to the OPT-175B model with 3-bit quantization, LUT-GEMM substantially accelerates token generation latency, achieving a remarkable 2.1x improvement on a single GPU when compared to OPTQ, which relies on the costly dequantization process.

Published as a conference paper at ICLR 2024
LUT-GEMM: QUANTIZED MATRIX MULTIPLICATION BASED ON LUTS FOR EFFICIENT INFERENCE IN LARGE- SCALE GENERATIVE LANGUAGE MODELS
Gunho Park Bacseong Park Minsub Kim", Sungjae Lee", Jeonghoon Kim",
Beomseok Kwon², Se Jung Kwon', Byeongwook Kim², Youngjoo Lee', Dongsoo Lee²
1 Pohang University of Science and Technology
2 NAVER Cloud
[gunho1123 12hh, young joo lee))postech..c.
[baeseong park, minssub. kim, sung-jae lee, jeonghoon. samuel, beomseok. kwon, sejung. kwon, byeonguk. kim, dongsoo lee 1erggooo.eeeeeaeerrorrr @ com
ABSTRACT
Recent advances in self-supervised learning and the Transformer architecture have significantly improved natural language processing (NLP), achieving remarkably low perplexity. However, the growing size of NLP models introduces memory wall problem during the generation phase. To mitigate this issue, recent efforts have focused on quantizing model weights to sub-4-bit precision while preserving full precision for activations, resulting practical speed-ups during inference single GPU. However, these improvements primarily stem from reduced memory movement, which necessitates essoucce-intensvee dequantization process rather than actual computational reduction. In this paper, we introduce LUT-GEMM, an efficient kernel for quantized matrix multiplication, which only eliminates the resource-intensive dequantization process but also reduces computational costs compared to previous kernels for weight-only quantization. Furthermore, we pro- posed group-wise quantization to offer flexible trade-off between compression ratio and accuracy. The impact of LUT-GEMM facilitated by implementing high compression ratios through low-bit quantization and efficient LUTTbbased opera- tions. We show experimentally that when applied to the OPT-175B model with 3-bit quantization, LUT-GEMM substantially accelerates token generation latency, achieving a remarkable 2. x improvement on single GPU when compared OPTQ, which relies on the costly dequantization process. The code is available ottp://///.....oooo//arareee////
INTRODUCTION
Recent years have observed large-scale language models (LLMs) presenting state-of-the art perfor- mance on various natural language process (NI.P) tasks. Such rapid progress in NL.P performance has been highly facilitated by the self-supervised learning methods, avoiding expensive manual labeling (Devlin et al. 2019 Baevski et al.) 2020; Chen et al. 2020). Leveraging extensive training datasets, architectures like the Transformer model (Vaswani et al. 2017). leading to notable increases in model parameters.
these models benefit from efficient sequence-to-sequees
Previous studies Brown et al. 2020 Kaplan et al. 2020 Hoffmann et al. 2022 have reported that
LLM performance follows predictable power-law law scaling function of model size. Accordingly, in recent years, several large-scale generative language models, including GPT-3 (175B) (Brown)
al. 2020), HyperCI.OVA (204B) (Kim al. 2021a). Gopher (280B) (Rae al. 202T), D), Chinchilla (70B) (Hoffmann et al. 2022), Megatron Turing NLG (530B) (Smith et al.) 2022). PaLM (540B) (Chowdhery al.) (2022) and LLaMA (65B) (Touvron et al. (2023), have been proposed further advance state-of-the-art performance. However, models with billions of parameters cannot be accommodated on single GPU due the limited memory size of GPUs, which is sacrificed to
"Equal contribution

---

Published as a conference paper at ICLR 2024
W (INTT) A (INTB)
W (s 4bit) (FP16)
( A (FP16)
W (S 4bit)
A (FP16)
Mstmul (INT8)
Dequant
Pre- computation
O (INT32)
W (FP16)
LUT-GEMM
Quant
Matmul (FP16)
Index
LUT
(INT8)
O (FP16)
(FP16)
(8) W8/AB Quant.
(b) W4/A16 Quant.
(c) W41A16 Quant.
(On-the-fly Dequant.)
(LUT-GEMM, Ours)
Figure 1:
Three matrix multiplication schemes for W8/A8 or W4/A16 (i.e., weight-only) quantization
format. Our proposed LUT-GEMM can adopt W4 and A16 without requiring an additional dequanti- zation step.
enhance memory bandwidth (Migacz 2017 Yu et al.) 2017). To address such a concern, researchers have proposed to use model parallelism, which distributes computations over multiple GPUs through GPU-to-GPU communication (Shoeybi et al.) 2019 Narayanan et al.) 2021). Nevertheless, it is worth noting that model parallelism introduces additional overheads, stemming from the inter-GPU communication. Consequently, the performance gains achieved through model parallelism exhibit a sub-linear relationship with the number of GPUs employed.
To mitigate the challenges related to model parallelism, parameter quantization (Xu et al. 2018; Choi jet al. 2017| McDonnell (018)) presents a practical solution for minimizing model size, reducing the number of GPUs required for inference. Among the various quantization schemes, the preferred choice is quantizing both activations and weights to exploit integer-based arithmetic units (Jacob) eet 2018 8 Wu et al.) 2018 Lin et al. 2016). Nonetheless, this quantization method is practically limited to 8 bits, and non-limeat operations (e.g., softmax and normalization) may yield imprecise results (Kim et al. 202 Bhandare et al. 2019). Moreover, to fully utilize integer arithmetic units, it
is essential to implement onythh-fly activation quantization and dequantization, along with an accurate estimation of the activation distribution (Yao et al. 2022 Dettmers al., (022).
Recent research has proposed 4-bit weight-only quantization as an approach for memory compression
(Frantar et al. 2022 Lin et al./ 2023 Dettmers al.) 2023 Kim et al., 2023). involving on-the- fly conversion to pree-iini While this sacrifices the computational benefits of using integer arithmetic, empirical findings in LLMs suggest that weight-only quantization can achieve significantly higher compression ratios for a given target accuracy compared to quantizing both weights and activations (Zeng et al., 2022). Various weight-only quantization methods have been proposed to improve the compression ratio while preserving accuracy, often accompanied by dedicated kernels for
practical acceleration through quantization (Jeon et al. 2020 Frantar et al. 2022 Lin et al. 2023).
In this paper, we present LUT-GEMM, a kernel designed to facilitate quantized matrix multiplications with quantized weights and full- precisiecn activations. As shown in Figure [] LUT-GEMM addresses two issues prevalent in previous quantization approaches: 1) accuracy degradation due to quantized activations and 2) the need for additional dequantization implementation. LUT-GEMM inherently accommodates quantized weights and full-precision activations, enabling the acceleration of the inference process while preserving the desired level of precision. Specifically, LUT-GEMM employs the binary-coding quantization (BCQ) format (Rastegari al. 2016) to capitalize on simple arithmetic operations. It is worth noting that BCQ was initially proposed to support non-uniform quantization, which relies on customized hardware for bit-level operations. To our knowledge, we are the first to show that prior uniform quantization can be reformulated in the form of BCQ, allowing LUT-GEMM to support both non-uniform and uniform quantization formats. Consequently, LUT-GEMM can execute a wide range of weight-only quantization schemes for matrix multiplications, achieving low inference latency and eliminating the need for on-the-fly fly dequantization.
Our major contributions in this work include the following: 1) We verify that BCQ is capable of representing both uniform and non-uniform weight quantization. 2) We show that LUT-GEMM. using the BCQ format, offers broad spectrum of latency and accuracy trade-offs, leveraging GPU- specific hardware utilization methods to implement various BCQ configurations efficiently. 3) For LLMs, we demonstrate that LUT-GEMM, which utilizes quantized weights without a dequantization process, can considerably accelerate matrix multiplications with small quantization bits while power consumption is greatly saved by reducing the number of GPUs. 4) Assuming 3-bit BCQ format for weights of OPT-175B served by a single GPU, experimental results show that LUT-GEMM accelerates token generation latency by 2.1 x compared to the OPTQ (Frantar et al. 2022
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND
2.1 GPU-ACCELFRATEDD GENERATIVE LMS
For large LMs, the processing time of matrix multiplications dominates the entire inference latency because of higher time complexity compared to activation functions, normalization layers, and so on DDetteers et al., 2022). Specifically, matrix multiplications account for at least 75% of the processing time for various LM sizes and input token lengths (see Appendix [A]. Note that due to the limited memory capacity of single GPU, large LMs may need multiple GPUs, resulting in increased communication latency. GPUs are commonly adopted to accelerate inference as GPUs embed lots of arithmetic units and support multiple threads, critical for speeding up matrix multiplications (Nraaaaaa et al.) 2021] Migacz 2017). However, extracting high performance from GPUs depends on arithmetic intensity, and therefore, the batch size should be large enough to ensure high reuse ratio from main memory (Markidis et al. 2018).
2.2 QUANTIZATION METHODS AND LIMITATIONS
Various research efforts have been made to enhance the serviceability of large and heavy deep neural
networks by improving their latency and throughput. These efforts include quantization (Rastegari
et al.
2016
Jacob et al., 2018 Nagel et al. 2017 Xu al. 2018 Chung et al. 2020p; pruning (Han
al.
2016 Zhu & Gupta 2017 Gale et al. 2019). knowledge distillation (Hinton et al. 2015 Polino
cct al.
2018), and low-rank approximation (N. Sainath et al. 2013 Chen et al. 2018 8 Edalati et al.
2021).
Among these, quantization is the most extensively researched field, which involves using faster and more efficient computing units and reducing memory usage. Uniform quantization using an INT8 operator is particularly well-studied among the various quantization formats and is currently
being actively applied in LLMs (Yao et al. 2022: Dettmers et al. 2022 Xiao et al., 2022). )
INT8 arithmetic units, commonly found in contemporary computing systems, offer reduced latency (thanks to their low hardware complexity) and decreased memory usage of up to 50% compared to FP16. Thus, present NVIDIA GPUs employ Tensor cores that support INT8 operations accelerate computations (Markidis et al. 2018). However, the utilization of INT8 mandates the quantization of activations, which can pose a significant challenge in achieving compressed and accelerated models employing INT8 units. For instance, when scaling factors are determined offline, activation quantization may lead to a considerable reduction in model accuracy as outliers are approximated.
To address declining accuracy in quantizing LLMss token-based dynamic quantization has emerged as a crucial technique (Yao et al. 2022 Dettmers et al., 2022). The I.L.M.nt8() method (Dettmers) et al. 2022) addresses the systematic appearance of outliers in I.LMs by proposing a decomposition method that conducts most computations in INTS and dequantizes a limited number of outliers to FP16, resulting in only a marginal latency decrease in LLMs. SmoothQuant (Xiao et al., 2022) takes an advanced approach by mathematically transferring activation variance (challenging to quantize due to outliers) to weights, which are relatively easier to quantize. This technique enables efficient computation using INT8 arithmetic units (i.e., W8A8), even when employing static quantization of activations. However, it should be noted that applying SmoothQuant in novel L.LMs is necessary to empirically observe and validate the occurrence of outliers before applying the migration technique.
The utilization of INT8 precision introduces variability in its efficacy, primarily influenced by the specific characteristics of each phase within LLM inference. While the summarization phase exhibits a high likelihood of weight reuse, resulting in a compute-bound operation, the generation phase, due to its autoregressive nature, primarily involves memory-bound workload. This intrinsic memory limitation restricts hardware throughput even with INT8 precision. Consequently, services generating a large number of tokens may experience only marginal performance gains with INT8 adoption.
Recent studies have focused on the inefficiency of the generation step and, in response, proposed the utilization of the W4A16 format (Frantar et al., 2022 Zeng et al. [2022] Dettmers et al. [023] Kim ct al. 2023), which compresses model weights into 4-bit integers without quantizing the activations as weights typically dominate memory size compared to activations. Given that current computing
systems cannot accelerate the W4A16 format, OPTQ (Frantar et al. 2022) and AWQ (Lin et al. 2023) solve such issue by dynamically dequantizing weights to FP16 and then performing matrix multiplication. Leveraging the memoory-boun nature of the generation step, this approach enhances generation latency by reducing memory movement despite the dequantization overheads.
3

---

Published as a conference paper at ICLR 2024
2.3 BINARY- CADING QUANTIZATION
Binary-coding quantization (BCQ) initially introduced by Xu et al. (0018) presents a compelling alternative conventional uniform quantization methods. For instance, when a weight vector w (of size n) is quantized into q-bit by BCQ, w is approximated to be Ei= a,b; where ag E R is a scaling factor and by E (-1,+1 +1 3 is a binary vector. this paper, we have chosen to employ BCQ as our primary quantization technique for weights while retaining activations in full precision to address the challenges mentioned earlier. Moreover, we introduce an extension to BCQ, enhancing its capabilities to encompass both uniform quantization and group-wise quantization. This extension, outlined in the subsequent section, not only broadens BCQ's 's utility but also enables the applicability of LUT-GEMM to various quantization methods.
DESIGN METHODOLOGY OF LUT-GEMM
LUT-GEMM is devised to develop pegh-performance, energy-efficient inference systems for LLMs. To achieve this objective, LUT-GEMM incorporates several innovative approaches. Firstly, employ lookup table (LUT)) based computation technique to mitigate redundant calculations caused by digitized binary weights after BCQ. Furthermore, since most non-uniform quantization methods involve complex operations with limited parallelism and often lack hardware support, we design LUT-GEMM to efficiently support BCQ formats. Our proposed kernel, LUT-GEMM, directly utilizes BCQ formats without additional overhead, such as dequantization. Secondly, we expand conventional BCQ methods by introducing bias term, significantly enhancing representational capability. This simple yet profound enhancement enables the representation of both non-uniform and uniform quantization methods within the extended BCQ format, providing us with the flexibility leverage various quantization techniques based on the specific requirements of LLMs. Finally, We further refine the implementation details the binary-coding between compression ratio and quantization error to better exploit the characteristics of LLMs. As result, LUT-GEMM demonstrates reduced latency and/or a decreased number of GPUs required for LLM inference while inherently accommodating various weight-only quantization methods.
quantization scheme, enabling a trade-off'
3.1 LUT BASED QUANTIZED MATRIX MULTIPLICATION
Our quantization scheme, which utilizes BCQ format for weight-only quantization while maintaining full precision for activations, leads to duplicate and redundant partial computations in naive matrix multiplications. To illustrate, assume that a binary matrix B 1-1,+114k and an activation vector IR. are given as
B=
[1
322
33
274
355
Then, computing Bæ (that is to be multiplied by scaling factors) would repeat (m) + 2h .a) three times and ( (-x4- - 2z +x) two times. Such redundant computations are caused by digitized elements of B, and thus, we expect more duplicated computations the size matrices increases according to the growth of model size. Moreover, loading each element of B requires bit-level memory accesses that can be slow for commercial CPUs and GPUs.
To efficiently perform Bx:" while avoiding bit-level memory accesses, we can pre-compute all possible combinations of full-precision activations and binary patterns. Note that a LUT has been widely used to save processing time when numerous computations yield outputs within a restricted set (de Queiroz& Stein 2004 Meher 20100; Jeon et al. 2020, Xu et al. 202T). LUTT-based computation is justified, especially when retrieving a value from a LUT is much faster than carrying out the original calculations. BCQ format (without quantizing activations that require heavy modifications in training codes and model structure (Wu et al. 2018 Jacob al. 2018)) is also useful to be implemented by LUT-based approaches. To construct a LUT, a hyperparameter ji is introduced, representing the sub-vector length of æ. For example, to address redundant computation observed in Equation] I we can pre-compute 8 (=23) possible values with every three elements in x and store those values in LUT.. Hence, i is 3 in this example. Once 2 values of L.T generated by
4

---

Published as a conference paper at ICLR 2024
Big centIor
Bu
(0000011100
18/0009
Packed (UINTSS)
Scate Factors (FP18)
Input Veclore (FP16)
R
201 Das
TBx
TB
Br, Da
LUT3
Scale Factor
Makris Ac R
Rinwry Weightt Matria
Output
Marri
Cc.
Ba
10k
Tike
Figure 2: The overview of LUT-GEMM implementation on GPUs. In this example, we assume 8, la- = 4./ 4, bo = 32, and y 1. 1." denotes element-wise multiplication and "." indicates a tensor product.
using a sub-vector of æ, arithmetic operations to obtain partial dot products (of Bæ replaced with LUT retrieval operations while a key is given by concatenating binary elements of B. To complete Bæ T computation, the final step, those partial products are summed and then multiplied by scaling factors. When the row dimension of B is enlarged (as generative LMs get larger), the utilization of LUT increases due more occurrences redundant computations.
To quantify the reduction in computation, assume that a q-bit quantized (m x n) binary matrix B, (for 1. 2......... .Q) is multiplied with an (n 1) input vector æ using p-width LUTs. The computational complexity of the LUT-bbsed approach can be described as follows, when mq >> 24:
C =
Cbustd Cbustd Cread O
2
11
11
where Civita and Cread represent the complexity of building the LUT and reading a value from the LUT, respectively. Consequently, compared to the complexity of conventional matrix multiplication (mm), LUT-GEMM can achieve computational savings of times by leveraging LUTs.
j
3.2 LUT BASED IMPLEMENTATION ON GPU
In addition to the LUT-based scheme (eliminating redundant computations and bit-level memory accesses), our strategy for optimizing single-batch operations GPUs is follows: 1) To improve parallelism, we create as many threads as possible while each thread is allowed to perform independent LUT accesses. 2) Binary weights accessed by thread share a common scaling factor such that operations related to scaling factors do not degrade the performance of a thread. 3) we allocate small resources
thread, then LUT utilization can be low, and synchronization overhead can increase. As such, we need to optimize thread configurations empirically.
Figure 2 illustrates the overall LUT-GEMM implementation scheme GPUs. For the sake of simplicity, we formulate the proposed quantized matrix multiplication as Ei= (A; (B; x)), FP16
where A is (m 1) FP16 scaling matrix, B an (m n) FP16 binary matrix, input vector of size n. and the operator o indicates clement-wise multiplication. For LUT-GEMM. we assign LUTs to thread block (TB) of GPU. Then, the size of submatrix of B allocated each TB becomes (t,, X tw) where f. = 1 x Small t, can increase the number of available threads while large in enhances LUT utilization inside TB. Thus, ta empirically determined (2048 is a practical number for saaee-ccale LMs). For > 1, the entire process of Figure can be iterated times while intermediate results are accumulated. See Appendix B for implementation details.
3.3 REPRESENTATIONAL CAPARILITY OF LUT-GEMM
LUT-GEMM extends the BCQ format to support both non-uniform and uniform quantization methods, which are widely used compressing LLM. this section, we introduce extended BCQ format to enhance representational capability of LUT- LUT-GEMM.
Asymmetric Binary-Coding Quantization Conventional BCQ methods are basically limited to exhibiting symmetric quantization shape with respect the zero point. Another constraint of BCQ
5

---

Published as a conference paper at ICLR 2024
methods stems from their inability to represent the value of zero due to the symmetric quantization properties. To enhance the representational capability of BCQ, we extend the conventional BCQ method by including bias term follows:
[oc b:) 22 b, E|1,++1J",
where weight vector (of size n) is decomposed into binary vector b, with associated scaling factors ag. As a result, the modified BCQ format, along with the inclusion of the bias term, can represent asymmetry quantization centered around z, as illustrated in Figure Ba).
Uniform Quantization Incorporating a bias term into the BCQ format, we have discovered that the extended BCQ format can effectively represent uniform quantization by carefully adjusting the scaling factors and bias term, as visually depicted in Figure Bb... See Appendix [ for details of how asymmetric uniform quantization can be converted into symmetric BCQ with a bias term. In Figure we can see that for q-bit quantization, the uniform quantization method employs a single scaling factor, while the non-uniform quantization technique calls for the use of q distinct scaling factors. Consequently, due to the incorporation of the extended binary- quantization format, our proposed LUT-baased matrix multiplication scheme applicable to both uniform quantization and binary-coding-basee baased non-uniform quantization methods.
2
2's
z
7
200
2(ar-0o)
8
8
8
5
S
(a) Non-uniform Quantization
(b) Uniform Quantization
Figure 3: Extension of binary-coding quantization to support both non-uniform and uniform quanti-
zation formats, achieved by including bias term (q = 3).
3.4 LATENCY-ACCUR.N TRADE-OFF FOR IMPROVED APPLICABILITY
As the hidden size increases rapidly (e.g.. dmoddll = 12288 for GPT-3 175B) according to the advent of large-scale LMs, would more difficult compute proper scaling factor shared by larger number of weights. As an alternative to row-wise quantization, group-wise quantization in which scaling factor can be shared by an arbitrary number weights widely used minimize quantization eтror (Frantar et al.) 2022. Lin et al. (023). To examine the latency variance of LUT-GEMM with respect to group size g. we perform matrix multiplications (using an (m n.) matrix and an (n x 1) matrix) when values vary as shown in Figure a). We observe a sufficiently large g, such as 128 and 256, can result in fast LUT-GEMM while accuracy improvement by group-wise is substantial. To gain insights into the underlying mechanisms, because single-batch operations are primarily memory-bound and latency is proportional to the memory footprint. Let S, and Sa represent the space complexity of binary weights and scaling factors, respectively. Then the overall space complexity S can be described as
analyze the memory footprint of LUT-GEMM
11-m- m 9 16 m - = m n (1+ + S. = - (1 (1-
16 (+11)
S S=S.+
(4)
As a consequence, when g 16. becomes independent g and can be approximated as O (m n q). To verify our claim that latency of LUT-GEMM is proportional to memory foot- print (when running single-batch operations), we explore various (q, g) pairs and their corresponding compression ratios and measure matrix multiplication latency when m Figure Note that even if the number quantization bits q is smaller, the memory footprint can be large when the group size is small (i.e., more scaling factors employed, see Appendix D) Therefore, the additional search parameter g allows a fine-grained search space compression ratio that is not available by alone. Across all available compression ratios, latency is a function of compression ratio. For instance, two different pairs (gi.g)) and (92. 92) exhibit a similar memory footprint, then expect similar latency by LUT-GEMM.
12288 as depicted in

---

Published as a conference paper at ICLR 2024
Normalized latency
m n 12288
0.4
Latenoy (ms)
0.3
1.5
0.2
0.1
1.0
400
16
32 64
128
256
512
1024
1
2
8
g
Compression ratio
(a)
(b)
Figure 4: (a) Normalized LUT-GEMM latency when a (m n) weight matrix is quantized by 3-bit with different values. (b) Relationship between latency and compression ratio when LUT-GEMM performs quantized matrix multiplications with 12288 and various (q, 9) pairs.
4 EXPERIMENTAL RESULTS
In this section, we present the experimental results obtained by utilizing LUT-GEMM across various levels of complexity, ranging from single-layer experimental setup to the complete model level. Initially, we examine the influence of LUT-GEMM into the inefficiency of tensor parallelism. Finally, we analyze the end-to-end latency of OPT models (Zhang al. 2022 to determine the overall effect of LUT-GEMM on performance.
a specific layer, followed by investigation
4.1 KERNEL EVALUATION
Latency Comparisons with Various Kernels Table I shows latency measurements for the first layer of the Feed-Forward Network (FFN) in the OPT-175B model (Zhang al. 2022). The measured kernels include cuBLAS (for FP-FP or INT-INT), OPTQ (Frantar et al.) 2022). AWQ (Lin et al. 2023) (for FP-INT), and LUT-GEMM (for FP-INT or FP-BCQ), Note that OPTQ and AWQ kernels involve dequantization followed by GEMM, while LUT-GEMM accepts quantized weights directly, eliminating the need for dequantization. We can observe that the latency of the INT8-INTSS (with cuBLAS) implementation only slightly improves latency over FP-FP since cuBLAS is not well- optimized for single batch operations. While OPTQ and AWQ achieve lower latency than cuBLAS due to reduced memory movement but are slower than LUT-GEMM due to dequantization overhead. Moreover, LUT-GEMM exhibits 2.6x reduction computation compared to the previous GEMM kernels. Thus, the proposed LUT-GEMM kernel, which operates directly on quantized weights and reduces computational complexity, achieves the lowest latency among the kernels considered.
Table Latency comparison of the first FFN layer on OPT-175B model with various precision and corresponding kernel selections with m 12288 and g 128 on A100-80GB-GPU.
Precision for Data Type
Latency (ms)
Kernel
Weight
Input
Output
(Speed up)
(4m m))
(mu 1)
(Am x 1)
cuBLAS
FP32
FP32
FP32
1,4015 (x0.52)
cuBLAS
FP16
FP16
FP16
0.7256 (x (x1.00)
cuBLAS
INT8
INT8
INT32
0.6345 (x (x1.14)
LUT-GEMM
INT8
FP16
FP16
0.4620 (x1.5)
OPTQ Frantar et 2022)
INT3
FP16
FPI6
0.3599 (x2.02)
AWQ Lin 2023
INT4
FP16
FP16
0.3238 (x2.24)
LUT-GEMM
INT4
FP16
FP16
0.2688 (x2.70)
LUT-GEMM
I
FP16
FP16
0.2250 (x3.222
LUT-GEMM supports both non non-uniform and uniform quantization.
Comparison with FP16 Tensor Parallelism Table Sablerissmarizee the profiling results of matrix multiplications performed by using cuBLAS (with tensor parallelism) LUT-GEMM. GPU power and other metrics are collected by using mvidia-smi utility (Tiwari et al.) 2015) Ali et al. 2020). We notice that tensor parallelism with additional GPUs in cuBLAS brings about significant decrease in GPU utilization, memory bandwidth (B W) utilization, and computation latency ratios. As evidenced by the increase in the latency ratio of communication, such reductions in utilization indicate that
7

---

Published as a conference paper at ICLR 2024
some GPUs can be temporarily idle until all GPUs synchronized. Accordingly, the speed-up that can be obtained by tensor parallelism a lot smaller than the number of GPUs. As a result, cuBLAS with more GPUs causes increased energy consumption for matrix multiplications. On the other hand, LUT-GEMM (with one GPU) can offer high speed-up (that cannot be achieved by tensor parallelism) while retaining high GPU/memory utilization. Combining low latency and reduced number of GPUs, thus, LUT-GEMM substantially saves energy consumption for matrix multiplications.
Table 2: Profiling results of matrix multiplications (with an (4m x m) matrix and an (m 1) matrix).
For LUT-GEMM, y m and g is or 4.
Kemel-GPUs-n--q
Comm. Ratio (%)
Speed
GPU BW
Memory
Avg. Power
Total
Up
Util. (%)
Util. (%)
(W/GPU)
Energy (mJ)
cuBLAS-I-12288-16
0.00
1.00
96.57
98.37
215.70
161.85 (x 1.00)
LuBLAS-8-12288-1
38.58
3.45
52,48
24.27
172.44
299.96 (x 1.85)
BIT-GEMM-1-122888
0.00
4.85
88.63
58.38
280.85
43.49 (x0.27)
LUT-GEMM-1-222 ---22888-4
0.00
3.23
92.51
77.98
292.76
68.09 (x0.42)
4.2 END-TO-END LATENCY
We now evaluate the end-to-end latency of inference with single batch size, considering various LLaMA models with quantized weights while preserving full precision activations. Table iilustraes the end-to-end latency per token and perplexity when weights are uniformly quantized using AWQ method (Lin et al. 2023) Additionally, the evaluation for OPT family models and LLaMA family models can be found in Appendix Table 6 and Table respectively, providing a comprehensive overview. Latency measurements are conducted within the Faster Resterrrransoomer framework, exploring different GPU configurations assess potential speed-up gains from model parallelism. From our observations, we can conclude the following: 1) Reducing the group size (g) effectively decreases perplexity, even when employing a simple RTN quantization scheme, the cost of a marginal increase in latency. 2) Increasing the number of GPUs (and, consequently, parallelism) does not significantly reduce latency due to various overheads such GPU-to-GPU communication cost, as described in Appendix A In the case of LUT-GEMM, where matrix multiplications are accelerated, relative GPU-to-GPU communication overheads become more pronounced compared cuBLAS, making model parallelism less effective. This highlights the prominence of communication overheads for high-performance matrix multiplication engines.
In the case of the LLaMA-30M model, end-to-end inference with FP16 weights can be executed on a single GPU (See Appendix Figure g for a latency comparison various model sizes on a single GPU). However, for the LLaMA-65B model with FP16 weights, the model size exceeds the memory capacity of a single GPU (80GB for A100), necessitating model parallelism techniques. Nevertheless, when the weights of the LLaMA-65B model are quantized to 3 or 4 bits, demonstrated to be viable solution in (Frantar et al. 2022), inference can be accommodated on single GPU. Assuming 3-bit quantization and the implementation of LUT-GEMM with 1=128, a speed-up of 2.41 for LLaMA-30B (using one GPU) and 2.04x for LLaMA-66B (using two GPUs) is achievable. Note that when fine-tuning performed after constructing a pre-trained model, more efficient quantization techniques such as Alpha'Tuning (K won et al., 2022) can enable even 2-bit quantization, as shown in Appendix Table 8 The efficacy of LUT-GEMM is further enhanced when combined with advanced quantization methods to reduce the number of quantization bits.
Table 3: Comparison of perplexity and end-to-end lateney per token with cuBLAS and LUT-GEMM implementations with AWQ quantization method and model parallelism on multiple GPUs.
Quant.
Perplexity
Latency (ms)
Model
Kernel-q-9
method
Wiki2
I-GPU
2-GPU
4-GPU
cuBLAS- 16-NAA
FP16
4.10
43.6
25.1
16.9
LLaMA 30B
LUT-GEMM--4122
AWQ
4.23
20.7
15.5
12.2
LUT-GEMM--3-128
AWQ
4.88
18.1
13.7
11.2
CuBLAS- 16-N//
FP16
3.53
00M
46.9
29.1
LLaMA
LUT-GEMM-4-128
AWQ
3.67
35.7
25.4
19.9
65B
LUT--GEM--3-128
AWQ
4.24
31.3
23.0
18.3
8

---

Published as a conference paper at ICLR 2024
5 ACCELERATING QUANTIZED OPT-175B
Table provides comparison of the end-to-end latency for generating token in OPT-175B, representative large-scale LM, using the FasterTrrang Testerrransformer framework. LUT-GEMM demonstrates its ability to decrease the number of GPUs needed for running inference, while concurrently reducing latency as y diminishes as the number of GPUs rises. For OPT-175B with FP16, minimum of 8 GPUs is necessary for executing inference. However, upon utilizing the BCQ format for quantization, LUT-GEMM is able to perform inference using just single GPU, while maintaining comparable overall latency. It should be noted that, when comparing identical 3-bit (weight-only and row-wise) quantization scenarios, the latency for token generation using LUT-GEMM is 2.1 x lower than that of the OPTQ library. This significant reduction in latency can be primarily attributed to LUT-GEMM's ability directly accept quantized weights, thereby eliminating the need for dequantization.
Table 4: End-to-end latency per token for OPT-175B model. The latency is measured A100 80GB.
Latency per token (ms)
GPUs
Baseline
OPTQ
LUT-GEMM
FP16
3-bit
1-bit
2-bit
3-bit
4-bit
OOM
106.5
30.4
40.1
51.6
OOM
OOM
N/A
25.2
30.1
35.8
41.2
OOM
N/A
20.3
23.8
27.2
30.1
42.4
N/A
20.1
22.4
24.2
25.8
Let us demonstrate that the flexible features of LUT-GEMM (attributable to the extended BCQ format) accelerate existing uniform quantization methods. shows the perplexity of OPT-1755 for various and g configurations, as obtained using the OPTQ method (See Appendix Table for a comprehensive accuracy evaluation at g = 3). The table also displays the corresponding latency (per generated token) achieved by LUT-GEMM (excluding FP16). The results clearly indicate that LUT-GEMM provides lower latency as g decreases, although an excessively small may have marginal adverse impact on latency. All in all, by integrating LUT-GEMM and OPTQ at the expense of an acceptable increase in perplexity, it is possible to reduce the number of GPUs required for running OPT-175B inference from eight GPUs to a single GPU while ensuring comparable latency. It crucial to note that such an improvement in performance cannot achieved when dequantization is included.
Table 5: Perplexity of quantized OPT-175B using OPTQ and end-to-end latency per token by LUT-GEMM for various 9 and configurations. of -.' indicates row-wise quantization.
Model size(GB)
Keree
9
PPL
(Comp. Ratio)
Lateney(ms)
cuBLAS
16
N/A
8.34
347.9 (x 1.00)
42.4 (8-GPU)
4
8.37
87.0 (x4.00)
OOM (1-GPU)
8.68
65.3 (x5.33)
51.6 (1-GPU)
LUT-GEMM
32
8.94
54.4 (x6.40)
55.2 (1-GPU)
2
64
9.18
48.9 (x7.11)
47.5 (1-GPU)
2
128
9.58
46.2 (x7.53)
46.5 (I-GPU)
PPL numbers extracted from the OPTQ reference
(Frantar et al. 2022
SUMMARY AND LIMITATIONS
In this paper, we introduce LUT-GEMM, a highly efficient matrix multiplication kernel designed to operate directly on quantized weights, thereby eliminating the need for an additional dequantization step. Leveraging an extended BCQ format, LUT-GEMM exhibits the capability to process both uniformly and non-uniformly quantized models, achieving 2. speedup over GPTQ. However, it is important to note some limitations: Our method primarily focuses single-batch inference and exhibits diminishing performance gains as the batch size increases. This limitation is primarily attributed to the constrained memory bandwidth between and LUTs in the shared memory. Nevertheless, we anticipate that this constraint will gradually diminish in significance with the advent of advanced hardware solutions.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
In the Supplementary Materials, we furnish code for replicating our "Kernel Evaluation" eepauaationeeeprimeeett Specifically, this entails:
Offering our CUDA kernel alongside a concise benchmarking script for single matrix-vector products.
Supplying README document that presents example commands and instructions for executing all the scripts.
REFERENCES
Ghazanfar Ali, Sridutt Bhalachandra, Nicholas Wright, Alan Sill, and Yong Chen. Evaluation of power controls and counters general-purpose graphics processing units (gpus). 2020.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:1 1244-1244660 2020.
Aishwarya Bhandare, Vamsi Sripathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Saletore. Efficient 8-bit quantization of transformer neural machine language translation model. arXiv: M.Xiv:/906.005555 00532, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 202.
Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsich. GroupReduce: Block-wise low-rank approximation for neural language model shrinking. In Advances Neural Information Processing Systems, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Yoojin Choi, Mostafa EI-Khamy, and Jungwon Lee. Towards the limit of network quantization. In International Conference on Learning Representations (ICLR), 2017.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson. Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Heiir-Hellsttem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL hittps://arxiv org/ abs 2bs/2204.02311
Insoo Chung, Byeongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baeseong Park, Sangha Kim, and Dongsoo Lee. Extremely low bit transformer quantization for on-device neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020,
pp. 4812-4826. 2020.
R.I. de Queiroz and P.A. Stein. LUT filters for quantized processing of signals. IEEE Transactions on Signal Processing, 52(3):687-693, 2004.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O) 8-bit matrix multiplication for transformers scale. arXiv preprint arXiv:2208.07339 2022.
10

---

Published as a conference paper at ICLR 2024
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A spanse-quantized represen- tation for near-lossless Ilm weight compression. arXiv preprint arXiv:2306.03007, 306.0078 2023.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Ali Edalati, Marzieh Tahaei, Ahmad Rashid, Vahid Partovi Nia, James J. Clark, and Mehdi Reza- gholizadeh. Kronecker decomposition for gpt compression, 2021.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2022.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: pttp:::02200990.0 09574, 2019.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. International Conference on Learning Representations (ICIR), 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203 2022.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-rrithmeticonnl inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018.
Yongkweon Jeon, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Jeongin Yun, and Dongsoo Lee. Biggemm: matrix multiplication with lookup table for binary-coding-baseee quantized dnns. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-14. IEEE, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08361 2020.
Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, et al. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3405-3424, 2021a.
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer- only bert quantization. In International conference on machine learning, pp. 5506-5518. PMLR, 2021b.
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint 13Xi:23006.06229 2023.
Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung- Woo Ha, Nako Sung, and Dongsoo Lee. Alphatuning: Quantization- aware parameter-efficieet adaptation of large-scale pre- trairrd language models. In Findings of EMNLP 2022, 2022.
11

---

Published as a conference paper at ICLR 2024
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on dlement-wise division for post-training quantization. arXiv preprint arXiv:2306.0033// 003177 2023.
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolu- tional networks. In International Conference on Machine Learning, pp. 2849-2858, 2016.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq; Activation- aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00977 00978, 2023.
Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng, and Jeffrey S Vetter. Nvidia tensor core programmability, performance precision. 2018 IEEE international parallel and distributed processing symposium workshops (IPDPSW), pp. 522-531. IEEE, 2018.
Mark D. McDonnell. Training wide residual networks for deployment using a single bit for each weight. In International Conference on Learning Representations (ICLR), 2018.
Pramod Kumar Meher. LUT optimization for memory-basee based computation. IEEE Transactions on Circuits and Systems II: Express Briefs, 57(4):285-228 2010.
Szymon Migacz. 8-bit inference with TensorRT. In NVIDIA GPU Technology conference, 2017.
Tara N. Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In ICASSP, pp. 6655 6659,, 2013.
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), pp. 7696-7705, 2017.
Deepak Narayanan, Mohammad Shocybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient aaree-scale language model training on gpu clusters using megatron-Im. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15, 2021.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raguel Ferúández. The lambada dataset, August 2016. URLpttps:
org/ 10. / 2630551
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. In International Conference on Learning Representations (ICLR), 2018
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esmm Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong. Daniel Toyama, Cyprien de Masson d' dutuume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. arXiv:2112.114466 2021.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XetR-Net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
12

---

Published as a conference paper at ICLR 2024
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron- Im: Training multi- bullii-illionn parameter language models using model parallelism. arXiv preprint arXiv:1999 08053, 2019.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron train megatron megatron-turrnn nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.1111 19900, 2022.
Devesh Tiwari, Saurabh Gupta, George Gallarno, Jim Rogers, and Don Maxwell. Reliability lessons learned from gpu experience with the titan supercomputer at oak ridge leadership computing facility, In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-12, 2015.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Bric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 19771, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. International Conference on Learning Representations (ICLR), 2018.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2022. URL httL:http::/// //axxiv org/ htre/bb//20 22 1121111 10438
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. Alternating multi-bit quantization for recurrent neural networks. International Conference on Learning Representations (ICLR), 2018.
Shiyu Xu, Qi Wang, Xingbo Wang, Shihang Wang, and Terry Tao Ye. Multiplication through single look-up-table (LUT) in CNN inference computation. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, pp. 1-1, 2021.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206. 0861, 2022.
Jiecao Yu, Andrew Lukefahr, David Palframan, Ganesh Dasika, Reetuparna Das, and Scott Mahlke. Scalpel: Customizing DNN pruning the underlying hardware parallelism. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 548-560, 2017.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. Glm- 130b: An open bilingual pre-trained model, 2022.
URL htttt:t//aaxiiit /
02414
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. 22X.:22005.0166.. 2022.
Michael Zhu and Suyog Gupta. To prune, or to prune: exploring the efficacy of pruning for model compression. CoRR, abs/1710.01878, 2017.
13

---

Published as a conference paper at ICLR 2024
MatMul
Communication
Others OPT-66B (2 GPU)
OPT-6.7B
OPT-13B
OPT-30B
OPT-175B (8 GPU)
Occupancy (%)
(1 GPU)
(1 GPU)
(1 GPU)
100
2
2
n
2
2
2
2
2
2
2
2
2
2
2
2
Number input tokens
Figure 5: Latency breakdown of various OPT models. The overall performance is dominated by MatMul and communication operations. Experiments are conducted with A100 80GB and FasterTransformen (v4.0) framework.
A LLM INFERENCE LATENCY BREAKDOWN
Figure presents detailed breakdown of latency for various large-scale generative language models (LMs), specifically OPT models ranging from 6.7B to 175B parameters. This analysis consid- different numbers of input tokens and was conducted on A100 (80GB) GPÜ using the inference framework developed by Nvidia The OPT models are built upon the Transformer architecture Vaswani al. (2017) and consist of identical layers. Each layer incorporates multi-heaa heali- attention and feed-forward network, both involving four primary linear computations with relatively higher time complexity compared to other non-linear operations. Conse- quently, matrix multiplications constitute a substantial portion, accounting for at least 75% of the processing time across various LM sizes and input token lengths. Note that we can observe the emergence of GPU communication overhead in larger models. This is primarily attributable to the limited memory capacity of a single GPU, which necessitates communication between GPUs to accommodate the larger models.
B DETAILED IMPLEMENTATION
Each TB first conducts pre-computation using partial values assigned in order to fill up the number of LUTs. Then LUTs can be shared by all threads inside TB (so mitigate costly global memory accesses) and multiple rows of a submatrix of B can be processed by multiple threads (so as improve throughput). When threads finish retrieving and summing LUT values, scaling factors are fetched (only once for each thread) and multiplied to produce partial outputs. Finally, Txp partial outputs are accumulated across TBs (through at omi cAdd operations, as illustrated in Figure 2 to generate the final outputs. LUTs are stored in shared memory inside GPU and the shared memory presents high bandwidth (e.g., 19TB/s for A100). Thus, high memory accesses for LUTs (while multiple FLOPs can be replaced with one LUT access) enable fast matrix computations. As for the memory size of LUTs, only 1KB is required for every 8 hidden dimensions and the shared memory size is more than a few megabytes (e.g., 20MB for A 100 with 192KB per SM and 108 SMs available). Thus, the whole LUT's be safely stored in shared memory. To illustrate, the hidden dimension can be up to 324,000 for A100 while 12,288 is the hidden dimension for GPT-3 175B.
C CONVERSION OF UNIFORM QUANTIZATION INTO BCQ
q-bit uniformly quantized weight v with scaling factor s can be expressed (prior to conversion)
as follows:
i 2 + 6, [0.
(5)
14

---

Published as a conference paper at ICLR 2024
Then, ii can be rewritten as:
9-1
2
2b: + 2 =
Σ
(2b, - 1) +
1
8
2
+
+2
2 1=0
2
2
(6)
1=0
1=0
Given that binary weight b in the BCQ format can be defined as b 26 1, we obtain
Σ (2'- by)
2 2- +).
which can be interpreted as a special case of BCQ, where 2' and 2 = Σi === 2
ΣE Ory + 2 in Equation 3
In detail, to construct the binary weights and scaling factors from the pre-trained weights, we introduce a two-step methodology. Initially, scaling factors and integer weights are generated via uniform quantization method (Frantar et al. 2022 Lin et al. 2023 Lee et al. (223) described in Equation Subsequently. the integer weights are transformed into binary weights using bitwise operations, described in Equation 7 More precisely, the scaling factors a, and binary weights b; in BCQ format for the i-th bit position computed as 2'-1s and 26, 1 respectively. Here, and b; denote the scaling factor and the binary value the i-th bit position of the integer weight in uniform quantization, respectively.
D IMPACT ON COMPRESSION RATIO
Let q be the number of quantization bits. For a given q. a smaller group size g can lower quantization error the expense of increased memory footprint for scaling factors. Then, the target quantization error serves as constraint for determining a compromise between and thus resulting in a range of achievable compression ratios. In other words, due the introduction of g, we can control the amount of scaling factors and binary vectors as a trade-off process. Note that the memory footprint of conventional row-wise quantization techniques dominated by the size of binary vectors because the size of scaling factors can usually be ignored if the column width of a matrix is large enough, Compared to the conventional scheme, our proposed group-wise BCQ provides new wide search space for quantization formats to meet target compression ratio.
Figure @ shows an example with two (g. q) configurations to quantize an (4 8) matrix. Indeed, even if the number of quantization bits is smaller, the memory footprint can be large when the group size small (i.e., more scaling factors are employed).
(a)
(b)
Scale Factors (bits)
4 4 16
4xxxxxxx 16
Binary Weights (bits)
4 1xx8x33 3
4
x 4
Scale Factor
Binary Weight B (1-bit)
Scale Factor (16-bit) (b) g 4, q= 4, m 4 8
Binary Weight (1-bil)
Total (bits)
864
640
A (16-bit)
(a) g = 2, g 3, 8
(c) Memory Footprint
Figure 6: Group-wise BCQ example with two different (g. q) configurations to quantize an (4 x 8)
matrix. Assuming a scaling factor of 16 bits, smaller can yield larger memory footprint if also small.
E EXPLORATION OF COMPRESSION RATIO
To study the capability of group-wise BCQ to enlarge search space for compression, conduct experiments on three pre-trained OPT models Zhang et al. (002)), which are publicly available" Specifically, apply post-training quantization (with iterative solver introduced in Xu al.
15

---

Published as a conference paper at ICLR 2024
(2018) to pre-trained OPT models while 9 and q vary. Then, each quantized model is evaluated on the LAMBADA Paperno et al. (2016) dataset to find the relationship between compression ratio and accuracy. Figure 7 |sowss accuracy and compression ratid when try various g values and values. From Figure 7 we observe that compared to the conventional row-wise quantization, group-wise BCQ offers new optimal configurations. Thus, achieve the best compression ratio (or minimum accuracy degradation). it is necessary to explore different and values simultaneously for given target. Note that for OPT-13B and OPT-30B, as discussed the limits of row-wise quantization for large-scale LMs, a small value is critical to achieving low accuracy degradation (while latency heavily affected by a small g). All all, the effects q and on accuracy differ with each model such that and y are hyper-paramettrss to be optimized.
g 32
g 64 Og= 128 Ag 256
g 512 O row-wise
75
30B, FP16
70
13B, FP16
Accuracy (%)
6.7B, FP16
60
55
6.7B, q
13B,
q
-
30B, q 3
50
2.0
2.5
3.0
3.5
4.0
4.5
5.0
5.5
Compression ratio
Figure 7: Accuracy and compression ratio with the various combinations of quantization bits (g) and group size (g). Three pre-trained OPT models are quantized (by post-training BCQ method) and then evaluated on the LAMBADA dataset.
F ADDITIONAL EXPERIMENTAL RESULTS
This section contains additional experimental results.
Table 6: Comparison of perplexity and end-to-end latency per token for OPT family models.
Quant.
Perplexity
Latency (ms)
Model
Kemel-q-9
method
Wiki2
PTB
LAMBADA
1-GPU
2-GPU
4-GPU
cuBLAS- T6- N/A
FP16
9.56
11.84
15.84
40.5
23.5
14.7
LUT-GEMM-4-3
RTN
9.71
12.02
16.16
18.5
14.3
11.9
OPT
LUT-GEMM
RTN
9.88
17.8
13.9
11.8
30B
LUT-GEMM-3-2
RTN
10.78
13.89
19.21
16.7
13.3
11.6
LUT-GEMM-3444
RTN
14.61
24.63
31.13
15.7
12.6
11.2
CuBLAS-I6-N/A
FP16
9.34
11.36
15.47
OOM
48.4
28.3
LUT-GEMM-4-32
RTN
9.54
33.5
18.5
OPT
LUT-GEMM-4-64
RTN
9.54
11.63
15.91
31.9
23.2
17.8
66B
LUT GEMM-3-33
RTN
18.82
35.76
42.16
30.5
21.5
16.9
LUT LUT-GEMM-3-44
RTN
51.15
130.54
130.19
27.5
20.9
16.0
"Calculated FP16 model size divided by each quantized model size.
16

---

Published as a conference paper at ICLR 2024
Table 7: Comparison of perplexity and end-to-end latency per token for LLaMA family models.
Quant.
Perplexity Wiki2
Latency (ms)
Model
Kernel-q-g
method
I-GPU
2-GPU
4-GPU
CuBLAS-I6 16-N/A
FP16
5.68
10.0
7.6
5.6
LLaMA
LUT-GEMM-4-IT
AWQ
5.96
6.1
5,4
5.7
7B
LUT-GEMM-3
AWQ
7.01
5.5
5.1
5.3
cuBLAS-
FP16
5.09
18.2
12.0
8.6
LLaMA
LUT-GEMM-4-128
AWQ
5.25
10.4
8.0
6.9
13B
LUT-GEMM---128
AWQ
5.88
9.3
7.1
6.4
cuBLAS- 16-N/A
4.10
16.9
LLaMA
LUT-GEMM-4-128
AWQ
4.23
20.7
15.5
12.2
30B
LUT
AWQ
4.88
18.1
13.7
11.2
CuBLAS-I6-N/A
FP16
3.53
OOM
46.9
29.1
LLaMA
LUT-GEMM-4-128
AWQ
35.7
25.4
19.9
65B
LUT-GEMM-3-128
AWQ 4.24
31.3
23.0
18.3
PPL numbers extracted from the AWQ reference Lin et al.
2023
125M
350M
2.7B
6.7B
13B
30B
66B
1756
40
Latency (ms)
20
10
FP16
3-bit
4-bit
10
10
10 10
10"
Model size (bytes)
Figure 8: End-to-end end latency comparison between FP16 and quantized OPT models single A100 80GB GPU with g = 128. Note that the latency data for the OPT-66B and OPT-1753 models in FP16, well as the OPT-175B model in 4-bit quantization, omitted due Out Of Memory (OOM) constraints.
Table 8: Accuracy achieved with 2-bit AlphaTuning on MNLI dataset.
Model
Method
g
Ace. (%)
mm Acc. (%)
Full fine-tune
87.5
87,4
2048
85.0
84.9
1024
85.1
85.1
OPT-1.3B
256
85.3
85.6
AlphaTTuning
128
85.7
86.2
64
86.3
86.6
32
86.5
87.1
Full fine-tune
88.5
88.4
1280
85.5
86.0
256
86.8
87.7
OPT-2.7B
AlphaTuning
128
87.3
87.8
64
87.4
87.9
32
87.6
88.4
17

---

Published as a conference paper at ICLR 2024
Table 9: Accuracy of quantized OPT-175B using OPTQ (without group quantization) and the
corresponding end-to-end latency per token by LUT-GEMM.
Perplexity
Accuracy (%)
Model size (GB)
Kernel 4
Latency (ms)
Wiki2
PTB
C4
PIQA
(Comp. ratio)
cuBLAS 16
8.34
12.01
10.13
81.07
347.9 (x1.00)
(8--GU
LUT-GEMM
3
8.68
12.86
10.67
80.03
65.3 (x5.33)
51.6 (I-GPU)
PPL and accuracy numbers extracted from the OPTQ reference (Frantar al. 2022).
18


Reviews:
[{"review_id": "j3lGWF117I", "reviewer": "Reviewer_cNRT", "summary": "Binary Code Quantization (BCQ) applied to large language model weights by LUT-GEMM marks an innovative step, where weight decomposition into quantized matrices and scaling factors is an offline process. This offers a flexibility to balance between compression and accuracy by varying group sizes and quantization bits. At execution time, activations are split into sub-vectors with corresponding look-up tables for sub-sum calculations, replacing the typical multiply-accumulate operations. This \"multiplication\" of binary matrices with activations, followed by scaling, optimizes the model's runtime efficiency. UT-GEMM's application on well-known models, including GPT-3, illustrates significant enhancements in performance, memory usage, energy consumption, and a reduction in necessary GPUs.", "questions": "How can the method be extended to larger batches or scenarios when multiple single requests are batched into one?\n\nIs the ability to represent uniform quantization that important in this method? Uniform methods are typically strictly worse as they over-allocate precision to the edges of the distribution.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Extensive empirical evidence presented through various data visualizations confirms the efficiency of LUT-GEMM method.\n\nThe research goes further than most in addressing the real-world deployment challenges on standard hardware, which is a standout feature.\n\nImpressive experiment section focusing on the largest open-source models and fitting them onto a single GPU.\n\nDetailed latency insights provided in the paper lay a solid foundation for understanding the efficiency gains.\n\nWell-written motivation and related work sections. Discussions show a deep understanding of GPU programming.", "weaknesses": "The achievement of deploying huge models on a singular GPU is undermined by the lack of reported accuracy metrics at this scale.\n\nThe paper lacks a direct accuracy and latency comparison among different models, although the OPT model family is a good choice. \n\nThe paper does seem to have some limited technical novelty since it adapts and extends prior method in minor to moderate ways, but the analysis and implementation seem to make up for it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pk1RmtmQ32", "reviewer": "Reviewer_UZqJ", "summary": "This paper proposes to replace FP16 GEMM calls in weight-only-quantization neural network inference with look-up-table-based GEMMs. The proposed technique is designed to help scenarios where the bit width of weight-only-quantization is 4 or less and the weight matrix size is large enough, as is the case for large language models. The core idea is that, when the bit width of weights is low, one can pre-compute multiplication results with the activation tensor for all combinations and store them, and hence most of the multiplications are replace by table look up; the amount of sum operations remain about the same; the amount of memory reads remain about the same. Experimental results include kernel-level latency comparison in Table 2 and end-to-end latency comparison on OPT in Tables 3-5. Source code is included in the supplementary.", "questions": "Please see the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The core idea of the paper makes sense and in fact is more general and beyond the BCQ format used this paper. The key question on rating the originality is the relation to prior works. For example, how does this work relates to BiQGEMM (Jeon et al., 2020)? Section 3.1 does acknowledge this and a few other prior works but it's unclear what are the distinctions. If a satisfactory answer could be provided, then originality could be a big strength for this paper.\n\nThe experimental setup is good, particularly that perplexity test is included.\n\nIncluding source code is a big plus, although I have not verified it.", "weaknesses": "As said above, the originality question is the key. Please provide detailed narrative on the differentiation. Including prior kernels in evaluations would be even stronger.\n\nThe last row of Table 2 suggests 4X speed up with 4-bit quantization at kernel level. However this does not seems to translate to the 4-bit end2end latency in Tables 3 and 4, not anywhere close to 4X. Can you explain why?\n\nA critical property of the proposal is that the benefit gets larger as the matrix dimension increases. Right now the results only show end2end latency for OPT 66B and 175B. Could you add a plot of latency as a function of OPT model sizes, all the way from the smallest to 175B?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ZSSNiVgpa", "reviewer": "Reviewer_banf", "summary": "This paper proposes an quantization optimization method for Transformer-like model on GEMM calculation, which uses integer weight quantization and floating-point activation values, and uses a lookup table computation scheme to achieve inference acceleration. The paper provides specific GPU implementation methods and performance analysis, explains the implementation methods of uniform quantization and non-uniform quantization in this scheme. In the experimental section, the author compared with the basic cuBLAS implementation, and the results showed that the proposed method has much better performance and does not cause significant loss of inference accuracy.", "questions": "1. The speed of inference is important, but it is only one part. I hope the author can compare the advantages and disadvantages of other quantization acceleration methods, including compression ratio of GPU memory requirements, the need for post training calibration, etc. \n2. The author claims that the BCQ quantization used is suitable for both uniform and non uniform quantization, but the experiment only considered uniform quantization. Theoretically, non-uniform quantization has better representation ability to achieve higher accuracy. Although testing the performance of non-uniform quantization may be difficult on current hardware, it is still possible to compare the accuracy differences between the two methods.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The work of this paper is of great significance, especially for optimizing the inference performance of LLM, which is currently highly concerned. Compared to the baseline cuBLAS implementation, the performance of this method has significantly improved.\n2. The design, experiment, and analysis of this paper are relatively solid. For the algorithm design of GPU, this paper provides detailed explanations and time complexity analysis. This paper carefully considers the performance of different compression ratios under different bit and grouping parameters g. \n3. CUDA kernel code and performance test code is provided.", "weaknesses": "1. The novelty of this paper is ordinary. The main parts, including BCQ quantification and group-wise quantization optimization, both come from existing research. This paper proposes to convert uniform quantization into BCQ format, which is not a complex transformation. \n2. The author mentioned that the optimization here currently only focuses on the inference of a single batch, which may limit its use. \n3. The LUT-Based method requires a significant amount of memory capacity in exchange for efficiency. In Table 2, the 4-bit quantified LUT-GEMM storage footprint exceeds the 16 bit model of the cuBLAS baseline. In fact, storage resources are also the main focus of quantization in large language models, not just performance. This paper seems to focus mainly on computational efficiency, but lacks a comparison between memory resource usage. \n4. The experimental data in this paper is not enough, and the main baseline for comparison is 16 bit cuBLAS, lacking a comprehensive comparison of other quantization methods. The experiment only tested the quantization accuracy of PPL indicators using the OPT model on the LAMBADA dataset, which cannot fully demonstrate the effectiveness and universality of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hWhffvaKQ8", "reviewer": "Reviewer_kS8V", "summary": "This paper proposes an interesting post-training quantization method, without needing to dequantize the integer back to FP16. In particular, they leverage binary coding quantization (BCQ) to represent the LLM weights in both uniform and non-uniform fashion, resulting in a set of scaling factor matrices A and binary weight matrices B of +1 or -1. Therefore, the multiplication between weights W and activations X will become A(BX), where BX is composed of repetitive additions with partial sums being stored on the buffer as LUT to accelerate the computation.\n\nThe author also provides a customized CUDA kernel to deploy such LUT-based GEMM on GPUs, achieving a 2.1× improvement on a single GPU when compared to OPTQ using OPT-175B models with 3-bit weight-only quantization.", "questions": "See weaknesses.\n\nIn addition, the authors did not clearly describe how you derive the scaling factors matrices and binary matrices from the pre-trained weights. I think that part is necessary and it will be better if there is any theoretical analysis of the quantization error or variance.\n\nI am open to raising the score depending on the good answers.\n\n--\nUpdate: Thank you for the comprehensive response. Many of my concerns have been resolved, and I've increased the rating score.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Clear ideas and descriptions of representing weights with scaling factor matrices and binary weight matrices. The advantage of using LUT and grouped scaling factors is extensively evaluated. The way of leveraging fine-grained scaling factors enabled a non-uniform quantization format.\n\n2. Solid implementation. The analysis and GPU implementation stand this idea out. I understand the importance of having it work on GPUs but I still wonder how efficient is the implemented kernel as compared to the customized hardware accelerator.\n\n3. Plausible ablation studies. There are a lot of ablation studies to show the effectiveness of every proposed component.", "weaknesses": "One big issue in writing that prevents one from reimplementing the proposed method is how to construct the binary matrices and scaling factor matrices from the pre-trained weights. The attached code assumes randomized matrices so that part cannot provide any useful information regarding this.\n\nFor more technical questions:\n\n1. Regarding the LUT part, what is the repetition that we can exploit form? The quantitative measurements or ablation studies on this will provide more insights into how many benefits we can get for such LUT-based GEMM instead of vanilla additions. Any theoretical analysis on the upper bound of savings brought from LUT?\n\n2. Regarding the scaling factor part, how to determine the group size? In my understanding, if the group size is small, there are no benefits as you have as many matrices as used bits to represent weights. At what group size, do the benefits compensate the cost of the scaling factor matrices? Is there any formula to calculate this?\n\n3. Regarding the experiments, why only consider the OPT model? How about the speedups on other candidates like LLaMa at different scales? In that way, we can also understand the scalability of the used kernel better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)