any4: Learned 4-bit Numeric Representation for LLMs
Authors: Mostafa Elhoushi
Jeff Johnson
Venue: ICML 2025 poster (2025)
Keywords: Quantization
Compression
Acceleration
Primary area: deep_learning->large_language_models
TL;DR: We present any4, a 4-bit weight quantization solution that enables learned arbitrary numeric representation without requiring pre-processing of weights or activations.

Abstract: We present any4, a learned 4-bit weight quantization solution for large language models (LLMs) providing arbitrary numeric representations without requiring pre-processing of weights or activations. any4 yields higher accuracy compared to other related 4-bit numeric representation types: int4, fp4 and nf4, as evaluated on a range of model sizes, generations and families (Llama 2, Llama 3, Mistral and Mixtral). While any4 does not require preprocessing of weights or activations, it is also competitive with orthogonal techniques that require such preprocessing (e.g., AWQ and GPTQ). We also experiment with any3 and any2 and show competitiveness at lower bits. Additionally, we show that we can calibrate using a single curated diverse sample rather than hundreds of samples from a dataset as done in most quantization approaches. We also open source tinygemm, a latency optimized GPU matrix multiplication library for LLMs, that implements any4 using a GPU-efficient lookup table strategy along with other common quantization methods. We open source our code at https://github.com/facebookresearch/any4.

any4: Learned 4-bit Numeric Representation for LLMs
Mostafa Elhoushi
Jeff Johnson
"!
Abstract
FP10
INT4
FP4
NF4
AMYY
We present any4, learned 4-bit weight quantiza- tion solution for large language models (LLMs) providing arbitrary numeric representations with- out requiring pre-processing of weights or activa- tions. any4 yields higher accuracy compared other related 4-bit numeric representation types: int4, Ip4 and nf4, as evaluated model sizes, generations and families (Llama 2, Llama 3, Mistral and Mixtral). While any4 does not require preprocessing of weights or activations, it is also competitive with orthogo- nal techniques that require such preprocessing (e.g., AWQ and GPTQ). We also experiment with any3 and any2 and show competitiveness at lower bits. Additionally, we show that we can calibrate using single curated diverse sample rather than hundreds of samples from a dataset as done in most quantization approaches. We also open source tinygemm, a latency optimized GPU matrix multiplication library for LLMs, that implements any4 using GPU-efficient lookup table strategy along with other common quan- tization methods. We open source our code at http://githbb.oooffffeeboessssat
14
range of
C4 Papleaby
10
Unma3.2
Lama3
Limoa
Unmo33 70B
Model
Figure 1: Perplexity by quantizing various Llama3 model sizes. Our proposed any4 is the most accurate across nu- meric formats.
as 4-bit integer (int4), floating point (fp4), or custom distri- butions such as NormalFloat4 (nf4) (Dettmers al., 2023)) along with quantization grouping (Dai et al., 2021) are used to increase accuracy. Pre-processing weights and/or activa- tions (e.g., AWQ (Lin et al., 2024), GPTQ (Frantar et al., 2023), or weight Hadamard transforms (Ashkboos et al., 2024b; Liu et al., 2024) can aid with accuracy as well. In this paper, we present new learned numeric representation, any4, that does not require online or offline modification of weights activations. any4 quantization accuracy outper- forms other numeric representation types, and is compet- itive with orthogonal quantization algorithms that prepro- cess weights and/or activations (orthogonality implying that some of these techniques can be applied together with any4 representation). Accuracy was evaluated on wide range of model sizes, generations and families.
1. Introduction
Reduced neural network parameter sizes important for efficient inference, whether at datacenter scale, where ac- celerators can be provisioned based more upon arithmetic throughput rather than memory requirements, or with edge devices, where smaller, slower memories could be used improving battery lifetime while meeting performance con- straints. Given training is typically done in high dynamic range floating point arithmetic, techniques lossily com- press weights must deal with the possibility of varying scale factors and outliers. Various weight numeric formats, such
Equal contribution 'FAIR Meta. Correspondence to: Mostafa Elhoushi <m.clhoushi@e org.. Jeff Johnson <jhj @meta.com>.
2. Background
Trained neural network weights tend to be roughly Gaussian in nature but with heavier tails (Goodfellow et al., 2016). In attempting to lossily compress weights via quantization (yielding fewer reproduction values than the original do-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

any4: Learned 4-bit Numeric Representation for LLMs
main), being able closely match the weight distribution with post-quantization possible reproduction values is im- portant for accuracy.
cost amortized across many values (Darvish Rouhani et al., 2020). Grouped quantization is attempt at this. Instead of forcing single scalar value itself to be the entire representa- tion, we can define groups of contiguous values along row or column of the matrix. A common offset and scale factor is defined for a group of values such that the reconstruc- tion error is improved, with typical group sizes in practice being 32- 256. Other variants include Shared Microexpo- nents (Rouhani et al., 2023), providing group-wis, troup-wise shared exponent value (multiplicative scale) to adjust per-scalar 4 bit floating point values (MX4) in lieu of a scale and offset.
2.1. Uniform Integer Quantization
Some of the first neural network quantization works con- cerned uniform integer quantization (Jacob et al., 2018). Given a set of values to quantize, we obtain the maximum absolute value, and set that to the extreme value (e.g.. -128 / +127 for int8 and -8 /+ for int4 quantization), with zero be- ing preserved (int8/int4 zero dequantizes to original domain zero). Each increment between int8/int4 values corresponds to fixed increment (scale) in the original floating point domain.
2.4.
Quantization
Thus far we have discussed uniform (linear) and floating- point (log/linear) distributions. But we can go further and have quantization reproduction values match the seen distri- butions more closely.
This allows for more efficient (chip area and power) hard- ware circuits, as integer multiply-add is much simpler than floating point multiply-add. However, uniform integer quan- tization is best suited to representing samples from a uni- form distribution, which is a mismatch with neural network properties. Increased bitwidth (more dense uniform sam- ples) is needed for accuracy due to the expected distribution mismatch, indicating that there is waste in memory storage.
NormalFloat4 (nf4) (Dettmers et al., 2023) attempts to do exactly this by having the reproduction values (fixed ahead of time) match Gaussian distribution exactly. However, with an even number of reproduction values (e.g., 2" for n bits), we cannot represent a Gaussian symmetrically if we wish to preserve zero. So nf4 is asymmetric, using one of the 16 values to represent zero This results in higher accuracy, especially for partially sparse matrices.
2.2. Floating Point Quantization
Floating point quantization (reducing fractional precision and dynamic range via rounding) is another mechanism. Unlike integer quantization, reproduction values are now non-uniformly spaced. Floating point arithmetic is a piece- wise linear distribution of values: the steps between floating point exponents are geometric in nature (multiply or divide by 2 each increment), but within given exponent value, the spacing of reproduction values is linear (as given by the significand bits). This is slightly closer mapping as Gaussian distribution with zero mean has most of the mass of the distribution at smaller exponent values more densely sampled by floating point than linear distributions on the number line, while within an exponent the spacing of values is still linear.
AbnormalFloat4 (af4) (Yoshida, 2023) is a variant of nf4 which adjusts the distribution based on quantization group size. The larger the quantization group, the larger the ex- pected maximum absolute value of Gaussian distribution samples, but the mass of the distribution would still be close to O. Mapping the nf4 distribution based on the seen ab- solute maximum value would result in much of the mass of the distribution (values closer to the mean) not being as accurately represented. af4 adjusts the distribution based on group size to take this into account.
2.4.1. ARBITRARY NON-UNIFORM QUANTIZATION: ANY4
Such quantization makes sense with hardware support for reduced bit width floating point types (e.g., fp8 formats with Nvidia's H100 GPU and fp4 with Nvidia's 1100 GPU). In lieu of native conversion instructions, bit manipulation can usually convert or round a n-bit fpn value to the nearest standard fp16/bf16 value (thus, fp4 can be emulated on devices with higher bit width floating point support).
Instead of trying to match an a priori data distribution as nf4/af4 do, we can instead learn the distribution from the seen data itself. This was explored in signal process- ing (Lloyd, 1982a; Max, 1960) and any4 explores this for L.L.s. For each set of values along each row of a matrix, we can perform k-means (Lloyd, 1982b; MacQueen et al., 1967) or neural network- based clustering, so each row of the matrix has its own 4-bit quantization code, providing indices into per-row codebook or lookup table (LUTT containing arbitrary floating point dequantization values. This adds little overhead quantization: for each row of a 4x4096 matrix, any4 will add 16 bfloat 6/ values, for an overhead of (16 x sizeof([b]fooat x 8 bits/byte) / 4096 columns = 0.0625 bits for each matrix entry. Like existing
2.3. Grouped Quantization
As the bitwidth (and thus the number of possible quanti- zation reproduction values) decreases, it can be useful to introduce metadata pertaining to groups of values to the quantization to improve accuracy, with metadata storage
2

---

any4: Learned 4-bit Numeric Representation for LLMs
4-bit techniques, for higher accuracy add quantization groups (e.g., each set of g contiguous shared 16-bit scale and zero point). Thus, per-scalar quan- tization group overhead for 128 in our example would be ((4096 128) x (2 x 16)) 4096 = 0.25 bits, yielding 0.0625 0.25 + 4 = 4.3125 bits for any4 representation. Note that standard int4 grouped quantization already 4.25 bits/entry here, with extension to any4 only adding 0.0625 bits/entry of LUT overhead.
For symmetric quantization:
values has
max max(abs(ujjc))
djeG
Qma
0
where G is set of indices of a tensor, and 8 are scaling factors, Q and Qmar are the minimum and maximum values of the lower precision numeric format.
In addition, the likely most efficient way to implement nf4 and af4 in software itself is via the same mechanism as any4: using a LUT, as there no efficient programmatic way to convert a 4-bit integer to an nf4/af4 value using a small number of instructions. To support nf4/af4, our CUDA implementation also allows using a single 16 entry any4 LUT for entire matrix instead of LUT per each matrix row. This paper solely evaluates the latter.
Scaling could be applied different granularities:
Tensorwise: where G is the set of all indices of the tensor. Hence, all elements in tensor, w. share the same scale factors: as = a, Bi.5 = 3, Vi
Rowwise:
where G the set of all indices of a row Elements in each row of a tensor share the same scale factors: aij = Ois 8.. = B,, Vj.
2.5. Quantization Process
Columnwise: where G is the set of all indices of column. Elements in each column of tensor share the same scale factors: 0i. = O.j, 8r.j = Bj, Vi.
Vanilla quantization happens in 2 steps: scaling followed by rounding.
2. SCALING
Groupwise: where G is the set of non-overlapping con- secutive indices along a row (or column), of size 1 x g. where group size, g. is a scalar hyperparameter. Elc- ments in each group, Gk. share the same scale factors: Olijj Gi Bi_ 3r,Gu: Vj s.t. kg S j < k(g+ 1). Values of 64 128 for g usually provide sweet spot between accuracy and overhead for 4-bit quantization.
Numeric formats have different numeric ranges, and high precision numeric formats usually have orders of magnitude larger ranges from low precision numeric formats, e-g.. fp32 ranges from -3.4 x 1038 to +3.44 1038 while int4 ranges from -7 to +8. Moreover, the numeric range of a given tensor could be orders of magnitude different from low precision format (e.g., most weight values range from -0.01 to +0.01 while int4 ranges from -7 to +8). Hence, directly rounding each element a tensor to its nearest value a numeric format will waste most of the bits and lead to high reconstruction error.
Blockwise: where G is the set of indices within two- dimensional block of size b x b. where, b, is a scalar hyperparameter. Elements in each block, GR. of ten- sor share the same scale factors: 0z.j aG 3o.5 = BG Vi,j s.t. kb S i < k(b 1), lb < j < 1(b 1).
Instead, most approaches scale a tensor, or subset of ten- sor, to the range of lower precision numeric format. Given weight tensor w, and an index i, the scaled weight tensor, g, can be expressed as:
In our work, we focus weight-only groupwise quanti- zation (along the reduction dimension) and, unless stated otherwise, use a default group size g of 128.
W: B
2.5.2, ROUNDING
18,,
After scaling, the next step is round the scaled value to the nearest value in the low-precision quantization format:
Scale factors and B, are high precision scalar values that are calculated for each group of indices, G. For asymmetric quantization':
WQ roundg(wss)
(4)
And dequantize: dequant(ww)) auu + 3.
0jea max(w)ea) - min(wsed)
Qmaz Q min
3. Related Work
Bica min(w
Quantization has long been researched to run on CPUs and custom chips (Xie & Jabri, 1992). Various techniques can be categorized into:
Note that some quantization literature scale slightly differently Wi
from us: Ajeg = (w and ws. oq
B
3

---

any4: Learned 4-bit Numeric Representation for LLMs
Weights vs. Activations vs. Gradients vs. Opti- mizer States Quantization can be applied weights only (AWQ (Lin et al., 2024), GPTQ (Frantar et al., 2023)), weights and activations (SmoothQuant (Xiao et al., 2023), LLLM.int80) (Dettmers et al., 2022a)), KV cache (KVQuant (Hooper et al., 2024)), and can be applied to gradients for training (TinyScript (Fu et al., 2020)) and opti- mization states (8-bit Optimizers (Dettmers et al., 2022b)). Auto-regressive decoding with batch size and sequence length is highly memory bound process (a big portion of compute time is spent in loading weights compared to processing activations), thus 4-bit weight only quantization leads to better speedup than 8-bit weight and 8-bit activa- tion quantization (Py Porch, 2024). Moreover, 4-bit weight only quantization leads to better accuracy-speed tradeoff compared 4-bit weight and 4-bit activation quantization. In this research, we focus quantizing weights only.
Pre-processing Weights and/or Activations While many quantization algorithms simply round each high precision value to value in the quantized set of possible values (Round to Nearest (RTN), stochastic rounding (Xia et al., 2021), or adaptive rounding (Nagel et al., 2020)), other algorithms perform some offline or online processing of weights and/or activations. Instead of keeping outlier ac- tivations or sensitive weights, AWQ (Lin al., 2024) and SmoothQuant (Xiao et al., 2023) mitigate their effects by di- viding outlier channels by a scaling factor and compensating by multiplying weights with the same factor. Other quantiza- tion approaches mitigate outliers by applying matrix trans- formations on weights and activations, e.g., QulP (Chee et al., 2023), QuaRot (Ashkboos et al., 2024a) and Spin- Quant (Liu et al., 2024). Another line of research follows an iterative procedure of quantizing weights in subsets, modify- ing unquantized elements to mitigate the errors introduced after quantizing each subset, e.g., GPTQ (Frantar et al., 2023).
Post-Training Quantization (PTQ) vs. Quantization Aware Training (QAT) PTQ refers to quantization on a trained model without the need for further training. QAT refers to quantization during training, whether training a model from scratch, e.g., FP8-LM (Peng et al., 2023), or continually training finetuning a trained model, e-g-. QLoRA (Dettmers et al., 2023). This work falls under PTQ as it does not require further training of model.
A common trend is to use a combination of techniques. QulP cascades incoherence processing with adaptive round- ing, QTIP (Tseng et al., 2024) uses Hadamard transforms to remove outliers, vector quantization for numeric representa- tion and other techniques, while SqueezeLLM preserves a portion of outlier/sensitive values in high precision and ap- plies scalar quantization. In this work, we opt for a one-shot quantization algorithm that does not require any online or offline pre-processing or transformations on weights and/or activations, and focus on the aspect of learning quantization from data with efficient inference in hardware, achieving SOTA accuracies compared to other numeric format ap- proaches and is competitive with orthogonal approaches that pre- pre-process weights and activations. We leave it to fu- ture work to combine any4 with such orthogonal techniques.
Numeric Representation While integer quantization is the most commonly used numeric representation, other numeric representations, that have been explained above, are also used for inference and/or training: fp8 (Wang et al., 2018), fp6 (Gernigon et al., 2023), fp4 (Sun et al., 2020), nf4, and af4 (Yoshida, 2023).
Lookup Table (LUT) Representation While most research quantize to pre-defined numeric formats, other approaches use a dynamic format that is specified for each tensor or subset of elements of tensor using a upok-uup-tble (LUT) (a.k.a. codebook). In scalar quantization techniques, e.g., DeepCompression for CNNs (Han et al., 2016), GOBO for BERT (Zadeh et al., 2020), SqueezeL for LLMs (Kim et al., 2023), LUTs map scalar quantized values to scalar high precision values. In vector quantization techniques (Stock et al. for CNNs (Stock et al., 2020), AQLM for I.MSs (Fgiazarian et al., 2024)), LUTss map vectors of quan- tized values to vectors of high precision values.
4. Proposed Solution
4.1. any4 Algorithm
In any4 quantization, we first apply group-wise scaling, then try to find the optimal numeric representation for each row of a weight matrix. Naively applying K-means clustering on scaled weights will lead to a sub-optimal quantization scheme. This is because K-means clustering will minimize the reconstruction error of the weight matrix rather than the output of multiplying weights with sample inputs, and even for weight reconstruction, K-means clustering will minimize the reconstruction error of the scaled weight matrix rather than the original weight matrix.
Preserving Outlier/Sensitive Values LLM.int80 (Dettmers et al., 2022a) found that keeping < 0.1% of outlier activa- tions and their corresponding weights in high precision min- imizes drop in accuracy. SqueezeLLM (Kim et al., 2023) found that kceping 0.40% outlier weights and an additional 0.05% sensitive weights, determined by a Hessian metric, minimizes accuracy drops. In this work, we quantize all val- ues and keep outlier/sensitive values in higher precision.
We denote a weight matrix with dimensions of N x K as w. an input vector with dimensions M x K, where M = 1 without loss of generality, as æ and the output vector with dimensions M N as y. Matrix multiplication in high
4

---

any4: Learned 4-bit Numeric Representation for LLMs
precision can be expressed
The offset factors, Br. cancel each other out. Hence,
have:
y = w
(5)
and matrix multiplication with quantized weights as:
min E
Mal | min E
Σ 43
(acy Cki, wis, Ri)
y = adequan(wwq)
(6)
For the ith element of output y, this equivalent to:
(14)
y/ =
We now proceed to solve this by a K-Means-style alternating optimization procedure:
Us dequant(wq..)) )az
(8)
0. Initialize: for ith row of a weight matrix, randomly initialize set Q, to random set of 2" values:
Our goal is to find the set of 2" quantized values for row d:
Q: (wog Wo! ........ .....
Q: = {wag "a) ""?""""""""""" 1}
(15)
for r-bit quantization (anyn) that will minimize the expected mean square error in output activations for possible input activations:
1. E-Step: Given Q: and the row of scaled weights:
:us {wss 1. WSS.
(16)
min B
9 ull
(10)
we would like to deduce the best wQ.., for each cor- responding ws., that will minimize the expression defined in Eq. 14. Since in this step, the possible val- ues Q, are fixed and we are merely selecting from a set of discrete values, apply a local minimization step and re-write Eq. 14 to:
We choose a greedy approach minimize the mean of Frobenius norm of the error of the output activation vector by minimizing the absolute eгror of each of its elements:
min E 10: - vel min
B
Σ 22 Vj
dequant(wrq..)
uas.
min (oLyws,, an, ea, EQ.
min
(wi,s - dequant(wQ.,) )) Σ
(17)
Q:
a, min (ws., wQ.,)
wa EQ:
(11)
Again since ai are fixed in this step and are inde- pendent of wQ we can drop that term:
This way, focus dealing with finding the optimal quantization configuration for each row of the weight ma- trix. (Note that GPTQ opts to minimize output activations error different way such that all rows of the weight ma- trix are co-optimized together). Expanding the right hand side of the equation:
w
min SQ EQ.
(WsL. we
(18)
2. M-Step: After applying the E-Step above, each WQ will be set to one the 2" values the set Q, We refer to each set of indices i. that are associated with specific quantized value Q) as cluster. We can re- write Eq. 14 create separate sum term for elements each cluster:
min Q,
19:
Vil
min
(wis
(ac + (r,))
(12)
The high precision weights are mathematically equivalent to applying scaling factors on scaled weights (i.e., re-arrange Eqn. 1 to expand Way into way aisWs.. Bi.z):
min E 1 Q:
Vi
min E
Σ
Σ
as (ppt))
min R. [ 34l
Yj VgEQ!
(19)
= min E
Σ
(0s,) + Bi. was + Bio))az3
4:
To minimize the term, we can aim to set the difference for elements for each cluster to 0:
min
Σ
(as,)(ss., wQ,, )ay
Q
Σ MGE
(as, 042 Wy
(20)
(13)
5

---

any4: Learned 4-bit Numeric Representation for LLMs
The expression inside the expectation operation scalar value. Moreover, except for input activations all the other variables are deterministic and known offline. Hence, the expectation operator is only needed to be applied on input activations:
"Once upon time, a girl named Alice was living alone on an island. One day. she met a wizard ..."
Fiction:
"The United Nations held its Gen- eral Assembly meeting this year amid multi- ple world crises and wars. In his speech, the General Secretary called for
News:
Code: "public static void main(Stringll args) In wwt.emmmout.rinnlnn ("Hello world!"):'n
Σ Mas B|azy| ai YgE
Math: (5.2 + 2.7) 0.6 1.9 2.2
(21)
Facts: "The capital of Egypt is Cairo. It the largest city in the region and is home to..."
Re-writing:
Listing Calibration sample used to generate LUTss.
row group 0 1o q-0.27 pa-2.1
row group 1 gt)
Σ as, Bla5]= B|ay]
Σ
ai Wag [2yl
conotoeteductton cimension))
Q=0.00
Ba-Q.6
VyER?
VyER!
1.2
0.08
0.3
5.8
122
8,5
11.3
(22)
Σ
=
2.8
1.87
4yE
Nikhmber
((-1)) go
11-1) gt
Re-arranging:
(1) Original N K matrix
(2) Apply quantization groups
(example along rows, group size K/2)
E Exeeo Ws,, Bag
0.03
0.07
024
0.5
13.9
Wo?
EvJe ac. Bayl
(23)
any4
(16 entries]
frow examploi
0.03
0.07
0.24
0.5
13.9 row
(3) Clustering
Eqn. 23 states that the optimal value to represent group of scaled weights within a cluster is their aver- age weighted by the product of the scaling factor weight element and mean of the norm of activations applied that element.
-10
-0.06
row (o-1)
Sampled activations
(4) any4 for
10
16
2
We alternate between the E-Step and M-Step till the values of Qi converge.
N K matrix from (2)
nearest LUT entry
The equation of E-Step is equivalent the cluster assign- ment step of K-means clustering, while the equation of M-Step is equivalent the centroid update step of weighted K K-means. Hence, our mathematical formulations guides us to creating the LUT each row of scaled weight matrix by the algorithm depicted in Alg. 1. We also summarize our algorithm in Fig. We speedup the process by paralleliz- ing the loop over each linear weight's 's rows, enabling us to quantize Llama3 8B in 10 minutes.
(6) quantized matrix Index to closest pen-row any4 value (N x K matrix of 4-bit codes)
Figure 2: any4 quantization process
4.2. tinygemm Library
As part of this paper, we present tinygemm, a GEMM li- brary optimized for low-latency LLM inference at small batch sizes (1 to 16) for Nvidia GPU Ampere generation and later architectures. For matrix multiplication y = ww" where æ is of size M x and w of size N x (M and N being the outer dimensions and being the reduc- tion dimension), linear layers, the product of batch size
While most quantization papers use dataset like C4 to obtain set of calibration activations, we hand curate single calibration sample, as shown in Listing. 1, that covers diverse set of topics, and then obtain the mean of absolute of activations along the channel axis to represent E |ar)).
6

---

any4: Learned 4-bit Numeric Representation for LLMs
Llama3.2 1B
Perplexity +
Tasks 1
HumanEval Pass@1
MBPP
WikiText-2
C4
PTB
CodeParrot
MMLU
HellaSwag
GSM8K
BBH
Pass@1
FPI6
9.76
12.77
16.56
3.49
16.46%
21.4%
36.1%
47.7%
6.60%
31.1%
INT4
11.89
15.74
20.32
4.08
9.76%
11.4%
30.1%
44.7%
3.18%
26.2%
FP4
13.01
17.11
21.89
4.28
8.54%
5.8%
29.3%
43.6%
2.27%
23.3%
NF4
10.99
14.63
18.78
3.82
13.4% %
13.8%
33.3%
45.8%
2.65%
26.8%
ANY4
10.63
13.95
17.94
3.71
11.0%
18.6
32.9%
46.7%
3.71
29.0%
Llama3 8B
FP16
6.14
8.93
10.59
2.54
29.3%
41.4%
62.0%
60.1%
50.7%
62.8%
INT4
6.87
11.37
2.83
23.2%
35.4%
59.6%
58.6%
40.6%
58.5%
FP4
7.10
10.22
11.81
2.89
22.0%
36.8%
57.1%
58.5%
35.0%
53.2%
NF4
6.63
9.52
11.14
2.72
23.2 %
39.2%
60.7%
59.1
41.1%
59.0%
ANY4
6.51
9.40
11.07
2.68
21.3%
39.2%
61.0%
59.5%
41.7%
59.2%
Llama3 70B
FP16
2.86
6.77
8.16
1.91
17.7%
60.8%
75.4%
66.3%
80.6%
82.4%
INT4
3.63
7.97
8.86
2.21
18.3%
45.0%
73.0%
66.2%
73.9%
78.4%
FP4
3.94
7.76
8.99
2.17
22.0 %
50.8%
71.9%
65.6%
75.3%
77.9%
NF4
3.43
7.67
8.84
2.15
18.9%
39.6%
73.7%
66.1%
75.9%
79.3%
ANY4
3.20
7.01
8.33
1.99
17.1%
57.4%
75.1%
66.1%
78.5%
81.8%
Table I: Quantizing Llama3 models with various numeric formats. Appendix.
Results for Llama2 and Mistral/Mixtral are in the
and sequence length corresponds matrix dimension M. At M < 8, activation is itself much smaller than tensor core tile sizes (m 16,n 8, k 16) for 16-bit float Ampere+ mma "tensor core" exed-function matrix multi- plication instructions. In this case, each 8 x 16 tile of w (weights) is only used once (no data reuse). Thus, multistage asynchronous pipelining and data reuse concerns in typical high- perfforrrorm GPU GEMM kernels are reduced, as the problem is largely memory latency (or bandwidth) limited. Tensor cores still outperform manual (scalar) matrix multi- plication at M = 1 (GEMV/ matrix-vector multiplication) per our analysis. An early version of tinygemm, largely fo- cused on int4 grouped quantization for small batch sizes, has been part core PyTorch since late 2023, subsequently uti- lized by gpt-fast (PyTorch, 2023), torchao (PyTorch, 2024), and Hugging Face Transformers (Wolf al., 2020).
As there is little no weight reuse opportunity for small batch sizes, and loads into registers can be asynchronous they generally do not stall execution until the point of first use, tinygemm does not use shared memory in many cases. This strategy improves performance at small batch sizes, but is not applicable for larger sizes. To improve efficiency, when M S 8, we maintain weights on the left to use the 16 x 16 tile, computing y (wx""² flipping the order of matrices presented to mma with transpositions performed on the fly, and if M > 8, maintain weights the right for the 8 x 16 tile (y xw").
To implement int4, nf4, or any4 GEMM, we dequantize weights on the fly before mma multiplication. Speed im- proved by always ensuring that we can load matrix data using vectorized 16 byte loads in coalesced and contigu- ous fashion across the warp from global memory. In cases where single thread's quantized tile data is less than 16 bytes (a m m1666116 "B" tensor core layout with quantized 4-bit values only needs bytes loaded prior to dequantiza- tion per CUDA thread per mma), multiple tiles along the reduction dimension ("k-tiles" in tinygemm terminology) can be packed together ensure that wide data loads can be used in all cases.
Many inference works (especially in open source) concen- trate on M = performance, where latency is a concern. Even in this case, where we would be using only 1 or IG of tensor core throughput, we improve latency by laying out matrices in main (global) memory the exact format that mma expects per tile rather than standard row-major column-main major format. Typical tensor core GEMM kernels use shared memory (a small, high-speed user-controllable scratchpad memory) to transpose tiles of matrices into the desired format before multiplication can proceed. We avoid this by performing the transposition in advance, allowing matrix data pass directly from global memory to registers.
I
Instead of typical int4-to-flon fo-aaa dequantization (converting an integer in [-8, 7] to floating point via native instructions or bit manipulation). we can use 1 16-entry LUT per row containing arbitrary floating point values. tinygemm, this LUT is held in single register with lookup provided using

---

any4: Learned 4-bit Numeric Representation for LLMs
Llama3 8B
Quantization Algorithm
Numeric Format
Wiki Perplexity 4
Numeric
WikiText-2 Perplexity L
Numeric Format
Wik
Format
Perplexity
FPI6
RTN
INT4
6.9
INT3
17.1
INT2
1.9E3
GPTQ
INT4
6,5
INT3
8.2
INT2
2.1E2
4-bits
AWQ
INT4
6.6
3-bits
INT3
8.2
2-bits
INT2
1.7E6
QulP
INT4
6.5
INT3
7.5
INT2
85.1
RTN
ANY4
6.5
ANY3
8.0
ANY2
1.0E3
Llama3 70B
FPI6
2.9
RTN
INT4
3.6
INT3
11.8
INT2
4.6E5
GPTQ
INT4
3.3
INT3
5.2
INT2
11.9
4-bits
AWQ
INT4
3.3
3-bits
INT3
4.8
2-bits
INT2
1.7E6
QulP
INT4
3.4
INT3
4.7
INT2
13.0
RTN
ANY4
3.2
ANY3
4.6
ANY2
253.8
Table 2: Quantizing Llama3 models with various quantization algorithms for different bit widths.
GPU warp shuffle functionalityy, with the 4-bit quantization codes used as LUT indices. An alternative strategy would be to use shared memory LUT containing all possible 16 x 16 256 pairs of any4 reproduction values so that two packed any4 values (in byte) can be dequantized per lookup. While this amount of shared memory usage will likely not affect performance (via occupancy) that much, it does suffer shared memory bank conflict penalties in many circumstances.
better results.
We ran on different model families (Llama (Touvron et al., 2023a) and Mistral (Jiang et al., 2023)), differ- ent generations (Llama2 (Touvron al., 2023b) and Llama3 (Grattafiori et al., 2024)), and different sizes (from IB all the way to 70B). We provide results of Llama3 in Table 1, Llama2 in Table A1, and Mistral in Table A2. Our results show any4 has the best accuracies across all models.
Speed Comparisons We benchmark matrix multiplication of vector activation and square weight tensors from 1K to 16K A100 80GB GPU using PyTorch 2.3.0 and provide the speedups of our tinygemm library in Fig. 3, int4, nf4, and any4 were implemented using our tinygemm library. int4 kernels have the highest speedup. reaching close to 3 3x. nf4 and any4 speedups reach up to 2 lower than int4 because of the overhead of looking up the LUTs. Nevertheless, any4 has almost the same speedup as nf4, despite the latter requiring single LUT for whole tensor and the former requiring separate LUT for each row in the weight matrix.
5. Results
We quantize weights of all linear modules of all transformer layers: key, query, value, and output projections, up. down projections and gate for feed-forward networks (FFN), Fol- lowing most quantization papers, we keep weights of em- bedding and final classification layers sigesircissoon.
We evaluate both perplexity and downstream tasks. For perplexity, we ported the implementation of GPTQ for WikiText-2 (Merity al., 2017), C4 (Raffel et al., 2019), and Penn Treebank (Marcus et al., 1993) that is used by codebases of other quantization papers. To add coding do- main, we added perplexity on CodeParrot (CodeParrot).
Comparison with Orthogonal Quantization Techniques As explained in the Related Works section, our work pro- poses a new numeric representation applying RTN (round- to-nearest). Despite our work being orthogonal to others that transforms weights and/or activations to make them more rounding quantization friendly, compare any4 to GPTQ, AWQ, and QulP that use int4 in Table 2. Results of AWQ, GPTQ, and QuIP are obtained from (Huang et al., 2024). In 4-bit the results show that any4 has either the best or competitive performance. For future work, we can evaluate these orthogonal techniques together, replacing the int4 representation with any4.
For downstream tasks, we used Eleuther Harness (Gao al., 2024) for natural language tasks, and BigCode Har- ness (Ben Allal et al., 2022) for coding tasks. Accuracies downstream tasks tend to be noisy (Wang et al., 2024), while perplexity is less noisy indicator of a model's performance.
Comparison with Other Numeric Representations We first compare accuracy any4 with other numeric formats: int4, fp4, nf4. We use group-wise scaling with group size 128, and asymmetric scaling for all models, except for Llama3 70B where we found symmetric scaling leads to
3-bit and 2-hit Quantization Although main goal was b-t representation, ran experiments to see how any3 and
8

---

any4: Learned 4-bit Numeric Representation for LLMs
Ilama3.2 1B
Calibration Data
Number of Samples
Sequence Length per Sample
Perplexity &
WikiText-2
C4
PTB
CodeParrot
FP16
9.76
12.77
16.56
3.49
ANY4
WikiText-2
128
2048
10.70
14.08
18.02
3.74
ANY4
Pile
128
2048
10.70
13.99
18.26
3.74
ANY4
C4
128
4096
10.74
14.14
18.10
3.75
ANY4
C4
128
2048
10.67
14.05
17.97
3.74
ANY4
128
512
10.62
13.96
18.03
3.72
ANY4
Handwritten Prompt
10.63
13.95
17.94
3.71
Table 3: any4 quantization with different calibration data.
test split.
1414 NFC
ANY4
BF16
Group Size In Table 4 we ablate quantization group size from 64 to 1024. any4 always has the lowest perplexity across other 4-bit representations across all group sizes. noteworthy that fp4 and nf4 perplexity degenerates for large group sizes 1024, while any4 only increases marginally,
100
05
I Llama3.2 1B
00
600
614
4216
1024
1208
13512
14380
Group Size
64
128
256
512
1024
FP16
12.77
Figure Speedup our tinygemm CUDA kernels on 80GB A 100 on matrix multiplication of K input by K x K weight, w.r.t PyTorch's 's 6 implementation.
FP4
16.19
17.11
18.
20.43
2.3E6
NF4
14.27
14.63
14.98
15.38
7.8E5
ANY4
13.75
13.95
14.09
14.24
14.34
Table 4: C4 perplexity after quantizing with different group
sizes.
any2 perform compared to the prior orthogonal quantization techniques (Table 2). For 3-bit, any3 is either the best or competitive with other approaches. For 2-bit, QulP is the best, while any2 better than AWQ and competitive with GPTQ.
6. Conclusion & Future Work
We have presented any4, an algorithm to find an optimal low-bit numeric representation for each row in a weight matrix, as well as tinygemm, a matrix multiplication library for low-latency, low-bit inference. We have shown that accuracy of any4 superior to other 4-bit numeric formats with low memory overhead, and competitive with various orthogonal quantization techniques that involve further pre- processing. We would like to explore combining with these orthogonal techniques in the future.
5.1. Ablation Studies
Calibration Data
In Table 3 we ablate with different calibration datasets to calculate sample weighting Eqn. 23 of our any4 algorithm. The results show that proposed handwritten sample per- forms better than commonly used datasets in literature, de- spite being significantly smaller in number of tokens. Note that the handwritten sample or prompt has a fixed number of words translates to different number of tokens depend- ing on the tokenizer that changes with different models. Our prompt has 88 words only, which will in worst case translate to few hundred tokens. These results may indicate that single data sample with diverse topics could be enough or better to calibrate than using many long sample sequences. Our evaluation sequence length is 2048 (following (Lin et al., 2024; Frantar et al., 2023)), calibration is on training split of each dataset, and evaluation is on the validation or
A Acknowledgements
We would like thank Newsha Ardalani for help in running experiments; Daniel Haziza, Francisco Massa, Luca Wehrst- edt, Bram Wasti, Steven Li, and Lin Xiao for discussions.
Impact Statement
This paper presents a work that quantizes pretrained models. The input to the algorithm is model's pretrained weights,

---

any4: Learned 4-bit Numeric Representation for LLMs
architecture, and calibration dataset (which in our case was single hand-written prompt). We have not evaluated if the quantization algorithm increases or decreases any societal impact of the underlying model. One factor that may introduce bias into the model is the calibration dataset. We leave for future work to analyze the effect of different calibration datasets (or prompts in our case) on bias and truthfulness.
Darvish Rouhani, B., Lo, D., Zhao, R., Liu, M., Fowers, J., Ovtcharov, K., Vinogradsky, A., Massengill, S., Yang. L.., Bittner, R., Forin, A., Zhu, H., Na. T., Patel, P., Che, S., Chand Koppaka, L., SONG. X. Som, S., Das, K., T, S., Reinhardt, S., Lanka, S., Chung, E., and Burger, D. Push- ing the limits of narrow precision inferencing at cloud scale with microsoft floating point. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. and Lin, H. (eds.). Advances in Neural Information Processing Systems, vol- ume 33, pp. 10271-10281. Curran Associates, Inc., 2020. wwrrips..eerrprcooommmm aper appr_EEleee/ppeee////////1/////2 fea Bea1
References
Arthur, D. and Vassilvitskii. S. k-means++: the advantages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '07, pp. 1027-1035, USA, 2007. Society for Indus- trial and Applied Mathematics. ISBN 9780898716245.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3.int80:: 8-bit matrix multiplication for transformers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30318- 30332. Curran Associates, Inc., 2022a. URL https: / //proceedings. hcw.psssss.u.ir......pp..r.ere http:ccccpppppeiepee/e /paper//002//11ee//3344222222220
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in ro- tated LLMs. In The Tiihtt- Annual Conference on Neural Information Processing Systems, 2024a. URL net / forum fo dddffq
W38v1X.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit optimizers via block-wise quantization. In Interna- tional Conference on Learning Representations, 2022b. URL https: /oppr net/ fortt/foummmt d= shpkpVXzo3h.
Ashkboos, S., Mohtashami, A., Croci, M. L.,, Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in ro- tated Ilms, 2024b. URL bettps://ariiv www.aawwww...aaaaaee..o.eaoo s/2404. 00456.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized lims. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (cds.), Advances in Neural Information Processing Systems, volume 36, pp. 10088-10115. Cur- ran Associates, Inc., 2023, URL htt.s: http:tt::///roceddd ngs . www.eeeew...ccc.ooooooo htt.:errrrps.ccccppeeeeeeeeeeees 3/FLle//EeeS77111880000000008880 490-Paper-Conference. pdf.
Ben Allal, L.,, Muennighoff, N., Kumar Umapathi, L., Lip- kin, B., and von Werra, L. A framework for the evaluation of code generation models. https: //github. com /bigcode project /beeco/biiood evaluation- arness, 2022.
Chee, J., Cai, Y., Kuleshov, V., and Sa, C. QulP: 2- bit quantization of large language models with guar- antees. shirth-seventh Conference on Neural In- formation Processing Systems, 2023. URL https: //openreview.neee //ogenneviee..et//foommiicccee 2ttp://roormmm2dxxxxgxvxvvvv
Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. In Forty-first International Conference Machine Learn- ing, 2024. URL https: opwnreview.eeeeee//o/pe...aaa.we net for um?id=5mCaIIRTIn:
CodeParrot. Codeparnov/cooeearrrtcelll URL https:
///huggingffacc..ecmc co/
/ coodeparot-cceeat
Dai, S., Venkatesan, R., Ren, M., Zimmer, B., Dally, W., and Khailany. B. Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference. In Smola, A., Dimakis, A., and Stoica, I. (eds.), Proceedings of Machine Learning and Systems, volume 3, pp. 873- 884, 2021. URL wttp.tpppp//ppreeeenngooooo
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Confer- ence Learning Representations, 2023. URLhtttp. / //openreview.oo net.eewwweeeeeee.eeeeeeeeeeeeeee http:r/wwwwee.eeeeeooorr..e..... http:///oorrrmmttttctthh....
orc//pppee__flle/ppppe/////////p
Fu, F., Hu, Y., He, Y., Jiang. J., Shao, Y., Zhang, C., and Cui, B. Don't 't waste your bits! Squeeze activations and gradients for deep neural networks via TinyScript. In
33.f045sfee11111877888e533-3--23 df.
10

---

any4: Learned 4-bit Numeric Representation for LLMs
II, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2304-3314. PMLR, 13-18 Jul 2020. URL https://// proceedi ngs ml r. www.srrsss/eee.. vv1v 1tt.1///222c.mmmmmmmmmm
Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q. Dong, Q. Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang. R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan. S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speck- bacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V. Do, V. Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang. X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Gold- schlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y.. Zhang, Y., Li, Y., Mao, Y., Coudert, 7. D., Yan, Z., Chen, Z.,, Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Bocsenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poul- ton, A., Ryan, A., Ramchandani. A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape. B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhoferr C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Mont- gomery, E., Presani, E., Hahn, E., Wood, E., Le, E.-T., Brinkman, E., Arcaute, E.,, Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Cag- gioni, F., Kanayet, F,, Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G., Sizon, G., Guangyi, Zhang, Lakshminarayanan G., Inan, H., Shojanazeri, H., Zou, H., Wang. H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj. I., Molybog. I., Tufanov, I., Leontiadis, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.- Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhongn.... J., Jin, J., Yang. J., Cummings, J., Carvill, J., Shepard, J.. McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L. A. L.. Silva, L., Bell. L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M.,
Gao, L., Tow, J., Abbasi, B., Biderman. S., Black, S., DiPofi,
A., Foster, C., Golding, I.., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K. Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L.. Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024, URL httph::///// /tt::///zeeoo..oo org prrgg/ecorr// 2608602.
Gernigon, C., Filipp S.-I., Sentieys, O., Coggiola, C., and Bruno, M. Low-precision floating-point for efficient on- board deep neural network processing, 2023. URL http s://arxiv org/ abs 11172.
Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning.
MIT Press, 2016. URL http:: www. deeplearni ngbook. org. Book in preparation for MIT Press.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C. McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C. Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D. Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, F., Lobanova, E.,, Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang. F., Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang. G., Cucurell, G., Nguyen. H., Ko- revaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J.. Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J.. Chi, J.. Huang, J., Liu, J., Wang. J., Yu, J., Bitton, J. Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala- Yeary, L., van der Maaten, L.. Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M.. Singh, M., Paluri, M., Kardas, M., Tsimpoukelli M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M.. Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P.,

---

any4: Learned 4-bit Numeric Representation for LLMs
Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M.. Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev. N. P.. Dong, N.. Cheng, N., Chernoguz. O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P.. Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q. Alao, R., Rodriguez, R., Ayub, R.,. Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang, S., Zhang. S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satter- field, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T.. Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mi- hailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang. X., Wu, X., Wang. X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z. The llama 3 herd of models, 2024. URL https: / a arxiiv.ei org/ abs/ / 2407 21783.
and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F,, Lengyel, G., Lample, G., Saulnier, I.,, Lavaud, I., R., Lachaux, M.- A., Stock, P., Scao, T. L.. Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https:
//arxiv.ogg/abb/t.. wwt.:o///aabb/aaggg/ abs////0/...../ 06825.
Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M., and Keutzer, K. Squeezellm: Dense- and and-sparse quantization. arXiv, 2023.
Lin, J., Tang, J., Tang, H., Yang. S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation- aware weight quantization for Ilm compres- sion and acceleration. In MLSyx, 2024.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant: Llm quantization with learned rotations, 2024. URL https: 6406.
abs 2405..205.
Lloyd, S. Least squares quantization in pem. IEEE Transac- tions on Information Theory, 28(2):129-137, 1982a. doi: 10. 1.11109/TT.199 1982. 1056489.
Lloyd, S. Least squares quantization in pem. IEEE transac- tions on information theory, 28(2): 1ee11119-113 137, 1982b.
MacQueen, J. al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pp. 281-297. Oakland, CA. USA, 1967.
Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Confer- ence on Learning Representations (ICLR), 2016.
Marcus, M. P., Marcinkiewicz, M. A., and Santorini, B. Building a large annotated corpus of english: the penn treebank. Comput. Linguist. 19(2):313333) June 1993. ISSN 0891-2017.
Hooper, C. R. C., Kim, S., Mohammadzadeh, H., Mahoney, M. W. Shao, S., Keutzer, K., and Gholami, A. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL hwtpt::////..... opwnrevieeeeeeeeeeeeeeee net Forum Pid=OLXotew9D)
Max, J. Quantizing for minimum distortion. IRE Trans- actions on Information Theory, 6(1):7-12, 1960. doi: 191.///T..990.1157777.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https:/t//// open review.nets forum i www.ddyyyyydaa
Huang, W., Zheng, X., Ma. X., Qin, H., Lv, C., Chen, H., Luo, J., Qi, X., Liu, X., and Magno, M. An empirical study of Illma3 quantization: From mmss to milms, 2024. URL https://///. /awxiviiivvv..ia/aaa//aaaa orgg abs 2404 14047.
Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? adaptive rounding for post-training quantization. In Proceedings of the 37th In- ternational Conference on Machine Learning, ICML'20. JMIR.org, 2020.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization
12

---

any4: Learned 4-bit Numeric Representation for LLMs
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cour- napeau, D., Brucher, M., Perrot, M.. and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C.,, Goswami, V., Goyal, N., Hartshorn, A., Hosseiii, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y.. Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, L., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https: /arxiviiv///////////////////aav/ org/ s/2307 09288.
Machine Learning Research, 12:2825-2838 2011.
Peng, H., Wu, K., Wei, Y., Zhao, G., Yang. Y., Liu, Z., Xiong, Y., Yang, Z., Ni, B., Hu, J., Li, R., Zhang. M., Li, C., Ning, J., Wang, R., Zhang, Z., Liu, S., Chau, J., Hu, H., and Cheng, Fp8-Im: Training fp8 large language models, 2023.
PyTorch. gpt-fast, 2023. URL sattorch-labsgreg fast.
PyTorch. torchao, 2024. URL https::////gtt /pytorch/ao.
Tseng, A., Sun, Q., Hou, D., and Sa, C. D. QTIP: Quanti- zation with trellises and incoherence processing. The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024. URL https: /openrevi ew net/ /foumm id=7sdkkvvuYCC
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv e-prints, 2019.
Rouhani, B., Zhao, R., Elango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M., More, A., Melnick, L.. Golub, M., Varatkar, G., Shao, I.., Kolhe, G., Melts, D., Klar, J., L'Heureux, R., Perry, M., Burger, D., Chung, E., Deng, Z., Naghshineh, S., Park, J., and Naumov, M. With shared microexponents, a little shifting goes a long way, 2023. URL htttt::/ /tta::////xvvvi attaaaavvvvrrr.. abs / 2302002222 08007.
Wang, N., Choi, J., Brand, D., Chen, C.-Y,, and Gopalakr- ishnan, K. Training deep neural networks with 8-bit floating point numbers. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Gar- nett, R. (eds.), Advances in Neural Information Process- ing Systems, volume 31. Curran Associates, Inc., 2018. URL httpss:/pppoceeddngg..eeeee aper_=illee/pppe//////////e/e 7ef05ec771442::13339344
ad
Stock, P., Joulin, A., Gribonval, R., Graham, B., and Jégou, H. And the bit goes down: Revisiting the quantiza- tion of neural networks. In International Conference on Learning Representations, 2020. URL https: ///penrevvice //ppnreviewee.c.c/mmeeeae wwwpreevewwweeeeeeee formm det.rrmmmeedrreeeeeeee.e.e
Wang. S. I., Gu, A.. Madaan, L., Hupkes, D., Liu, J., Wei, Y., Jain, N., Lai, Y., Sootla, S., Press, O., Rozière, B., and Synnaeve, G. Eval-Arena: noise and errors on I1m evaluations. https: / /tte://iitti //ggttttbbb.... wwwm/rru-evvall eval arena, 2024.
Sun, X., Wang, N., Chen, C.-Y., Ni, J., Agrawal, A., Cui, X., Venkataramani, S., El Maghraoui, K., Srinivasan, V. V., and Gopalakrishnan, K. Ultra-low precision 4-bit training of deep neural networks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1796-1807. Curran Associates, Inc., 2020. URL https://proceeddingg nww..ss......ppp...... cc/ ccppa fiLle//paperr2000/1ii1//1//
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, L., Gugger, S., Drame, M.. Lhoest, Q., and Rush, A. M. Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38- 45, Online, October 2020. Association for Computational Linguistics. URL https: www. ac aeb.org// n nthology emn wmw.p-demos..........mmmmmmmmmmm 6.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023a. URL https:: / arxiv. .rgg org/abs/ abs/ 2302. 13971.
Xia, L., Anthonissen, M., Hochstenbach, M., and Koren, B. A simple and efficient stochastic rounding method for training neural networks in low precision, 2021. URL https:/ /ttp///aavvvv//e///ee//eee/eaavl http:///arri..oor//////0//....
13

---

any4: Learned 4-bit Numeric Representation for LLMs
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Xie, Y. and Jabri, M. Analysis of the effects of quantization in multilayer neural networks using statistical model. IEEE Transactions on Neural Networks, 3(2):334-332) 1992. doi: 10.1 109/72.
Yoshida, D. Nf4 isn't 't information theoretically optimal (and that's good), 2023. URL https::/aaxxx...... s/2306.0690 06965.
Zadeh, A. H., Edo, I., Awad, O. M., and Moshovos, A. GOBO: Quantizing Attention-Baset NLP Models for Low Latency and Energy Efficient Inference In 2020 53rd Annual IEEE/ACM International Symposium Microarchitecture (MICRO), pp. 811-824, Los Alami- tos, CA, USA, October 2020. IEEE Computer Soci- cty. doi: 10.100//MCCR0000222.0000000000.0 URL h:tpps///ob...eeeceoouurrrco... 10.1109/AMCc0022266.000000000000
14

---

any4: Learned 4-bit Numeric Representation for LLMs
Appendix
A. Solution Details
provide here more details about our proposed any4 algorithm.
A.1. Algorithm
We summarize our any4 quantization algorithm Alg.
Algorithm
any4 quantization algorithm.
module2input cal ibrate (model, sample_data)
for module in model:
module. weight moaule.weight ()
#Q =
alpha
beta []
for i in range shape [0])
wSi, alphai, betai scale (w[i, :1)
xi module2input /moduuuuuet/ [module]]] [i]
wQli, = k
samples-wSi,
vabs (xi.rean
(alpen
aptan (betai)
module. wernlle.weigtt.dee wQ
alpul alpha
module.beta beta
B. Further Results
B.1. Comparison with Other Numeric Formats
We compare our any4 numeric format with other numeric formats for the Llama2 family of models in Table Al and for Mistral-7B and Mixtral-7B Table A2.
Mistral-7B Instruct v0.2
Perplexity
Tasks
WikiText-2
C4
PTB
CodeParrot
MMLU
HellaSwag
GSMSK
BigBench
FP16
5.95
8.82
21.77
2.63
58.7%
66.1%
41.7%
51.7
INT4
6.14
9.03
22.02
2.70
57.1%
65.1%
39.7%
50.4%
FP4
6.19
9.10
21.62
2.70
56.6%
64.7%
38.2%
47.7%
NF4
6.06
8.93
24.72
2.66
58.0%
65.5 %
38.5%
51.8%
ANY4
6.00
8.85
23.24
2.64
58.6 %
65.4%
41.1%
51.7%
Mixtral-8x78 Instruct v0.1
FP16
4.14
7.18
16.47
2,20
68.2%
67 .6%
64.8%
68.1
INT4
4.45
7.45
16.84
2.26
66.5%
66.3%
57.8%
61.8%
FP4
4.46
7.48
18.42
2.27
66.8%
66.5%
59.4%
62.8%
NF4
4.30
7.32
15.00
2.24
67.6%
67.2% %
61.0%
66.5%
ANY4
4.27
7.27
16.14
2.22
67.7 %
67.1%
62.8%
65.8%
Table A2: Quantizing Mistral and Mixtral with various numerie formats.
15

---

any4: Learned 4-bit Numeric Representation for LLMs
Llama2
Perplexity +
Tasks t
HumanEval Pass @
MBPP Pass@
WikiText-2
C4
PTB
CodeParrot
MMLU
HellaSwag
GSM8K
BBH
FPI6
5.47
6.97
20.83
2.54
17.1%
20.0%
41.3%
57.2%
13.6%
39.8%
INT4
5.74
7.30
24.00
2.63
10.4%
18.2%
38.1%
56.4%
10.6%
36.5%
FP4
5.83
7.37
22.57
2.65
11.0%
16.8%
36.5%
56.6%
11.2%
35.5%
NF4
5.66
22.82
2.60
11.6%
19.2%
37.4%
56.8%
12.0%
36.8%
ANY4
21.23
2.57
14.0%
18.4%
40.3%
56.7%
12.7 %
36.9%
Llama2 13B
FP16
4.88
6.47
28.93
2.40
19.5%
18.4%
50.5%
60.0%
23.2%
47.4%
INT4
5.05
30.79
2.45
15.2%
16.4%
48.8%
59.3%
20.8%
44.2%
FP4
5.07
6.67
30.96
2.46
15.2%
16.2%
49.5%
59.3%
19.3%
43.0%
NF4
4.99
6.58
31.17
2.43
15.9%
16.0%
49.9% %
59.9%
22.1% %
44.6%
ANY4
4.97
6,55
28.83
2.42
15.2%
18.0%
49.3%
59.5%
21.6%
44.6 %
Llama2 70B
FPI6
3.32
5.52
14.44
2.11
31.7%
37.4%
65.2%
64.8%
53.3%
67.1%
INT4
3,46
5.61
14.71
2.14
26.8%
37.8%
64.4%
64.7%
51.4%
65.0%
FP4
3.53
5.67
14.34
2.16
28.0%
30.6%
64.1%
64.0%
51.6%
65.0%
NF4
5.61
14.65
2.14
29.9%
37.2%
64.5%
63.9%
50.6%
65.4%
ANY4
3.40
5.58
14.64
2.13
26.8%
35.8%
64.8%
64.5%
51.6%
66.6%
Table A1: Quantizing Llama2 models with various numeric formats.
C. Further Ablation Studies
C.1. Minimization Terms
In Table A3 we ablate using different terms to minimize when learning (using K-means clustering) the LUT of each row in the weight matrix. First row shows the results of optimizing weights directly. The other 2 rows show the results of using the 2 additional terms of Equation 14 in our paper, i.c., multiplying with activations and scales. These results confirm that our derivation that lead to all the terms Equation 14 is essential for optimal accuracy.
Llama3.2 1B
Perplexity
Term Minimize
Wiki'eext-2
C4
PTB
CodeParrot
Weights Only
(ws.., u&,,,)
6.680
9.619
11.186
2.751
Weights x Activations
(ws,,, ).) WS, a.s UQ. 2.))
6.496
9.375
11.055
2.675
Weights Activations Group Scales
6.487
9.366
11.034
2.680
[Ours]
Table A3: Perplexity after quantizing Llama3.2 with LUTs created by minimizing different terms.
C.2. K-Means Initialization
We seikit (Pedregosa et al., 2011) to implement K-means clustering, that is core to any4's quantization algorithm. By default, scikit initializes cluster centroids using k-means++ algorithm (Arthur & Vassilvitskii, 2007), but it also supports random initialization, as well as initializing with a vector of pre-defined values. In Table A4 we ablate K-means initialization on Llama 3.2 IB by evaluating k-means++ and random initialization, as well as seeding with uniform int4 values (i.e., integer values -7 to 8), and nf4 values (ranging from to +1). We see that k-means++ performs clearly the best, while uniform int4 initialization performs the worst.
16

---

any4: Learned 4-bit Numeric Representation for LLMs
Ilama3.2 1B
Perplexity +
K-Meam Initialization
WikiText-2
C4
PTB
FPI6
9.76
12.77
16.56
ANY4
k-means++
10.63
13.95
17.94
ANY4
random
10.66
13.97
18.17
ANY4
int4
10.83
14.21
18.69
ANY4
nf4
10.65
13.96
18.21
Table A4: any4 quantization with K-means clustering initialzied with different algorithms and values.
17


Reviews:
[{"review_id": "0qALXg4ef1", "reviewer": "Reviewer_pDZ9", "summary": "The paper introduces a method for finding the optimal 4-bit quantization codebook for quantizing pre-trained language models. This is done by applying the LLoyd-Max algorithm to each *row* of the weight matrices of the model, thus finding an optimal (per-row) quantization look-up table. In order to find the optimal quantization values, the authors define the mean-squared error criteria not in terms of the weights themselves, but in terms of the outputs of the input-weight matrix multiplication, using a short hand-crafted input text sequence to achieve the calibration. They show that the obtained representation, called any4, performs favorably compared to quantizing to int4, fp4, and nf4 formats.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and constructive feedback, which will help improve the paper.\n\nPlease find below our remarks:\n- **Optimizing Weights Directly:** Please find the results in Table A below. First row shows the results of optimizing weights directly. The other 2 rows show the results of using the 2 additional terms of Equation 14 in our paper, i.e., multiplying with activations and scales. These results confirm that our derivation (Eq. 14) is essential for optimal performance. \n\n| | Term to Minimize in Equation 14 | WikiText-2 | C4 | PTB  | CodeParrot |\n|--|--|--|--|--|--|\n| Optimizing Weights Only | $( w_{S_{i,j}} - w_{Q_{i,j}} )$ | 6.680 | 9.619 | 11.186 | 2.751 |\n| Optimizing Weights * Activations | $( w_{S_{i,j}} x_{j} - w_{Q_{i,j}} x_{j} )$ | 6.496 | 9.375 | 11.055 | 2.675 |\n| Optimizing Weights * Activations * Group Scales [Ours] | $(\\alpha_{i,j} w_{S_{i,j}} x_{j} - \\alpha_{i,j} w_{Q_{i,j}} x_{j} )$ | 6.487 | 9.366 | 11.034 | 2.680 |\n\n**Table A:** *PPL Results on Any4 Quantization on Llama3.2 1B (Lower is Better)*\n\n- **Lack of Novelty:** Please see our response to Reviewer U2Ld\n- **Significant Improvement for Llama3:** Quantizing Llama 3 is known to be more challenging than Llama 1 and 2, often resulting in larger accuracy drops [A]. This is likely due to its larger pretraining dataset (8T tokens vs. 2T for Llama 2 [B, C]), which makes it better trained—and therefore harder to compress without loss. In contrast, older models are relatively undertrained and easier to quantize, so the impact of advanced quantization methods appears smaller. Thus, the stronger results on Llama 3 highlight the effectiveness of our approach, especially on newer, more robust models where quantization is harder and matters more.\n- **PTQ Results using NF4:** We provide in Table B the results of combining AWQ with NF4 as well as results of combining AWQ with ANY4. These results show that combining preprocessing-based methods like AWQ with our numeric format leads to further improvements.\n\n| Model | Quantization Algorithm | Numeric Format | WikiText-2 PPL↓ |\n|--|--|--|--|\n| Llama3 8B | | FP16  | 6.14 |\n| | RTN | INT4 | 6.87 |\n| | RTN | NF4 | 6.63 |\n| | RTN | ANY4 | 6.51 |\n| | AWQ | INT4 | 6.53 |\n| | AWQ | NF4 | 6.51  |\n| | AWQ | ANY4 | 6.38 |\n\n**Table B:** *Combining AWQ with Different Numeric Formats*\n\n- **Algorithm Time Comparison:** We have measured the time to quantize a Llama2 7B and found it to be approximately 10 minutes, which is similar to the time we have measured to run both AWQ and GPTQ. Therefore, we conclude that quantization times of different approaches are similar.\n- **Different Group Sizes:**\n  - We used group size 128 as it is the default in many quantization papers (e.g., AWQ, GPTQ).\n  - In the Appendix we have provided results for group sizes 64, 128, 256, 512, 1024 for Llama3.2 1B.\n  - As requested by the reviewer, we also provide here results for group size 32, as well as the other group sizes we already had in the Appendix\n  - Please note the bitsandbytes library that we use for FP4 and NF4 quantization doesn't support group size 32.\n| | 32 | 64 | 128 | 256 | 512 | 1024 |\n|--|-|-|-|-|-|-|\n| FP4  | N/A   | 16.19 | 17.11 | 18.12 | 20.43 | 2.3E6 |\n| NF4  | N/A   | 14.27 | 14.63 | 14.98 | 15.38 | 7.8E5 |\n| ANY4 | 13.54 | 13.75 | 13.95 | 14.09 | 14.24 | 14.34 |\n\n**Table C:** *Llama3.2 1B Perplexity on C4 (Lower is Better)*\n\n- Other:\n  - **Formatting and Typos:** We thank the reviewer for the detailed comments and we will apply them to the camera ready paper.\n  - **Equation 17 following Equation 14:** We thank the reviewer for highlighting this important point. While Eq. (14) defines a global row-wise objective, Eq. (17) corresponds to a local minimization step within a K-Means-style alternating optimization procedure.\n    - Specifically, Eq. (17) performs the E-step, assigning each weight $w_{s_{i,j}}$ to its nearest codebook value in $Q_i$, treating activations $x_j$ as constants. This simplifies to a local nearest-neighbor assignment. The M-step (Eqs. (19)–(20)) then updates $Q_i$ by minimizing the total reconstruction error across the row.\n    - Thus, while Eq. (17) does not solve the global objective in Eq. (14) directly, it is part of an iterative process that does. We will revise the text to clarify this decomposition and explicitly note that Eq. (17) performs local minimization within the broader alternating optimization scheme.\n  - **Downstream Task Performance:** Please check \"Perplexity is a Less Noisy Indicator then Downstream Tasks\" in our response to Reviewer yZMB.\n\n**References**\n[A] “How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study”, https://arxiv.org/abs/2404.14047v1, April 2024\n\n[B] “Scaling Laws for Floating Point Quantization Training”, https://arxiv.org/abs/2501.02423, January 2025\n\n[C] “The Llama 3 Herd of Models”, https://arxiv.org/abs/2407.21783, July 2024", "claims_and_evidence": "The main claim of the paper is that using the proposed any4 representation leads to better perplexity and task performance than quantizing to standard int4, fp4, and nf4 formats. The authors test their method on Llama 3 models of different sizes (1B, 3B, 8B, and 70B) shown in Figure 1 and  Table 1, as well as Llama 2 (7B, 13B, and 70B) in Table A1, and Mistral-7B and Mixtral-8x7B in Table A2. Overall, any4 does show improved perplexity, which tends to lead to better downstream task performance (but is not always the case). For these results, the authors fixed the group size (for the common scaling factors) to 128. The results do seem to indicate that the difference in performance is more pronounced for Llama 3 family of models, compared to Llama 2 and Mistral.\n\nThe authors also compare perplexity results (on WikiText-2) of their method vs. using post-training quantization techniques with integer formats in Table 2, showing it performs same/better for 4 bits and LLama 3 70B for 3 bits (the increase in perplexity for 3 and 2 bits is overall quite high for all techniques, so I am not completely sure about the utility of those particular results). It would be interesting to see PTQ results using NF4 (as this was the most competitive format otherwise), but I understand these results were taken from Huang et al., 2024 where integer formats were used.", "methods_and_evaluation_criteria": "The authors evaluate their method using the perplexity metric on WikiText-2, C4, and Penn Treebank, as well as on downstream tasks within EleutherAI’s Harness and BigCode’s Harness. These datasets and tasks are in line with what is generally used in literature, and I believe provide a good evaluation basis for quantization approaches.", "theoretical_claims": "The main theoretical derivation in the paper is for the k-means/Lloyd-Max algorithm for finding the optimal quantization codebook given the set of weight elements. The algorithm is fairly standard, and I believe its usage within this work is appropriate. I believe the final algorithm is correct, I just wanted to ask a clarifying question:\n* The authors claim that Eq. (17) follows from (14); this wasn’t quite clear to me, as (14) involves the summation across all row elements, while in (17) we are choosing the closest quantized value for each element in the row independently. Couldn’t the optimal weights for (14) differ from (17), i.e., (17) is an approximation but not the exact solution to (14)?", "experimental_designs_or_analyses": "/", "relation_to_broader_scientific_literature": "I believe the authors did a fine job of relating their work to the prior literature; they use the well-known Lloyd-Max algorithm to find optimal quantization codebooks for each row of every weight matrix within the model, and compare this to quantizing to standard numerical data types, as well as the commonly used post-training quantization techniques.", "essential_references_not_discussed": "I am not aware of any essential references that were not mentioned in the manuscript.", "other_strengths_and_weaknesses": "Strengths:\n* The proposed method overall showcases better perplexity/task performance compared to the standard numerical formats.\n* Good literature review — I believe the authors provided a good coverage of different quantization techniques.\n* Providing an open-source GPU implementation of the method is a very welcome contribution.\n\nWeaknesses:\n* I am slightly concerned that the paper lacks novelty; the main contributions of the paper are using the standard Lloyd-Max procedure per row of weight matrices, as well as defining the problem in terms of the matrix multiplication outputs using a short hand-written calibration dataset. It doesn’t seem too surprising that finding an optimal quantization codebook would lead to better performance compared to pre-defined quantization formats; in addition, the improvement does not seem to be as significant for models other than Llama 3 family.", "other_comments_or_suggestions": "* Some links within the paper (to figures and tables) seem to be broken (for example Fig. 4.1, line 307).\n* The authors might want to consider moving some of the derivation on pages 5 and 6 to the appendix to improve the flow of the main paper, but I leave this to the authors’ preference.\n* It might be useful to specify for Eq. (6) what dequant function is (before it is expanded later in (12)).\n* I believe it would be clearer to use the norm notation in the equations only for the vector equations (i.e. (10)), and not for the scalar equations such as (11).\n* It might be useful to add a more detailed caption for Figure 2.\n* I think the presentation of Table 2 could be made clearer (and “Numeric Format” column for 4-bits is wrong). For example, maybe consider removing “Numeric Format” columns altogether, and indicating next to the PTQ methods that they are associated with int format, while the last RTN row is associated with the any format.\n* (Minor) There are two bolded values in Table 1, MBPP column, Llama 3.2 1B", "questions_for_authors": "The following questions are more of minor/clarifying nature:\n* As the authors compare their technique with a few other PTQ techniques, I was wondering if they could provide a comparison  (could just be a comment/approximate) in terms of the cost of their k-means technique vs. running a PTQ algorithm?\n* Why is group size=128 used for the main results? (I also thought it might be useful to add group size=32 results, as this is the standard group size used for e.g. MXFP)\n* Would be interesting to know how much better is it to optimize outputs on a calibration dataset vs. optimizing the weights directly?\n* The results for any4 seem to be stronger for Llama 3 compared to other model families explored; do the authors have any comments/explanation for this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uuB7NyxCr7", "reviewer": "Reviewer_yZMB", "summary": "The paper presents any4, a new 4-bit numerical representation for post-training quantization of LLMs. The paper states that any4 does not require any additional pre-processing of weights or activations and for most LLMs, can find the optimal 4-bit representation with a single sample. The authors define the basis for the any4 format - using weight-only group quantization. They derive the rest of the math which forms the basis of their k-means based algorithm. This results in an LUT representation, each having 16 elements (for 4-bits). The authors further present tinygemm, a library that has efficient implementations for the different numerical formats (primarily matrix multiplies are achieved via efficient dequantization of weights to 16-bits and then multiplying with the activations). This is followed up by experimental results for different models (Llama 2/3, Mistral), showcasing both perplexity and downstream few-shot evaluation results.\n\n---------\n\n### Update after rebuttal\n\nAfter reviewing the rebuttal and the responses to follow-up questions, I've decided to retain my score. There are several reasons: \n- The difference between the LUT compression row-wise and group-wise quantization introduces a disparity in my mind. It seems like throwing away many of the benefits of group quantization to save memory with row-wise LUTs. It might have been nice to explore other forms of quantization like power of 2 scaling from MX formats to save on memory (uses only a scale factor with 8-bit scales). \n- The RTN explanation still feels lacking. In the end, the authors do apply changes like group wise quantization and an LUT lookup table - for which I'd categorize this method as an LUT method. \n- The overall novelty of the algorithm is limited. LUT compression has been explored in various forms before - and with upcoming fp4 formats in GPUs, having 4-bit codebooks is an expensive choice.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their constructive review, including **“higher confidence in the presented method”** due to presenting results on generation and coding tasks, and thank the reviewer for checking the **“soundness of [the] math”** of the **“full derivation of [the] method”**, as well as for the important suggestions to improve the quality of the paper. \n\nPlease find below our remarks:\n- **Benefit of Group Quantization with Row-wise LUT:** the reason we chose row-wise (look up table) LUT is to minimize the storage overhead. \n  - Each LUT consists of 2^n_bit FP16 values. So for 4-bits, each LUT consists of 16 FP16 values. Having such a LUT for each group of 128 values is a high overhead, while having it for each row (in Llama2 7B each row has 4096 values) the overhead is negligible.\n  - On the other hand, grouping requires 2 FP16 values (one for scale and one for offset) for each group. So we can afford having relatively small group sizes for grouping. \n- **Comparison with SqueezeLLM:** In the paper we haven’t compared with SqueezeLLM because SqueezeLLM keeps a portion (albeit small) of it’s weights in high precision (in their code they keep 10 rows of each weight matrix in high precision as well as 0.45% of outlier and sensitive values in the matrix). Nevertheless, we compare it here as requested by the reviewer in the Table below. \n  - Despite that any4 does not rely on storing rows or portions of its weights in high precision, its perplexity is competitive with SqueezeLLM.\n  - Moreover, we would like to highlight that storing rows or outlier/sensitive values in high precision is orthogonal to any4 and could be combined with it. \n  - The numbers reported below were copied from the SqueezeLLM paper. We tried to run the SqueezeLLM code to quantize Llama3 models, to compare with any4, but we hit into runtime errors in their code.\n\n| Model      | Quantization | WikiText-2 PPL↓ | C4 PPL↓ |\n|------------|--------------|-----------------|---------|\n| Llama2 7B  | SqueezeLLM   | 5.57            | 7.08    |\n|            | ANY          | 5.59            | 7.10    |\n| Llama2 13B | SqueezeLLM   | 4.96            | 6.54    |\n|            | ANY          | 4.97            | 6.55    |\n| Llama2 70B | SqueezeLLM   | 3.39            | 5.57    |\n|            | ANY          | 3.40            | 5.58    |\n\n**Table D:** *Comparison of any4 with SqueezeLLM*\n\n- **Any4 Under RTN Umbrella:** We categorized ANY4 under round-to-nearest (RTN) because in RTN given a list of possible values (whether those values are a LUT as in ANY4, or predefined as in INT4, FP4, or NF4), we round each weight to the nearest one.\n  - Any computation done in ANY4 algorithm is used to obtain the optimal values of the LUT, rather than modifying the values of the weights or activations. While quantization algorithms do modify the values of weights and/or activations:\n    - AWQ scales down the activations and scales up the weights.\n    - GPTQ quantizes each column in a weight matrix sequentially, and every time a column is rounded to the nearest number, the remaining unquantized weight values are modified to mitigate the reconstruction error of rounding previous columns.\n- **Accuracy on All Tasks:** \n  - **Perplexity is a Less Noisy Indicator then Downstream Tasks:** Accuracies on downstream tasks, especially generation tasks, could be noisy. EvalArena [D] studies the noisiness of such downstream tasks. E.g., HumanEval and MBPP appear in EvalArena with a significantly low signal to noise ratio, which explains why results in Table A1 may show ANY4 not always outperforming others on those specific tasks.. On the other hand, perplexity is a less noisy metric as it measures average loss on all tokens in each sample (rather than evaluating a final token or evaluating a pass or fail for a whole sample), and any4 does consistently well on perplexity in Table A1.\n  - **4-bit vs. 3-bit vs. 2-bit:** Table 1 compares perplexity on various bitwidths with quantization algorithms that do process weights and/or activations. While the results show QuIP tends to be better than our approach on lower bits (always better on 2-bits and sometimes better on 3-bits), our approach is always in the top for 4-bits.\n  - **Overall:** instances where any4 is not the top-performing method are a natural outcome of our extensive evaluation across a diverse set of models, generation types, model sizes, and tasks.\n- **Other:**\n  - **Typos and Formatting:** We thank the reviewer for pointing out the typos and issues in formatting. We have applied all the fixes.\n  - **Theoretical Claims:** Please see \"Limited Theoretical Foundation / Formal Proof\" in our response to Reviewer aGNw.\n\n**References**\n[D] EvalArena, https://crux-eval.github.io/eval-arena/", "claims_and_evidence": "I think one thing that is unclear is the differentiation between the group quantization approach which the authors state they use for their method vs the computation of the LUT across an entire row (both in their derivation (equations 15-23), but also in their Figure 2). This is somewhat confusing. What is the benefit of doing group quantization and then creating only row-wise LUT? The end result is many scalars for the actual quantization and then very few actually representable values to represent them. Is this not creating a mismatch in the overall quantization process?", "methods_and_evaluation_criteria": "Yes, the presented evaluations are consistent with other literature in the field, and do fair comparisons against other methods.", "theoretical_claims": "While there are no theoertical claims, the authors present the full derivation of their method. I have checked the soundness of their math.", "experimental_designs_or_analyses": "Yes, the experimental analysis and presented ablations are sound.", "relation_to_broader_scientific_literature": "The paper presents a new 4-bit numerical representation, which is similar to previous efforts such as NF4 [1] and FP4 [2]. For the new method, the authors work with group quantization, which has been tackled in previous works such as GPTQ [3], and their benchmarks are commonly reported in quantization papers [3]. \n\n[1] NF4: https://arxiv.org/abs/2305.14314\n[2] FP4: https://arxiv.org/abs/2310.16836\n[3] GPTQ: https://arxiv.org/abs/2210.17323", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Addn. Strenghts:\n- The authors present benchmarks on generative tasks like MBPP - which gives higher confidence in the presented method\n\nWeaknesses:\n- Despite using a scalar based 4-bit LUT quantization method, the authors do not compare their results with SqueezeLLM, which is the closest method to their paper.\n- Broadly classifying their algorithm under the RTN umbrella is not correct. There are additional transformations going on to ensure high quality of their resulting models - Table 2 needs to fix this.\n- While the overall method does show some benefits on certain tasks, the results are varied for different methods and different tasks. For example in table 2, sometimes their method is better, and sometimes its not. Same with Table A1.", "other_comments_or_suggestions": "1. Table 2 - for the numeric format column, this should talk about the format and not the method used\n2. Lines 307-308, there is no figure 4.1 - please correct this to point to Figure 2 instead\n3. Table A2 needs the correct highlighting for results of Mistral-8x7B model", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "DPdYYIlO2J", "reviewer": "Reviewer_aGNw", "summary": "The paper introduces \"any4,\" a newly proposed learned 4-bit numeric representation aimed at optimizing the quantization of weights in large language models (LLMs). Any4 enhances accuracy compared to traditional 4-bit formats such as int4, fp4, and nf4, and does not require preprocessing of weights or activations. Furthermore, it shows competitive results against other techniques that perform such preprocessing, like Adaptive Weight Quantization (AWQ) and Generalized Post-Training Quantization (GPTQ). The authors also present tinygemm, a latency-optimized GPU matrix multiplication library designed for implementing any4 effectively. The results indicate that any4 achieves superior accuracy across various model sizes and types.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their constructive review and comments, including that the approach was **“rigorously benchmark[ed]”**, the claims are **“supported by experimental results and methodological explanations”**, and that it’s implementation **“enhances the paper’s real-world applicability”**, as well as the important suggestions to improve the quality of the paper. \n\nPlease find below our remarks regarding the reviewer’s requests and comments:\n- **Limited Theoretical Foundation / Formal Proof:** We would like to highlight that our approach is based on a theoretical derivation that we laid out in Section 4.1 of the paper. That mathematical derivation to minimize the reconstruction error of the output of each of the model’s linear layers, eventually lead to a modified k-Means clustering that guides us to apply clustering on the product of weights, mean activations, and group scaling factors (as described in Equation 14). \n  - To prove the claims of our formal proof, we have provided in this rebuttal Table A (in the response to Reviewer pDZ9), empirical results that prove that each term in Equation 14 was necessary to minimize the loss of the model.\n- **Calibration Dataset Concerns**: \n  - In Table A5 in the Appendix, we show that our single sample calibration set outperforms other datasets such as WikiText-2, C4, and The Pile.\n    - The results show that our approach of a single curated sample with sentences covering diverse topics (fiction, non-fiction, code, and math), as shown in Section A.2 in the Appendix could be sufficient to calibrate a quantization algorithm, in contrast to large datasets where each sample typically covers a single domain.\n  - Nevertheless, our approach can work by calibrating with arbitrary datasets with arbitrary number of samples.\n- **Calibration Cost:** \n  - Applying our single-sample calibration takes only 1–3 seconds, while applying a larger number of samples (e.g., 128) that is required by other algorithms could take 10 or 20 seconds. \n  - We have found that the total time to run our quantization algorithm on Llama2 7B is 10 minutes, which is similar to the time reported by other quantization algorithms: AWQ and GPTQ.\n- **Interaction with Other Acceleration Methods:** Overall, our quantization algorithm is orthogonal to other optimizations.\n  - **Torch.compile:** We have already integrated our INT4 implementation of our tinygemm library with torch.compile. We still haven’t yet integrated the ANY4 implementation within the library with torch.compile, but we believe it is straightforward.\n  - **FlashAttention:** FlashAttention in general (and FlashAttentionv3 in particular) only optimizes activation-activation matrix multiplication in the attention operation (i.e., Y = softmax(QK.T / sqrt(d) )V) while our work focuses on quantizing weight-activation matrix multiplication (for attention that would be Q=XWq, K=XWk, V=XWv, O=XWo, and we also quantize all the matrix multiplications in feed forward layers). Hence, from the perspective of end-to-end speedup of a model, our work is orthogonal to FlashAttention.\n- **Other**\n  - **Moving Results from Appendix to Main Body:** We agree with this proposal and plan to do that in the Camera Ready version if the paper is accepted.\n  - **Discussions on Other Optimization Approaches:** We can add discussions to the paper but we can mention here:\n    - Quantization in general is orthogonal to other optimization approaches like pruning and neural architecture search.\n    - We have focused in this paper on Post-Training Quantization (PTQ) that does not require retraining or fine-tuning of the model, while pruning requires extensive fine-tuning or continual pretraining, and NAS requires training from scratch or continual pretraining.\n- **Strengthening Evaluation:**\n  - **Ensuring Statistical Validation:** For the camera ready paper, we plan to re-run some of the text generation experiments with different seeds and report the average, and measure perplexity on different random subsets of the respective datasets and report their averages. However, we have noticed that in most papers on quantization, and on LLMs in general, measurements on perplexity as well as text generation are only provided for a default seed.\n  - **Consistent Experimental Conditions:** For perplexity, we have re-used the same evaluation script that is used in [GPTQ](https://github.com/IST-DASLab/gptq/blob/2d65066eeb06a5c9ff5184d8cebdf33662c67faf/llama.py#L206), that is also used in [AWQ](https://github.com/mit-han-lab/llm-awq/blob/aacd3b8923b080d58734001b3e7842c8ca3e6967/awq/entry.py#L300), and used the exact same seed to select the subsets of data. For downstream tasks, we have used LM Evaluation Harness and BigCode Evaluation Harness with the default settings provided in each library. We believe that our experimental conditions are consistent but we welcome further suggestions to improve consistency.", "claims_and_evidence": "Overall, the claims about Any4 quantization are generally well supported by experimental results and methodological explanations. The performance improvements and efficiency gains are backed by perplexity results and design choices.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper for the any4 quantization technique are appropriate for addressing the challenges in optimizing large language models (LLMs).\n\n1. Methodology: It employs group-wise scaling and K-means clustering for quantization, aiming to improve efficiency and accuracy in weight representation.\n\n2. Evaluation Metrics: Using perplexity and assessing downstream task performance provide a reliable indication of the model's effectiveness in real-world applications.\n\n3. Benchmark Datasets: The inclusion of diverse datasets (e.g., WikiText-2, C4) allows for comprehensive evaluation across different contexts, reinforcing the applicability of the proposed method.\n\nOverall, the methods and evaluation criteria are well-tailored to the problem at hand, offering both theoretical advancement and practical usability in model deployment.", "theoretical_claims": "It does not explicitly detail any formal proofs or theoretical claims.", "experimental_designs_or_analyses": "It compares with other numeric formats (int4, fp4, nf4) and orthogonal quantization techniques (AWQ, GPTQ, QuIP). While the methodology is well-structured, the evaluation could be strengthened by ensuring statistical validation and consistent experimental conditions. Additionally, the ablation studies, currently in the Appendix, provide valuable insights and would be more impactful if moved into the main content for better visibility and discussion.", "relation_to_broader_scientific_literature": "The paper introduces the any4 learned numeric representation, which improves traditional quantization techniques like int4 and fp4 without requiring preprocessing. It consistently outperforms or matches existing methods, using a single diverse sample for calibration instead of many. Results on group size and initialization enhance understanding of numeric stability. The paper suggests future work combining any4 with other techniques, contributing valuable insights to neural network efficiency and quantization research.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "### Strengths:\n\n1. Originality – The any4 representation introduces a novel quantization approach that eliminates the need for preprocessing weights or activations, setting it apart from existing methods and encouraging further exploration in adaptive quantization.\n\n2. Significance – With growing demand for efficient neural networks, any4 effectively reduces parameter size while maintaining high accuracy, making it valuable for both cloud and edge deployment of large language models.\n\n3. Practical Implementation – The inclusion of tinygemm, a GPU-optimized matrix multiplication library, enhances the paper’s real-world applicability, achieving real speedup in GPUs.\n\n4. Clear Evaluation – The paper rigorously benchmarks any4 against multiple numeric formats and quantization techniques, providing strong empirical evidence of its advantages.\n\n### Weaknesses:\n\n1. Limited Theoretical Foundation – While the experiments are thorough, a deeper theoretical analysis of any4’s effectiveness could strengthen its claims.\n\n2. Calibration Dataset Concerns – Relying on a single curated calibration dataset may introduce biases or limit generalizability; a broader dataset selection could improve robustness.\n\n3. Comparative Analysis Gaps – While the paper compares any4 with several quantization methods, expanding the discussion to include other optimization approaches, such as neural architecture search, could provide better context.", "other_comments_or_suggestions": "It has almost three pages of related work and background, which is too detailed and is not really necessary. This structure limits the space in the experiment section, and actually, ablation studies and other content can be moved to the main content. Additionally, consider condensing the related work to focus on key contributions that directly inform the development of any4, allowing for a more concise and impactful presentation of experimental results.", "questions_for_authors": "1. What justifies the effectiveness of using only a single calibration sample? Would using more samples further improve performance? Additionally, how does the calibration cost of your method compare to other quantization techniques?\n\n2. How does tinygemm interact with other acceleration methods, such as torch.compile, which can automatically optimize execution? Furthermore, when used in self-attention, how does its performance compare to FlashAttention (https://pytorch.org/blog/flashattention-3/)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "EK72B0Rddb", "reviewer": "Reviewer_U2Ld", "summary": "This paper introduces a 4-bit weight quantization method called any4, designed for Large Language Models (LLMs).  The authors claim that this method offers arbitrary numeric representations without the need for preprocessing weights or activations.  The paper compares any4 with other 4-bit numeric representation types like int4, fp4, and nf4, using various LLMs (Llama 2, Llama 3, Mistral, and Mixtral) of different sizes and families.  The key finding is that any4 outperforms other numeric formats in terms of accuracy.  The paper also demonstrates that any4 is competitive with techniques like AWQ and GPTQ, which require weight/activation preprocessing.  Additionally, the authors show the competitiveness of any3 and any2 at lower bit widths and highlight the ability to calibrate any4 using a single, curated diverse sample instead of hundreds of dataset samples.  The paper also introduces tinygemm, an open-source GPU matrix multiplication library optimized for LLMs.  tinygemm efficiently implements any4 using a lookup table strategy on GPUs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback, particularly recognizing that the paper **“presents a compelling case”**, and **“provides strong evidence to support its claims and makes a valuable contribution to the field of LLM quantization”**, as well as the important suggestions to improve the quality of the paper. \n\nWe address the reviewer’s comments as follows:\n- **Calibration Sample Construction:** The calibration text sample (that we have provided in Section A.2 in the Appendix) was manually crafted with sentences spanning diverse domains: Fiction, News, Code, Math, Facts, and concatenating them together. Our intuition was that having a single sample with many diverse topics would be sufficient, rather than extracting many samples from a Common Crawl dataset and hoping that they will cover diverse domains.\n- **Combining any4 with AWQ or GPTQ:** We have attempted to combine any4 with AWQ during the rebuttal period and provided results in Table B (in response to Reviewer pDZ9) in this rebuttal. We show that indeed combining AWQ with ANY4 improves the results of AWQ.\n- **Native FP4 Support in New Nvidia B200 GPUs:** We acknowledge that new hardware is supporting FP4 as a native format for computation but we emphasize that our approach is particularly **effective for PTQ** and **can enable arbitrary number of bits**:\n  - In our experiments, we observed that FP4 leads to lower Post-Training Quantization (PTQ) accuracy compared to NF4 and our proposed ANY4 format. i.e., FP4 does not perform well for zero-shot quantizing without training.\n  - Nevertheless, FP4 works well for Quantized Aware Training (QAT). Native support for computing in FP4 enables training models faster with FP4. However, our paper explicitly focuses on post-training quantization (PTQ), a practical setting where model weights are quantized without retraining. \n  - Furthermore, our method also achieves strong results with ANY3, a 3-bit format for which there is currently no native hardware support—highlighting the broader applicability of our lookup-table-based quantization approach beyond current hardware capabilities. Moreover, it could be used for 6-bit quantization [H] which other papers are exploring.\n- **Clarification on ANY2:** We agree ANY2 is not competitive at this stage; we included it to illustrate the extensibility of our method across bit widths, even at the extreme low end. \n  - Moreover, QuIP which performs well on 2-bit quantization is orthogonal to our approach and hence may be integrated in future work to enhance 2-bit quantization performance.\n- **Novelty:** Our method is novel in that it is the first to apply scalar LUT-based quantization minimizing output error in LLMs without any finetuning, unlike prior work [F, G] that provided a formulation in reducing output error rather than weight error but required finetuning.\n\n**References**\n\n[E] “Ultra-Low Precision 4-bit Training of Deep Neural Networks”, Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, Kailash Gopalakrishnan, https://papers.nips.cc/paper_files/paper/2020/hash/13b919438259814cd5be8cb45877d577-Abstract.html, NeurIPS 2020 \n\n[F] “And the Bit Goes Down: Revisiting the Quantization of Neural Networks”, Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, Hervé Jégou, https://arxiv.org/abs/1907.05686, ICLR 2020\n\n[G] “Extreme Compression of Large Language Models via Additive Quantization”, Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, Dan Alistarh, https://arxiv.org/abs/2401.06118, ICML 2024\n\n[H] \"FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design\", Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song, https://arxiv.org/abs/2401.14112", "claims_and_evidence": "Overall, the paper presents a compelling case for the any4 quantization method. The claims are generally well-supported by evidence from experiments and comparisons with existing techniques. However, there are a few areas where further clarification or investigation could strengthen the paper:\n\nStrengths:\n\n- Clear evidence of any4's superior accuracy compared to {int4, fp4, and nf4} in terms of perplexity and downstream task accuracy.\n- Competitive performance with preprocessing techniques such as {AWQ, GPTQ, and QuIP}\n- Effectiveness at lower bit widths (any3)\n- Efficient calibration: The ability to calibrate any4 with a single, diverse sample is a significant advantage.\n- Open-source implementation: The tinygemm library provides a practical implementation of any4, allowing for further research and adoption.\n\nAreas for improvement:\n\n- Calibration sample details: the paper shows using a single, curated diverse sample for calibration, but does not explain how it was constructed or why they thought this was sufficient. More information about this process would be beneficial.\n- Further exploration of any4 with other techniques: While the paper demonstrates any4's competitiveness with existing techniques, it doesn't explore combining any4 with those techniques (e.g., using any4 as the numeric format for GPTQ or AWQ). This could be a promising avenue for future work.\n- Any2: the paper claims any3 and any2 are the effective, but looking at the results, any2 doesn't really work... \n\n\nDespite these minor points, the paper provides strong evidence to support its claims and makes a valuable contribution to the field of LLM quantization.", "methods_and_evaluation_criteria": "Yes\nThe paper made sensible choices (row-wise grouping), focused on downstream accuracy, showed performance across a large set of model sizes, compared to other low precision quant techniques, and had an easy calibration setup. This makes sense for the problem or application at hand.", "theoretical_claims": "Yes.\nThe theoretical claim in the paper:\n```\nEqn. 23 states that the optimal value to represent a group of scaled weights within a cluster is their average weighted by the product of the scaling factor of a weight element and mean of the norm of activations applied to that element.\n```\nThis just follows from their equations.", "experimental_designs_or_analyses": "Yes\n\nExperimental Design:\n- They used a wide range of models (Mistral, Llama 2, Llama 3), and sizes (from 1B to 70B)\n- They compared with a good set of numeric formats (int4, fp4, nf4) and quantization techniques (AWQ, GPTQ, QuIP)/\n- They ablate different group sizes\n\nAnalysis:The paper evaluates speedup vs perplexity and downstream tasks\n\nPotential Issues:\n- limited exploration of combining any4 with e.g. GPTQ or AWQ\n\nDespite these minor points, the experimental designs and analyses in the paper are generally sound and provide convincing evidence to support the claims made about the any4 quantization method.", "relation_to_broader_scientific_literature": "Accelerating llm inference via compressed representations is an active area of research in the community.", "essential_references_not_discussed": "not that i can immediately think of", "other_strengths_and_weaknesses": "The work shows improved performance, but does have performance degradation. Its fast but this is on old hardware (A100s). When B200s come out with native fp4 support, this will not work anymore (unless you can write fp4 codebook with your learned codebook).", "other_comments_or_suggestions": "This is just a learned compression codebook (LUT) which optimizes for reconstructing the output (instead of minimizing mse of weights error). \nI've personally talked about this with 2 separate orgs; in my mind its not novel, but I guess I cant think of anyone who has published it already...", "questions_for_authors": "none", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Despite low scores, reviewers all point to the value of this contribution, and it is clear that it is timely and sufficiently novel for ICML Low-precision computation is relevant across a wide range of ML workloads, and hence is of interest to a large subset of the ICML audience. As noted in review, the implications of this work will change with newer hardware architectures, but in my view, this strengthens the case for timely publication.
Decision: Accept (poster)