QBB: Quantization with Binary Bases for LLMs
Authors: Adrian Bulat
Yassine Ouali
Georgios Tzimiropoulos
Venue: NeurIPS 2024 poster (2024)
Keywords: LLMs; Binarization; Network quantization;
Primary area: natural_language_processing

Abstract: Current post-training quantization methods for LLMs compress the weights down to 4-bits, with moderate to low degradation in accuracy. However, further reducing the number of bits or accelerating the network while avoiding large accuracy drops, especially for smaller, sub 7B models, remains an actively researched and open problem. To address this, in this work, we introduce Quantization with Binary Bases (QBB), a new approach for low-bit quantization that effectively removes (nearly) all multiplications, reducing the implementation to summations. Our novel approach works by decomposing the original weights into a set of binary (1-bit) matrices using an iterative process. For a given layer, starting from a weight matrix, we first construct an initial approximation using an analytical solution, where each new binary matrix, paired with a scaling vector, approximates the residual error of the previous estimation. Secondly, using gradient descent and a progressive learning curriculum, we find the optimal set of binary matrices and scaling vectors that minimize the $\ell_2$ distance between the produced approximation and original weights. Thirdly, as previous steps are input agnostic, we holistically optimize the scaling vectors alone, calibrating them in student-teacher fashion, with the teacher providing both the data, 
 by autoregressive generation starting from a random token, and the target logits.
 When evaluated across multiple LLM families, our approach matches and outperforms all prior works, setting a new state-of-the-art result using a summation-only based approach.

QBB: Quantization with Binary Bases for LLMs
Adrian Bulat 1.2
Yassine Ouali'
Georgios Tzimiropoulos
Samsung Al Cambridge
Tecchical University of lasi 'Queen Mary University of London
Abstract
Current post-training quantization methods for LLMs compress the weights down to 4-bits, with moderate low degradation in accuracy. However, further reducing the number bits or accelerating the network while avoiding large accuracy drops, especially for smaller, sub 7B models, remains actively researched and open problem. To address this, in this work, we introduce Quantization with Binary Bases (QBB), a new approach for low-bit quantization that effectively removes (nearly) all multiplications, reducing the implementation to summations. Our novel approach works by decomposing the original weights into a set of binary (1-bit) matrices using an iterative process, For a given layer, starting from weight matrix, we first construct an initial approximation using an analytical solution, where each new binary matrix, paired with scaling vector, approximates the residual error of the previous estimation. Secondly, using gradient descent and a progressive learning curriculum, we find the optimal set of binary matrices and scaling vectors that minimize the l distance between the produced approximation and original weights. Thirdly, as previous steps are input agnostic, we holistically optimize the scaling vectors alone, calibrating them in student-teacher fashion, with the teacher providing both the data, by autoregressive generation starting from random token, and the target logits. When evaluated across multiple LLM families, our approach matches and outperforms all prior works, setting a new state-of-the-a result using a summation-one based approach.
Introduction
Large Language Models (LLMs) demonstrated proficiency in natural language understanding and generation across multiple domains, exhibiting strong in-context learning abilities [3; 52; 53]. This remarkable performance can be in part attributed to the ever- ever-increasing model and dataset sizes [25; 91, with the current generation of models being trained on Trillion-sized (tokens) datasets [53], using Billions of parameters [53: 24; 29]. This, in turn, made deploying such models, even consumer-gras -rande GPUs, problematic. A promising and flexible direction that disentangles the training and the deployment process, while being hardware friendly, is given by post-training and/or data- free quantization [16; 32; 48]. Within this area, current methods have successfully achieved 4x compression (i.e. 4-bit quantization) with small drops in accuracy [16; 32], especially for larger (7B+) models. However, compressing the model further and/or applying said methods to smaller models yields unsatisfactory results [16; 32]. Importantly, on the inference side, this class of PTQ approaches replaces 16b multiplications with mixed-precision (i.e., 16b-4b) ones, which have limited hardware support and rely on software-bait basaed solutions. Our aim to alleviate these limitations. enabling higher compression rates while removing (nearly) all multiplications.
To this end, departing from prior work, focus the most extreme case of quantization, that of binarization (which, as we will show, enables us to remove costly multiplications. As binarizing the weights directly, without retraining, results in unsatisfactory performance, we propose approximate the weights W using a set of binary matrices B, i E {1, .N with corresponding fp16 scaling vectors a; The main case of interest studied is that of N 4. We note that binarising the weights
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

is particularly interesting case as the matrix multiplications can be implemented using masked selection/data loading and summations, removing all the multiplications, but 1, that with the scaling vector.
As binarising the weights directly is suboptimal (see Sec. 4), we introduce new iterative approach for estimating the optimal binary matrices B, and scaling vectors o;; Firstly, for a given weight W belonging to the I-th layer, we construct an initial approximation using an analytical solution, whereby each new additional binary matrix approximates the residual error of the previous estimation(s). Secondly, using gradient descent, we iteratively adjust the binary weights and scaling vectors one by one by minimizing the &z norm between the target weights and the produced estimation. As until now the process is agnostic to the input distribution of the data, we plug the approximated weights back into the model, optimizing in the interest of speed and generalizability the scaling vectors only, This step uses a teacher-student approach and does not require any training dataset. Instead, we generate synthetic data starting from random tokens by adapting [38] to be more sample efficient. Note that this bypasses the need for any real data. Our approach matches and outperforms prior 4-bit quantization methods across wide variety of LLMs models (e.g: LLaMA-2, Phi-2, etc.) while removing all multiplication (except the final one with the scaling vector), being fast to calibrate and being amenable, in certain conditions to higher compression rates (see Sec. 4.6), setting new state-of-ther arr result.
2 Related work
LLMs quantization is particular application of network quantization [18; 8; 22; 42; 56] to LLMs. As for the former, broadly speaking, these methods can be split into two categories: Quantization- Aware Training (QAT)- - which requires retraining training in quantized form, and Post-Training Quantization (PTQ), which can be applied directly, without additional retraining. As retraining such models, especially without losing generality, is challenging. most of the work focused on [1;; 49; 32; 30J, although a recent line of work tackles the latter too using data-free strategies [38] or parameters efficient fine-tuning [14]. One of the most prominent recent lines of work was the introduction of GPTQ [16] that proposes one-shot low-bit weight quantization method based on approximate second-order information. Follow-up work improves upon it by considering the outliers [27; 133. the effect of the activations on the weight quantization performance [32; 30], by training the quantization parameters [48] or by grouping and reordering the parameters [62]. Methodologically, our approach stands in between PQT and QAT, as we support both layer-wise input agnostic binarization and fast data-free recalibration. In either case, our approach doesn't require a training corpus.
Binarization also known as I-bit quantization, represents the most extreme case of quantization. In this instance, when both operands are binary, the multiplications can be replaced with bitwise operations, while when one is binary and the other real, the multiplications become summations. These desirable properties make binarization a particularly interesting case of quantization. The current base formulation can be traced to [10; 11], which proposes to binarize the values using the sign function, whose gradient is estimated using
straight-thooug estimator [2]. Subsequent works
primarily focus on training aware aware solutions for vision models [46; 33; 34; 5; 4; 15; 58; 64; 63; 26; 6; 40; 35; 61; 45]. For example, Rastegari et al. [46] introduces analytically computed scaling vectors for both the input and the weights, while [4] building on top, proposes to learn them holistically, via backpropagation instead. Liu et al. [34] argues for the use of real-valued wown-sampling layers and the use of double skip connections, while [5] replaces ReLU with PReLUs and makes use of two-staged optimization.
Many works have also studied network binarization in the context of transformers [1; 44; 36; 37], focusing on either full retraining or fine-tuning on labeled datasets using mostly smaller models (c.g.. BERT, BART). More relevant to our work are the recent attempts to adapt and apply binarization to LLMs [55; 39; 47; 60). BitNet [55] swaps all linear layers with BitLinear, their 1-bit weight's replacement, which are then trained from scratch. PB-LLM [47] explores the partial binarization of LLMs. in which the salient weights are stored in higher bits, and the rest are binarized, with a fine-tuning step (i.e., a QAT step) to recover the original LLM performance. BitNet b1.58 [39] encodes every weight using three states (-1,0, instead of two, which are then trained from scratch similar to BitNet [55]. Finally, OneBit [60] proposes a 1-bit QAT method that decomposes the weights into 1-bit weights and two full-precision scaling vectors, which are then fine-tuned with
2

---

Step
Step 2
Step
B
10
sign(-)
sieo
aigu(.)
W
Cout
IWI.
02
Figure 1: Binary weights initialization phase: starting from the target weight W, we construct N binary matrices B, and scaling vectors a analytically. The 1st binary weight and scaling vector are obtained using Eq. 2 while the subsequent ones use Eq. 3.
Step
Step N
B
B-
B.
B
B
B2
$
....
Froben
1
02
DCy
01
ON
W
W
W
W
4
Figure 2: Iterative weights binarization: Starting from the weights initialized as shown in Fig. 1, we update the binary weights one-by-one, where at each step E (1... .N., only B, is updated by minimizing the loss function in eq. (4), while the rest are frozen. Note that all of the scaling vectors ar are updated at every step.
distillation loss for better approximation. However, all these approaches require full retraining or fine-tuning on labeled datasets with a large training overhead. In contrast, our approach is applicable post-training, doesn't require any labeled (nor unlabeled) datasets, is fast optimize, makes use of multiple binary bases, and scales well to billion-sized models.
3 Method
3.1 Layer-wise input-agnostic weights binarization
Given linear layer, represented as f(X, W. b) X W b, with X Rex the input tensor, W E R Krout the weight matrix and, optionally, E Rrout the bias term. n represents the number of tokens, Can and Cout the number of input, and respectively output channels. Note that considering the batch size to be and the bias O for brevity, as the formulations are independent of them. Our goal is to then quantize the elements of W with W such that X W ~ X KWW. Unlike prior works [16; 32] that focus primarily on and 4-bit quantization, herein, we focus on compressing weights using set of binary bases, without comprising the model's accuracy and importantly, without requiring any training or calibration dataset(s), enabling the process a potential implementation based (nearly) solely summations.
For our purposes, let us consider the most extreme case of quantization, that of binarization, whereby the values are represented using 1-bit. In this instance, the real-valued weights W can be approxi- mated W W aB, where B sign (W) is binary matrix and Roott a scaling vector. Given set real-valued weights that are binarized, following Rastegari et al. [46], the optimal value for be computed analytically
cout WWle, vool with WWle,, as the e -n-or computed
the columns of W. Note, that in order to avoid spurious states (L.e. states which the function
(-1, <<0 0
is ill-defined 0), the sign function used is defined as: sign(x:
1, x 2

---

In essence, this relation tells us that we can find the optimal values for B and a by solely looking at W, i.c. without requiring any training data. The downside that the quantization errors are very high due to reduced representation power, which completely degrades the model's performance. TThis perhaps not surprising, as the number of representable states for one given value drops from 216 for 16-bits to 2 for 4 bits and, finally, 2 for the binary case. This discrepancy is even more noticeable when considering the total number of unique permutations representable on such a weight matrix.
To address this, we propose to represent W with a linear combination of N binary matrices:
W Σ a,B,
(1)
In this case, finding the optimal values for Oi, B; {I,..., N no longer be achieved using the analytical solution of Rastegari et al. [46]. As such, we propose a new iterative optimization approach in which the weights and the scaling vectors are iteratively learned via gradient descent. The exact optimization process requires, however, careful considerations. The gradients of sign(.) are 0 almost everywhere. taking the form of the Dirac Delta function. Hence, to make training possible, in practice they are approximated using ttraight-through estimator [2] taking the form of a clipped identity function [46; 11]. These gradients are however noisy with the training prone to oscillations and instabilities: for a value infinitesimally close to O, there exists full state transition to either of the gradient, hence any noise, even a machine representation error, can result in large swings. To alleviate this, we study (i) how to initialize the weights and the scales, and (ii) how to stabilize the optimization process.
extremes, i.e. - or +1, for any
Weights and scale initialization: Given an initial estimation ay B₁ computed analytically by directly approximating the weight matrix W as:
B = sign W).a: =
W
Cout
we then initialize the subsequent o;, i,, i E (2... NJ values by approximating the residual error A at step i:
B, = signg Ai); a,
|| | (Alle,,,0
A,
W
Σ Bg
Cout
This sets the underlying distribution behind each B, matrix to the direction that would minimize the overall quantization error. See Fig. I for an illustration.
Iterative weights and scales optimization: Starting from the above initialization, we aim to minimize
the following loss function:
N L2 = W Σ a B
where both a and B, are trainable parameters, with B, maintained the binary state using the sign function. However, the direct, naive approach of optimizing all parameters jointly leads to training instability. To alleviate this, we devise an iterative training procedure, akin to block coordinate descent, in which the weights are trained one-by-one. Specifically, and as shown in Fig. 2, for N binary matrices, we have N training steps. During each training step, only one binary weight is updated, while the rest stay frozen. We note, however, that the scales are trained in all cases. This approach prevents us from concomitantly changing too many binary weights, effectively simplifying the problem at each step to learning a binary correction factor to existing approximation.
Progressive weights quantization: To further simplify the approximation of full precissenn weights using binary bases, we follow previous binarization approaches [5; 40] and progressively reduce the gap between full-precision and binary weights Specifically, instead of directly approximating full-precision weights, we first quantize them to lower precision using an off-the-shelf method (e.g.. 4-bit using GPTQ). Then, we apply the proposed method with these quantized weights as a starting point (i.e. initialization followed by iterative optimization).

---

3.2 Data-free holistic binarization
Until now, our binarization process was W on per-layer basis. However, this does not factor two important aspects: (1) the effect of cumulative errors resulted from imperfect approximations of the previous layer's weights and (2) the effect on the overall model's output objectives, given that a local approximation isn't necessarily the optimal global solution under quantization induced errors.
considering only the bonae-bin weights
To maintain the generalization properties of the target LLM,, instead of making use of downstream datasets, we generate synthetic data using the (full-precision) model itself, following Liu et al. [38]. Specifically, the 1st token is randomly selected, with the model generating the subsequent ones up to a certain length or until the EOS token encountered. Note that the next token is selected stochastically to increase the diversity of the output sequences.
Once the data is generated. we directly it to train the model with the standard next-token CE loss. However, as the synthetic data is noisy, forcing these labels into the quantized model is suboptimal. Instead, we directly use the produced logits p" soft ground truth, effectively performing Knowledge Distillation [20]. In practice, we use the MSE loss between the student (quantized model) and teacher (full precision model) logits, which is analogous cross-entropy based distillation at high-temperature [20]. The loss is defined as:
IITT. -PS.P... 1
LASE
where V represents the size of the vocabulary and the number of tokens.
By inspecting the per-layer quantization (Sec. 3.1), we notice that different layers within different regions of the model have different quantization errors that sometimes different by up to an order of magnitude. Hence, training them only with a signal applied at the very end the model is suboptimal. Thus, also apply layer-wiss wise MSE loss between the corresponding features of the teacher and student and the end of every transformer block:
Creat
1TT ft
(6)
where f.T and fif are the features after the /-th transformer module of the i-th token produced by the teacher, and respectively student (quantized) model.
The overall loss is then computed as:
Caistil = 81 LNSE + 82 Loaat
with 8 and scaling factors for balancing the two loss terms.
It's generally accepted that higher accuracy gaps between the student-teacher pair hinder the learning process, as the student has a difficult job of matching the teacher's output. To alleviate this, herein, we randomly swap parts of the teacher model with blocks from the student model. Effectively, this creates an instant family of models, with a performance situated between that of student and teacher models. The sampling varied, with increasing difficulty (i.e. fewer blocks swapped) as the training progresses.
Efficient Calibration: To improve the efficiency of our calibration process, firstly, we only fine-tune the scaling vectors a making the training process compute and time efficient, keeping everything else, including the binary weights B, frozen. Secondly, speed up the training further and make more sample-efficient, we propose a simple filtering step of the generated training data. Specifically, we only train self-generated sequences where the quantized student and full-precision teacher differ the most, instead of directly using all the generated samples as in Liu et al. [38]. This is done by first scoring the generated sequences using Eq. 5. then keeping the top-K samples with the highest student-teacher discrepancy. This reduces the training time without any impact on the final results.
4 Ablation studies and analysis
4.1 Impact of the proposed components
5

---

Table Impact the main components measured for Phi-2 (2.7B) and LLaMA-2 (7B) model in terms of perplexity on WikiTex12. N/A indicates failure (> 10" perplexity).
Method
Phi-2 (2.7B)
LLaMA-2 (7B)
without Sec. 3.1 3.2
N/A
N/A
Sec. 3.1
12.32
5.37
Sec. 3.2
10.48
5.21
In this section, we showcase the importance of each proposed component, focusing in particu- lar on what takes make the binarization sta-
0.0035
w
0.0030
ble. For breakdown of the KD subcomponents, see Sec. 4.3. the results from Tab. show, both proposed components, i.e. Layer-wise in- put agnostic weights binarization (Sec. 3.1) and Data-free holistic distillation (Sec. 3.2) improve the model's accuracy and are critical for achiev- ing convergence. Notably, the data-free calibra- tion is particularly important for smaller models, which are generally harder to quantize.
0.0025
0.0020
with cascaded init
w/o cascaded init
0.0015
random init
0.0010
0.0005
0.0000
20
40
60
80
100
120
Moreover, to validate that binarization is inher- ently unstable, we first ablate the initialization step. Fig. 3 we plot, on layer-by-layer basis, the & quantization error for different weight ini- tialization strategies: random initialization the weights are sampled from normal distribution; w/o cascaded initialization all weights attempt to minimize the global error at initialization; with cascaded initialization each weight approximates the remaining residual error (Eqs. 2 and 3). We can immediately observe that random initialization results in eтrors that are an order of magnitude higher. Furthermore, the proposed cascaded initial- ization results in reconstruction errors that are average 2x smaller than the direct alternative, that minimizes the global error instead of the residual one, as proposed in our work. Interestingly, there few layers (the ones harder quantize) for which the two are behaving similarly. Such layers tend to have more outliers, which makes the finding of the optimal solution (that minimizes the average reconstruction error), more prone local minimums.
Layer ID
Figure 3: Per-layer reconstruction error for a Phi-2 model when varying the initialization used: ran- dom, without the cascaded (residual) init, and with.
4.2 Binary weights and process analysis
To offer some insights on the binarization pro- cess, in Fig. 4 show the proportion of the weights that change state between initialization and the end of the iterative training process, on per-layer basis, for each binary base. In- terestingly, the first and last binary matrix dergo minimal changes, with typically only 0.01 0.2% of the weights transitioning state. In comparison, the second, and particu- larly the third base change approximately 40% and respectively 50% of their bases. Moreover, we can observe that for layer IDs 6 and 8, while the proportion of transitions decreases for the second one, the amount increases for the third, which appears to compensate in order to main- tain the same overall proportion of bit flips when aggregated across all bases. These phenomena can be tied to how the initialization is performed: the first binary matrix directly approximates the
Proportion bi flips
0.3-
different
0.7
83-
Layer
Figure 4: Proportion of binary weights that changed their state (bit flips) after applying the proposed iterative training process, shown for the first 20 layers of a Phi-2 model for all bases.

---

full precision weights W, hence it is already close to its optimal solution while the last one has to approximates relatively small residual error, bases give relative similar results (see Fig. 6).
4.3 Effect of distillation
Knowledge Distillation [20] was shown improve the performance of smaller models by the virtue of providing guidance either at the feature level [19] or at the output space, by providing (soft) pseudo-labels. Herein, we ablate various design choices, summarizing the results in Tab. In particular, we consider the following options: (1) CE using directly the synthetic data as hard labels and then calibrating the model using the standard next token prediction Cross- Eessooy Loss; (2) KD Using the soft probabilities provided by the teacher, this is identical to standard Knowledge Distillation
[20]: (3) KD-MSE- Compute a lg (MSE) loss between the logits of the student and teacher; (4) KD-MSE with swap- gradually vary the strength of teacher, from weaker to stronger, by randomly swapping some block in the teacher model with ones from the student and then apply the KD-MSE loss. Note that the per-layer feature loss (eq. 6) is used for all cases except (1) CE, i.e. when the teacher model is not utilised. As the results show, the KD-MSE with swap outperforms all other variants. Morcover, for this use case, the KD-MSE variant outperforms the vanilla KD loss.
Table 2: Effect of different Knowledge Distillation strategies measured using a Phi-2 (2.7B) model in terms of perplexity WikiText2.
CE
KD
KD-MSE
KD-MSE with swap
11.82
10.61
10.48
10.40
PPL vs Number Training Samples
4.4 Effect of data filtering
Random Data
11.4
During the calibration step, in order to make the training more sample efficient, we proposed to filter the randomly self-generated data and only keep the sequences with the highest teacher- student discrepancy as measured by the MSE be- tween their output logits (i.e. Eq. 5). As shown in Fig. 5, this simple filtering step results in faster convergence for the same amount of train- ing samples (i.e. lower perplexity) compared to directly training on the randomly generated sequences.
11.2
13.1
11.0
10.9
10.8
100
200
400
800
1600
12000
6400
96000
Number of Samples
Figure 5: The PPL on WikiText-2 with Phi-2 (2.7b) trained on variable number of training samples. Here, we compare the performance when using the generated data directly, when filtering is applied.
4.5 How many binary bases should we use?
0.025
Throughout this work, we mainly focus on the N case it provides an optimal balance be- tween accuracy, compute cost and storage. For explore
0.020
clarity and completeness, herein, how the performance varies for different values of N. As study case, we take the Phi-2 (2.7B)
error
0.015
model and plot the initial reconstruction errors, after applying the method described in Sec. 3.1. We do so on a layer-by-laye basis, as this will also reveal wherever certain layers are harder to quantize. Looking at the results shown in Fig. 6 we can make the following observations: (1) There is notable performance drop and phase transition for N < 2; (2) Perhaps unsurprisingly, not all layers are made equal, with different lay- ers having different approximation errors, and hence exhibiting variable difficulty levels from
0.010
M
2
0.005
0.000
40
80
100
120
Layer ID
Figure 6: Per-layer reconstruction error after the initial layer-by-layer optimization for Phi- 2 model. Notice that the performance is generally stable for N 2 3.
7

---

quantization point of view; This could be indicative for future quantization efforts; (3) For the layers located in the middle, a pattern emerges, with every block being (nearly) equally hard/easy to quantize. This aligns well with the idea that transformers learn similar representation across most of their layers, unlike, let's say, a convolutional model; (4) For N 5, we can observe improvements, especially for the most difficult layers, while the easier one has a similar value as for N = 4.
4.5.1 Effect of iterative optimisation
Herein, we measure the impact of the proposed iterative training strategy presented in Sec. 3.1. As the results from Fig. 7 updating all weights
.00200
00175
00150
jointly leads to signigicantly worse results, lead- ing to an average error that is one order of mag- nitude higher.
00125
iterative
00100
simultaneous
00075
4.5.2 Effect
of progressive weights quantization
00050
C0025
In line with prior work on binarization [40], aim to gradually increase the quantization level, as this simplifies the problem by reducing the apparent difficulty of the target. To this end, in Tab. 3 we measure the impact of progressive quantization and showcase that our approach is amenable to different quantization targets. On one hand, we can observe that attempting to directly quantize the fp16 weights results de- graded performance, on the other, our approach works with target weights produced by different approaches, in this case: QPTQ and OmniQuant Moreover, we can observe that better int4 weights also translate in better binary bases, with our method incorporating the improved scale and outlier handling from OmniQuant for example.
.00000
0
20
40
60
80
100
120
Layer ID
Figure 7: Per-layer reconstruction error after the initial layer-by-layee optimization for Phi- 2 model. Notice that the performance is generally stable for N 2 3.
Table 3: Effect of progressive weight quantization strategies measured using a LLama-2 (7B)
model in terms of perplexity WikiText2.
FP16
GPTQ
OmniQuant
Starting perf:
5.12
5.61
5.58
Ours:
7.10
5.49
5.21
4.6 Discussion on efficiency
Current low-bit post-training quantization methods result in implementations that replace the full/half precision multiplications present within linear layer with ones performed using operators represented on fewer bits, most frequently 16b-4b sectivation-eightee
When the number of bits for at least one of the operands drops the most extreme case, i.e., 1-bit, all these multiplications Depending on the implementation, they become either a masked selection ((0, 1}) or a conditional sign set (111,1}) Herein, we opt in for the former, as depending on the target hardware, it is possible to reduce the number of summations too, by implementing a masked load, given that the binary matrices generally have sparsity level between 50-80%. Getting back the core message, by removing all the multiplications inside the linear layer, except for the final one with the scaling vector, the power consumption could be reduced, and on custom hardware, it could translate into a smaller die size and faster execution [51; 50; 31; 57], as in principle an adder circuit can be implemented using a flip-flop. While different implementation and technological nodes will result in some variations, taking as an example [21], on 45mm node, and depending on the bit-width of the activation, an adder consumes 8x less energy compared with a multiplier. While left for future studies, further decreasing the activations too to 1-bit can enable even faster implementations based solely on bit-wise operations [4; 10]. Minery-wise, as we learn a sum of binary bases, certain implementations, although not trivial to construct, can exploit either left-most significant bit alignment to compress
removed.
8

---

the weights to (N (N + 1) bits, as a sum of N binary matrices results in N + states, or can make use of codebooks given the cross-matrix similarities and sparsity levels. All in all, we highlight that such representation, which removes (nearly) all the multiplications, opens the door to important energy savings, which are especially important a world where the total energy consumption of Al models is expected to reach 85-135 TWh by 2027 [12] (if current trends follow) this equates to the entire power consumption of the Netherlands.
5 Results
We compare our approach with the current state-of-the-art for low-bit quantization terms of perplexity score on the main benchmark for quantization WWikiTex22 [41], focusing mainly on the LLaMA-2 [53] 17, 13, B family of models. However, we also include results for LLaMA [52] {7, 13, 30, 65 B and Phi-2 [23] 2.7B models. This allows us effectively cover wide range of model architectures and sizes, from the "smaller", and hence harder to quantize, Phi-2 (2.7B parameters) model to the large 70B LLaMA-2 model.
Training details. We implement our method using PyTorch [43], sourcing the pretrained models from Hugging Face [59] repo, as provided by their respective authors. The evaluation code is based on [17]. During the input-agnostic quantization part, presented in Sec. 3.1 and using a single A GPU, we optimize each set of binary matrices and scaling vectors, layer by layer, using the following hyperparameters: Adam optimizer [28], 15000 iterations, no weight decay, an initial learning rate of le 4 decayed to 0 using cosine scheduler. For the data-free distillation step, presented in Sec. 3.2, we fine-tune tune the scaling vectors only for epochs using an Adam optimizer. a cosine learning rate scheduler, no weights decay, and an initial learning rate set to 2.5c 4. For added stability, we clip the gradients with a norm higher than 1.
Baselines. For weight-only quantization, we compare with the primary baseline which is the vanilla Kound-to-neared (RTN) quantization, addition to current state-of-the-an methods for both W4A16 (i.e. weights in 4-bits and activations in 16-bits) and W3A16 quantization. Specifically, we consider GPTQ [16], SpQR [13], QulP# [54], and OmniQuant [48]. Here, we report the results for both per-channel weight quantization and group-wise weight quantization with group size of 128 (i.e. denoted as g 128).
Outcome. As the results from Tab. 4 show, our approach matches or outperforms prior PTQ approaches across all model sizes.
Broader impact and limitations
As our work removes nearly all multiplications, it could improve the energy efficiency of the current generation of AI models, reducing costs while being more environmentally friendly.
In terms of limitations, closed datasets, hence any potential biases present in the original data, affecting the original models would likely propagate the quantized models too. Therefore, deploying such models. Moreover, the benefits and gains mentioned are theoretical, as no custom implementation was constructed. Additional effort may be required to translate the theoretical gains into practice.
work builds on top of existing pre-trained LLMs, some trained on
recommend caution before
6 Conclusions
In this work, we introduced Quantization with Binary Bases (QBB), a new approach for low-bit quantization that effectively removes (nearly) all multiplications, reducing the implementation to summations. Our new approach decomposes the original weights into a set of binary matrices and scaling vector(s) using an iterative process, whereby starting from an initial approximation obtained using an analytical solution, each new binary matrix and scaling vector is optimized in progressive manner using gradient descent by minimizing the f2 distance between the produced approximation and the original weights. Finally, as the previous steps were input agnostic, the scaling factors are optimized holistically, calibrating them student-teacher fashion, with the teacher providing both the data, by autoregressive generation starting from random token, and the target logits. When

---

Table 4: Weights only quantization results of LLaMA and LLaMA-2 models in terms of perplexity on WikiText2.
LLAMA
LLAMA-2
Method
Quantization
7B
13B
30B
70B
7B
13B
70B
FP16
5.68
5.09
4.10
3.53
5.12
4.57
3.12
RTN
6.43
5.55
4.57
3.87
6.11
5.20
3.67
GPTQ [16]
6.13
5.40
4.48
3.83
5.83
5.13
3.58
AWQ [32]
W4A16
6.08
5.34
4.39
6.15
5.12
OmniQuant [48]
5.86
5.21
4.25
3.71
5.74
5.02
3.47
QuIP [54]
5.76
5.17
4.18
3.60
5.56
4.95
3,38
RTN
5.96
5.25
4.23
3.67
5.72
4.98
3,46
GPTQ [16]
5.85
5.20
4.23
3.65
5.61
4,98
3.42
W4A16g
AWQ [32]
5.81
5.20
4.21
5.62
4.97
OmniQuant [48]
5.77
5.17
4.19
3.62
5.58
4.95
3.40
RTN
25.73
11.39
10.68
539.48
10.68
7.52
GPTQ [16]
8.06
6.76
5.84
5.06
8.37
6.44
4.82
AWQ [32]
W3A16
11.88
7,45
10.07
5.21
24.00
10.45
OmniQuant [48]
6.49
5.68
4.74
6.58
5.58
3,92
QulP* [54]
5.98
5.31
4.36
3.78
5.79
5.10
3,56
7.01
5.88
4.87
4.24
6.66
5,51
3.97
GPTQ [16]
6.55
5.62
4.80
4.17
6.29
5.42
3.85
W3A16g128
AWQ [32]
6.46
5.51
4.63
3.99
6.24
5.32
OmniQuant [48]
6.15
5,44
4.56
5.28
3.78
QBB (Ours)
W(4x
5.69
5.14
4.19
3.64
5.21
4.63
3.20
evaluated across multiple LLM families, our approach matches and outperforms all prior works, setting new state-of-the-arr result using summation-only based approach.
10

---

References
[1] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arX1::2012.15701, 2020. 2
[2] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
2, 4
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[4] Adrian Bulat and Georgios Tzimiropoulos. XNOR-Net++: Improved binary neural networks.
In BMVC, 2019. 2,
[5] Adrian Bulat, Georgios Tzimiropoulos, Jean Kossaifi, and Maja Pantic. Improved train- ing of binary networks for human pose estimation and image recognition. arXiv preprint 194.00090 2019. 2,4
[6] Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. BATS: Binary ArchitecTure search. ECCV, 2020. 2
[7] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
[8] Yoojin Choi, Mostafa EI-Khamy, and Jungwon Lee. Towards the limit of network quantization. arXiv preprint arXiv: 16/2.01543, 2016. 2
[9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):
1-113, 2023. 1
[10] Matthieu Courbariaux, Yoshua Bengio, and Cean-Pierre David. BinaryConnect: Training deep neural networks with binary weights during propagations. In NeurlPS, pages 3123-3133 3131,, 2015. 2,
[11] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to++ or-1. arXiv preprint arXiv: 1602: 1602.02830. 2016. 2, 4
[12] Alex de Vries. The growing energy footprint of artificial intelligence. Joule, 7(10):219-21194 2023.
[13] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Salch Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless Ilm weight compression. arXiv preprint 1xXiv:330600077 2023. 2, 9
[14] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024. 2
[15] Ruizhou Ding, Ting- Wu Chin, Zeye Liu, and Diana Marculeseu. Regularizing activation distribution for training binarized deep networks. In CVPR, 2019. 2
[16] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, arXiv preprint arXiv:2210. 17323, 2022. 1,
2, 9, 10
11

---

[17] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. A framework org/ records//
for few-shot language model evaluation, 12 2023. URL 10256836.
[18) Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.61155 2014. 2
[19] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. comprehensive overhaul of feature distillation. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 1921-1930, 2019.
[20] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 100.005503.023//, 2015, 5,
[21] Mark Horowitz. 1.1 computing's s energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technicul papers (ISSCC), pages 10-14.
IEEE, 2014. 8
[22] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-rrithmeticooe
inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018.
[23] Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023. 9
[24] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint ar arXiv:2310.06825, 2023. 1
[25] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar arXix:2001.0836,, 2020. 1
[26] Dahyun Kim, Kunal Pratap Singh, and Jonghyun Choi. Learning architectures for binary networks. In ECCV, 2020. 2
[27] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and and-ssare quantization. arXiv preprint arXiv:2306.07629, 2023. 2
[28] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1411691412.6980, 2014. 9
[29] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 1 176b-parameter open-access multilingual language model. 2023. 1
[30] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint arXiv:2306.0l 02272, 2023. 2
[31] Wenshuo Li, Xinghao Chen, Jinyu Bai, Xuefei Ning, and Yunhe Wang. Searching for energy- efficient hybrid codvol-conollution neural networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1943-1952, 2022. 8
[32] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Im compression and acceleration. arXiv preprint 27Xiv.2306.00978 2023. 2, 3, 10
12

---

[33] Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network.
In NeurIPS. pages 345-353. 2017. 2
[34] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-Real Net: Enhancing the performance of 1-bit CNNs with improved representational capability and advanced training algorithm. In ECCV, pages 747-763, 2018. 2
[35] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28, 2020, Proceedings, Part XIV 16, pages 143-159. Springer, 2020. 2
[36] Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krish- namoorthi, and Yashar Mehdad. Bit: Robustly binarized multi-distilled transformer. Advances in neural information processing systems, 35:14303-14316, 2022.
[37] Zechun Liu, Barlas Oguz, Aasish Pappu, Yangyang Shi, and Raghuraman Krishnamoorthi. Binary and ternary natural language generation. arXiv preprint arXiv:2306.0184/ 2023. 2
[38] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quanti- zation aware training for large language models. arXiv preprint (rxXiv2330..78800 2023. 2, 5
[39] Shuming Ma, Hongyu Wang. Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.
2024. 2
[40] Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to- binary convolutions. ICLR, 2020. 2, 4,
[41] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: /009:v/60907844/ 2016. 9
[42] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019. 2
[43] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. 9
[44] Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. Bibert: Accurate fully binarized bert. arXiv preprint arX.v:2203.06330, 2022. 2
[45] Haotong Qin, Mingyuan Zhang, Yifu Ding, Aoyu Li, Zhongang Cai, Ziwei Liu, Fisher Yu, and Xianglong Liu. Bibench: Benchmarking and analyzing network binarization. In International Conference on Machine Learning, pages
PMLR, 2023. 2
[46] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pages 525-542. Springer, 2016. 3.4
[47] Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large language models. arXiv preprint arXiv:2310.0000 2023. 2
[48] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu. Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantiza- tion for large language models. arXiv preprint arXiv:2308.13137 2023. 2, 9, 10
[49] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pages 31094-31116 PMLR, 2023. 2
13

---

[50] Han Shu, Jiahao Wang, Hanting Chen, Lin Li, Yujuu Yang, and Yunhe Wang. Adder attention for vision transformer. Advances in Neural Information Processing Systems, 34: 199999, 2021. 8
[51] Dehua Song, Yunhe Wang. Hanting Chen, Chang Xu, Chunjing Xu, and DaCheng Tao. Addersr: Towards energy efficient image super-resolutinn In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15648-15657, 2021.
[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1,9
[53] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Stti::23000092888 2023. 1,9
[54] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipit: Even better 1lm quantization with hadamard incoherence and lattice codebooks. arXiv preprint MXXv:22002.04396 2024. 9, 10
[55] Hongyu Wang. Shuming Ma, Li Dong, Shaohan Huang. Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang. Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310. 1453, 2023. 2
[56] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8612-8620, 2019. 2
[57] Yunhe Wang, Mingqiang Huang, Kai Han, Hanting Chen, Wei Zhang, Chunjing Xu, and Dacheng Tao. Addernet and its minimalist hardware design for energy-efficient artificial intelligence. arXiv preprint arXiv:2
2021. 8
[58] Ziwei Wang, Jiwen Lu, Chenxin Tao, Jie Zhou, and Qi Tian. Learning channel-wist interactions for binary convolutional neural networks. In CVPR, pages 568-577, 2019. 2
[59] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et al. Huggingface's trans- formers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.0377/ 2019, 9
[60] Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. arXiv preprint arXiv:2402.11295 2024.7
[61] Chunyu Yuan and Sos Agaian. A comprchensive review of binary neural network. Artificial Intelligence Review, 56(11): 19991913111,, 2023. 2
[62] Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder- post-training quantization for large language models. arXiv preprint arXiv:2304.01089 2023.
[63] Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network more networks per bit? In CVPR, pages 4923-4932, 2019. 2
[64] Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Structured binary neural networks for accurate image classification and semantic segmentation. In CVPR, pages 413-422, 2019. 2
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes].
Justification: The contributions presented in the introduction and abstract align with the method described in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please the section titled "Broader impact and limitations".
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Not applicable
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the details are presented in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification:
No code was included with the paper at submission time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppccceeccccccccccccctlt / ssissssoicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Yes, they are listed in Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No bar plots are provided, however, the process is deterministic, hence subsequent runs should result in very similar outcomes.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see the Results section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpp///////ri..../// httppppp.ccppppcpateeeeee httpppulcc/thhhcccueeleeee..eeee wtw.iiheeeeelllleccccccc ines?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please the Broader Impact section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All prior resources used are appropiately cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets, Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No such data present.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No such data present.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "TO1rJpho4N", "reviewer": "Reviewer_vSy1", "summary": "This research brings the sum of binary bases to the post-training quantization of the large language models. The authors propose a three-step algorithm. In step1: taking the sign of the full-precision weights and use the norm of the weights as the scalings, step2: adjusting the binary bases using gradient descent, step 3: adjusting scales using teacher-student (knowledge distillation).\n\nStep 1 was initiated in computer vision [49], and step 2 introduced in [36]. Set of binary bases for inference was suggested in [36] and later in LQ-Net. https://arxiv.org/pdf/1807.10029\n\nThe full binary base for efficient inference is not novel, authors tried to adapt [36] for post-training of large language models, which necessitates step 3 for a better performance. The paper needs to clarify its connection with [36] and LQNet. I was confused in the middle of reading the paper about the main contribution of the paper. My final judgment is \"this paper is an adaptation of existing ideas,\" but I congratulate the authors on putting considerable effort into bringing it to post-training and large language model context.", "questions": "- Did you try your method on the fixed-point activations? if yes please specify, if not please make it clear in the paper.\n- Why competing methods are on 4 and 3 bits but yours is 2.3? Is it a mistake or 2.3 is the average bit to be 2<2.3<3. I think it is fair to report your result on W4 and W3 just like the other baselines and then show it on a lower lower average bit.", "limitations": "The paper focuses on LLamA models. I would prefer a similar comparison for other mainstream language models as well.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes step 3 to make the performance close to the state-of-the-art post-training such as omi-quant, GPTQ, AWQ.", "weaknesses": "I doubt that Table 3 fairly shows the strength and the weakness of the method. First explain what W4A16g128 mean. I suppose it means weight quantized to 4 bit fixed point, activation to floating point 16 (half precision or brain float?) or maybe fixed point, g128 referees to the grouping size of the weight quantization for the fixed point representation. \n- SpQR baseline is missing in Table 3. https://arxiv.org/abs/2306.03078", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and comments. We hope to have addressed their remaining concerns below.\n\n**Q1.** _\"On doubts that Table 3 fairly shows the strength&weakness of the method.  First explain what W4A16g128 mean._\n\n**A1.** We believe that the comparison is fair, as we align our setting to that of prior methods, e.g. AWQ. Regarding the meaning of W4A16g128, you are correct. The example denotes a quantization scheme that makes use of 4-bit fixed point quantization for the weights, with a grouping size of 128 and half-precision (i.e. fp16) activations. We will detail this in the updated manuscript.\n\n**Q2.**  _SpQR baseline is missing in Table 3._\n\n**A2.** As suggested, we will include the baseline in Table 3, thank you! We note that the conclusions don't change (i.e.: LLaMA-7B @ pp on Wiki2, 5.69 (Ours) vs 5.87 (SPQR));\n\n**Q3.** _Did you try your method on the fixed-point activations? if yes please specify, if not please make it clear in the paper._\n\n**A3.** We performed an ablation in Table 5. Our approach only affects the weights, hence it's largely agnostic to the quantization strategy used for the activations. We will make this clearer.\n\n**Q4.** _On why no results with W4 and W3 just like the other baselines and then show it on a lower average bit._\n\n**A4.** Performance doesn't improve significantly beyond this to justify the costs, this is a consequence of initializing our model from W4 quantized weights.\n\n**Q5.** The paper focuses on LLamA models. I would prefer a similar comparison for other mainstream language models as well.\n\n**A5.** We mainly focused on LLamA (v1 and v2) models as this facilitated the comparison with prior work. We also already include results for the Phi-2 model (Table 1), and in addition, we tested it on Mistral-7B where we saw similar results (i.e. +0.1 perplexity improvement compared with the direct GPTQ baseline)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uiZSJSg4lN", "reviewer": "Reviewer_XBMG", "summary": "This paper proposed QBB by discomposing the original weights into 4 binary matrics and scaling factor. \nTo compensate the error, two techniques are further proposed: \n1. use gradient descent to find the optimal set of binary matrices and scaling vectors\n2. use knowledge distillation to optimize the scaling vector only.", "questions": "See weakness.", "limitations": "This paper missed some important reference. The experiments part should be put more efforts.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper present a thorough analysis, including detailed ablation studies and analysis of different components.\n- This paper has clear presentation: Well-organized, with effective use of figures and tables to support the content.", "weaknesses": "- The technique of decomposing the weights into several binary matrics is similar to BiLLM [1]. In BiLLM, they proposed residual approximation method, which is just the same to QBB when N=2. And this paper was published on arxiv on Feb. 2024. \n\n- In the related work, QBB listed the PB-LLM, which is a mixed-precision framework that achieved around 2-bit quantization. However, the experiments in Table3 did not involve it. Additionally, BitDistiller [2] also conducted experiment on 2-bit weight quantization. \n\n- This paper did not provide the code in supplementary material.\n\n- In QBB, two additional adjustments were conducted to minimize the error, which involves additional training cost. However, I haven't found any experiments w.r.t. training cost. Correct me if I am wrong. \n\n- From my understanding, the computational cost of QBB is between PTQ and QAT. You should compare with PTQ and QAT method separately. For QAT, LLM-QAT and Bitdistiller should be involved. \n\n- From line 154 to 157, you proposed progressive weights quantization technique by first quantizing weights to 4-bit and then applying the QBB. There is no additional process between these two procedure. From my point, I think it is useless and uneffective. There is no experiment that can prove that your progressive weights quantization is better than directly quantizing. \n\n- From Figure 6, when we set the N to 5, it is more stable than 4. Why not just use 5?\n\n- In section 3.2, \"Data-free\" is not properly used. Because I find that you utilized the generated data by LLM. It is not completely data-free. Please refer to [3].\n\n- Teacher-student swaps can easily cause training collapses, especially for floating-point & low-bit models. I am more curious about any strategies employed  to alleviate this problem. \n\n[1] https://arxiv.org/pdf/2402.04291 \n[2] https://arxiv.org/abs/2402.10631\n[3] https://arxiv.org/abs/1906.04721", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and comments. We hope to have addressed their remaining concerns below.\n\n**Q1.** _On: The technique of decomposing the weights into several binary matrics is similar to BiLLM [1], published on arxiv on Feb. 2024._\n\n**A1.** Thank you for pointing out [1], we weren't aware of it. Kindly note that [1] was not published in a peer-reviewed venue at the time of the NeurIPS deadline. Nevertheless, we will cite and discuss it in our updated manuscript.\n\nRegarding, the differences with [1]: [1] focuses on applying different quantization strategies for the salient and non-salient weights, identified using a structural search based on the Hessian Matrix. The salient columns then use a secondary residual binarization (i.e. to better preserve the information in the salient channels). The remaining non-uniformly distributed weights are split intro two groups by searching the optimal split. The work aims generally to exploit the sparsity of information present in the weights, allocating the bits accordingly and focusing on extreme quantization levels.\n\nIn contrast, our approach directly approximates the weights using a sum of binary matrices, that are learned on a layer-by-layer basis, without using any calibration data (as opposed to [1]), by gradient descent). Each new binary weight is trained in a cascaded manner. Our approach generalizes to different numbers of binary matrices, doesn't make use of any splitting algorithm, nor aims to identify/tie the binarization process to the weights' saliency.\n\nThereafter, unlike [1]  we follow a data-free calibration of the scaling factors using self-generated sequence that starts from randomly sampled tokens. Finally, the calibration process is placed within a newly adapted distillation strategy.\n\nWe believe that these are significant methodological differences that distant our work from [1] (an arxiv paper).\n\n **Q2.** _On comparison with PB-LLM & BitDistiller[2] that achieve 2-bit W quantization._\n\n**A2.** As we didn't focus on 2-bits or less, we haven't included such methods in the Tables. However, following your suggestion, we will include [2] in the result tables (i.e. 5.97[2] vs 5.21 (this work) on a LLamA-7B-2), noting that the results won't be directly comparable.\n\n**Q3.** _This paper did not provide the code in supplementary material._\n\n**A3.** Due to internal processes and timelines, unfortunately this wasn't possible to do at the time of submission.\n    \n**Q4.** _In QBB, two additional adjustments were conducted to minimize the error, which involves additional training cost. However, I haven't found any experiments w.r.t. training cost._\n\n**A4.** Depending on the model choice, the training take 4-12 hours on A100 GPU. This can likely be reduced by early stopping, as in many cases the process converges long before the end of the scheduler. However, for simplicity, we kept the same value.\n \n**Q5.** _The computational cost of QBB is between PTQ and QAT. You should compare with PTQ and QAT: LLM-QAT and Bitdistiller should be involved._\n\n**A5.** We haven't included a comparison with the two works mentioned as our work only performs a calibration process using a data-free approach, unlike LLM-QAT which performs a full model finetuning. E.g. LLM-QAT requires 8 A100 GPUs at a batch size = 1 per GPU to perform full model finetuning. In comparison, our approach can be run on a single GPU.  Nevertheless, we see the potential usefulness of such comparisons, and following your request, we will include them in the updated manuscript.\n\n**Q6.** _From L154-157, you proposed progressive weights quantization technique by first quantizing weights to 4-bit and then applying the QBB. There is no additional process between this two procedure. From my point, I think it is useless and uneffective. On no experiment to show this._\n\n**A6.** We ablate the impact of different initialization strategies in Table 4, where we showcase both the importance of starting from 4-bit weights and the method used to construct the 4-bit weights. The results show this step to be necessary. The intuition behind this is that the gap between the sum(binary weights) and fp16 ones are larger, making the optimisation process more challenging.\n    \n**Q7.** _From Figure 6, when we set the N to 5, it is more stable than 4. Why not just use 5?_\n\n**A7.** We didn't as the additional cost didn't justify to us the additional costs (see Fig. 6).\n    \n**Q8.** _In section 3.2, \"Data-free\" is not properly used. Because I find that you utilized the generated data by LLM. It is not completely data-free. Please refer to [3]._\n\n**A8.** We kindly disagree: the data-free assumption is not violated. The sequences used for training are produced by the model itself, starting from a random token. There is no real (i.e. collected) data used in the process, nor techniques based on manual intervention (e.g. prompting). As the starting token is random, this process is more akin to self-distillation on random inputs.\nWe note that such discriminative models, such as those used in [3] (i.e. MobileNet archs) are not comparable to generative models such as LLMs, as they are from different families of models, and the former are unable to produce sequences of data given a random sample. Hence, this is not a violation of the setting, but simply a difference between the capability of vision models vs an LLM when exposed to randomly constructed points.\n\n**Q9.** _Teacher-student swaps can easily cause training collapses, especially for floating-point & low-bit models. I am more curious about any strategies employed to alleviate this problem._\n\n**A9.** In our case, the first stage aims to independently (i.e. on a layer-by-layer basis) approximate the weights of each layer, using a set of binary weights. Post approximation, these layers are expected to already behave similarly with the target weights. Thanks to this, the discrepancies between such layers is small, making the swapping stable in general.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fXJWwAReIv", "reviewer": "Reviewer_L58J", "summary": "This paper introduces Quantization with Binary Bases (QBB), a novel method designed to reduce the computational complexity of large language models (LLMs) by replacing most multiplications with summations. QBB decomposes original weights into a set of binary (1-bit) matrices through an iterative process, optimizing them to minimize quantization error. Additionally, the quantized model is refined via Knowledge Distillation. Tested across multiple LLM families, QBB demonstrates superior performance compared to existing methods.", "questions": "Please refer to Weakness.", "limitations": "Please refer to Weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This method's utilization of binary matrices and scaling vectors to approximate original weights marks a significant advancement, effectively reducing computational complexity.\n2. Combining post-training quantization with knowledge distillation is a commendable approach. The performance has notably improved through the application of Knowledge Distillation after quantization.\n3. The paper offers extensive experimental results that showcase the effectiveness of QBB across diverse LLMs and datasets.", "weaknesses": "1. I believe comparing it with non-uniform quantization methods, such as QuIP#, would enhance this paper, as it does not utilize uniform quantization.\n2. Regarding progressive weights quantization, I'm unclear whether starting with a quantized model means that $W$ in Equation 4 represents quantized weights rather than full-precision weights. If so, this approach may not be optimal for the quantization step.\n3. For Training details, does each layer need 15000 iterations? Can you provide the details of runtime and overhead?\n4. How were the values of $s_1$ and $s_2$ specified in Equation 7?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and comments. We hope to have addressed their remaining concerns below.\n\n**Q1.** _I believe comparing it with non-uniform quantization methods, such as QuIP#, would enhance this paper, as it does not utilize uniform quantization._\n\n**A1.** Thank you for your suggestion. We already report results for QuIP# in Table 3. We will expand this to include more configurations, with different levels of quantization, making this aspect also more clear.\n\n**Q2.** _Regarding progressive weights quantization, I'm unclear whether starting with a quantized model means that_\n\n**A2.** By this, we mean that instead of directly binarizing the full/half precision weights, we start the process from a model quantized using an off-the-shelf 4-bit quantization method. This reduces the initial informational gap and is well aligned with the idea of gradually dropping the number of bits used in literature. Please see also Table 4.\n\n**Q3.** _in Equation 4 represents quantized weights rather than full-precision weights. If so, this approach may not be optimal for the quantization step._\n\n**A3.** In eq. 4, W represents the to-be-approximates weights, they can be full precision or quantized to a higher number of bits. B_i represents the constructed binary matrices. As num_bits(W) is always higher than num_bits(B) this is generally not a concern. Moreover, this steps only corrects the remaining error that is incurred after the first step.\n\n**Q4.** _For Training details, does each layer need 15000 iterations? Can you provide the details of runtime and overhead?_\n\n**A4.** Yes, this is performed on a layer-by-layer basis, we could likely reduce this significantly using an early stopping scheduler, but for simplicity, we opted for the same number of iterations in all cases. In terms of overhead, this takes a few hours (4-12) on a A100, depending on the model size. However, as the process is only run once, there is no additional overhead at test/runtime.\n\n**Q5.** _How were the values of and specified in Equation 7?_\n\nThey were selected empirically using one of the model variants (i.e. the smallest), and then used for the rest of the models too.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rLvbaRQE1L", "reviewer": "Reviewer_waud", "summary": "This paper proposes a PTQ technique which decompose the model weitghs into a set of binary matrices. An interactive binarization process and a progressive model distillation procedure are conducted to reduce the quantization error. The paper claims to set a new SOTA of a summation-only based approach.", "questions": "1. Please revise the discussion in Sec 4.6 to provide detailed memory and computation savings comparing to regular linear W4A16 quantized models.\n2. Please explain in detail about the quantization configuration of the reported W2.3 quantized model", "limitations": "The limitation of the work is adequately addressed.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper works on the problem of quantizing large language model, which is an important research question with practical application and positive social impact.\n2. The paper is overall well-written, the method is clearly derived and well-illustrated with figures. The motivation and the methods are easy to follow.\n3. Good performance are reported both in terms of preplexity and zero-shot performance on multiple models.", "weaknesses": "1. From the novelty prespective, the proposed binary decomposition method is largely similar to previous nonlinear quantization methods line LQ-Net (https://arxiv.org/pdf/1807.10029 ECCV'18). The similarity and difference between the proposed method and LQ-Net should be discussed and the performance should be compared.\n2. The paper claims the achieved model can be efficient by removing all the multiplications. However, I don't believe this can be achieved from the formulation of quantized weights in Equ. (1). Although no multiplication is needed to multiply the binary part B with the activation, the multiplication of scaling factor alpha to the activation is still needed, and can be costly when N is large. In fact, the proposed quantization scheme is effectively an N-bit quantization with non-uniform quantization bins, which theortically shouldn't be more efficient than an N-bit linear quantization with integer weights.\n3. Deriving from my second point, the efficiency discussion in Sec 4.6 is not concrete. An estimation on exactly how memory and computation is saved comparing to a regular linear W4A16 quantized model should be provided.\n4. The comparison is mainly done against PTQ methods with linear weight quantization, yet the proposed method performs non-linear quantization of the weight and requires several epochs of finetuning. This will result in both higher training cost and inference cost comparing to previous methods, making the comparison results unfair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and comments. We hope to have addressed their remaining concerns below.\n\n**Q1.** _From the novelty prespective, the proposed binary decomposition method is largely similar to previous nonlinear quantization methods line LQ-Net ([https://arxiv.org/pdf/1807.10029](https://arxiv.org/pdf/1807.10029) ECCV'18). The similarities and differences between the proposed method and LQ-Net should be discussed, and the performance should be compared._\n\n**A1.** We disagree, as the similarities are largely superficial. While LQ-Net uses multiple binary bases to emulate n-bit quantization, unlike our approach: they are fully trained in a supervised manner on the task at hand, hence their solution is neither data-free nor is applicable as post-training quantization step; they follow a different formulation, in which the binary basis is found by looking up the quantization interval while the learnable floating basis by solving a linear regression problem. Moreover, no form of distillation, such as the strategy proposed in this work is used.\nIn contrast, our process follows a two-step approach, in which we first construct a series of binary bases that closely approximate the original weights, without requiring any activations, on a layer-by-layer basis, directly using gradient descent. We then calibrate the scaling factors using a data-free approach and distillation strategy.\n\nIn terms of a numerical comparison, the current LQ-Net formulation would require retraining the model from scratch/finetuning it on specific datasets.\nThank you for pointing out this work. We will include an ample discussion making the difference clear.\n\n**Q2.** _The paper claims the achieved model can be efficient by removing all the multiplications. However, I don't believe this can be achieved from the formulation of quantized weights in Equ. (1). Although no multiplication is needed to multiply the binary part B with the activation, the multiplication of scaling factor alpha to the activation is still needed and can be costly when N is large. In fact, the proposed quantization scheme is effectively an N-bit quantization with non-uniform quantization bins, which theoretically shouldn't be more efficient than an N-bit linear quantization with integer weights._\n\n**A2.** The reviewer is correct that we don't remove _all_ multiplications, however, we don't claim to remove all multiplications, but _nearly_ all multiplications. The remaining multiplications are not too different from most quantization approaches which will also scale the output using some scalars. Do note however that the 4b-16bit multiplications, occurring for the matmul itself, are removed, and as noted in Sec 4.6, depending on the hardware implementation they can be significantly more energy efficient, requiring fewer physical gates.\n\n**Q3.** _Deriving from my second point, the efficiency discussion in Sec 4.6 is not concrete. An estimation of exactly how memory and computation are saved compared to a regular linear W4A16 quantized model should be provided._\n\n**A3.** Most of the savings are expected to be from the significant reduction in multiplications, converting it to a nearly summation-only approach. This could primarily result in significant energy savings (as pointed out 4-8x). For this aspect we kindly point out to the works cited in Sec. 4.6 [24; 34; 54; 55; 61;] which have a more ample and practical analysis on this, as they specialise on summation-based strategies. In terms of memory savings, on a memory-pressured system, an intuitive direct saving could be by loading one binary basis at a time into memory, overlapping the computation of the first computation with the loading of the data of the second, thus diminishing the peak memory used.\nTo obtain exact measurements, an implementation is needed, which as we mentioned in the limitation section, wasn't part of the present work.\n\n**Q4.** _The comparison is mainly done against PTQ methods with linear weight quantization, yet the proposed method performs non-linear quantization of the weight and requires several epochs of finetuning. This will result in both higher training cost and inference cost comparing to previous methods, making the comparison results unfair._\n\n**A4.** There is no additional cost at runtime as the calibration process is performed prior to test time deployment. During calibration, indeed, the process, altogether, depending on the model, requires 4-12hours. As the process is expected to be run once, can be performed on commodity hardware, and requires no data, we consider this time to not be a major concern in practice.\n\n\n**Q5.** _Please explain in detail about the quantization configuration of the reported W2.3 quantized model_\n\n**A5.** The quantization configuration is based on N=4 strategy under the assumption of left-most significant placement (detailed in Sec. 4.6.) We also observe some patterns in terms of quantization errors (see Fig. 6) which should make it possible to adjust the levels further depending on this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel approach called "Quantization with Binary Bases (QBB)" for low-bit quantization of large language models (LLMs). The method decomposes original weights into binary matrices, significantly reducing computational complexity by replacing most multiplications with summations. The authors demonstrate that QBB achieves state-of-the-art performance across various LLMs.

Reviewers acknowledged the technical soundness of the approach but raised concerns about its novelty, particularly in relation to existing methods like LQ-Net. They also questioned the paper's claims regarding computational efficiency and the fairness of comparisons with linear quantization methods. Some reviewers suggested additional experiments and comparisons to strengthen the paper's contributions.

In response, the authors provided detailed clarifications and additional comparisons, addressing the reviewers' concerns. Most reviewers agreed that the paper offers a solid contribution to the field, particularly in its innovative application of binary matrices to LLM quantization. Considering the paper's strengths, including its thorough experimentation and practical implications, the AC recommends acceptance.
Decision: Accept (poster)