OneBit: Towards Extremely Low-bit Large Language Models
Authors: Yuzhuang Xu
Xu Han
Zonghan Yang
Shuo Wang
Qingfu Zhu
Zhiyuan Liu
Weidong Liu
Wanxiang Che
Venue: NeurIPS 2024 poster (2024)
Keywords: model quantization
weight-only quantization
extremely low-bit
onebit
Primary area: natural_language_processing
TL;DR: This paper propose a novel model structure for 1-bit weight quantization and a corresponding parameter initialization method. Extensive experiments demonstrate that it has clear advantages over representative strong baselines.

Abstract: Model quantification uses low bit-width values to represent the weight matrices of existing models to be quantized, which is a promising approach to reduce both storage and computational overheads of deploying highly anticipated LLMs. However, current quantization methods suffer severe performance degradation when the bit-width is extremely reduced, and thus focus on utilizing 4-bit or 8-bit values to quantize models. This paper boldly quantizes the weight matrices of LLMs to 1-bit, paving the way for the extremely low bit-width deployment of LLMs. For this target, we introduce a 1-bit model compressing framework named OneBit, including a novel 1-bit parameter representation method to better quantize LLMs as well as an effective parameter initialization method based on matrix decomposition to improve the convergence speed of the quantization framework. Sufficient experimental results indicate that OneBit achieves good performance (at least 81% of the non-quantized performance on LLaMA models) with robust training processes when only using 1-bit weight matrices.

OneBit: Towards Extremely Low-bit Large Language Models
Yuzhuang Xu
Xu Han
Zonghan Yang² Shuo Wang? Wanxiang Che
Qingfu Zhu
Zhiyuan Liu
Weidong Liu
Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China
Department of Computer Science & Technology. Tsinghua University, Beijing, China xyz21thu com, car hit edu..dddueee cn
Abstract
Model quantification uses low bit-width values to represent the weight matrices existing models to be quantized, which a promising approach to reduce both storage and computational overheads of deploying highly anticipated LLMs. However, current quantization methods suffer severe performance degradation when the bit-width is extremely reduced, and thus focus utilizing 4-bit 8-hit values to quantize models. This paper boldly quantizes the weight matrices of LLMs I-bit, paving the way for the extremely low bit-width deployment LLMs. For this target, we introduce 1-bit model compressing framework named OneBit, including novel 1-bit parameter representation method to better quantize LLMs as well as an effective parameter initialization method based on matrix decomposition improve the convergence speed of the quantization framework. Sufficient experimental results indicate that OneBit achieves good performance (at least of the non-quantized performance on I.LaMA models) with robust training processes when only using 1-bit weight matrices. Code and checkpoints are available at wtttttt://// /tttps:://iiaaie /gtb........... wwt. conm//uyuuhuungg//Deell
Introduction
Transformer 36 has emerged as the pivotal architecture in large language models (LLMs), fun- damentally reshaping the approach to natural language processing in deep learning era 6 34 4 Despite their popularity, deploying mansformer-based LLMs presents significant challenges due to their computational intensity and considerable memory requirements as the parameters of LLMs become more and more. For instance, even moderately-sized L.LMs like LLaMAA-3B 34 require around 26GB of memory to load its all parameters in FP16 format. Such overheads make deploying LLMs difficult beyond toigh-h- GPUs like the 1 let alone on mobile devices. The high demand for resources not only drives up usage costs, but also restricts their wider application.
have been devoted reducing the computational and memory over- heads of LLMs, while still preserving most of their original model capabilities. Among these efforts, quantization has gained widespread attention, particularly Post-Tras DrssnTrainnng Quantization (PTQ), benefit- ted from its lower transferring costs. Seminal studies such as GPTQ (14), SpQR [12]. and AWQ [20] successfully compress the weight matrices of LLMs 4-bit values while maintaining the main abilities of LLMs. Efficient quantization represents significant advances in LLM optimization, by achieving balance between time and space efficiency as well model performance.
Numerous efforts 10 14
Unfortunately, the efficacy of PTQ rapidly diminishes when the quantization bit-width is extremely low, shown Figure Existing PTQ methods managed compress weight matrices down to at least 3-bit 9 Recent researches hope to leverage Quantization-Awar Training (QAT) to
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

16
GPTQ
14
LLM-QAT
Perplexity
OmniQuant
Ours
16
8
2
weight bits
Figure 1: The perplexity (lower scores mean better performance) of existing widely-used low- bit quantization methods LLaMA-7B, reported Wikitext2 [23]. All the examined previous approaches suffer from significant performance degradation when quantizing models to 2-bit values. Our 1-bit quantization method can outperform these 2-bit baselines.
overcome the bottlenecks faced by PTQ. LLM-QAT [2] introduces a few learnable parameters into the quantization process, achieving notable results. OmniQuant [30], integrating learnable equivalent transformation, presents promising results in 2-bit quantization. However, existing methods decline when compressing model weights to bit, struggling to maintain effectiveness, This mainly stems from the drastic precision loss at extremely low bit-width representation in weight matrix W. significantly increasing loss in linear projection WX, which is the core operator within LLMs.
In this paper, propose novel Linear layer and Sign-Value-Indene Value-Independent Decomposition (SVID) for weight matrices to represent LLMs using approximately 1-bit values. our novel layer archi- tecture, each original high-bit weight matrix is represented one sign matrix (#1) and two value vectors. The value vectors provide necessary floating-poin precision in linear projection at little cost and help the model to be trained easily. The sign matrix maintains the high rank of the original weight matrix with a small space cost, thereby preserving high information capacity. SVID offers better parameter initialization for 1-bit models from the non-quantized model and we employ quantization-awae aware knowledge distillation to transfer the capabilities of the original model to the proposed 1 I-bit counterpart, Experiments demonstrate that our method performs well the W1A16 16 (1-bit weight and 16-bit activation) quantization level. Furthermore, our 1 1-bit model is more amenable training and knowledge transfer than previous works. summary, our contributions are 3-fold:
We propose a novel and efficient 1-bit model architecture for LLMs, which can improve both the time and space efficiency during model inference. Moreover, our architecture is more stable during quantizing LLMs.
We propose SVID to decompose high-bit bit matrices into low-bit ones, which is essential for the initialization of our 1-bit architecture. Experiments demonstrate that the SVID-based initialization can improve the model performance and convergence speed.
Extensive experiments demonstrate that our method works well model sizes from 1.3B to 13B in OPT, LLaMA, and LLaMM2, showcasing its generalizability.
Related Work
2.1 Large Language Model Compression
Quantization, pruning, and knowledge distillation (KD) are the mainstream methods for model compression. Quantization compresses model weights into low-bit values 14 20 M. For data type alignment in computation and reducing memory, it also involves quantizing activation 10 39 and key-value cache [30]. Pruning simplifies model complexity by removing unimportant weights or modules, thereby sparsifying the original larger models larger teacher model [16 achieving the purpose of compressing the larger one. Beyond these methods, low-rank factorization approximates the original weight matrix W with the product of two lower-rank matrices [40] and also achieves promising results. Our work belongs to quantization, using KD for knowledge transfer from the original LLM and uniquely
[3 B1 22). KD trains smaller student
model under the guidance

---

focusing on extremely low bit-width quantization. More details about model compression can refer existing survies 37
2.2 Large Language Model Quantization
Since this paper aims to obtain extremely low-bit LLMs, here we thus introduce more details about LLM quantization. Quantization stands as popular and crucial method for model compression, capable of achieving significant compression ratio with relatively small loss. It can be classified into Post-Training Quantization (PTQ) and Quantization-Awar Training (QAT) according when
quantization is applied.
PTQ directly converts trained models into lower-bit counterparts using accurate solvers and limited calibration data without additional training. Typically, GPTQ [14] row-wisely quantizes weight matrices and adjusts remaining weights to compensate for the precision loss caused by quantization, achieving nearly lossless 4-bit weight quantization. Moreover, numerous studies observed the effect of "outliers" in quantization [TO, 18 20 LLM.int8() [IO] suggests mnxeddprecisiie decomposition to ensure the accuracy of a few outliers in activations. SmoothQuant [39] reduces the difficulty of quantization by smoothing the outliers of activation. SpQR [12] identifies sensitive weights to ensure their precision, while quantizing other weights to lower bit-width.
QAT integrates quantization steps within the model, applying them during training fine-tuning. It allows the model to better adapt to the reduced precision induced by quantization, leading to improved performance compared to PTQ. LLM-QAT [2I] introduces small number of learnable parameters into quantization and employs KD using data generated by the original model itself. OmniQuant (30) classify it QAT) further introduces learnable equivalent transformation, achieving acceptable results 2-bit weight quantization. Contemporary work QulP# (35 combines randomized Hadamard transform, vector quantization techniques, and fine-tuning achieve better performance in 2-bit level. PEQA 117 and QLoRA [I]] focus on fine-tuning limited number of extra parameters to mitigate the precision loss caused by sub-4bit weight quantization. Our work is closely related to QAT, but due to the unique challenges posed by 1-bit quantization, our representation and initialization methods of quantized weights are distinct from any existing work.
3 Methodology
This section demonstrates our 1-bit architecture of the Linear layer to be quantized and discuss how initialize the quantized model to achieve better performance in knowledge distillation. We start with a short review of classical weight quantization methods in Section 3.1 and then formulate our OneBit from Section 3.2 to Section 3.4 in detail.
3.1 Background
The main idea of model quantization is to compress each weight matrix W within models in FP32 or FP16 format low-bit counterpart. Specifically, often quantize the weight matrices of Linear layers in transformer to 8, 4, and even 2 bits.
The majority of quantization studies primarily employ the round-to-neares, (RTN) method, by which the weight w is rounded the nearest value in the quantization grid. It can be formulated as
v Clip (1=T + +, 2,,0,2"11).
where s denotes the quantization scale parameter, z denotes the zero point parameter, and N is the quantization bit-width. Clip(-) truncates the result in the range of 0 to 2 - 1. With the bit-width being lower and lower, the quantization grid also becomes sparser. When we quantize a LLM to 1-bit values, there are only 2 available numbers to be chosen in the quantized model. Existing study 9 points out that quantization based on the RTN method may get their best performance at the 4-bit level. Further quantizing 2-bit values following this paradigm would result in substantial degradation [3 as shown in Figure
Furthermore, when N equals 1, quantization based on RTN method is essentially equivalent to setting a threshold, with weight w on either side of it being converted to corresponding integer value tt In such scenario, the parameters and in Eq. (I) effectively lose their practical
3

---

LayerNNorm
X
X
WT
g
WT
h
(a) FP16 Linear Layer
(b) Our Binary Quantized Linear Layer
The main idea of our method OneBit. The left is the original FP16 Linear Layer, in which both the activation X and the weight matrix W are in FP16 format. The right is our proposed architecture. Only value vectors g and h are in FP16 format, and the weight matrix consists of + instead, which can represented in INT1.
Figure 2:
significance. Consequently, when quantizing weights to bit, the element- wise RTN operation drastically undermines the precision of the weight matrix W, leading to poor performance of the quantized model.
3.2 1-bit Linear Layer Architecture
Due the severe precision loss of 1-bit weight quantization, converting weight matrices in Linear layers directly from FP32/16 to 1-bit format based on RTN challenging. Wang et al. [38] explore this possibility by studying the capabilities of purely 1-bit weight matrices, training the 1-bit model from scratch. In the WIA16 setting, their Linear layers are designed as
Mean ()]
W+1 Sign
W
Mcan Abs W
Mcan (W))
(2)
= 17- LayerNorm WT 11>
where W denotes the quantized weight matrix with the shape quantized matrix. is the input of Linear layer and Y is the output. Sign(.), Mcan(-) and Abs(.) functions return the sign matrix, average and absolute value matrix. Unfortunately, this approach reduces computational demands but also leads to marked decrease in performance (38]. Moreover, due training difficulties, experiments show that this method is challenging use for quantizing existing models and can only be applied to training models from scratch.
and W denotes the 1-bit
Inspired by Wang al. (38, we also quantize the weight matrix using the function Sign(-), and the element of the quantized matrix is set to +1 or 11 as well. Morcover, we also notice that although W maintains high rank of W, the missed floating-poinl precision still destroys the model performance. Therefore, different from previous work, we introduce value vectors with an FP16 format to compromise the precision loss in the quantization process. During training, proposed Linear layers are designed as
W +1 Sign (W),
Y = [X X g)
WT o h
Z = Layer (Y),
where g and h are the two FP16 value vectors. During inference, W+1 is packed with an INTI format, and Sign(-) will not be used, as shown in Figure 2 Note that specify the calculation order using brackets in Eq. ( for minimizing the time and space cost. The main difference between Wang al. (38) and OneBit the extra parameter g and h. Even if additional parameters brought in, the benefits far outweigh its small cost. For instance, when we quantize one weight matrix with the shape 4096 x 4096, the average bit-width of the quantized result is 1.0073. A.e. for the details.
3.3 Sign- Value- Valuaallle-Ideeedenn Decomposition
In our proposed 1-bit architecture, the weight matrix W mathematically divided into two compo- nents: sign matrix W +1 in INTI format and two value vector g/h in FP16 format. To initialize

---

the 1-bit model with the help of the fully trained weight, we introduce the Sign- Vauuu--eeddeeedddnn Decomposition (SVID) of the weight matrix W, which can be formulated as W sign W value- Here we have W value = |W| and W sign Sign W.. For W value: we further approximately decom- Siign
pose it into the outer product of two vectors and b, which is also known as rank- approximation. Hence, our proposed matrix decomposition method can be represented as
W
W sign (ab").
We can employ some widely used matrix decomposition methods to perform the rank-1 approximation, such as SVD [ and NMF [25].
Proposition 1
Given the weight matrix W and input X, the Linear layer can be reformulated as the following according to SVID:
XWT [[X W sign T
We prove this approximation in Appendix A. This bridges the gap between the architecture of the quantized model and its original weights. indicates that if we assign W Wsign to W a to and bT to g, the quantized model is an approximate initialization of the original model. Moreover, compared to restoring the original matrix W first (such as in Eq. ( the computational order in Eq. (5) saves approximately one matrix W in FP16 format in memory as there no need restore W in FP16 format.
The main objective of SVID is to involve the sign matrix W sign in approximating matrix W, rather than solely relying on value vectors in FP16 format. To substantiate the role of the sign matrix Waign in matrix approximation, we present the following proposition.
Proposition 2 the way W abT E and |W| abT Eg, where E, denotes the error matrices. In terms of the Frobenius-norm, the SVID is closer to the original matrix W:
Given matrices W and W W = W sign
W We decompose these matrices in
w Wsign c abr
<
W ab
(6)
F
We also prove this proposition in Appendix A. matrix W sign in matrix approximation.
It clearly demonstrates the practical role of the sign
Note that, given the predominantly low precision of most parameters, approximate the weight matrix W accurately. SVID is not aimed to precisely replicate the original model's 's parameters, but provide an effective starting point for further training, leveraging the extensive training of the original model. Details on transferring knowledge from the original model to the quantized counterpart are in Section 3.4
quite challenging to
3.4 Knowledge Transfer
We employ quantization-awwe aware knowledge distillation to transfer knowledge from the original model (i.e. teacher model) to the quantized one (i.e. student model). In the student model, the element in matrix W and vectors g/h in Eq. (3) will be trained. We cross-entropy based logits and mean-square-eroon based hidden state of the frel-precision teacher model to direct the quantized student model 32 Language modeling loss is not used. The cross-entropy is defined as
LCE
ΣΣ This 1=1
1
p.. (o:) log!. (o:),
where denotes the number of classes and n, denotes the number of training samples in the current batch. T and S are the teacher model and student model, respectively. The error hidden states is defined as
MMSE
(8)
12
5

---

where ny denotes the number of layers and denotes the hidden state. Hence the final objective function can be formulated as
LKD Lce + aLMSE
where a is the eypem-erraeeter that balances the importance the cross- moss-entropy loss and the features in the intermediate layers. Please refer to f.6 further discussions of this part.
4 Experiments
We experiment with 1-bit weight-only quantizaton and maintain 16-bit activation (W1A16) this work. We evaluate our approach by performing experiments on OPT-1.3B/2.7B models, LLaMA- 7B/13B models and LLaMA2-7B/1 models, and present results on various tasks.
4.1
Settings
Data For the training data of quantization aware knowledge distillation, follow Liu et al. [1] to synthesize corpus using next token generation from the original teacher model. It randomizes the first token from vocabulary and generates the next token iteratively until reaching either the <EOS> token or the maximum length. Specially, the top-1 predictions are selected deterministically for the first 3 to 5 tokens, followed by stochastic sampling for the remaining tokens. We utilized LLaMA-7B to generate a total of 132k data entries, each with maximum length of 2,048.
Training Details Every KD experiment learns the training data 50 epochs, from which 2048- token segments are selected. We employ NMF in scikit-learn to decompose the weight matrices in SVID. The quantized student models are optimized by Adam [19] with B1 0.9, B3 0.98. The learning rate for all experiments is scheduled by cosine strategy. We use NVIDIA A A100 GPUs and maintain FP16 precision while training quantized models. For additional details such as learning rate, please refer to Table
Table 1: Training details of knowledge distillation.
Models
learning rate
# GPUs
OPT-1-3B
4e-4
1.0
1
x 8
OPT-2.7B
2e-4
1.0
LLaMA-7B
4e-4
1.0
1 8
LLLMA-11B
2e-4
1.0
ILaMA2-7B
le-4
1.0
1 x 8
LLaMA2-13B
2c-4
1.0
2 x
Baselines To our knowledge, there is previous work exploring the 1-bit quantization of LLMs from a knowledge transfer perspective. To this end, we relax the quantization bit-width of baselines to 2 bits (W2A16) while maintaining the WIAI6 16 setting in our method. We compare our method with GPTQ [4], LLM-QAT (T and OmniQuant 30) To ensure a fair comparison in terms of space usage, basclines do not employ grouped quantization. Additionally, we included the results of vanilla transformers with FP16 precision as reference. While the recent work BitNet (38) also introduced one 1-bit model architecture, it only worked for training models from scratch. We also analyze its capability to transfer knowledge from the original models in Appendix A.5
Evaluation Metrics Basically, we evaluate quantized models by testing the perplexity the validation set, specifically on WikiText2 [23] and C4 (28). Lower perplexity indicates that the compressed model better preserving the output distribution of the original model. Furthermore. accuracies of zero-shot tasks including Winogrande [29), HellaSwar wag [4]] PIQA [4, BoolQ [7]. and ARC X are also reported. They evaluate if the capabilities of the original model on downstream tasks are retained. We utilize the open- sourced toolkit LLMM-Evluttion-He -arnessHasss to perform the perplexity test and all zero-shot tasks.
httpe://ecikit--eerr.org

---

Table 2: Main results of evaluation experiment. We report the perplexity and zero-shot accuracy. "FP16" is the transformer with FP16 parameters and we refer to it as the upper-bound of all the methods. The best score is bolded.
Perplexity())
Zero-shot Accuracy(1)
Models
Methods
Wiki2
C4
Wino.
Hella.
PIQA
BoolQ ARC-e ARC-c
Avg.
FP16
14.63
14.72
59.67
53.73
72.42
57.68
50.80
29.69
54.00
GPTQ
9.5e3
3.8e3
49.33
25.57
52.07
39.60
26.68
23.63
36.15
OPT-1.3B
LLM-QAT
4.9e3
2.1c3
49.72
25.72
50.05
37.83
25.76
25.09
35.70
OmniQuant
42.43
55.64
51.85
33.39
60.94
56.45
38.76
23.38
44.13
OneBit
25.42
22.95
51.14
34.26
62.57
59.45
41.25
24.06
45.46
FP16
12.47
13.17
60.93
60.59
74.81
60.28
54.34
31.31
57.04
GPTQ
8.7e3
3.9e3
49.88
26.47
49.84
39.88
25.76
26.02
36.31
OPT-2.7B
LLM-QAT
3.7e3
1.4e3
52.09
25.47
49.29
37.83
24.92
25.60
35.87
OmniQuant
30.25
41.31
51.62
38.21
62.19
54.25
40.82
24.74
45.31
OneBit
21.86
20.76
51.67
38.18
63.87
54.28
43.39
24.40
45.97
FP16
5.68
7.08
66.85
72.99
77.37
73.21
52.53
41.38
64.06
GPTQ
1.9e3
7.8e2
49.41
25.63
49.95
43.79
25.84
27.47
37.02
LLaMA-7B
LLM-QAT
7.1e2
3.0e2
51.78
24.76
50.87
37.83
26.26
25.51
36.17
OmniQuant
15.34
26.21
52.96
43.68
62.79
58.69
41.54
29.35
48.17
OneBit
10.19
11.40
58.48
51.54
68.01
57.28
42.47
30.20
51.33
FP16
5.09
6.61
70.17
76.24
79.05
68.47
59.85
44.54
66.39
GPTQ
3.2e3
9.9e2
50.67
25.27
50.00
42.39
26.14
27.39
36.98
LLaMA-13B
LLM-QAT
1.8e3
1.2e3
51.62
25.40
50.33
37.83
27.02
26.87
36.51
OmniQuant
13.43
19.33
53.83
54.16
68.99
62.20
45.50
30.38
52.51
OneBit
9.18
10.25
62.90
56.78
70.67
64.16
44.53
32.00
55.17
4.2
Main Results
Table@[compate our method with other typical strong baselines on different models. Due to space limitations, results of LLaMA2-7B/13B are listed in Appendix A.3] In various model sizes, 1-bit weight quantization method obviously outperforms others under the W2A16 setting. Moreover, the effectiveness of QAT based methods consistently improves as the model size increases, whereas the result of the PTQ method, GPTQ, may degrade when model size increases (c.g., from 7B to 13B on LLaMA). This demonstrates that QAT-based method can achieve stable results extremely low-bit quantization. Specifically, our method approaches the performance of FP16 more closely as the model size increases. For instance, when scaling from LLaMA- (on C4) of the FP16 model decreases by only 0.47, whereas our method sees reduction of 1.15.
LLaMA 13BB, the perplexity
For perplexity, only our method achieves comparable results to the strongest FP16 baseline. For instance, our method achieves 9.18 the Wiki2 dataset on I.aMA-13B model and the FP16 baseline is 5.09. The performance loss of other methods is sigmificant, even though they use 2-bit quantization, which is more than our bit. For GPTQ and LLM-QAT, the performance degradation after quantization is pretty severe. As for OmniQuant, even though it is the strongest baseline under the W2A16 16 setting, it still suffers greater performance loss compared to our W1A16 setting.
For zero-shot accuracy, although all methods inevitably have some degradation, method achieves the closest performance to the FP16 baseline among most models. On the 0.T-113B/2. model, our method shows smaller performance loss most tasks such PIQA and ARC-e. Additionally, the loss other tasks is negligible compared with the second-best baseline, OmniQuant. On the L- 7B model, method also notably outperforms OmniQuant in most tasks except BoolQ/ARC-e, averaging about 4% improvement overall.
4.3 Problem Solving Ability
We have demonstrated the superior performance our method under the W1A16 setting, compared to other representative baselines. Although all methods inevitably face performance degradation in
7

---

Winograde
STEM
Hellaswag
Humanities
23
16 L4
24
Boo Q
Others
Average
Social Sciences
ARC. c
Average
Pythia 1.08 TinyLLaMA-1
LowRank Laaaa
Pythia- 1 TinyLLaMA-1.18
OneBit-7B
(a) Common sense reasoning tasks
(b) General world knowledge (MMLU)
18
2.2
2.2
16.0
16.0
16.0
16
2.0
2.0
Memory (GB)
1.8
12
Blt-width
1.6
8
2:88
1.0
Total Memory
Average Bit-width
Pythia-1.08
LowRank Llama
TinyLLaMA-1.18
OneBit-78
(c) Memory footprint and bit-width
Figure Comparison of model capabilities and compressive degree.
1-bit weight quantization, it remains of interest how our method fares in solving practical problems among the various approaches to reducing model size. For instance, directly training smaller models
42 or employing low-rank decomposition to reduce the number of parameters.
To this end, consider crucial abilities of LLMss commonsense reasoning and world knowledge. For commonsense reasoning, we use the 6 tasks (Hellaswag, etc.) and settings described in Section 4.2] For world knowledge, we examine using the Massive Multi-task Language Understanding (MMLU;
15), a benchmark that covers wide domains and knowledge. We compare the following 4 models:
Pythia-1.0B [3] A well-trained model released by EleutherAl whose memory footprint is 1.54x that of our OneBit-7B model.
TinyL.LMA-1..1 [42] A model with the same structure as the LLaMA models, which undergoes continued training. To compare fairly, we use checkpoint at training steps, which 2x that of our OneBit-7B model.
LowRank LLaMA (24) Decompose every weight matrix Linear layers to two low-rank matrices and learn from the original LLaMA-7B model by KD in the same setting of OneBit-7B.
OneBit-7B The model that we use in Section which is built with OneBit.
Figure and 3b demonstrate common sense reasoning ability and general world knowledge of different models. We can observe that, although other models have more parameters and are more thoroughly trained than ours, our model still has advantages common sense reasoning. This reflects the benefits inherited from the larger 7B model. In terms of world knowledge, despite significant loss social sciences, our model outperforms the fully trained Pythia- IB in other domains. These results demonstrate the practical usability of OneBit.

---

Table 3: Compression ratio of LLaMA models.
Models
FP16 (GB)
OneBit (GB)
Ratio (%)
LLaMA-7B
13.5
1.3
90.4
LLaMA- 13B
26.0
2.2
91.5
LLaMA-300
65.1
4.9
92.5
LLaMA-655
130.6
9.2
93.4
25
OPT-1.38
Baseline (FP16)
OneBit (WIA16)
Perplexity Wiki2
OPT-2.78
20
same 420, 0.67
better PRL
15
OPT-1.38
OPT-2.78
same PPL, 0.22x size
LLaMA-73
10
LLaMA-138
LLaMA-7B
LLaMA-138
24
Model Size (GB)
Figure 4: Tradeoff between size and PPL.
5
Analysis and Discussion
5.1 Efficiency
evident that extremely low-bit quantization of weights can significantly reduce the memory footprint of models. As shown in T the actual compression ratio increases as the model size increases. This is particularly meaningful for larger models, making it possible to fit the model into GPU. While there performance loss, Figure 4 illustrates that our method achieves good trade-off between space occupancy and model performance. For example, we can achieve comparable performance FP16 with only 0.2x the model space. Furthermore, quantizing #1 also aids in accelerating matrix multiplication on CPUs. It is because the floating-pon point multiplication of elements two matrices can converted into much faster bit operations on these chips. Thus the substantial reduction in memory overhead makes these low-bit LLMs meet the requirements for deployment on PCs and smartphones.
5.2 Robustness
Existing work 338 has already noted the instability within QAT. Extremely low-bit quantization makes the training process highly sensitive to the learning rate, making it difficult for the model to converge when the rate is too small or too large. This is primarily due to the large magnitude of gradients generated as the weight elements fluctuate between +1 and -1, leading to substantial fluctuations in the output of Linear layers. Experiments demonstrate that OneBit shows more stable training process and is not sensitive to learning rates. Please refer Appendix A.5 for details.
5.3 Effect of Different Components
The variable components our method primarily include Post-LayerNorm, value vectors, and parameter initialization.
We discover that there might be floating-point overflow during the QAT process. As depth increases, the activation can become progressively larger. tackle it using Post-LayerNorm instead of Pre- Pre-LayyrNorm. In contrast, Pre-LayerNorm may occasionally be ineffective.
Post-LayerNorna

---

Singular Value Decomposition (SVD)
Non-negative Matrix Factorization (NMF)
Only Copy Original Weight
Training Loss
80
60
40
20
0
1000
2000
3000
4000
5000
6000
Training Steps
Figure 5: Training process of OneBit-7B.
Value Vectors The main structural difference between OneBit and BitNet (38] the two value vectors, which are demonstrated to be effective in Section 4.2 They facilitate stable training and the knowledge transfer process. Please refer to Appendix A.3 for more details of comparison.
Parameter Initialization In our proposed SVID, both NMF and SVD can be used decompose [W] and we recommend using the former. This is because we find that NMF may make the training more faster to converge. Figure shows that initializing by NMF facilitates better performance.
Conclusion
We propose novel model structure for 1-bit weight quantization and a corresponding parameter initialization method to address the difficulty in 1-bit quantization. Extensive experiments on LLMs of various sizes and series demonstrate that OneBit has clear advantages over representative strong baselines and achieves good tradeoff between model size and performance. We further analyze the capabilities of such extremely low-bit quantized models and provide guidance for future research.
Acknowledgments
We gratefully acknowledge the support of the National Natural Science Foundation of China (NSFC) via grant 62236004, 62206078, 62441603 and 62476073. This work was also supported by the National Key Research and Development Program of China under grant 2023YFB4503000.
References
[1] R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem. On- policy distillation of language models: Learning from self-generated mistakes. In Proceedings of the Twelfth International Conference Learning Representations (ICLR), 2024.
[2] E. Beltrami. Sulle funzioni bilineari, giomale di mathematiche ad studenti delle uninersita.
11, 98-106.(an english translation by boley is available as university minnesota, department of computer science). Technical report, Technical Report 90-37, 1990.
[3] S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. Orien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of International Conference on Machine Learning (ICML), pages 2397-2430. 2023.
[4] Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. PIQA: Reasoning about physical commonsense in natural language. Proceedings of the AAAI conference on artificial intelligence (AAAI), pages 7432-7439, 2020.
10

---

[5] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners, Advances in Neural Information Processing System (NeurIPS), 33:1877-1901, 2020.
[6] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprini arXiv:2303. 12000..227.2.. 2023.
[7] C. Clark, K. Lee, M.-W. Chang. T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2924-2936, 2019.
[8] Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? Try ARC, the A12 Reasoning Challenge. arXiv preprint 1803 05457, 2018.
[9] T. Dettmers and L. Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In Proceedings of International Conference Machine Learning (ICML), pages 7750-7774, 2023.
[10] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing System (NeurIPS), 35: 30318-30332, 2022.
[11] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing System (NeurIPS), 36:10085 10115, 2023.
[12] T. Dettmers, R. Svirschevski, V. Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh. SpQR: A sparse-quantized representation for near- lossless Ilm weight compression. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024.
[13] E. Frantar and D. Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In Proceedings of International Conference on Machine Learning (ICML), pages 10323-10337, 2023.
[14] E. Frantar, Ashkboos, T. Hoefler, and D. Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17322 2022.
[15] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In Proceedings of the Ninth International Conference on Learming Representations (ICLR), 2021.
[16] C.-Y. Hsieh, C.-L. Li, C.-k. Yeh, H. Nakhost, Y. Fujii. A. Ratner, R. Krishna, C.-Y. Lee, and T. Pfister. Distilling bys-byystee! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics (ACL), pages 8003-8011 2023.
[17] Kim, J. H. Lee, S. Kim. J. Park, K. M. Yoo, Kwon, and D. Lee. Memory- fine-tuning of compressed large language models via sub-4-bit integer quantization. Advances in Neural Information Processing System (NeurIPS), 36:36187-36209 2023.
[18] S. Kim, C. Hooper, Gholami, Z. Dong, X. Li, Shen, M. W. Mahoney, and Keutzer. SqueezeLLM: Dense-and-spare quantization. In Proceedings of International Conference on Machine Learning (ICML), pages 23901-23923, 2024.
[19] D. Kingma and Ba. Adam: A method for stochastic optimization. In Proceedings of the Third International Conference on Learning Representations (ICLR), 2015.
[20] J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han. AWQ: Activation-awarc weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
11

---

[21] Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y. Mehdad, Y. Shi, R. Krishnamoorthi, and V. Chandra. LLM-QAT: Data-free quantization aware training for large language models. In Findings of the Association for Computational Linguistics (ACL.,, pages 467 2024.
[22] X. Ma, G. Fang, and X. Wang. LLM-Pruner: On the structural pruning of large language models. Advances in Neural Information Processing System (NeurIPS), 36:21 702-21720, 2023.
[23] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In Proceedings of the Fifth International Conference on Learning Representations (ICLR), 2017.
[24] M. B. Noach and Y. Goldberg. Compressing pre-trained language models by matrix decomposi- tion. In Proceedings of the Ist Conference of the Asia- cia-Paaifiic Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing (AACL-IJCNLP), pages 884-889 2020.
[25] P. Paatero and U. Tapper. Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values. Emvironmetrics, 5(2):111-126 1994.
[26] B. Peng. C. Li, He, M. Galley, and J. Gao. Instruction tuning with gpt-4. arXiv preprint urXiv:2304.03277, 2023.
[27] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[28] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21()):5485-5511, 2020.
[29] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[30] W. Shao, M. Chen, 7. Zhang, P. Xu, L. Zhao, 7. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo. Om- niQuant: Omnidirectionally calibrated quantization for large language models. In Proceedings of the Twelfth International Conference Learning Representations' (ICLR), 2024.
[31] M. Sun, Z. Liu, A. Bair, and Kolter. A simple and effective pruning approach for large language models. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024.
[32] S. Sun, Y. Cheng, Z. Gan, and J. Liu. Patient knowledge distillation for BERT model com- pression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL.P), pages 4323 4332, 2019.
[33] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Ii, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An cntructiio-follooinn llama model. https://githhsn staaford_aal aaafford_alpacala 2023.
[34] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint 23X.::23022.177 2023.
[35] A. Tseng. J. Chee, Q. Sun, V. Kuleshov, and C. De Sa. QulP#: Even better Ilm quantization with hadamard incoherence and lattice codebooks. In Proceedings of International Conference on Machine Learning (ICML). pages 48630-48656, 2024.
[36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing System (NeurlPS), 30, 2017.
[37] Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, J. Liu, Z. Qu, S. Yan, Y. Zhu, Q. Zhang, M. Chowdhury, and M. Zhang. Efficient large language models: survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
12

---

[38] H. Wang, S. Ma, L. Dong, S. Huang. H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei. BitNet: Scaling 1-bit transformers for large language models, arXiv preprint arXhv:2300.11553, 2023.
[39] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of International Conference Machine Learning (ICML), pages 38087-38099, 2023.
[40] M. Xu, L. Xu, and D. P. Mandic. TensorGPT: Efficient compression of the embedding layer in Ilms based on the tensor-train decomposition. arXiv preprini arXiv:2307.0003 230,,0050,, 2023.
[41] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4791-4800, 2019.
[42] P. Zhang, G. Zeng. T. Wang, and W. Lu. Tlnyl.lama: An open-source small language model. arXiv preprint arXiv:2401.02:5 2024.
[43] X. Zhu, J. Li, Liu, C. Ma, and W. Wang. survey on model compression for large language models. arXiv preprint arXhv:2308.06633 2023.
13

---

A Appendix
A.1 Proofs Propositions
In this section, provide the necessary and detailed proofs for the propositions presented in this paper. All symbols have the same definition as the main text.
Given the weight matrix W and input X, the Linear layer can be reformulated as the following according to SVID:
Proposition 1
XW ((x@b") W sign C a..
Proof
From Eq. 4 have Waj Sij azbj, where 8ij is the element of W sigu- Hence have
(XW")
Σ Iisk zu
Σ J
=
if
Σ Bisk Sykzbb
Σ
Taa Tab Bubbs
E(x in Kk
[(xxb") WT
sign 13
sign. =[(xxbT) WT C 1g
This proposition is proved.
Lemma Leti; (W) denote the i-th biggest singular value of matrix W. The following inequality holds:
σ1 ( ( 2 01 (W).
Proof According to the definition of induced norm, there are
σ1 (W) = | 2 max
|x ||2
x.|[x|==1
a1 [W]) = ||| [
max
||| |y||2-
v.llvlla=1
Note that for Vx, ||x||2 = and we have
2
|x
Σ
Σ
Juisy |ussyla0r/]
2
Σ
Way Wayo0yl
Σ
Σ
WiyKi
[x
Therefore
[ >
Wx||2
v.lly v
x.||x|[a=1
This lemma is proved.
14

---

sign WW We decompose these matrices in the way W ab E₁ and [W] äb Eg, where E, denotes the error matrices. In terms of the Frobenius-norm, the SVID closer to the original matrix W:
Proposition 2 Given matrices W and W W = W
W Wign ab
W
ab
Proof Here we consider SVD to prove it. For SVD, the norm of the error matrix E in the rank-1 approximation is the sum of the squares of all singular values except for the largest one. We have
E
P
of (W),
1=2 E
P
Σ
of ([WI).
i=2
Based on WW] = ||| we have
Σ of (W)
Σ oス (W))
According to Lemma 1, we can conclude
E 1 S |
From the equation in this proposition, we can formulate
W sign
W
W
O
cb
+ W sign
E2.
sign
Hence we have
W W sign
@
ab
W
sign ① Eg.
Therefore
W sign Egll
=
1
E
where 8ij 11 is the element of W sign- Hence the inequation in this proposition is proved.
A.2 Details Baselines
In this subsection, we provide the essential details of the baselines in this work:
We employ the open-source code released by the author. Both OPT models and I.LaMA models take 128 2048-token samples from the C4 dataset to calibrate the quantized model. For LLaMA models, apply the activation order heuristic according to the recommendation from the code.
GPTQ [1
LLM-QAT 21::
We reimplement this method to adapt the W2A16 setting, as LLM-QAT is not designed for 2-bit weight quantization. We also do not quantize the KV Cache. When quantizing the weight matrix Linear layer, we use symmetric MinMax quantization in which the zero-point is set to 0. The training crrperpprammters refer to the training details in Section 4.1
the same ours. Please
We employ the open-source code released by the author. Both OPT models and LLaMA models take 128 3048-tokce samples from the WikiText2 dataset to calibrate the quantized model. The learning rate for learnable weight clipping and equivalent transformation is set to 5e-3 and le-2, respectively. We use batch size of 1 and train 40 epochs for each model. For OPT models, both learnable weight clipping and equivalent transformation are leveraged. For LLaMA models, only learnable weight clipping is used.
OmniQuant 30
15

---

A.3 Results of LLaMA2
Table compares the results on LLaMA2-7B/13B. Obviously, our method has advantages in both perplexity and zero-shot accuracy. It also reflects that the advantages of our method are more pronounced in larger models. For instance, when scaling from LLaMA2-7B to LLAMA2-13B, the perplexity of the FP16 model decreases by around only 0.5, whereas our method reduces it by around 1.0 both Wiki2 and C4 datasets.
Table 4: Results of LLaMA2. We bold the best scores.
Perplexity(L)
Zero-shot Accuracy(t)
Models
Methods
Wiki2
C4
Wino.
Hella.
PIQA
BoolQ
ARC-e
ARC-c
Avg.
FP16
5.47
6.97
67.09
72.94
76.88
71.10
53.58
40.61
63.70
GPTQ
7.7e3
NAN
50.28
26.19
49.46
42.97
26.77
28.58
37.38
LLMMA2-7B
LLM-QAT
1.1e3
6.6e2
49.08
25.10
50.12
37.83
26.26
26.96
35.89
OmniQues Quant
31.21
64.34
51.22
33.87
56.53
59.14
33.63
24.32
43.12
OneBit
9.73
11.11
58.41
52.58
68.12
63.06
41.58
29.61
52.23
FP16
4.88
6.47
69.77
76.62
79.05
68.99
57.95
44.20
66.10
GPTQ
2.1e3
3.2c2
51.85
25.67
51.74
40.61
25.46
27.30
37.11
LLaMA2-13B
LLM-QAT
5.1e2
1.1e3
51.38
24.37
49.08
39.85
27.15
24.32
36.03
OmniQuan
16.88
27.02
53.20
50.34
62.24
62.05
40.66
29.61
49.68
OneBit
8.76
10.15
61.72
56.43
70.13
65.20
43.10
33.62
55.03
A.4 Instrution Following Ability
Instruction following is an important ability of LLLMs 5 Beyond the discussion on model abilities and efficiency before, we also focus on the instruction following ability of extremely low-bit models, which is closely related to their practical usability. this subsection, we empirically study this capability of our quantized model. We fine-tune the model for 3 epochs using the alpaca_eo _n___2k___ 52k dataset and alpaca templates [33] then observe the generation both zero-shot and few-shot settings before and after fine-tuning During training. the learning rate is set to le-7 and the batch size to 32. Other parameters are consistent with Section 4.1
Table demonstrates the content generation and instruction following abilities of our 7B model. Under zero-shot setting, the model without SFT produced verbose, repetitive, and low-quality text. However, once experienced to SFT, our model is able to smoothly output high-quality content, exhibiting excellent instruction following ability. For the few-shot setting, our model exhibits instruction following ability both before and after SFT.
A.5 Comparison with BitNet
Recently, BitNet (38 introduces a 1-bit model architecture and applies the architecture to train models from scratch, demonstrating the fcasibility and application value of the 1-bit model structure. In this paper, we attempt to combine 1-bit quantization with knowledge distillation to quantize the LLaMA-7B model. Unfortunately, despite following the suggestion to use larger learning rates, the behavior remains unstable during training.
Figure 6 shows that the training process of BitNet may suffer from instability during knowledge distillation. We conjecture that it because the gradient pretty large when the weight elements fluctuate between +1 and -1, further aggravating the output of the Linear layer.
As more effective measure, the value vectors we propose for quantization necessary floating-pont numerical precision but also limit the fluctuation range of the matrix multi- plication results after guantization. This can be understood from forward and backward computation, respectively.
only supplement the
Forward stability. Quantized matrix multiplication is more prone overflow than FP16 coun- terparts in response minor perturbations of input activations. This is because the magnitude of
16

---

Table 5: Instruction following cases in zero-shot and few-shot settings before and after Supervised Fine- Tine-Tuning (SFT). The few-shot examples are from MMLU benchmark.
Prompt:
List several Windows operating systems developed by Microsoft corporation.
Generation:
Microsoft Windows 10 is a free operating system developed by Microsoft for use on personal computers and tablets. was first released to the public on 29 April 2014.
Microsofi Windows 10 free operating system developed by Microsoft for use on personal computers and tablets. It first relcased to the public 29 April 2014. Windows 10 is the successor to Windows 8.1.
Zero-shot w/o SFT
Micmosott Windows 10 is free operating system developed by Microsoft for use personal computers and tablets. It was first released to the public on 29 April 2014. Windows 10 is the successor to Windows 8. Windows 10 is the successor to Windows 8.1.
Microsoft Windows 10 is a free operating system developed by Microsoft for use on personal computers and tahlets.
Prompt:
Below is an instruction that describes task. Write a response that appropriately completes the request.
### Instruction:
List several Windows operating systems developed by Microsoft corporation.
### Response:
Zero-shot w/ SFT
Generation:
Microsoft has developed several Windows operating systems, including Windows 10, Windows 8, Windows 7, Windows XP, Windows 2000, Windows NT, Windows ME, Windows 95, and Windows I. Windows 10 is the latest version of the Windows operating system. It was released in 2019. Windows 8 was released in 2011. Windows 7 was released in 2009. Windows XP was released in 2001. Windows NT was released in 1997, Windows ME was released in 1999. Windows 95 was released in 1995. Windows was released in...
Prompt:
The following are multiple choice questions (with answers) about college biology. Which of the following contain DNA sequences required for the segregation chromosomes in mitosis and meiosis?
Splicensomes Telomeres B. Centromeres C. Nucleosomes
Answer: B. The Barr body found within female mammalian cells' nuclei example of: A. euchromatin B. heterochromatin C. cistron D. pseudogenes Answer: B. Cleavage symmetry arising from early cell divisions that occur
oblique angle to the polar axis is known A. meroblastic cleavage B. radial cleavage C. spiral cleavage D. superficial cleavage Answer: C, Which of the following is NOT source of atmospheric carbon? A. Respiration B. Photosynthesis C. Bacterial docomposition D. Combustion of fossil fucls Answer: B. The concept of punctuated equilibrium refers to: A. oscillating ecological successional stages B. ccological succession arrested by sudden environmental changes, c-g., fire C. persistent bursts speciation followed by relatively unchanging lineages Answer: D. Which of the following second messenger that stimulates release of calcium ions into the cytoplasm? A. Prostaglandins B. Calmodulin
Few-shot w/o SFT
predator-prey relationships in relatively stable environments
C. Cyclic AMP D. Inositol triphosphate Answer: Generation:
B
Prompt:
Below is an instruction that describes task. Write a response that appropriately completes the request.
### Instruction:
The following are multiple choice questions (with answers) about college biology. Which of the following contain DNA sequences required for the segregation chromosomes in mitosis and meiosis? A. Telomeres B. Centromeres C. Nucleosomes D. Spliceosomes Answer: B. The Barr body found within female mammalian cells' nuclei
example of: A. euchromatin B. heterochromatin C. cistron D. pseudogenes Answer: B. Cleavage symmetry arising from carly cell divisions that occur
oblique angle to the polar axis is known A. meroblastic cleavage B. radial cleavage C. spiral cleavage D. superficial cleavage Answer: C. Which of the following is NOT source of atmospheric carbon? A. Respiration B. Photosynthesis C. Bacterial docomposition D. Combustion of fossil fucls Answer: B. The concept of punctuated equilibrium refers to: A. oscillating ecological successional stages B. ecological succession arrested by sudden environmental changes, e.g., fire C. persistent bursts speciation followed by relatively unchanging lineages Answer: D. Which of the following second messenger that stimulates release of calcium ions into the cytoplasm? A. Prostaglandins B. Calmodulin C. Cyclic AMP D. Inositol triphosphate Answer: ### Response:
Few-shot w/ SFT
predator-prey relationships in relatively stable environments
Generation:
D
17

---

elements in quantized matrices, particularly the value +1. is far greater than the parameters of most FP16 matrices. By multiplying by value vectors of magnitude similar to that of the FPI6 model, the range of variation in model output activations can be restored to the level of FP16. Furthermore, we also avoid the increasingly large "drift phenomenon" of activations through Post-LayerNorm.
Backward stability. Since Sign(.) function is not differentiable, when the elements of the matrix change, their gradient may become infinite. Similar forward stability, by multiplying two numer- ically smaller value vectors, we avoid layer-by-layer accumulation and explosion during gradient back-propagation. Moreover, we implement the derivative function of Sign(-) using the derivative of the hyperbolic tangent function, thereby avoiding the problem of gradient explosion at the zero point of every weight.
350
350
300
300
Training Loss
Training Loss
250
MN
150
150
100
100
50
50
100
200
250
300
350
100
200
300
400
500
600
Training Steps
Training Steps
(a) learning rate 4c-4
(b) learning rate 6c-4
350
350
300
300
Training Loss
Training Loss
200
200
150
150
100
100
100D
Training Steps
Training Steps
(c) learning rate 8c-4
(d) learning rate 10c-4
350
350
Training Loss
Training Lass
200
200
150-
150
50
600
1200
100
200
700 Training Steps
800
Training Steps
(c) learning rate 12c-4
(f) learning rate 15c-4
Figure 6: Training comparisons among different learning rates when BitNet performs knowledge distillation from LLaMA-7B. Here we choose the same W1A16 setting as ours. The weight matrices in BitNet are directly copied from the original LLaMA-7B model.
18

---

Table 6: Ablation study of different loss LLaMA-7B. "ATTN" means attention score alignment.
Perplexity(1) Wiki2
Zero-shot Accuracy(t) PIQA
Loss Setting
C4
Wino.
Hella.
BoolQ
ARC-e
ARC-c
Avg.
LKD
13.48
14.57
50.83
35,14
62.89
60,46
37.33
26.37
45,50
LKD + EMSE (a = 1)
10.19
11.40
58.48
51.54
68.01
57.28
42.47
30.20
51.33
CKD + LSE (a = 10)
10.38
11,56
60.30
50.73
67.46
62.51
41.71
29.61
52.05
LKD + LMSE + CATTN
NAN
NAN
A.6 Discussion on Knowledge Distillation
Although knowledge distillation is not the main contribution of this paper, we nevertheless provide the rationale behind certain settings used in our experiments to explain the necessity of these configurations.
We firstly explain the role of different loss functions in guiding the process of knowledge transfer. Fundamentally, distillation loss alone can achieve satisfactory transfer process (comparing to other baselines). Additionally, as shown in the Table [6] aligning the hidden states between layers can result in a quantized model with better perplexity. However, further incorporating attention score alignment on this basis leads to the model failing to converge. LLM-QA [21 has conducted similar experiments on quantization-awe - knowledge distillation loss and concluded that using only the distillation loss yields the best results. The difference in conclusions may stem from two factors. On novel model architecture, which differs from theirs, the optimal
one hand, due to our adoption usage of loss functions may be different as well. On the other hand, as we focus on extremely low bit-width compression, each layer of the model suffers significant information loss compared the teacher model. The regularization of hidden states between layers may help reduce the variance in the learning process, thus demonstrating stronger generalization.
Furthermore, we also discuss the cost of our quantization method. Using LLaMA-7B as an example, quantizing the model with our method requires approximately days on 8 A100-80GB GPUs. In comparison, training the LLaMA-7B model from scratch consumes 82,432 GPU hours [34.. The quantization time, being less than 2% of the pretraining time, is still an acceptable cost.
A.7 Average Bit-width of Linear Layer
This subsection formulates the calculation the average bit-width of Linear layers. Assume there is weight matrix with shape of 4096 x 4096 in such layer, the number of bits in every component
4096 4096,
16 x x 4096 x 2,
where the first is for the 1-bit quantized weight matrix and the second is for the two FPI6 value vectors. Hence the overall number of bits is 16, 908, 288. Moreover, the number of parameters is 4096 4096 + 2 4096 x 1 16, 785, 408. Therefore, the average bit-width of this Linear layer 16, 908, 288 + 16, 785, 408 x 1.0073.
B
Limitations
Although our proposed method significantly reduces the memory footprint of LLMs, bringing hope for efficient deployment of them, there are still some limitations. Firstly, compared to the original model, our extremely low-bit quantization inevitably incurs performance loss. Additionally, we are yet to understand the mathematical principles behind the optimal parameters of the 1-bit quantized model, thus capability transfer can only be achieved through the relatively costly process of KD. Fortunately, this cost is one-time expense. Moreover, due to the unique nature of 1-bit quantization, our method can not naturally extended to higher bit-width. Lastly, we have not considered the activation quantization and leave it future work.
19

---

C Ethics Statement
In this study, we employ models that are publicly available and open source. We affirm that the use of these models aligns with their original intended purposes. These models have been utilized strictly within the scope of academic and research-based activities, adhering to ethical guidelines and ensuring compliance with open-source licenses.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We thoroughly elaborate on several limitations of this paper in Sec, B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
We provide proof' of all propositions and lemma in A.1.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All experimental results can be reproduced easily, we release the main frame- work in the submitted code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

reproducible, they Answer: [Yes]
Justification: We release the main framework in the submitted code. Moreover, everyone fetch all code, instructions, and data of this work in open-source repo this review process is complete.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm cc/
public/guidess htwps//ooooouuaasssonoooeoo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/publ httppss.ccpublli/guudeeee wttp:///////ooeeeessiiiisssissee
ssioiiooiiiiioioccicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Reviewers and readers refer Sec. 4.1 and A.2 for these details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All random settings have been provided in the paper (e.g., Sec. 4.1). In addition, all information will be included in the open source code after this review process.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
23

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have shown these in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://neurigge http://p......p..c..cccccccaaaaa
Answer: [Yes]
Justification:
We understand and fully comply with our ethical principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We describe the positive aspects of this job and generally there are no negative societal impacts from this job.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse in Ethics Statement, Sec. C.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite these works accurately.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide necessary usage instructions the code. Due to space limitations, models not available. Anyone can easily access the model and instructions after the review.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowd-sourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowd-sourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "sve16TFUTU", "reviewer": "Reviewer_9tLG", "summary": "This paper proposes OneBit, which quantizes the LLM weight matrices to 1-bit and achieves good performance and improved convergence speed by using two additional vectors with FP16 per one linear layer.", "questions": "1. Would it be possible to compare the inference speed of this method and other methods? I am curious about the potential delay in inference caused by using additional FP16 vectors.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is generally well-written and easy to follow.\n\n2. The memory required for the model part is less than other methods.\n\n3. Their methods nicely outperforms other methods in many tasks.", "weaknesses": "1. It will be great if they compare with other one-bit based quantization methods such as BitNet.\n\n2. I recommend authors to add empirical results for larger models like Llama-70b.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you have invested in reviewing our paper.\n\n**Weakness 1**: \"It will be great if they compare with other one-bit based quantization methods such as BitNet.\"\n\nConverting the **\"pre-trained model\"** into a low-bit representation is **the focus of almost all research on model quantization** [1,2,3], and we also follow this point. BitNet, as another well-known work, differs from model quantization by proposing a possible model structure with 1-bit weights (they focus on training low-bit model from scratch). Therefore, BitNet and our model quantization essentially address different research problems. In fact, we are also curious whether the BitNet structure can be used for quantizing existing models, and **we have discussed the relevant results in Appendix A.5 and Fig. 6**. The conclusion shows that, compared to BitNet, our model structure achieves stable ability transfer and training process, whereas BitNet fails to converge during quantization-aware distillation. Hence, we cannot provide benchmark results of BitNet in the main text.\n\n**Weakness 2**: \"I recommend authors to add empirical results for larger models like Llama-70b.\"\n\nDue to the **limitation of computational resources**, we are currently unable to perform experiments on 70B-scale models, although we have been trying to get the resources. However, our existing results **indicate an exciting trend**: \"the larger the model, the smaller the performance gap between the FP16 precision and the W1A16 quantized model.\" This conclusion is mentioned in lines **L238~L240** of our paper.\n\n**Question**: \"Would it be possible to compare the inference speed of this method and other methods? I am curious about the potential delay in inference caused by using additional FP16 vectors.\"\n\nIt is possible to compare the inference speed of ours **with FP16 baseline**. In a 4k-length inference test, if the weights are simulated in FP16 format containing only ±1, the OneBit model, which includes FP16 vectors, takes approximately 1.2x the duration of Llama-7b without these vectors. In fact, thanks to the **broadcasting mechanism** of tensors, the element-wise multiplication of matrices and vectors can be performed very quickly. Therefore, **the additional vector incurs minimal time overhead**.\n\nWe look forward to hearing from you and hope to address any concerns you may have about our work. Please let us know if you have any further questions.\n\nReference:\n\n[1] Frantar E, Ashkboos S, Hoefler T, et al. OPTQ: Accurate quantization for generative pre-trained transformers[C]//The Eleventh International Conference on Learning Representations. 2022.\n\n[2] Xiao G, Lin J, Seznec M, et al. Smoothquant: Accurate and efficient post-training quantization for large language models[C]//International Conference on Machine Learning. PMLR, 2023: 38087-38099.\n\n[3] Liu Z, Oguz B, Zhao C, et al. LLM-QAT: Data-free quantization aware training for large language models[J]. arXiv preprint arXiv:2305.17888, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sjklsUdDvt", "reviewer": "Reviewer_gg38", "summary": "This paper proposes OneBits, a novel quantization-aware training methodology for 1-bit large language models (LLMs). OneBits introduces two key contributions for training 1-bit models. First, it presents a new 1-bit binary quantization linear design that separates the weight matrix into sign and value components. The sign is packed into INT1, while the value is decomposed using a 1-rank decomposition factor added to the linear operation. Second, to train the 1-bit models in the linear layers of BitNets, OneBits modifies the traditional quantization-aware training (QAT) method by augmenting the cross-entropy loss function with an additional term for the reconstruction error of each layer, resulting in the final objective loss function.\n\nUsing the proposed approach, OneBits is applied to various decoder-only LLM models. The comparisons between OneBits (W1A16) and other methods like LLM-QAT, AWQ, and OmniQuant (W2A16) demonstrate that OneBits achieves superior performance in common sense reasoning tasks.", "questions": "- While the OneBit method has demonstrated independent evaluation of zero-shot and few-shot performance, showing effectiveness compared to LLM-QAT and OmniQuant, it does not provide evidence of a single OneBit model performing well in both zero-shot and few-shot scenarios simultaneously. It would be valuable to present combined performance results, such as including MMLU results in Table 2 for a comprehensive comparison.\n- When comparing the quality of output generated by OneBit LLM to other models using metrics like AlpacaEval, what trends or patterns emerge regarding the quality of generated data?", "limitations": "Since OneBit LLM is applied only to weights, this method is likely to be effective in improving latency and throughput, particularly in scenarios involving small batch sizes during the generation phase. In this paper, activation quantization has not been considered, and further research in this area is necessary to optimize performance.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a final objective loss function that combines the final cross-entropy loss with the reconstruction error of each layer using a Quantization-aware Knowledge Transfer method. The effectiveness of incorporating the reconstruction error is demonstrated through an ablation study (Table 6).\n- Unlike the traditional 1-bit linear design in BitNet, the authors introduce a new 1-bit binary quantization linear design that includes scaling factors (g/h) for each input/output channel of the weight matrix. They also propose an initialization method from a pretrained model using Sign-Value Independent Decomposition (SVID).\n- To initialize the scaling factors (g/h) for each input/output channel, the paper explores various 1-rank decomposition methods for value in SVID, including Singular Value Decomposition (SVD) and Non-negative Matrix Factorization (NMF). Experimental results indicate that the 1-rank decomposition of value using NMF is more effective than SVD.", "weaknesses": "- While the paper demonstrates zero-shot performance in terms of PPL and CSR, it lacks experiments on how the same model maintains performance in few-shot scenarios, such as the MMLU benchmark.\n- If the quality of this generated data is poor, it could negatively impact the performance of the OneBit LLM. The paper does not clearly explain why self-generated data was used instead of public datasets like C4.\n- The analysis of OneBit LLM's benefits in terms of inference latency and throughput relative to accuracy is insufficient. A detailed examination of these metrics would provide a more comprehensive understanding of the advantages of using OneBit LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you have invested in reviewing our paper.\n\n**Weakness 1**: \"It might lack a few-shot benchmark result such as MMLU.\"\n\nAlthough few-shot evaluation is not a necessary component in most model quantization research [1,2,3], we still evaluated the **5-shot** performance of OneBit-7B on MMLU in Sec. 4.3. **Please refer to Sec. 4.3 and Fig. 3(b).**\n\n**Weakness 2**: \"Poor self-generated data may negatively impact the performance of OneBit. The reason for using it instead of public dataset C4 is not clearly explained.\"\n\nHere we **follow LLM-QAT (L204)** to perform knowledge distillation. In LLM-QAT [5], the authors **have demonstrated the effectiveness of using self-generated data**, which **provide a comprehensive coverage of sample-able tokens**. Using external data may introduce **bias**. In fact, in terms of content, the quality of self-generated data (which we have open-sourced on other platforms) is indeed inferior to the carefully cleaned real data. However, it can **maximize the ability transfer of the FP16 teacher in the student model**. We take Llama-7B and PPL as examples to compare the effects of the two types of data, where C4 has the same amount of data after sampling as the self-generated data.\n\n| data source | Wiki2 | C4 |\n| -- | -- | -- |\n| sampled-C4 | 15.01 | 12.29 |\n| self-generated | **10.19** | **11.40** |\n\n**Weakness 3**: \"A detailed examination of inference latency may be insufficient.\"\n\nThe potential advantage of our method benefits from a special multiplication, **INT1(W) * FP16(A)**, in which the traditional FP16*FP16 can be quickly and efficiently replaced by setting the sign bit of FP16 activation. For example, we have **W=(0.1, -0.3, -0.2)** and **A=(0.8, 0.2, 0.7)**, and traditional W*A maybe 0.1*0.8+(-0.3)*0.2+(-0.2)*0.7 and it can be quantized to Pos(0.8)+Neg(0.2)+Neg(0.7). Here **Pos(·)** represents the **machine instruction** for setting the sign bit of a floating-point number to '+', while **Neg(·)** represents the opposite. Unfortunately, since this computation method is **not yet perfectly supported at the GPU hardware level**, we cannot demonstrate how fast OneBit truly is on the device. All experiments have been conducted using FP16 format containing only ±1 for simulation. We provide an possible efficient implementation of the OneBit Linear Layer in the supplemental material. If we **simply simulate INT1 using FP16 format**, the inference time is approximately **1.2x** that of the original model. If we **aim for extreme space compression**, i.e., expanding the compressed weight during inference, the inference latency is approximately **2.2x** that of the original model. It is worth noting that **most weight-only quantization studies introduce additional inference latency**. Additionally, we sincerely hope that our work, along with recent similar efforts, will encourage device providers to support this faster computation method at the hardware level.\n\n**Question 1**: \"It would be valuable to present combined performance results, such as including MMLU results in Table 2.\"\n\nThank you for your suggestion! We did not take this for 2 reasons: first, other than OmniQuant, the W2A16 baselines perform poorly on MMLU. Second, we wanted to compare our method’s performance in general knowledge using MMLU, hence we included it in Section 4.3.\n\n**Question 2**: \"What trends or patterns emerge regarding the quality of generated data comparing to other models?\"\n\nAs shown in PPL (Tab.2) and the content in Tab. 5, OneBit can fluently output content as long as it has not forgotten the knowledge in that domain, showing no significant difference from the original model. However, once OneBit forgets the knowledge of a certain domain, it tends to output a minimal number of tokens followed by '\\n', and then stops outputting.\n\n**Limitation**: \"Activation quantization has not been considered, and further research in this area is necessary to optimize performance.\"\n\nTo date, weight quantization [1,4] and weight-activation[2,3,5] quantization **remain 2 distinct research paths**. The reason and difficulty lie in the fact that activation quantization also compromises the model’s capabilities, with significant compression of activations causing severe degradation of the model’s performance. Therefore, the strongest baseline for W-quantization is currently W1A16, while WA-quantization can generally achieve W4A4. We are working towards further compressing activations, but we think that not considering activations may not be a limitation.\n\nWe look forward to hearing from you and hope to address any concerns you may have about our work. Please let us know if you have any further questions.\n\nReference:\n\n[1] Frantar E, Ashkboos S, Hoefler T, et al. OPTQ: Accurate quantization for generative pre-trained transformers[C]//The Eleventh International Conference on Learning Representations. 2022.\n\n[2] Xiao G, Lin J, Seznec M, et al. Smoothquant: Accurate and efficient post-training quantization for large language models[C]//International Conference on Machine Learning. PMLR, 2023: 38087-38099.\n\n[3] Shao W, Chen M, Zhang Z, et al. OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models[C]//The Twelfth International Conference on Learning Representations. 2024.\n\n[4] Lin J, Tang J, Tang H, et al. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration[J]. Proceedings of Machine Learning and Systems, 2024, 6: 87-100.\n\n[5] Liu Z, Oguz B, Zhao C, et al. LLM-QAT: Data-free quantization aware training for large language models[J]. arXiv preprint arXiv:2305.17888, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pRXcCiHZyi", "reviewer": "Reviewer_mFhw", "summary": "This paper explores an innovative 1-bit quantization framework for Large Language Models (LLMs) to significantly reduce their memory and computational demands. Traditional methods face severe performance drops with reduced bit-width; however, this paper introduces a novel quantization and initialization approach that maintains at least 81% of the original model’s performance, even with extreme bit-width reduction. The proposed method, which includes a specialized matrix decomposition and parameter initialization, demonstrates strong performance and robustness in experiments, establishing a new direction for deploying LLMs on resource-constrained environments.", "questions": "1. The experiments described in Section 4.3 may not provide an appropriate comparison. Assessing the proposed method alongside directly training smaller models or utilizing low-rank decomposition to minimize parameter counts involves fundamentally different approaches to reducing model size. Additionally, the proposed method incorporates knowledge distillation, which is not employed in the baseline methods being compared, potentially skewing performance comparisons.\n\n2. An essential ablation study is notably absent from the discussion. The proposed method incorporates two additional floating-point vectors, $\\mathbf{a}$ and $\\mathbf{b}$, for binary quantization. Yet, the impact of these vectors on performance enhancement remains unclear, highlighting a gap in the evaluation of the method’s effectiveness.\n\n3. To offer a more comprehensive evaluation, I recommend including assessments on generative tasks, such as code generation. This would provide deeper insights into the versatility and practical applicability of the proposed method across different domains.", "limitations": "The authors have discussed the limitations of their study.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to follow. The proposed method shows good performance, achieving at least 81% of the unquantized model’s efficacy, a significant achievement given the drastic reduction in model complexity and size.", "weaknesses": "A significant issue discussed in the paper is the lack of a specialized CUDA kernel for optimizing binary operations, which hinders accurate evaluation of the additional computational costs associated with the two FP vectors $\\mathbf{a}$ and $\\mathbf{b}$. This limitation complicates the assessment of their impact on overall performance. Furthermore, despite the inclusion of $\\mathbf{a}$ and $\\mathbf{b}$, there remains a considerable performance decline compared to FP16 models, challenging the practical applicability of this approach in real-world settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you have invested in reviewing our paper.\n\n**Weakness**: \"It might lack **specialized CUDA kernel** for optimizing binary operations and the additional computational costs associated with the two FP vectors a and b may be not clear. Moreover, the **performance decline** challenging the practical applicability in real-world settings.\"\n\n- For CUDA kernel and inference time:\n\nThe potential advantage of our method benefits from a special multiplication, **INT1(W) * FP16(A)**, in which the traditional FP16*FP16 can be quickly and efficiently replaced by setting the sign bit of FP16 activation. For example, we have **W=(0.1, -0.3, -0.2)** and **A=(0.8, 0.2, 0.7)**, and traditional W*A maybe 0.1*0.8+(-0.3)*0.2+(-0.2)*0.7 and it can be quantized to Pos(0.8)+Neg(0.2)+Neg(0.7). Here **Pos(·)** represents the machine instruction for setting the sign bit of a floating-point number to '+', while **Neg(·)** represents the opposite. Unfortunately, since this computation method is **not yet perfectly supported at the GPU hardware level**, we cannot demonstrate how fast OneBit truly is on the device by carefully designing CUDA kernel.\n\nAll experiments have been conducted using FP16 format containing only ±1 for simulation. We provide an possible efficient implementation of the OneBit Linear Layer in the supplemental material (with parallel tensor computation). If we **simply simulate INT1 using FP16 format**, the inference time is approximately **1.2x** that of the original model. In fact, thanks to the **broadcasting mechanism** of tensors, the element-wise multiplication of matrices and vectors can be performed very quickly. Therefore, the **additional vector incurs minimal time overhead**. If we **aim for extreme space compression**, i.e., expanding the compressed weight during inference, the inference latency is approximately **2.2x** that of the original model. It is worth noting that **most weight-only quantization studies introduce additional inference latency**. Additionally, we sincerely hope that our work, along with recent similar efforts, will encourage device providers to support this faster computation method at the hardware level.\n\n- For practical applicability:\n\nEven though our proposed quantization is lossy compression, we strive to demonstrate its practical value in this paper. In Section 4, we show our method’s **excellent performance on benchmarks** by comparing it with strong baselines. In Appendix A.4, we demonstrate the practicality of our method through SFT and **instruction-following tests**. Moreover, compared to contemporaneously published method [1], our approach proves to be superior in both effectiveness and capability, underscoring the research and practical value of our method.\n\n**Question 1**: \"The proposed quantization method and training smaller models or utilizing low-rank decomposition are different approaches, and their comparison may not appropriate. Additionally, comparing with the baselines, which not employe knowledge distillation, may be appropriate as well.\"\n\nWe understand the reviewer's concern. It is important to clarify that our comparisons with these different compression methods are **not intended to defeat them**, but rather to **demonstrate the model’s capabilities from another perspective**—specifically, that it **performs better than smaller models of the similar scale**. Therefore, we did not include this comparison as the main result in Sec 4.2, but rather **as a separate subsection**, \"Problem Solving Ability\".\n\nAdditionally, regarding the second concern, firstly it is crucial for us to **compare with strong baselines in model quantization, regardless of the methods they employ**. Among the baselines we selected, **there are also methods based on knowledge distillation (training)**, such as LLM-QAT [2], which was once a strong baseline. **Please refer to Sections 4.1 & 4.2.**\n\n**Question 2**: \"An essential ablation study is notably absent from the discussion. The impact of these vectors a/b on performance enhancement remains unclear.\"\n\nIn fact, the **comparison with BitNet is essentially an ablation study** concerning the a/b vectors, as the **main difference** between our model structure and BitNet lies in the introduction of the FP16 a/b vectors. Due to space limitation, the main discussion is placed in Appendix A.5, with the conclusions only presented in Sections 5.2 & 5.3 of the main text. We will address this issue in the revised version. From the discussion in A.5, we demonstrate that, **once these two vectors are deleted (BitNet), the 1-bit weight W-only model fails to converge during quantization-aware distillation.** Hence, our model structure (with a/b vectors) achieves stable ability transfer and training process, which demonstrate its necessity.\n\n**Question 3**: \"Including generative tasks would provide deeper insights into the versatility and practical applicability of the proposed method across different domains.\"\n\nThank you for your valuable suggestions! We will consider adding more examples of generative tasks in the revised version's Appendix to demonstrate the practical value of our method.\n\nWe look forward to hearing from you and hope to address any concerns you may have about our work. Please let us know if you have any further questions.\n\nReference:\n\n[1] Huang W, Liu Y, Qin H, et al. BiLLM: Pushing the Limit of Post-Training Quantization for LLMs[C]//Forty-first International Conference on Machine Learning. 2024.\n\n[2] Liu Z, Oguz B, Zhao C, et al. LLM-QAT: Data-free quantization aware training for large language models[J]. arXiv preprint arXiv:2305.17888, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Apt94FwsZp", "reviewer": "Reviewer_a9Ag", "summary": "This paper presents OneBit, a framework for quantizing large language models (LLMs) to 1-bit weight matrices. Unlike existing methods that rely on 4-bit or 8-bit quantization to avoid severe performance degradation, OneBit introduces a novel 1-bit parameter representation and an effective parameter initialization method based on matrix decomposition.", "questions": "See above.", "limitations": "See above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes an aggressive compression method, exploring the feasibility and challenges of compressing LLMs to 1-bit. This is a highly meaningful research direction, and similar work should be encouraged for publication. However, I have some concerns about this approach, which I will detail in the questions section.\n\n- The paper is well-organized and well-written.\n\n- The experiments compare the proposed method with current popular low-bit quantization methods and demonstrate superior results. Additionally, the experiments address an important question in the current LLM research field: whether to use quantized large models or directly use smaller models.", "weaknesses": "- Entropy Encoding: Intuitively, 1-bit compressed models should be suitable for entropy encoding. Previous work has demonstrated that quantized LLMs still have compressibility [1,2]. Have the authors tried using popular entropy encoders to further compress these weights?\n\n- Inference Calculations: During inference, do g and h participate in the calculations? Does this mean that 1-bit net is not entirely integer-based computation? Does each layer have its own G and h?\n\n- Dequantization: Is dequantization required between layers during inference?\n\n- Knowledge Distillation (KD): In line 191, it is mentioned that KD does not use LM loss. Why is that?\n\n- Comparison with 1.58bitnet: Have the authors considered comparing their method with 1.58bitnet [3]?\n\n- Table 2 Details: Does Table 2 show the performance of W1A16 or W2A16? If it is W2A16, where is the performance of W1A16 reported? Why are there no experimental results for 16B and 60B models in Table 2? I suspect that extreme quantization has a more detrimental effect on larger models.\n\n- Comparison in Figure 3: Figure 3 compares the performance of 1-bit quantized 7B models with smaller models. Are these smaller models in full precision? How would the results compare with 8-bit smaller models?\n\n- Code Availability: Will the code be open-sourced?\n\n- Minor Issues: Figures 3 and 4 are not clear when printed in gray-scale.\n\nReferences:\n\n[1] Mao Y, Wang W, Du H, et al. On the compressibility of quantized large language models. arXiv preprint arXiv:2403.01384, 2024.\n\n[2] [2] Hershcovitch M, Choshen L, Wood A, et al. Lossless and Near-Lossless Compression for Foundation Models[J]. arXiv preprint arXiv:2404.15198, 2024.\n[3] Ma S, Wang H, Ma L, et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits[J]. arXiv preprint arXiv:2402.17764, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you have invested in reviewing our paper.\n\n**Question 1**: \"Have the authors tried using popular entropy encoders to further compress these weights?\"\n\nThis maybe a great intuition! Theoretically, a model quantized to 1-bit does have the potential for further compression using entropy coding. In fact, we are currently exploring this possibility, and it holds independent research value.\n\n**Question 2**: \"During inference, do g and h participate in the calculations? Does this mean that 1-bit net is not entirely integer-based computation? Does each layer have its own G and h?\"\n\nYes. The FP16 vectors g/h participate in the calculation during inference. Our proposed 1-bit net structure is indeed not entirely integer-based. The core of our idea is the minimal use of floating-point and their substantial benefits. Finally, each Linear layer has its own G and h.\n\n**Question 3**: \"Is dequantization required between layers during inference?\"\n\nNo. Dequantization is not required between layers during inference, because the original weight have been entirely converted into a quantized matrix W and two floating-point vectors g/h.\n\n**Question 4**: \"Why is it mentioned that KD does not use LM loss in L191?\"\n\nWe specifically mention in the paper that LM loss is not used to ensure accurate description and facilitate reproducibility of our results. The reason for not using LM loss is that this loss does not demonstrate beneficial effects in our experiments.\n\n**Question 5**: \"Have the authors considered comparing their method with Bitnet-b1.58?\"\n\nBitNet-b1.58 is a work slightly later than ours, aiming to propose a possible 1-2 bit model structure (examining the performance of training from scratch), which differs from model quantization that transforms a \"pre-trained model\" into a low-bit representation. However, we are also curious whether the BitNet-(1b/1.58b) structure can be used for quantizing existing models, and **we have discussed the relevant results of BitNet-1b in Appendix A.5 and Fig. 6**. BitNet-b1.58 is also similar to this. The conclusion shows that, compared to BitNet, our model structure achieves stable ability transfer and training process, whereas BitNet-(1b/1.58b) fails to converge during quantization-aware distillation.\n\n**Question 6**: \"**1.** Does Table 2 show the performance of W1A16 or W2A16? If it is W2A16, where is the performance of W1A16 reported? **2.** Why are there no experimental results for 16B and 60B models in Table 2? I suspect that extreme quantization has a more detrimental effect on larger models.\"\n\n1. Table 2 compares our method with four baselines: FP16, GPTQ, LLM-QAT, and OmniQuant. FP16 represents the non-quantized model, serving as the upper bound for all methods’ capabilities. As the first 1-bit weight quantization method, our approach, OneBit, uses W1A16 quantization. The baselines GPTQ, LLM-QAT, and OmniQuant use a W2A16 quantization level.\n\n2. Due to the **limitation of computational resources**, we are currently unable to perform experiments on larger scale models, although we have been trying to get the resources. However, our existing results **indicate an exciting trend**: \"the larger the model, the smaller the performance gap between the FP16 precision and the W1A16 quantized model.\" This conclusion is mentioned in lines **L238~L240** of our paper.\n\n**Question 7**: \"Figure 3 compares the performance of 1-bit quantized 7B models with smaller models. Are these smaller models in full precision? How would the results compare with 8-bit smaller models?\"\n\nYes, the smaller models are in full FP16 precision. The conclusion is that our 1-bit quantized model is better than the similar scaled full precision model. As the upperbound, the full precision model is better than the same 8-bit smaller models. Hence, we can directly conclude that our 1-bit quantized model is better than the 8-bit smaller models.\n\n**Question 8**: \"Code Availability: Will the code be open-sourced?\"\n\nWe have submitted the core OneBit Linear Layer python code in the supplemental material to NeurIPS. Additional, all the code, data, and checkpoints are fully open-sourced on other platforms. They will be made completely public after the peer review process is concluded.\n\n**Question 9**: \"Minor Issues: Figures 3 and 4 are not clear when printed in gray-scale.\"\n\nThank you for your reminder and suggestions! We will reconsider their line styles, colors, and plotting methods, and make adjustments in the revised version.\n\nWe look forward to hearing from you and hope to address any concerns you may have about our work. Please let us know if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces an aggressive quantization technique that is of interest to the community. However, I would recommend the authors moderate some of their claims. For instance, it is not, strictly speaking, 1-bit quantization when considering the two vector values.
Decision: Accept (poster)