Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models
Authors: Dongwon Jo
Taesu Kim
Yulhwa Kim
Jae-Joon Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Binarization
Quantization
Model Compression
Efficient LLM
Primary area: natural_language_processing
TL;DR: BinaryMoS introduces memory-efficient token-adaptive binarization for LLMs, reducing model size and enhancing representation ability of binarized weights

Abstract: Binarization, which converts weight parameters to binary values, has emerged as an effective strategy to reduce the size of large language models (LLMs). However, typical binarization techniques significantly diminish linguistic effectiveness of LLMs.
To address this issue, we introduce a novel binarization technique called Mixture of Scales (BinaryMoS). Unlike conventional methods, BinaryMoS employs multiple scaling experts for binary weights, dynamically merging these experts for each token to adaptively generate scaling factors. This token-adaptive approach boosts the representational power of binarized LLMs by enabling contextual adjustments to the values of binary weights. Moreover, because this adaptive process only involves the scaling factors rather than the entire weight matrix, BinaryMoS maintains compression efficiency similar to traditional static binarization methods. Our experimental results reveal that BinaryMoS surpasses conventional binarization techniques in various natural language processing tasks and even outperforms 2-bit quantization methods, all while maintaining similar model size to static binarization techniques.

Mixture of Scales: Memory-Eficieet Token-Adaptive Binarization for Large Language Models
Dongwon Jo
Taesu Kim
Yulhwa Kim 3 Jae-Joon Kim
Seoul National University 2 'SqueezeBits Inc. 3 "Sungkyunkwan University
{dongwonjo, kimjaejoon] @snu. ac ac. kr
(taesu. ( kiim)//sueezzettss..... com
(yulhvakim) Oskku edu
Abstract
Binarization, which converts weight parameters to binary values, has emerged as effective strategy reduce the size of large language models (LLMs). However, typical binarization techniques significantly diminish linguistic effectiveness LLMs. To address this issue, introduce novel binarization technique called Mixture of Scales (BinaryMoS). Unlike conventional methods, BinaryMoS em- ploys multiple scaling experts for binary weights, dynamically merging these experts for each token to adaptively generate scaling factors. This token-adaptive approach boosts the representational power of binarized LLMs by enabling contex- tual adjustments to the values of binary weights. Moreover, because this adaptive process only involves the scaling factors rather than the entire weight matrix, Bina- ryMoS maintains compression efficiency similar to traditional static binarization methods. Our experimental results reveal that BinaryMoS surpasses conventional binarization techniques in various natural language processing tasks and even out- performs 2- quantization methods, all while maintaining similar model size static binarization techniques.
Introduction
Though large language models (LLMs) have delivered impressive results in variety of natural language processing (NLP) tasks, their massive size often complicates deployment. One common method to compress LLMs through the quantization of weight parameters, which reduces model sizes by lowering the precision of weight values [ 8 2
30, 31 32
33
3 4 Existing quantization
approaches such as GPTQ 2 AWQ 3. and OWQ have successfully managed to reduce model sizes by converting 16-bit floating point weights to 4- bit representations, achieving a fourfold decrease in size, Binarization pushes this concept even further by reducing weight values 1-bit, resulting in a 16-fold size reduction.
However, such aggressive compression through binarization drastically limits the representational capacity of weights. leading to a significant degradation in the linguistic capabilities of LLMs. To address this limitation and improve the accuracy of binarized LLMs, recent research has actively explored binarization techniques tailored for LLMs (29 5 6 (7) Nonetheless, previous efforts often compromise the inherent advantages of binarization by introducing high memory overhead, and they continue to struggle to achieve sufficient accuracy with binarized LLMs.
In this paper, we propose novel binarization technique named Mixture of Scales (BinaryMoS). Typical binarization methods use scaling factors to control the effective values of binarized weights. Although these scaling factors occupy tiny fraction of the overall model size, they crucial
"Coresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Binary Weight
High-Prectaion (e.g. INT8) Weight
Non-Salient
Non-Salient
Scale
Scale
Input
Output
Input
Oulput
Scale
Scale
Salient
Sallent
(a) PB-LLM
(t)) BILLM
Router
Scale
Scale
Input
Output
Input
Scale
Scale
Output
(c) OneBit
(d) BinaryMoS
Figure 1: A brief overview of various LLM binarization methods. PB-LLM involves both binary weight matrix and a high-precision, sparse weight matrix, and BiLLM stores four types of binary weight matrices. OneBit simplifies the layer structure by introducing scaling factors for input and output dimensions respectively. BinaryMoS introduces multiple scaling experts to enhance the capacity of binarized models.
in reducing binarization error. BinaryMoS advances the functionality of these scaling factors by incorporating -daativeee scaling factors. Inspired by the Mixture of Experts (MoE) approach [26] 27 [34] which empolys multiple expert layers to enhance the model capacity, BinaryMoS adopts multiple scaling factors as experts to improve the representational capacity of binarized LLMs in memory-efficient way. During inference, BinaryMoS linearly combines these scaling experts based on the context generate token-adaptive scaling factors, thus dynamically adjusting the represented values of binarized weights to maximize the expressive power of the model. As result, BinaryMos, can improve the linguistic performance of binarized LLMs with minimal memory overhead.
2 Background
2.1 Binarization of LLMS
Binarization stands out as an extreme yet effective method for reducing model sizes in deep learning. This method achieves size reduction by transforming high-precision weight parameters into 1-bit values. The binarization process is typically governed by the following equation:
WB Sign W FP)
Here, Wyp E B is the full-precision weight matrix of linear layer where and represent the size of output and input dimension, respectively, and WB 1 denotes its binarized version. I represents scaling factors that responsible for adjusting the binary weight values along the output dimension. In general, the scaling factors are analytically derived as the absolute mean of FP weight values to minimize the L2 error between full-pree full-precision and binarized weights, and these scaling factors play a vital role in bridging the gap between the original full-prrcision weights and their binarized counterparts.
While binarization has been effectively applied traditional deep learning models Convolutional Neural Networks (CNNs) for image classification without losing accuracy 9
10 12 LLMs tend
be more sensitive to such extreme quantization, often experiencing significant accuracy degradation with standard binarization techniques. Therefore, various binarziation techniques tailored for LLMs

---

have been developed, as shown in Figure I PB-LLM 5 partially binarizes weight parameters while maintaining salient weight parameters as high-precision values (e.g.. 6 or INT8). However, this method results in considerable memory overhead. For instance, quantizing 10% of weight parameters as INT8 while binarizing the remaining 90% results in an average bit- of 1.7 bits for the weight parameters, which is closer to 2 bits than bit.
Furthermore, despite this partial binarization strategy of PB-LLM, the significant information loss inherent in binarization still causes considerable accuracy degradation. To reduce the binarization error and enhance accuracy, BiLLM [6] adopts a more refined approach assigning scaling factors. Assuming that weight parameters follow a bell-shaped distribution, BiLLM categorizes weight parameters based on their proximity to the mean value: concentrated weights, close the mean, and sparse weights, distant from the mean. Distinct scaling factors are then assigned to each group to minimize binarization errors. Then, to reduce the memory overhead associated with maintaining information of salient weights, BiLLM preserves this information by binarizing the difference between the binarized values and their full-precision counterparts. Consequently, each salient weight is represented by two I-bit values, effectively amounting to 2-bit representation. Despite significantly reducing binarization error, BiLLM complicates the structure of binarized LLMs, adding complexity to the inference process. This complexity arises from the need to manage additional sparse and salient weights alongside regular concentrated weights, requiring extra matrix multiplication during inference.
Meanwhile, unlike conventional binarization methods that typically employ scaling factors only for the output dimension of weights, OneBit (7 enhances the binarization process by incorporating scaling factors for both the input and output dimensions. This dual-dimension scaling approach addresses binarization errors across both dimensions, potentially enhancing model accuracy. Additionally, the size of each scaling vector is substantially smaller compared to the weight matrix, making this approach memory efficient. For instance, in linear layers with hidden dimension of h. the weight matrix size is h h, while each scaling vector is only x 1. Therefore, doubling these scaling factors adds a negligible memory overhead to the network. Moreover, as this approach of dual-dimensional scaling efficiently preserves enough information to significantly reduce binarization errors, One eliminates the need to store separate information for salient weights, thereby simplifying the model structure. The result of matrix multiplication Y of a linear layer using the OneBit approach can be defined as follows:
Y= - @ Sign(WFF)) o Sows [(X o S.,, SinnSigg((WF)) 3 Sout
Here, X E kx is the matrix of input activation where l represents batch size, while S.,. E 1xm and Sout E RR. xn denote the scaling factors for input and output dimensions, respectively. As outlined in Equation 2] processing scaling factors for both input and output dimension can be simplified to scaling input and output of the linear layer before and after matrix multiplication, respectively.
Despite advances in binarization techniques for LLMs, notable accuracy gap still exists between full- precision models and their binarized counterparts. Therefore, bridging this gap without sacrificing the fundamental benefits of binarization, particularly low memory usage, remains an important challenge in the field of LLM compression.
2.2 Mixture of Experts
The MoE approach is widely adopted strategy to boost the capabilites of deep learning models by integrating multiple specialized experts into a single framework (26] 27 34 Typically, the MoE approach for LLMs involves duplication of layers and selecting the appropriate layers among these duplicates for a specific task during inference. In the MoE setup, the router is a key to selecting the appropriate expert. It generally consists of linear layer followed by softmax function, which calculates and assigns scores to each expert. During the inference, only the experts with the highest score are selected and processed.
While integrating the MoE approach with binarized LLMs offers potential for improving model accuracy, it presents a substantial memory trade-off. The duplication of layers inherent in MoE increases the model size proportionally with the number of experts, thus diminishing the memory efficiency benefits gained from binarization. To address these challenges, we propose BinaryMoS, a novel binarization technique that aims to enhance model capacity while maintaining memory efficiency. This approach leverages scaling factors as experts, improving accuracy of binarized
3

---

Mutti-Heed Self-Attentio
Input
Gating Scora(6)
Limerr
Linser Layer IK projection)
Linser Layer projection)
W
Router
(a projection)
Sa Expert
(N
Attention Block
Linear Layer (Out projection)
Feed-Forwerd Forword Network
Matrix Multiplication
Linear Weighted Sum
Input
Linear (FC1)
Activation
Linear
(PC2)
Illustration of proposed BinaryMoS scheme. The proposed BinaryMoS introduce mixture of scale approach to generate token-adayy voken-adaptive scaling factors.
Figure 2:
LLMs without the extensive memory overhead associated with traditional MoE configurations. In the following section, we will delve deeper into how BinaryMoS operates and its benefits over conventional techniques.
Proposed BinaryMoS
3.1 Binarization with Mixture of Scale
An overview of the proposed BinaryMoS presented Figure 2 techniques that utilize single scaling vector per input output dimension, BinaryMoS integrates the concept of experts from the MoE framework into the scaling factors and utilizes multiple scaling experts for each dimension. As discussed in Section 2.1 although the size of scaling factors is relatively small, they play crucial role in preserving the accuracy of binarized models. Therefore, introducing multiple scaling experts incurs minimal memory overhead while effectively leveraging the advantages of the MoE strategy to enhance the capabilities of binarized models.
Unlike previous binarization
In the MoE framework, the number of experts selected corresponds directly to the number of layers processed. As a result, the typical MoE framework selects only one or two experts per inference stage to manage the increased processing burden associated with more experts being selected. On the other hand, the scaling factors of binarized LLMs are solely involved in linear operations with matrix multiplication, detailed in Equation This linearity allows for the efficient management of multiple scaling experts by linearly combining them before executing the matrix multiplication. Hence, instead of selecting only few experts, done in the conventional MoE framework, BinaryMoS dynamically generates instructions on how to combine these scaling experts based on the context. This approach overcomes the limitations of fixed expert choices typical MoE setups by enabling the creation of effectively infinite token-ades token-adaptive scaling factors through linear combinations. Consequently, by optimally utilizing the representational power multiple scaling experts, BinaryMoS maximizes the potential of binarized models while maintaining memory efficiency.
3.2 Router Design
In order to generate the -dkativee scaling factors, the proposed BinaryMoS designs the router for processing the following operations:
G Softmax(XXa,
S.n GS
Sout GSout
(4)
Here, WR E Raxx represents the weight parameters of router's linear layer, where denotes the number of experts. Sin E Rexm and Sout E R denote the scaling experts for input and output dimension, respectively. Initially, the router computes the gating score G, which represents

---

Table Comparison of memory requirements for deploying 6 and binarized models, with the number in parentheses denoting the compression ratio of binarized models over Floatl6 models.
Model
Float16
PB-LLM
BiLLM
OneBit
BinaryMoS
1-2MAA12---BB
13.51 GB
2.78 GB (4.86x)
2.28 GB (5.93x)
1.37 GB (9.86x)
1.40 GB (9.65x)
1LMMA-122-13B
26.20 GB
5.02 GB (5.22x)
4.06 GB (6.45x)
2.29 GB (11.44x)
2.33 GB (11.24x)
the significance of each scaling expert, using input activations and router weights, outlined in Equation 3 Notably, as the gating scores are generated with the softmax function, the sum of gating scores for the scaling experts equals 1. These scores are used to linearly combine the scaling experts, resulting in the creation of loken-adappive scaling factors Sin and Sout. as shown in Equation 4 Then, by replacing the static scaling factors Sin and Sout from Equation 2 with token-adaptive scaling factors, the result of matrix multiplication Y in a linear layer using the BinaryMoS approach can be revised as follows:
= [[x o (in))Sin(WFFP)) FP)] o Sout
(5)
We empirically find that using four scaling experts each for the input and output dimensions provides the optimal compromise between increasing model size and improving accuracy. Consequently, the proposed BinaryMoS utilizes four scaling experts for each dimension to enhance accuracy while maintaining efficiency.
3.3 Impact of BinaryMos LLM Compression
The proposed BinaryMoS introduces additional memory overhead due to multiple scaling experts and the weights of the router. However, this overhead is relatively minor. For instance, in the LLaMA-1/2-7B model [16] with hidden dimension h of 4096, the weight matrix for the linear layers is typically 4096 If BinaryMoS adopts 4 scaling experts, this translates to four a's, each of dimension 4096x 1, for both input and output dimensions. Additionally, the weights of the router would be 4096x4. Compared to the previous OneBit method, which requires a single a for both input and output dimensions, the additional components in BinaryMoS total 4096 10 parameters. The number of these extra parameters constitutes only 0.2% of the original weight parameters.
For a comprehensive examination of the impact of various binarization techniques, including Binary- MoS, on LLM compression, we evaluate the memory requirements of LLaMA models with F parameters and after applying different binarization methods, as detailed in Table Following stan- dard practice, all binarization techniques exclude the embedding layer and Im-head from binarization. Our analysis reveals that BinaryMoS significantly reduces the memory footprint of models, achieving compression ratios ranging from 9.65 x to 11 x As model size increases, the relative impact of additional parameters diminishes and the proportion of the unbinarized part decreases. Hence, can achieve higher compression ratios for larger models. For instance, the original LLaMA-1/2-132 model, requiring 26.20 GB for deployment, is impractical for edge devices due to its size. However, BinaryMoS reduces this model to just 2.33 GB, representing an 11 1.24-fold decrease in memory requirements. This significant reduction facilitates deployment on edge devices with typically limited memory capacities of 4 GB.
In contrast, PB-LLM and BiLLM methods achieve relatively lower compression ratios of around 5x and 6x, respectively. This is primarily due to two reasons: first, PB-LLM and BiLLM methods must retain salient weight information, increasing the average bitwidth of weight parameters. Second, the handling of sparse weight matrices in these methods introduces overhead in indexing sparse weight matrices, limiting the achievable compression ratio. OneBit achieves the highest compression ratio by only introducing dual-dimension scaling factors. Remarkably, BinaryMoS achieves a comparable compression ratio to OneBit, despite incorporating additional components for scaling experts. While the memory requirement of binarized models with BinaryMoS increases by only 2% compared to OneBit, the inclusion of scaling experts offers much greater potential to significantly improve perplexity.
This analysis demonstrates that although BinaryMoS introduces additional parameters, the relative increase in memory requirement is modest. This makes BonaryMoS a viable option for enhancing accurcy of binarized models without imposing significant memory burden.
5

---

3.4 Quantization-Aware Knowledge Distillation
Following training strategies adopted for network compression 13
we adopt the knowledge
distillation (KD) to transfer the knowledge of a full-precision teacher model to binarized student model. We employ the cross entropy (CE) loss distill the logit knowledge. This is calculated using
the following equation:
-oE=--EE Lor = TE ΣV Pe (X;) log (p.. (X,))
(6)
Here, S and T represent the student and teacher models respectively. denotes batch size, and c is the number of classes. Additionally, minimize the distributional discrepancies in layer outputs, we incorporate mean-squared error (MSE) based layer-to-layer (L2L) loss as follows:
L1.1 Σ MSE H H
In this loss H. and Hý are the output logits from the 1-th layer of the teacher and student models, respectively. The total loss function, integrating both CE and L2L distillation losses, is defined as:
L LCE + a L12L
(8)
where hyperparameter that balances the contributions of the CE and L2L losses. For the training of BinaryMoS, we empirically set a = 10.
Experiments
4.1 Experimental Settings
Models and Evaluation Datasets. our study, we evaluate BinaryMoS on various models, including those from the LLaMA-1 [16], LLaMA-2 (17), and OPT [[5] families. Specifically, utilize the OPT models with 125M and 1.3B parameters, and the LLaMA-1 and LLaMA-2 models with 7B and 13B parameters for evaluations. We measure language modeling capabilities of these models by evaluating their perplexity on the WikiText2 (24] and C4 (2 datasets. Additionally, we assess zero-shot accuracy on various Common Sense Reasoning Tasks such BoolQ [19], PIQA [20), HellaSwag (21), WinoGrande [22). ARC-e, ARC-c (23), utilizing the open-source LLM evaluation
framework, LMMEEvllationn-aanee
35
Training Details. We initialize the parameters of binarized models using those from pre-trained models, which serve teacher models for KD. For the training dataset, mixed dataset composed of the WikiTexx22 training dataset and a selected partition from the C4 training dataset, with a sequence length of 2048. The training is conducted over three epochs using the AdamW (18] optimizer, with hyperparameters set to 31 = 0.9, ,32 = 0.999, and zero weight decay. We implement cosine decay learning rate scheduler, preceded by warm-up phase constituting 0.03 of the total training duration. All training sessions are conducted on NVIDIA A 110 GPUs
Baselines. We compare BinaryMoS against previous LLM binarization methods, including PB- LLM [5), BiLLM (6), and OneBit (7), ensuring that all implementations adhere to the details provided in their respective papers. PB-LLM and BiLLM utilize the Post-Training Quantization (PTQ) approach for model calibration through the Optimal Brain Quantizer (OBQ) based method of GPTQ 21. For PB-LMM, which allows variable ratios of salient weights to enhance accuracy, we have set the ratio of salient weights to 10% to ensure the average bit width of weight parameters remains below 2 bits. OneBit employs Quantization-Aware Training (QAT) approach, and for fairesss, its training setup is aligned with that of BinaryMoS.. Given the significant accuracy improvements demonstrated by Bon over traditional binarization techniques, also include comparison with 2-bit quantization methods with PTQ approach, such as GPTQ [2] and OmniQuant (28) to broaden the evaluation scope.
4.2 Analysis on the Number of Scaling Experts
To determine the optimal number of scaling experts for BinaryMoS, which effectively maintains the accuracy of binarized LLMs while minimizing memory usage, conduct evaluations with

---

Table 2: The impact of the numbers of scaling experts on the proposed BinaryMoS. Quick assessment conducted using the LLaMA- 1-7B model trained on one-third of the training data.
Perplexity I Wiki2
Zero-shot Accuracy 1 ARC-e
Experts
C4
BoolQ
PIQA
Hella.
WinoG.
ARC-c
Average
9.33
12.54
60.27
67.84
46.77
52.09
38.38
27.98
48.89
9.19
12.18
62.69
68.55
48.36
55.09
40.23
28.92
50.64
8.92
11.85
60.51
67.46
49.95
55.24
41.16
29.35
50.61
9.17
12.28
58.68
67.46
47.51
53.67
39.52
29.43
49.38
(8)
Gating Score
0.0
0.2
0.4
0.6
0.8
1.0
Token
Sequence
Token N
(b)
Scaing Factor Expeetee [mpp
Scaling Output
Taken-Adaptive Scaling Factor (+t of Eipprt=<4
0.136
0.130
Scaling Factor Values
0.124
0.118
20
80
100
Scaling Factor for Input Dimension
0.152
0.144
0.136
0.128
20
40
8C
100
Index Input (Battom) and Output (Top) Dimension
Figure 3: (a) Gating scores of 4 scaling experts in 18th layer of LLaMA-1-77 1-7B model for each token in the input sequence. (b) Distribution of values of token-adaan addn-vdde scaling factors. The boxplot visually presents the distribution of token-adaptive scaling factors among processed tokens. The box spans the interquartile range, indicating the middle 50% of the scaling factors. Extending from the box are whiskers that reach the furthest data points within 1.5 times the interquartile range, providing insight into the overall range of the data.
LLaMA-1-7B using varying numbers of scaling experts. This evaluation is conducted using only one-third of the training data for quick assessment. As shown in Table 2 performance metrics, including perplexity and accuracy, generally improve the number of experts increases from 4. However, a further increase to experts leads to decline in model performance. This decline arises from the challenge of training routers appropriately assign scales tokens as the number of scales increases. Based on these observations, we choose to employ 4 experts in the BinaryMoS approach.
4.3 Analysis on the Token-Adaptive Scaling Factors
In this section, we explore the effectiveness of the proposed mixture of scale approach in generating token-adaptive scaling factors. To accomplish this, analyze the gating scores for scaling experts and the scaling factors derived from these scores. For this analysis, we utilize the LLaMA-1-7B model and input sequences sampled from the C4 dataset.
Figure B showcases the gating scores and resulting token- adaativee scaling factors for out projection of the 18th layer across tokens of the input sequence. The experimental results reveal substantial variation the gating scores for each expert across tokens. As depicted Figure bb) while conventional binarization methods with static scaling factors, akin to having single expert, offer fixed scaling factor, the scaling experts BinaryMoS successfully generate diverse range scaling factors. This highlights the efficacy of the mixture of scale approach, which adaptively determines the
7

---

Table Perplexity and zero-shot accuracy results of Floatló and binarized LLMs.
Perplexity Wiki2
Zero-shot Accuracy 1
Model
Method
Wbits
C4
BoolQ
PIQA
Hella.
WinoG.
ARC-c
ARC-c
Average
Float16
16
27.65
24.60
55.47
62.02
31.33
50.19
39.98
22.86
43.64
PB-LLM
3233.63
1509.33
37.83
50.60
26.67
50.43
27.02
23.63
36.02
OPT-125M
BiLLM
2989.53
1769.26
37.82
50.59
25.75
$1.30
27.65
23.63
36.12
OneBit
39.45
35.58
61.92
60.01
27.01
50.43
35.81
21.84
42.84
BinaryMoS
36.46
33.13
61.83
60.17
27.16
51.38
36.74
22.95
43.37
Float16
16
14.62
14.72
$7.82
72.42
53.70
59.51
50.97
29.52
53.99
PB-LLM
272.83
175.42
62.17
54.24
27.25
50.27
27.98
23.72
40.94
OPT-1.3B
BiLLM
69.45
63.92
61.92
59.52
33.81
49.32
34.38
22.35
43.55
OneBit
20.36
20.76
$7.85
66.53
39.21
54.61
42.80
23.97
47.50
BinaryMoS
18.45
18.83
60.34
68.66
41,99
53,99
44.87
26.19
49.34
Float16
16
5.68
7.08
73.21
77.42
72.99
66.85
52.53
41.38
64.06
PB-LLM
198.37
157.35
60.51
53.53
27.23
49.17
27.48
26.02
40.66
LLaMA-1-7B
BiLLM
41.66
48.15
62.23
58.65
34.64
51.14
33.08
25.68
44.24
OneBit
8.48
10.49
62.50
70.40
54.03
55.32
41.07
30.88
52.36
BinaryMoS
7.97
9.72
64.59
71.82
58.18
58.88
42.09
31.31
54.48
Float16
5.09
6.61
68.47
79.05
76.24
70.17
59.85
44.54
66.39
PB-LLM
35.83
39.79
62.17
58.70
33.97
52.17
31.86
23.63
43.75
L.AMA-1-13B
BiLLM
14.56
16.67
62.53
68.17
52.24
59.43
41.91
29.94
52.37
OneBit
7.65
9.56
63.30
71.98
60.61
59.43
42.85
32.42
55.10
BinaryMoS
7.16
8.81
63.82
73.88
64.05
60.93
44.28
33.11
56.68
Floa166
5,47
6.97
71.07
76.87
72.95
67.16
53.45
40.78
63.71
PB-LLM
76.75
85.92
62.17
52.82
26.87
50.11
26.89
24.31
40.53
LLaMA-2-7B
BiLLM
27.72
36.34
62.14
59.19
35.18
53.11
34.22
26.54
45.06
OneBit
8.60
10.74
63.06
70.40
54.24
56.67
40.82
29.35
52.42
BinaryMoS
7.88
9.75
65.02
71.55
59.41
56.18
41.84
30.03
54.01
Float16
16
4.88
6.47
68.99
79.05
76.62
69.77
57.95
44.20
66.10
PB-LLM
155.25
151.15
37.82
53.26
28.89
49.48
28.28
23.72
36.91
LLaMA-2-13B
BiLLM
20.71
27.19
62.20
62.51
38.05
56.35
40.69
27.73
47.92
OneBit
7.56
9.67
65.66
71.60
60.07
56,91
45.76
31.74
55.29
BinaryMoS
7.08
8.91
66.12
73.72
63.80
58.98
45.71
33.19
57.09
scaling factor for each token, leading to wider representation range. Consequently, can expect that BinaryMoS effectively enhances the capacity of binarized models and improves model accuracy.
4.4 Perplexity and Accuracy Results of Binarized Models
The perplexity and zero-shot accuracy results of previous binarization methods and the proposed BinaryMoS are presented in Table 3 BinaryMoS consistently outperforms earlier binarization techniques across all metrics, effectively narrowing the performance disparity with their Float counterparts.
In particular, smaller LLMs such as OPT-125M and OPT-1.3B typically face challenges in maintaining linguistic capabilities under model compression. Previous methods like PB-LLM and BiLLM result in significant increases in perplexity, often exceeding 1000 for the OPT- model. While OneBit made substantial improvements, perplexity increases remained above 10. BinaryMoS, however, significantly enhances these outcomes by keeping the increase in perplexity below 10. Moreover, boosts the accuracy of binarized models and diminishes the zero-shot accuracy gap to within 0.3% compared to F models. The distinct advantage of BinaryMoS over previous approaches, especially OneBit, lies in its use of scaling experts. This evaluation underlines the efficacy of the BinaryMoS with mixture of scales approach.

---

Table Perplexity and zero-shot accuracy results for 2-bit quantization methods and BinaryMoS.
Perplexity + (Wikitex12)
Method
Whits
OPT-125M
OPT-1.3B
LLaMA-1-7B LLaMA-1-138 LLaMA-2-7B LLaMA-2-133
GPTQ
2
660.52
125.29
45.73
15.20
40.23
32.87
OmniQuant
2
245.47
28.82
9.75
7.84
11.20
8.25
BinaryMoS
36.46
18.45
7.97
7.16
7.88
7.08
Perplexity (C4)
Method
Wbits
OPT-125M
OPT-1.3B
LLaMA-1-7B LLaMA-1-13B LLaMA-2-7B LLaMA-2-13B
GPTQ
213.60
45.43
27.87
15.15
31.37
26.23
OmniQuant
2
390.30
33.81
13.01
10.43
15.46
11.06
BinaryMoS
33.13
18.83
9.72
8.81
9.75
8.91
Average Zero-shot Accuracy T
Method
Wbits
OPT-125M
OPT-1.3B
LLaMA-1-7B LLaMA-1-13B LLaMA-2-7B LLaMA-2-13B
GPTQ
2
37.59
40.36
43.75
49.65
43.31
45.03
OmniQuant
2
36.54
46.43
51.58
56.42
49.54
54.24
BinaryMoS
43.37
49.34
54.48
56.68
54.01
57.09
4.5
Comparison between BinaryMoS and 2-bit Quantization
Since BinaryMoS consistently outperforms other binarization methods, we proceed to compare with conventional 2-bit quantization techniques, GPTQ and OmniQuant. While these two approaches entail lower calibration overhead for quantization due to their use the PTQ approach, they differ in their quantization methods. GPTQ and OmniQuant utilize group- wise quantization approach, employing groups 128 weights to finely quantize parameters and minimize quantization errors. Consequently, the memory demand during inference for these methods more than double that of BinaryMoS. The comparison results, presented in Table reveal that BinaryMoS even outperforms these 2-bit quantization methods, despite its lower memory requirement during inference. This once again underscores the effectivvness of integrating scaling experts.
5 Discussion and Future Work
BinaryMoS significantly improves the accuracy of binarized LLMs by increasing their representa- tional capability with mixture scales. This MoS approach holds promise for extension to multi-bit quantization, multi-bit quantization techniques also involve scaling factors for regulating quantiza- tion step size. However, this paper, our study does not delve into the effectiveness the mixture of scales on multi-bit quanization schemes, leaving this avenue for future exploration.
Though BinaryMoss MoS adopts the concept of MoF, does not fully leverage advanced training techniques These advanced methods optimize routing functions and balance token assignments among experts, thereby enhancing MoE model accuracy. Thus, investigating these training techniques is another topic for future research.
established in the field of MoE [26 27 34
6
Conclusion
This paper introduces BinaryMoS, novel binarization technique designed to enhance the representa- tion capability of binarized LLMs while preserving the fundamental advantage of binarization llow memory usage. BinaryMoS adopts the mixture of scale approach to dynamically adjust the scaling factors of binary weight values in a loken-adaptive manner. Given that scaling factors play a crucial role reducing binarization error occupy a small portion of binarized models, this approach effectively mitigates information loss associated with binarization with minimal memory overhead. Our experimental findings demonstrate that BinaryMoS surpasses existing binarization approaches and even outperforms 2-bit quantization methods in both perplexity and zero-shot tasks.

---

Acknowledgements
This work was supported in part by Institute of Information & communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (No. 2021-0-00105: Development of model compression framework for scalable on-device Al computing on Edge applications IITP- artificial intelligence semiconductor support program to nurture the best talents, No. 2021-0-01343: Artificial Intelligence Graduate School Program (Seoul National University). No. 2021-0-02206:: Artificial Intelligence Innovation Hub), in part by the Samsung Research Funding Center under Project SRFC-TC1603-53, and BK21 FOUR program Seoul National University.
0023-R5-2023-0025600
References
[1] Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang, "Evaluating Quantized Large Language Models", arXiv preprint 0rXiv:2402.18158 2024.
[2] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", International Conference on Learning Representations (ICLR), 2023.
[3] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han, "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", arXiv preprint arXiv:2306. 00978, 2023.
[4] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, Eunhyeok Park, "OWQ: Outlier-Aware Weight Quantization for Efficient Tine-Tuning and Inference of Large Language Models", Pro- ceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024.
[5] Yuzhang Shang, Zhihang Yuan, Qiang Wu, Zhen Dong, "PB-LLM: Partially Binarized Large Language Models", International Conference on Learning Representations (ICLR), 2024.
[6] Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, Xiaojuan Qi, "BiLLM: Pushing the Limit of Post- Dost-Training Quantization for LLMs", International Conference on Machine Learning (ICMI), 2024.
[7] Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, Wanxiang Che, "OneBit: Towards Extremely Low-bit Large Language Models", arXiv preprint 12Xiv:2402.11295, 2024.
[8] Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models". arXiv preprint arXiv:2308. 076333 2023.
[9] Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe, "Binary neural networks: A survey", arXiv preprint arXiv:2004.03333 2020.
[10] Zechun Liu, Wenhan Luo, Baoyuan Wu, Xin Yang, Wei Liu, and Kwang-Ting Cheng, "Bi- Real Net: Binarizing Deep Network Towards Real-Network Performance" Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[11] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi, "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks", Proceedings of the European Conference on Computer Vision (ECCV), 2016.
[12) Zechun Liu, Zhiqiang Shen, Marios Savvides, Kwang-Ting Cheng, "ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions", Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[13] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra, "LLM-QAT: Data-Free Quantiza- tion Aware Training for Large Language Models", arXiv preprint arXiv:2305.17888 2023,
10

---

[14] Wei Zhang, Lu Hou, Yichun Yin. Lifeng Shang, Xiao Chen, Xin Jiang. Qun Liu, "TemaryBERT: Distillation-aware Ultra-low Bit BERT", arXiv preprint htt:::2009.128122.. 2020/
[15] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, Luke Zeutleoyer, "OPT: Open Pre-trained Transformer Language Models", arXiv preprint arXiv:2205.01066 2022.
[16] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample, "LLaMA: Open and Efficient Foundation Language Models", arXiv preprint 2rXiv:2302.133777 2023.
[17] Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaci. Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others, "Llama 2: Open Foundation and Tune-Tuned Chat Models", arXiv preprint arXiv:2307.0922: 2023.
[18) Ilya Loshchilov, Frank Hutter, "Decoupled Weight Decay Regularization", International ference Learning Representations (ICIR), 2019.
innriiittiii Con-
[19] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova, "BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions", arXiv preprint arXiv:1905.10044 2019.
[20] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi, "PIQA: Reasoning about Physical Commonsense in Natural Language", Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
[21] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi, "HellaSwag: Can a Machine Really Finish Your Sentence?", arXiv preprint vrXiv:195.007830 2019.
[22] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi, "WinoGrande: An Adversarial Winograd Schema Challenge at Scale", arXiv preprint 1997 2019.
[23] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. Oyvind Tafjord, "Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge", arXiv preprint arXiv:1803.00 054577 2018
[24] Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher, "Pointer Sentinel Mixture Models", arXiv preprint arXiv: 1609.07600 2016.
[25] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu, "Exploring the Limits of Transfer Learning with a Unified Text-to-Teds Text Transformer", arXiv preprint arXiv: 10683, 2019.
[26] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, Jeff Dean, "Outrageously Large Neural Networks: The Sparsely-Gate Mfxture-of-FEperrs Layer", arXiv preprint arXiv: 171100657 2017.
[27] William Fedus, Barret Zoph, Noam Shazeer, "Switch Transformers: Scaling to Trillion Parame- ter Models with Simple and Efficient Sparsity", arXiv preprint arXiv:2101.0344 2021.
[28] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, Ping Luo, "OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models", International Conference on Learning Representations (ICLR), 2024.
[29] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei, "BitNet: Scaling 1-bit Transformers for Large Language Models", arXiv preprint arXiv:2310. 11.100./1453.. 2023.
[30] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer, "QLoRA: Efficient Fine- tuning of Quantized L.LMs", Advances in Neural Information Processing Systems, (NeurIPS), 2023.
11

---

[31] Tim Detimers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, Dan Alistarh, "SpQR: ASparse-Quamtized Representation for Near-Lossless LLM Weight Compression", arXiv preprini arXivv2306.03078. 2023.
[32] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong. Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer, "'QueezeLLM: Dense-and-Spare Quantization", International Conference on Machine Learning (ICML), 2024.
[33] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, Christopher De Sa, "QulP: 2-Bit Quantization of Large Language Models With Guarantees", Advances in Neural Information Processing Systems, (NeurIPS), 2023.
[34] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, William Fedus, "ST-MoE: Designing Stable and Transferable Sparse Expert Models", arXiv preprint 1rXiv:2202.08900 2022.
[35] Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christopher Meek, Ahmed Hassan Awadallah, Jianfeng Gao, "Few-Shot Learning Eval- uation in Natural Language Understanding", Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)., 2021.
[36] Taesu Kim, Jongho Lee, Daehyun Ahn, Sarang Kim, Jiwoong Choi, Minkyu Kim, Hyungjun Kim, "QUICK: Quantization-aware Interleaving and Conflict-free Kernll for Efficient LLM Inference", arXiv preprint arXiv:2402. 10076, 2024.
12

---

A Appendix
A.1 Ablation Study Datasets
To determine the optimal dataset for training binarized models, we conduct a comparative analysis using various training datasets, as summarized in 55 The results indicate that models trained solely on the WikiText2 dataset, due to its relatively small dataset size, tend to exhibit overfitting tendencies and struggle to generalize other datasets. While these models demonstrate considerable perplexity improvement on the WikiText2 evaluation, their perplexity on the C4 dataset and zero-shot accuracy is notably poor. Conversely, models trained exclusively on the C4 dataset perform well across a wide range of tasks, except for the evaluation on WikiTex12. Following the approach of previous research [3]. we also experiment with a generated dataset synthesized using the LLaMA-1- 7B model. Although this dataset generally performs satisfactorily across various language modeling tasks, its performance lags behind that of the C4 dataset. Therefore, enhance overall model performance, we opt to train the models on a mixed dataset comprising both C4 and WikiTeex22. Moreover, the accessibility of both C4 and WikiText2 as open-source datasets further facilitates their adoption for training purposes.
Table 5: Evaluation of binarized LLaMA-1-7B model trained with various training datasets. We train the model on subset of the dataset with the same training step. f: Generated dataset synthesized by LLaMA-1-7B model. j: Mixed dataset of Wikitext2 and C4.
Training
Perplexity L
Zero-shot Accuracy 1
Dataset
Wiki2
C4
BoolQ
PIQA
Hella. WinoG. ARC-e
ARC-c
Avg
Generated
12.54
13.04
60.51
66.
45.91
54.69
41,41
27.90
49.42
Wiki2
9.65
28.61
57.95
57.67
36.78
54.45
38.46
26.62
45.32
C4
13.76
11.97
60.33
67.79
49.69
53.74
39.26
29.52
50.06
Mixed |
8.92
11.85
60.51
67.46
49.95
55.24
41.16
29.35
50.61
A.2 Latency Measurement
To assess the latency of GEMV operation for our BinaryMoS, we have evaluated the latency of previous binarized models and the BinaryMoS by developing appropriate CUDA kernels for bit matrix multiplication, modifying the CUDA kernel for multi-bit matrix multiplication [36]. Additionally, we further customize the CUDA kernel of BinaryMoS to fuse scaling experts and routing operations on top of the 1 1-bit matrix multiplication CUDA kernel. We measure the latency of linear layers LLaMA-TH and LLaMA-131 with batch size of and results are presented in All experiments are conducted on NVIDIA A6000 GPUs.
Table 6
Previous methods like PB-LLM and BiLLM require extra matrix multiplications, making them very slow. OneBit, which employs the simplest binarization scheme, achieves significant improvement over the original 16 model and shows the minimum latency. Meanwhile, our BinaryMoS introduces additional operations for processing scaling experts, which require far fewer operations compared to matrix multiplication. Consequently, BinaryMoS also shows similar latency results to OneBit. This demonstrates that the multi-scaling factor module in BinaryMoS improves performance terms of perplexity and zero-shot accuracy with minimal overhead to latency.
Table 6: Latency ((see) linear layer in IL and L.L.MA- 1/2-13B.
Model Config
LLaMA-1/2-7B
LLaMA-1/2-13B
Weight Size
4096 4096
4096
11008
11008
4096
5120
5120
5120
13824
13824
5120
Float16
68.2
151.7
143.5
95.6
224.1
213.6
PB-LLM
96.1
177.5
168.3
122.7
243.7
234.7
BiLLM
87.1
96.4
104.2
95.2
124.2
131.0
OneBit
32.7
33.7
34.9
33.4
41.4
42.6
BinaryMoS
34.5
36.9
37.0
35.6
43.4
44.5
13

---

A.3 Experimental Results for LLaMA-1-30B
Table 7 provides further experimental results on the LLaMA-1-300 model. In line with the trends observed in 3 BinaryMoS consistently surpasses other binarization approaches for this 30B model. This assessment highlights the effectiveness of BinaryMoS for large-scale LLMs.
Table 7: Perplexity and zero-shot accuracy results F and binarized LLMs for LLaMA-1-30B
Perplexity | Wiki2
Zero-shot Accuracy
Model
Method
Whits
C4
BoolQ
PIQA
Hella.
WinoG.
ARC-e
ARC-c
Average
Float16
16
4.10
5.98
68.37
80.08
79.22
72.69
58.96
45.39
67.45
PB-LLM
32.24
34.59
62.18
60.22
33.75
53,91
33.50
24.40
44.66
LLaMA-1-30B
BiLLM
10.10
12.28
62.39
71.38
59.49
66.54
44.95
33.19
56.32
BinaryMoS
6.63
8.36
67.76
75.62
67.38
64.09
49.62
35.07
59,92
A.4
Generation Quality
Figure compares the generation quality between BinaryMoS and OneBit on LLaMA-1-131 model. BinaryMoS can generate contextually proper answers, whereas OneBit fails to generate correct answers. These results demonstrate that the BínaryMoS processes each token with token-adaptive scaling factors which contain contextual information and the improvement of BinaryMoS over OneBit is substantial enough to make binarized LLMs more applicable in practice.
Prompt: A cowboy rides
BinaryMoS: A cowboy rides wild, powerful horse around the prairie.
OneBit: A cowboy rides pistol.
Prompt: There are number of ways to reduce air pollution, such
BinaryMoS: There are number of ways reduce pollution, such as using clean- burning fuels like natural gas. Natural gas provides better emissions than coal, or oil.
OneBit: There are number of ways to reduce air pollution, such as cleaning machines more often for longer periods. Cleaning materials and products are less toxic.
Prompt: The capital of the state of New York is
BinaryMoS:
The capital of the state of New York is Albany, situated along the west bank the Hudson.
OneBit:
The capital of the state of New York located in the eastern part of the northern and the central part of the south region of the United States.
Figure 4: Comparison of generation quality the LLaMA-1-13B models with BinaryMoS and OneBit.
A.5 Limitations
While BinaryMoS effectively enhances the linguistic capabilities of binarized LLMss, its application to extremely large models such as LLaMA-2-70RB poses challenging due to the high training cost inherent QAT-based strategies. Hence, to facilitate the use of BinaryMoS in such large-scale models, it may be necessary to adapt into a PTQ-based approach or to integrate it with parameter- efficient training strategies in future work. Additionally, it is important to note that, despite the advancements brought about by by BinaryMoS, the reduction in linguistic performance of binarized LLMs coampred to their 6 counterparts remains substantial. Consequently, to make LLM binarization practical for real-world applications, further advancements in binarization techniques are required.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses limitations in the "Discussion and Future Work" and "I "Limitations" section, including challenges and areas for future exploration.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
The paper does not include theoretical results that require formal proofs. The focus is empirical evaluation of the proposed method.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides comprehensive details on the experimental settings, includ- ing models used, datasets, evaluation metrics, and training processes. It ensures that the steps reproduce the results are clear.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper provide open access to the data and code to faithfully reproduce the main experimental results, described in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wttpcc/pcccccccccccooooooop httai//gggudeee/eeeeuubbiiiii wwweiisssonPoPoiiiioioo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all necessary training details, ensuring that the experimental results can be understood and replicated.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not report error bars or statistical significance for the experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The paper provides sufficient information about the compute resources used
for the experiments, mentioning the use of NVIDIA A100 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwwaiipppppppppp///eeeea
Question:
Answer: [Yes]
Justification: The research conforms to the NeurlPS Code of Ethics, focusing on improving model efficiency and performance a responsible manner without evident ethical concerns.
Guidelines:
The answer NA
that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses societal impacts, highlighting the potential for efficient deployment of L.LMs on edge devices and acknowledging the importance of responsible use.
18

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not pose high risks for misuse that would require specific safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly credits existing assets, such as datasets and models, and follows appropriate licenses and terms use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets.paperssii paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce assets. It focuses on methodological improvements and uses existing datasets and models for evaluation.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve human subjects research, thus IRB approvals are not applicable.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "BeGewWQOjo", "reviewer": "Reviewer_scP1", "summary": "This paper introduces a method to compress large language models (LLMs) by quantizing weight values to 1-bit. The goal is to mitigate performance degradation seen in previous quantization methods applied to LLMs. They utilize ideas from Mixture-of-Experts and introduce token-adaptive scaling factors that help control the effective values of binarized weights. Their approach results in high compression ratios compared to earlier binarization methods, maintaining memory efficiency.", "questions": "See weaknesses", "limitations": "The authors claim to address the limitations of their work in the \"Discussion and Future Work\" section, but this discussion is incomplete. They do not mention latency requirements and potential failure cases of BinaryMoS are also omitted and not listed as limitations. Additionally, they should acknowledge that performance loss is still expectedly incurred compared to FP16.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Quantization helps lower the barriers to deploying large language models in compute-constrained environments. While previous approaches have been able to achieve this, it has come at the expense of linguistic utility.  BinaryMoS in an attempt to lower these barriers without these costs. \n\n- Through experiments on various benchmarks, they demonstrate the effectiveness of BinaryMoS in memory-efficient quantization \n\n- The paper offers a clear explanation of existing research work on binarization and this helps with clarity and readability.", "weaknesses": "- The motivation for extending the ideas from MoE to One-Bit is not entirely clear. Is the paper focused on applying binarization to MoE-style models? Please clarify this aspect. \n\n- The benefits of using 4 experts compared to 2 experts do not seem significant across tasks. Given the additional memory overhead introduced by these experts, wouldn't it be better to advocate for using two experts instead?\n\n- The analysis section on token-adaptive scaling factors does not adequately explain why BinaryMoS is preferable to One-Bit. While the variation in gating scores for experts across tokens is shown, there is no context provided about the sentences being analyzed. It seems possible that BinaryMoS is only useful and sensitive to certain domains and tasks and may not apply to every task.\n\n- The omission of latency measurements compared to other binarization methods raises questions about the robustness and efficiency of BinaryMoS. Since latency measurements are critical for real-world deployments of large language models, a discussion on this topic would be great", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive reviews, and here we address your comments in detail:\n\n**W1. Motivation of BinaryMoS**\n\nThe motivation stems from the fact that previous binarization methods, including OneBit, still have low representational power. To push the limits of binarized weights, we propose token-adaptive scaling factors inspired by MoE. Typically, MoE is used not for model compression techniques like quantization but to enhance the original model's capabilities by duplicating the weights of FFN layers according to the number of experts. Simply combining binarization methods with traditional MoE can enhance the model's capabilities, but the memory overhead is large, conflicting with the goal of extreme compression inherent in binarization. Therefore, we propose a novel binarization method that adopts the concept of using multiple experts from MoE and applies it to the scaling factors of binarization. In this way, while OneBit uses fixed scaling factors regardless of context, the proposed BinaryMoS can generate token-adaptive scaling factors which can improve the representation power of the binarized model.\n\nIn summary, our BinaryMoS scheme is inspired by MoE-style models, but it does not employ multiple experts with separate weights. Instead, it uses multiple scaling experts to generate token-adaptive scaling factors while fixing binarized weights, thereby retaining the memory efficiency of binarized models.\n\n\n**W2. Benefits of using 4-expert**\n\nAs the reviewer pointed out, in terms of zero-shot accuracy, the average accuracy of 2-experts and 4-experts is 50.64% and 50.61%, respectively, showing no significant difference. However, in terms of perplexity, the 4-expert configuration achieves remarkable improvements compared to the 2-expert configuration by lowering perplexity from 12.18 to 11.85 (reported in Table 2 of our paper). Hence, we can expect that 4-experts will generally achieve better language modeling capabilities compared to 2-experts.\nMoreover, due to the low memory cost of adopting scaling experts, as shown in Table A1-1, the model sizes of 2-expert and 4-expert BinaryMoS are similar, while both significantly reduce the model size compared to the original Float16 model. In other words, the cost of adopting the 4-expert configuration is low, while we can expect an improvement in language modeling capabilities. Based on these observations, we chose to employ 4-experts in BinaryMoS.\n\n**Table A1-1. Comparison of memory size for 2-expert and 4-expert BinaryMoS.**\n\n| **Model**     | **Float16** | **4-expert** | **2-expert** |\n|---------------|-------------|:------------:|:------------:|\n| LLaMA-1/2-7B  | 13.51 GB    |    1.40 GB   |    1.38 GB   |\n| LLaMA-1/2-13B | 26.20 GB    |    2.33 GB   |    2.30 GB   |\n\n**W3. Relationship between gating score and context**\n\nWhile Figure 3 shows the variation in gating scores for experts across tokens in an exemplary sentence, we observed a similar tendency throughout our experiments for various tasks reported in Section 4.4. Each token is assigned a different scaling factor with our token-adaptive binarization method. Although our experiments are still limited, we believe this trend could extend to other LLM tasks as well, since the concept of applying multiple scaling experts for weight binarization is general.\n\n\n**W4. Latency measurements compared to other binarization methods**\n\nThanks for pointing out the latency measurements. As you highlighted, the latency is a critical component for real-world deployment. However, previous binarization papers, including PB-LLM, BiLLM, and OneBit, have not reported their latency due to the lack of  a CUDA kernel for matrix multiplication between FP activation and 1-bit weights. Therefore, we first developed a custom CUDA kernel for 1-bit matrix multiplication by modifying the CUDA kernel for multi-bit matrix multiplication [R1-1, R1-2]. Then, we also developed a custom CUDA kernel for BinaryMoS by fusing the operations of scaling experts and routers on top of the 1-bit matrix multiplication CUDA kernel.  \n\nWe measured the latency of the linear layers in LLaMA-7B and LLaMA-13B (batch size: 1) and reported the latency results in **Table G-1 and G-2 of global response.** As illustrated in our paper, PB-LLM and BiLLM require extra matrix multiplications, so they tend to show similar or larger latency results compared to the original FP16 models. OneBit, which employs the simplest binarization scheme, achieves significant improvement over the original FP16 model and shows the minimum latency. Meanwhile, as our BinaryMoS introduces additional operations on processing scaling experts, which require far fewer operations compared to the matrix multiplication, our BinaryMoS also shows similar latency results as OneBit. This demonstrates that the multi-scaling factor module in BinaryMoS improves performance in terms of perplexity and zero-shot accuracy with minimal overhead to latency.\n\n[R1-1] Taesu Kim, et al., “QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference”, arxiv:2402.10076\n\n[R1-2] https://github.com/SqueezeBits/QUICK\n\n**L1. Limitation of BinaryMoS**\n\nThanks for the suggestion. \n\nFirst, we will include the latency results which we reported in the response to W4 and global response in the revised manuscript. \nRegarding other limitations, we will include the following parts in the limitation section of the revised manuscript.\n\n1. Our results are still limited to relatively small models up to 13B. We plan to work on larger models such as 30B.\n\n2. The performance loss from the FP16 model is still substantial, so further innovations regarding binarization are needed to reduce the performance gap.\n\nThank you again for your valuable feedback. If you have any additional questions, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BfxSPS2JjY", "reviewer": "Reviewer_s3b8", "summary": "This paper proposes a binarization technique for LLMs, inspired by the mixture-of-expert (MOE) model. In the proposed approach, multiple scaling factors for the binarized matrices are available, each treated as an expert just like in MOE. The model infers a weight combination of these scaling factors adaptively at each time step, hence the binarization strategy is different for each input token. The paper shows that this adaptive binarization technique does not increase much memory overhead as the traditional MOE does, but can effectively improve model quality, as compared with existing 1-bit or 2-bit quantization baselines.", "questions": "1. The performance of BinaryMoS depends critically on the choice of static experts S_in, S_out. How robust is the proposed approach is to random initialization of these experts? Does the magnitude of these scaling experts matter a lot?\n\n2. Are there any interesting patterns learned for expert weights? For examples, whenever one expert gets assigned larger weights, do the input sequences contain specific n-grams or syntactic structure? It would be great to find some insights into how and why the model learned to make expert choices, for better interpretability.\n\n3. Related to question 2, if there is no clear interpretable expert assignment patterns that can be found, what if you just determine expert weights randomly without training? This should be a baseline to validate the effectiveness of the learning procedure.\n\n4. How important is the knowledge distillation loss? It seems there is no analysis to the relative importance of distillation.\n\n5. How does the quantization performance scale with model size? If there is a clear scaling law of BinaryMoS which appears to be more efficient than full-precision training, then this approach should become a standard for very large model training and inference.", "limitations": "The paper addressed limitations in the \"Discussion and future work\" section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach effectively improved the quality of binarized LLMs, which has always been a challenging task. The introduction of adaptive binarization makes a lot of sense and deserves broader attention and experimentation.", "weaknesses": "More detailed analysis in terms of efficiency, scaling behaviors and ablation studies could have been given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive reviews, and here we address your comments in detail:\n\n**Q1. Importance of scale experts**\n\nAs you correctly pointed out, the choice of static experts $S_{in}$ and $S_{out}$ significantly influences the accuracy of BinaryMoS, so it is important to find proper $S_{in}$ and $S_{out}$. If we randomly set $S_{in}$ and $S_{out}$ as you questioned, BinaryMoS fails to achieve sufficient accuracy results. Therefore, during the training process of BinaryMoS, both $S_{in}$ and $S_{out}$ are also trained. \n\nThe term ‘static’ might cause confusion, but $S_{in}$ and $S_{out}$ are static only during the inference stage, not during the training stage. Moreover, before training, BinaryMoS adopts SVD-based $S_{in}$ and $S_{out}$ initialization methods proposed in OneBit, so that $S_{in}$ and $S_{out}$ are initialized to minimize the binarization error of pre-trained weights, rather than applying random initialization. \n\nWe will clarify that both $S_{in}$ and $S_{out}$ are also trained and their initialization methods in the revised version of the paper.\n\n**Q2. Linguistic pattern of assigning expert**\n\nThe weights of experts vary across input tokens, and we were not able to identify any interesting patterns learned for expert weights. However, we have included example sentences and the corresponding expert weights assigned to each token in Figure 1 of the attached PDF file of author rebuttal. This is provided in case others might find some interesting patterns. If you have any further insights or opinions on the patterns, please let us know.\n\n**Q3. Influence of random expert weights**\n\nThank you for the valuable insight. As you suggested, we measure the accuracy of BinaryMoS by setting expert weights randomly without training as shown in Table A2-1. The evaluation results demonstrate significant accuracy degradation, underscoring the importance of training to fully achieve the advantage of the proposed Mixture-of-Scale scheme. \nWe will update the analysis result in Table A2-1 and the above discussion in the Appendix of the revised paper. \n\n\n**Table A2-1. Accuracy evaluation of BinaryMoS model with random expert weights**\n\n| **Model**   | **PPL (Wiki2)** | **PPL (C4)** | **Avg acc** |\n|-------------|:---------------:|:------------:|:-----------:|\n| LLaMA-1-7B  |     1034.07     |    718.39    |    37.81    |\n| LLaMA-1-13B |      111.07     |    141.95    |    39.84    |\n\n\n**Q4. Importance of knowledge distillation**\n\nKnowledge distillation (KD) has been widely adopted to improve the accuracy of QAT techniques, and it has been also utilized in previous work such as OneBit. To verify the importance of KD loss, we compare the accuracy of both OneBit and BinaryMoS with and without KD in Table A2-2.\nThe binarized models without KD show worse perplexity and accuracy results compared to those with KD. However, regardless of the application of KD, BinaryMoS consistently outperforms OneBit, demonstrating the effectiveness of the proposed Mixture-of-Scale approach.\nWe will update the analysis result in Table D and the above discussion in the Appendix of the revised paper. \n\n**Table A2-2. Perplexity and averaged zero-shot accuracy results of OPT-1.3B model**\n\n| **Method** | **w/ KD** | **PPL (C4)** | **zero-shot acc. (avg.)** |\n|------------|-----------|-----------------|---------------------------|\n| OneBit     |    True   |      20.76      |           47.50           |\n| BinaryMoS  |    True   |      18.83      |           49.34           |\n| OneBit     |   False   |      22.95      |           45.46           |\n| BinaryMoS  |   False   |      20.19      |           46.05           |\n\n\n**Q5. Scaling law of BinaryMoS**\n\nAs you pointed out, if there is a clear scaling law for BinaryMoS, it could become a standard for compressing very large models. However, since BinaryMoS adopts QAT-based strategies, which require training the full weight parameters to adapt the model to quantization, it is very challenging to evaluate the effect of BinaryMoS on very large models, such as LLaMA 30B and 70B, due to high training cost. Hence, there is a difficulty in determining the scaling law of BinaryMoS at this point. Please note that this limitation is not specific to BinaryMoS but is a general limitation of QAT-based strategies, including previous works like OneBit.\nTo scale BinaryMoS to very large models, we need to find a way to integrate the BinaryMoS approach with parameter-efficient fine-tuning techniques (PEFT) such as LoRA [R2-1]. This integration would make the training procedure feasible for very large models. Therefore, we believe that this is an important future direction for BinaryMoS to extend its usability and effectiveness to much larger models.\n\n[R2-1] Yelysei Bondarenko, et al. “Low-Rank Quantization-Aware Training for LLMs”, arxiv:2406.06385.\n\nThank you again for your valuable feedback. If you have any additional questions, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MM9SV0PCUy", "reviewer": "Reviewer_fpKa", "summary": "This paper proposes using a mixture of scales for binarizing continuous latent weights. It includes a router implemented by a linear layer, outputting a softmax score which is then used as the combining matrix of the scale basis (called scale experts). The method uses two score basses, one for input $S_{in}$ and one for output $S_{out}$, each includes a number of scale vectors. The author conducted an experimental analysis of the number of experts to be used, and evaluated the proposed method on different language models with results showing performance gains by mixture of scales.", "questions": "Related to (4) where the utilized scales are obtained by combining the scale basis using $G$, scale experts $S_{in}$ and $S_{out}$ are of primary importance in the method. How are these scale experts are constructed and/or learnt?\n\nIf the scale experts are learnt, what can be the benefits of learning $S_{in, out}$ then compute $\\hat{S}_{in, out}$ instead of learning these scales directly?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed idea is pretty nice, simple, and shows performance gains.", "weaknesses": "Weak significance: Although the idea is nice and valid, its scientific and technological significance as well as its originality is low. The paper could be more suitable for a workshop than a major venue like NeurIPS. The experimental analysis of the number of scaling experts shows that the performance is not always improved with the number of experts. This also strengthens the above opinion of low significance. \n\nSeveral minor typos to be corrected, such as to use a unique term for MoS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviews, and hope our rebuttal could convince you and change your stance on our paper.\n\n**W1. Weak significance and low originality**\n\nIn this work, we propose a new binary LLM architecture to increase the representational power of binary models with negligible inference overhead by introducing the concept of Mixture-of-Scale, which can adaptively calculate proper scaling factors based on the context. We would like to highlight that good model architecture design is significant in the field of deep learning, as the model architecture determines the fundamental capability of the models. For this reason, many papers proposing modifications to model architecture are published in major venues. For example, Bi-Real Net, which applied a residual path to every convolution layer to enhance the accuracy of 1-bit convolutional neural networks (CNNs), has been published in ECCV [R3-1].\n\n[R3-1] Liu, Zechun, et al. \"Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\n**W2. Performance of the number of scale experts**\n\nWe agree that performance does not always improve as the number of experts increases. Meanwhile, we want to emphasize that the key point of the proposed BinaryMoS is the introduction of multiple scaling experts (e.g., 2, 4, 8) to generate token-adaptive scaling factors, not the specific number of experts. In this way, BinaryMoS can improve the accuracy of binary models compared to OneBit, which uses fixed scaling factors. As shown in Tables 3 and 4 of our paper, BinaryMoS significantly enhances the accuracy of binary models. This highlights the effectiveness of token-adaptive scaling factor generation in BinaryMoS for improving the accuracy of binary models.\n\n**Q1. Effectiveness of training scale experts**\n\nAs you correctly pointed out, the choice of scale experts $S_{in}$ and $S_{out}$ significantly influences the accuracy of BinaryMoS, so it is important to find proper $S_{in}$ and $S_{out}$. If we randomly set $S_{in}$ and $S_{out}$t as you questioned, BinaryMoS fails to achieve sufficient accuracy results. Therefore, during the training process of BinaryMoS, both $S_{in}$ and $S_{out}$ are also trained. Before training, BinaryMoS adopts SVD-based $S_{in}$ and $S_{out}$ initialization methods proposed in OneBit, so that $S_{in}$ and $S_{out}$ are initialized to minimize the binarization error of pre-trained weights, rather than applying random initialization. We will clarify that both $S_{in}$ and $S_{out}$ are also trained and their initialization methods in the revised version of the paper.\n\nIf we directly learn $\\hat{S_{in}}$ and $\\hat{S_{out}}$, it works exactly the same as OneBit. During inference, the scaling factors trained with this approach are fixed regardless of input context. Meanwhile, in the proposed BinaryMoS, on top of training $S_{in}$ and $S_{out}$, the router weight $W_R$ is also trained to generate gating score $G$ based on the input context $X$ so the token-adaptive scaling factors can be generated as described in Eq. (3) and (4) of our paper.\n\nThank you again for your valuable feedback. If you have any additional questions, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1sLrTdyoZN", "reviewer": "Reviewer_iCrR", "summary": "This paper studies the problem of post-training binarization of large language models. Built upon OneBit, this work propose BinaryMoS, to use a mixture of scaling weights for the linear layer of binarized LLMs. Instead of static scales in OneBit, BinaryMoS employs a set of scaling weight experts and adaptively combines them given the input representation of current tokens. As a result, BinaryMoS offers stronger representation capabilities while incurring minimal computation and memory overhead. Experimental results on various LLM benchmarks show the proposed technique compares favorably to other binarization methods for LLMs.", "questions": "1. While post-training binarization performs poorly compared to the original base model, methods like BitNet seem to bridging the gap by training low-bit models from scratch. Can your method extend to the scenario where the model is trained from scratch?", "limitations": "The authors have adequately addressed the limitations of the work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. BinaryMoS has minimal computational and memory overhead, and keeps the architecture simple without introducing sparse weight matrices.\n2. Mixture-of-expert technique could potentially applied to other quantization method.\n3. The method works for a wide range of LLMs.\n3. The proposed method out-performs other binarization techniques on several LLM benchmarks.", "weaknesses": "1. The paper lacks discussion of the cost of post-training adaptations. The training cost seems comparably high as it requires three epochs over the selected dataset.\n2. The improvement over OneBit is small, and the improved binarization still perform significantly lower than other less aggressively quantized method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive reviews, and here we address your comments in detail:\n\n**W1. High training cost**\n\nThough training three epochs over the selected dataset may seem costly, the datasets used for fine-tuning LLMs for quantization, particularly C4 and Wikitext2, are generally very small, making the training cost affordable. Our selected dataset consists of a total of 195M tokens, with 192M tokens from the C4 dataset and 3M tokens from the Wikitext2 dataset.\n\nWe will highlight the exact size of the selected dataset to clarify the training cost in the revised paper.\n\n**W2. Impact of BinaryMoS**\n\nBecause the accuracy improvement of BinaryMoS over OneBit is not as dramatic as the improvement of OneBit over PB-LLM and BiLLM, there may be concerns that the proposed solution's improvement is marginal. However, we would like to highlight that the improvement of BinaryMoS over OneBit is substantial enough to make binary LLMs more applicable in practice. For example, both PB-LLM and BiLLM fail to generate grammatically correct sentences, while both OneBit and BinaryMoS can generate grammatically correct sentences. However, as presented in Table A4-1, BinaryMoS can generate contextually proper answers, whereas OneBit fails to generate correct answers. This is because BinaryMoS processes each token with token-adaptive scaling factors which contain contextual information.\n\nWe will also update the above discussion and Table A4-1 in the Appendix of the revised paper.\n\n**Table A4-1. Comparison of generation quality on the LLaMA-1-13B models with BinaryMoS and OneBit.**\n| **Prompt: A cowboy rides a _**                                                                                                                                                |\n|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| **BinaryMoS**: A cowboy rides a **wild, powerful horse** around the prairie.                                                                                                        |\n| **OneBit**: A cowboy rides a pistol.                                                                                                                                            |\n| &nbsp;                                                                                                                                                                             |\n| **Prompt: There are a number of ways to reduce air pollution, such as _**                                                                                                      |\n| **BinaryMoS**: There are a number of ways to reduce air pollution, such as **using clean-burning fuels like natural gas.** Natural gas provides better emissions than coal, or oil. |\n| **OneBit**: There are a number of ways to reduce air pollution, such as cleaning machines more often for longer periods. Cleaning materials and products are less toxic.        |\n| &nbsp;                                                                                                                                                                             |\n| **Prompt: The capital of the state of New York is _**                                                                                                                          |\n| **BinaryMoS**: The capital of the state of New York is **Albany**, situated along the west bank of the Hudson.                                                                      | &nbsp;\n| **OneBit**: The capital of the state of New York is located in the eastern part of the northern and the central part of the south region of the United States.                  |\n\n\n**Q1. Training from scratch.**\n\nThank you for your question and sharing your insight. The main novelty of the proposed BinaryMoS is its ability to increase the representational power of binary models by introducing the concept of Mixture-of-Scale, which can adaptively calculate proper scaling factors based on the context. This BinaryMoS architecture can also be trained from scratch. As you highlighted, post-training binarization inherently performs poorly compared to training-from-scratch approaches like BitNet, as post-training binarization introduces much less training iterations. Hence, if we have enough training facilities, we can further improve accuracy of binary models by adopting training-from-scratch approaches for BinaryMoS.\n\nThank you again for your valuable feedback. If you have any additional questions, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

Thank you all for your valuable feedback on our work. 

We appreciate all the insightful questions and comments provided, and we have responded to each reviewer's comments as thoroughly as possible. Moreover, an additional one page PDF file is attached to provide supplementary figures.
We hope our responses will help resolve any concerns or curiosity.

Many reviewers have raised questions about the latency of BinaryMoS, so we have evaluated the latency of previous binary models and the proposed BinaryMoS by developing appropriate CUDA kernels for binary models. We measured the latency of the linear layers in LLaMA-7B and LLaMA-13B (batch size: 1) and reported the results in Tables G-1 and G-2. As illustrated in our paper, PB-LLM and BiLLM require extra matrix multiplications, making them very slow. OneBit, which employs the simplest binarization scheme, achieves significant improvement over the original FP16 model and shows the minimum latency. Meanwhile, our BinaryMoS introduces additional operations for processing scaling experts, which require far fewer operations compared to matrix multiplication. Consequently, BinaryMoS also shows similar latency results to OneBit. This demonstrates that the multi-scaling factor module in BinaryMoS improves performance in terms of perplexity and zero-shot accuracy with minimal overhead to latency.


**Table G-1. Latency (msec) of Linear Layer in LLaMA-1/2-7B.**
| **Model config** |                        |     **LLaMA-1/2-7B**    |                         |
|------------------|:----------------------:|:-----------------------:|:-----------------------:|
| **Weight Size**   | **4096 $\times$ 4096** | **4096 $\times$ 11008** | **11008 $\times$ 4096** |
| Float16          |         0.06815        |         0.15172         |         0.14346         |
| PB-LLM           |         0.09607        |         0.17751         |         0.16833         |
| BiLLM            |         0.08711        |         0.09638         |         0.10420         |
| OneBit           |         0.03266        |         0.03370         |         0.03494         |
| BinaryMoS        |         0.03449        |         0.03690         |         0.03695         |


**Table G-2. Latency (msec) of Linear Layer in LLaMA-1/2-13B.**
| **Model config** |                        |    **LLaMA-1/2-13B**   |                         |
|------------------|:----------------------:|:-----------------------:|:-----------------------:|
| **Weight Size**   | **5120 $\times$ 5120** | **5120 $\times$ 13824** | **13824 $\times$ 5120** |
| Float16          |         0.09558        |         0.22408         |         0.21355         |
| PB-LLM           |         0.12273        |         0.24367         |         0.23466         |
| BiLLM            |         0.09523        |         0.12421         |         0.13095         |
| OneBit           |         0.03338        |         0.04144         |         0.04258         |
| BinaryMoS        |         0.03561        |         0.04339         |         0.04445         |


Thanks again for your valuable comments. If you have any further questions or comments, please let us know.

Meta-review: Paper proposes BinaryMoS, a token-adaptive binarization method for LLMs that leverages the concept of MoE for scaling factors. This allows for contextually adjusting binary weight values, thereby enhancing the representation power compared to traditional static binarization techniques.

While one reviewer maintains a lower score (no engagement), their concerns are primarily focused on the paper's significance and presentation, which can be further addressed through the suggested improvements (eg., scaling to larger models, subject to resource availablity ofc, to understand its scalability; investigating the reln between learned scaling factors, input tokens and their impact on performance). The remaining reviewers, after the rebuttal, view the paper favorably, recognizing its technical contribution and potential impact.

I believe this paper makes a valuable contribution to the field of LLM compression by introducing a novel and effective binarization method. The proposed method achieves high enough accuracy (could be considered as SOTA) for binary LLMs while maintaining memory efficiency and low latency. And the paper is well-written and the results are convincing.
Decision: Accept (poster)