Title: Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Models

Binarization

Quantization

Model Compression

Efficient LLM
Abstract: Binarization, which converts weight parameters to binary values, has emerged as an effective strategy to reduce the size of large language models (LLMs). However, typical binarization techniques significantly diminish linguistic effectiveness of LLMs.
To address this issue, we introduce a novel binarization technique called Mixture of Scales (BinaryMoS). Unlike conventional methods, BinaryMoS employs multiple scaling experts for binary weights, dynamically merging these experts for each token to adaptively generate scaling factors. This token-adaptive approach boosts the representational power of binarized LLMs by enabling contextual adjustments to the values of binary weights. Moreover, because this adaptive process only involves the scaling factors rather than the entire weight matrix, BinaryMoS maintains compression efficiency similar to traditional static binarization methods. Our experimental results reveal that BinaryMoS surpasses conventional binarization techniques in various natural language processing tasks and even outperforms 2-bit quantization methods, all while maintaining similar model size to static binarization techniques.

Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models

Abstract

Binarization, which converts weight parameters to binary values, has emerged as an effective strategy to reduce the size of large language models (LLMs). However, typical binarization techniques significantly diminish linguistic effectiveness of LLMs. To address this issue, we introduce a novel binarization technique called Mixture of Scales (BinaryMoS). Unlike conventional methods, BinaryMoS employs multiple scaling experts for binary weights, dynamically merging these experts for each token to adaptively generate scaling factors. This token-adaptive approach boosts the representational power of binarized LLMs by enabling contextual adjustments to the values of binary weights. Moreover, because this adaptive process only involves the scaling factors rather than the entire weight matrix, BinaryMoS maintains compression efficiency similar to traditional static binarization methods. Our experimental results reveal that BinaryMoS surpasses conventional binarization techniques in various natural language processing tasks and even outperforms 2-bit quantization methods, all while maintaining similar model size to static binarization techniques.

1 Introduction

Though large language models (LLMs) have delivered impressive results in a variety of natural language processing (NLP) tasks, their massive size often complicates deployment. One common method to compress LLMs is through the quantization of weight parameters, which reduces model sizes by lowering the precision of weight values [1, 8, 2, 30, 31, 32, 33, 3, 4]. Existing quantization approaches such as GPTQ [2], AWQ [3], and OWQ [4] have successfully managed to reduce model sizes by converting 16-bit floating point weights to 4-bit representations, achieving a fourfold decrease in size. Binarization pushes this concept even further by reducing weight values to 1-bit, resulting in a 16-fold size reduction.

However, such aggressive compression through binarization drastically limits the representational capacity of weights, leading to a significant degradation in the linguistic capabilities of LLMs. To address this limitation and improve the accuracy of binarized LLMs, recent research has actively explored binarization techniques tailored for LLMs [29, 5, 6, 7]. Nonetheless, previous efforts often compromise the inherent advantages of binarization by introducing high memory overhead, and they continue to struggle to achieve sufficient accuracy with binarized LLMs.

In this paper, we propose a novel binarization technique named as Mixture of Scales (BinaryMoS). Typical binarization methods use scaling factors to control the effective values of binarized weights. Although these scaling factors occupy a tiny fraction of the overall model size, they are crucial

BinaryWeight

High-Precision(e.g.INT8)Weight

Non-Salient

Non-Salient

Scale

Scale

Input

indno

+

Input

indino

Scale

Scale

Salient

Salient

(a) PB-LLM

(b) BiLLM

Router

Scale

Scale

Input

Input

Z

Scale

Scale

Output

(c) OneBit

(d)BinaryMos

in reducing binarization error. BinaryMoS advances the functionality of these scaling factors by incorporating token-adaptive scaling factors. Inspired by the Mixture of Experts (MoE) approach [26, 27, 34], which empolys multiple expert layers to enhance the model capacity, BinaryMoS adopts multiple scaling factors as experts to improve the representational capacity of binarized LLMs in a memory-efficient way. During inference, BinaryMoS linearly combines these scaling experts based on the context to generate token-adaptive scaling factors, thus dynamically adjusting the represented values of binarized weights to maximize the expressive power of the model. As a result, BinaryMoS can improve the linguistic performance of binarized LLMs with minimal memory overhead.

2 Background

2.1 Binarization of LLMs

Binarization stands out as an extreme yet effective method for reducing model sizes in deep learning. This method achieves size reduction by transforming high-precision weight parameters into 1-bit values. The binarization process is typically governed by the following equation:

Here, W FP ∈ R n × m is the full-precision weight matrix of a linear layer where n and m represent the size of output and input dimension, respectively, and W B ∈ R n × m denotes its binarized version. α ∈ R n represents scaling factors that are responsible for adjusting the binary weight values along the output dimension. In general, the scaling factors are analytically derived as the absolute mean of FP weight values to minimize the L 2 error between full-precision and binarized weights, and these scaling factors play a vital role in bridging the gap between the original full-precision weights and their binarized counterparts.

While binarization has been effectively applied in traditional deep learning models like Convolutional Neural Networks (CNNs) for image classification without losing accuracy [9, 11, 10, 12], LLMs tend to be more sensitive to such extreme quantization, often experiencing significant accuracy degradation with standard binarization techniques. Therefore, various binarziation techniques tailored for LLMs

have been developed, as shown in Figure 1. PB-LLM [5] partially binarizes weight parameters while maintaining salient weight parameters as high-precision values (e.g., Float16 or INT8). However, this method results in considerable memory overhead. For instance, quantizing 10% of weight parameters as INT8 while binarizing the remaining 90% results in an average bit-width of 1.7 bits for the weight parameters, which is closer to 2 bits than 1 bit.

Furthermore, despite this partial binarization strategy of PB-LLM, the significant information loss inherent in binarization still causes considerable accuracy degradation. To reduce the binarization error and enhance accuracy, BiLLM [6] adopts a more refined approach to assigning scaling factors. Assuming that weight parameters follow a bell-shaped distribution, BiLLM categorizes weight parameters based on their proximity to the mean value: concentrated weights, close to the mean, and sparse weights, distant from the mean. Distinct scaling factors are then assigned to each group to minimize binarization errors. Then, to reduce the memory overhead associated with maintaining information of salient weights, BiLLM preserves this information by binarizing the difference between the binarized values and their full-precision counterparts. Consequently, each salient weight is represented by two 1-bit values, effectively amounting to a 2-bit representation. Despite significantly reducing binarization error, BiLLM complicates the structure of binarized LLMs, adding complexity to the inference process. This complexity arises from the need to manage additional sparse and salient weights alongside regular concentrated weights, requiring extra matrix multiplication during inference.

Meanwhile, unlike conventional binarization methods that typically employ scaling factors only for the output dimension of weights, OneBit [7] enhances the binarization process by incorporating scaling factors for both the input and output dimensions. This dual-dimension scaling approach addresses binarization errors across both dimensions, potentially enhancing model accuracy. Additionally, the size of each scaling vector is substantially smaller compared to the weight matrix, making this approach memory efficient. For instance, in linear layers with a hidden dimension of h , the weight matrix size is h × h , while each scaling vector is only h × 1 . Therefore, doubling these scaling factors adds a negligible memory overhead to the network. Moreover, as this approach of dual-dimensional scaling efficiently preserves enough information to significantly reduce binarization errors, OneBit eliminates the need to store separate information for salient weights, thereby simplifying the model structure. The result of matrix multiplication Y of a linear layer using the OneBit approach can be defined as follows:

Here, X ∈ R k × m is the matrix of input activation where k represents batch size, while S in ∈ R 1 × m and S out ∈ R 1 × n denote the scaling factors for input and output dimensions, respectively. As outlined in Equation 2, processing scaling factors for both input and output dimension can be simplified to scaling input and output of the linear layer before and after matrix multiplication, respectively.

Despite advances in binarization techniques for LLMs, a notable accuracy gap still exists between fullprecision models and their binarized counterparts. Therefore, bridging this gap without sacrificing the fundamental benefits of binarization, particularly low memory usage, remains an important challenge in the field of LLM compression.

2.2 Mixture of Experts

The MoE approach is a widely adopted strategy to boost the capabilites of deep learning models by integrating multiple specialized experts into a single framework [26, 27, 34]. Typically, the MoE approach for LLMs involves duplication of layers and selecting the appropriate layers among these duplicates for a specific task during inference. In the MoE setup, the router is a key to selecting the appropriate expert. It generally consists of a linear layer followed by a softmax function, which calculates and assigns scores to each expert. During the inference, only the experts with the highest score are selected and processed.

While integrating the MoE approach with binarized LLMs offers potential for improving model accuracy, it presents a substantial memory trade-off. The duplication of layers inherent in MoE increases the model size proportionally with the number of experts, thus diminishing the memory efficiency benefits gained from binarization. To address these challenges, we propose BinaryMoS, a novel binarization technique that aims to enhance model capacity while maintaining memory efficiency. This approach leverages scaling factors as experts, improving accuracy of binarized

Multi-HeadSelf-Attention

93

Input

9

92

94

GatingScore(G)

Linear Layer

Linear Layer

Linear Layer

WB

Router

(Qprojection)

(Kprojection)

(Vprojection)

+1

-1

L+

SinScaleExpert

K

+1

-1

-1

+1

(N = 4)

AttentionBlock

-1

-1

+1

-1

Sout

+1

+1

+1

-1

LinearLayer

+1

+1

+1

-1

-1

(Out projection)

out

Feed-ForwardNetwork

MatrixMultiplication

LinearWeightedSum

Input

Linear Layer

(FC1)

Activation

LinearLayer

(FC2)

LLMs without the extensive memory overhead associated with traditional MoE configurations. In the following section, we will delve deeper into how BinaryMoS operates and its benefits over conventional techniques.

3 Proposed BinaryMoS

3.1 Binarization with Mixture of Scale

An overview of the proposed BinaryMoS is presented in Figure 2. Unlike previous binarization techniques that utilize a single scaling vector per input or output dimension, BinaryMoS integrates the concept of experts from the MoE framework into the scaling factors and utilizes multiple scaling experts for each dimension. As discussed in Section 2.1, although the size of scaling factors is relatively small, they play a crucial role in preserving the accuracy of binarized models. Therefore, introducing multiple scaling experts incurs minimal memory overhead while effectively leveraging the advantages of the MoE strategy to enhance the capabilities of binarized models.

In the MoE framework, the number of experts selected corresponds directly to the number of layers processed. As a result, the typical MoE framework selects only one or two experts per inference stage to manage the increased processing burden associated with more experts being selected. On the other hand, the scaling factors of binarized LLMs are solely involved in linear operations with matrix multiplication, as detailed in Equation 2. This linearity allows for the efficient management of multiple scaling experts by linearly combining them before executing the matrix multiplication. Hence, instead of selecting only a few experts, as done in the conventional MoE framework, BinaryMoS dynamically generates instructions on how to combine these scaling experts based on the context. This approach overcomes the limitations of fixed expert choices in typical MoE setups by enabling the creation of effectively infinite token-adaptive scaling factors through linear combinations. Consequently, by optimally utilizing the representational power of multiple scaling experts, BinaryMoS maximizes the potential of binarized models while maintaining memory efficiency.

3.2 Router Design

In order to generate the token-adaptive scaling factors, the proposed BinaryMoS designs the router for processing the following operations:

Here, W R ∈ R m × e represents the weight parameters of router's linear layer, where e denotes the number of experts. S in ∈ R e × m and S out ∈ R e × n denote the scaling experts for input and output dimension, respectively. Initially, the router computes the gating score G , which represents

the significance of each scaling expert, using input activations and router weights, as outlined in Equation 3. Notably, as the gating scores are generated with the softmax function, the sum of gating scores for the scaling experts equals 1. These scores are used to linearly combine the scaling experts, resulting in the creation of token-adaptive scaling factors ˆ S in and ˆ S out , as shown in Equation 4. Then, by replacing the static scaling factors S in and S out from Equation 2 with token-adaptive scaling factors, the result of matrix multiplication ˆ Y in a linear layer using the BinaryMoS approach can be revised as follows:

We empirically find that using four scaling experts each for the input and output dimensions provides the optimal compromise between increasing model size and improving accuracy. Consequently, the proposed BinaryMoS utilizes four scaling experts for each dimension to enhance accuracy while maintaining efficiency.

3.3 Impact of BinaryMos on LLM Compression

The proposed BinaryMoS introduces additional memory overhead due to multiple scaling experts and the weights of the router. However, this overhead is relatively minor. For instance, in the LLaMA-1/2-7B model [16] with a hidden dimension h of 4096, the weight matrix for the linear layers is typically 4096 × 4096. If BinaryMoS adopts 4 scaling experts, this translates to four α 's, each of dimension 4096 × 1, for both input and output dimensions. Additionally, the weights of the router would be 4096 × 4. Compared to the previous OneBit method, which requires a single α for both input and output dimensions, the additional components in BinaryMoS total 4096 × 10 parameters. The number of these extra parameters constitutes only 0.2% of the original weight parameters.

For a comprehensive examination of the impact of various binarization techniques, including BinaryMoS, on LLM compression, we evaluate the memory requirements of LLaMA models with Float16 parameters and after applying different binarization methods, as detailed in Table 1. Following standard practice, all binarization techniques exclude the embedding layer and lm-head from binarization. Our analysis reveals that BinaryMoS significantly reduces the memory footprint of models, achieving compression ratios ranging from 9.65 × to 11.24 × . As model size increases, the relative impact of additional parameters diminishes and the proportion of the unbinarized part decreases. Hence, we can achieve higher compression ratios for larger models. For instance, the original LLaMA-1/2-13B model, requiring 26.20 GB for deployment, is impractical for edge devices due to its size. However, BinaryMoS reduces this model to just 2.33 GB, representing an 11.24-fold decrease in memory requirements. This significant reduction facilitates deployment on edge devices with typically limited memory capacities of 4 GB.

In contrast, PB-LLM and BiLLM methods achieve relatively lower compression ratios of around 5 × and 6 × , respectively. This is primarily due to two reasons: first, PB-LLM and BiLLM methods must retain salient weight information, increasing the average bitwidth of weight parameters. Second, the handling of sparse weight matrices in these methods introduces overhead in indexing sparse weight matrices, limiting the achievable compression ratio. OneBit achieves the highest compression ratio by only introducing dual-dimension scaling factors. Remarkably, BinaryMoS achieves a comparable compression ratio to OneBit, despite incorporating additional components for scaling experts. While the memory requirement of binarized models with BinaryMoS increases by only 2% compared to OneBit, the inclusion of scaling experts offers much greater potential to significantly improve perplexity.

This analysis demonstrates that although BinaryMoS introduces additional parameters, the relative increase in memory requirement is modest. This makes BinaryMoS a viable option for enhancing accurcy of binarized models without imposing a significant memory burden.

3.4 Quantization-Aware Knowledge Distillation

Following training strategies adopted for network compression [13, 14], we adopt the knowledge distillation (KD) to transfer the knowledge of a full-precision teacher model to a binarized student model. We employ the cross entropy (CE) loss to distill the logit knowledge. This is calculated using the following equation:

Here, S and T represent the student and teacher models respectively. n denotes batch size, and c is the number of classes. Additionally, to minimize the distributional discrepancies in layer outputs, we incorporate a mean-squared error (MSE) based layer-to-layer (L2L) loss as follows:

In this loss, H T l and H S l are the output logits from the l -th layer of the teacher and student models, respectively. The total loss function, integrating both CE and L2L distillation losses, is defined as:

where α is a hyperparameter that balances the contributions of the CE and L2L losses. For the training of BinaryMoS, we empirically set α = 10 .

4 Experiments

4.1 Experimental Settings

Models and Evaluation Datasets. In our study, we evaluate BinaryMoS on various models, including those from the LLaMA-1 [16], LLaMA-2 [17], and OPT [15] families. Specifically, we utilize the OPT models with 125M and 1.3B parameters, and the LLaMA-1 and LLaMA-2 models with 7B and 13B parameters for our evaluations. We measure language modeling capabilities of these models by evaluating their perplexity on the WikiText2 [24] and C4 [25] datasets. Additionally, we assess zero-shot accuracy on various Common Sense Reasoning Tasks such as BoolQ [19], PIQA [20], HellaSwag [21], WinoGrande [22], ARC-e, ARC-c [23]), utilizing the open-source LLM evaluation framework, LM-Evaluation-Harness [35].

Training Details. We initialize the parameters of binarized models using those from pre-trained models, which serve as teacher models for KD. For the training dataset, a mixed dataset composed of the WikiText2 training dataset and a selected partition from the C4 training dataset, with a sequence length of 2048. The training is conducted over three epochs using the AdamW [18] optimizer, with hyperparameters set to β 1 = 0 . 9 , β 2 = 0 . 999 , and zero weight decay. We implement a cosine decay learning rate scheduler, preceded by a warm-up phase constituting 0.03 of the total training duration. All training sessions are conducted on NVIDIA A100 GPUs.

Baselines. We compare BinaryMoS against previous LLM binarization methods, including PBLLM [5], BiLLM [6], and OneBit [7], ensuring that all implementations adhere to the details provided in their respective papers. PB-LLM and BiLLM utilize the Post-Training Quantization (PTQ) approach for model calibration through the Optimal Brain Quantizer (OBQ) based method of GPTQ [2]. For PB-LLM, which allows variable ratios of salient weights to enhance accuracy, we have set the ratio of salient weights to 10% to ensure the average bit width of weight parameters remains below 2 bits. OneBit employs a Quantization-Aware Training (QAT) approach, and for fairness, its training setup is aligned with that of BinaryMoS. Given the significant accuracy improvements demonstrated by BinaryMoS over traditional binarization techniques, we also include a comparison with 2-bit quantization methods with PTQ approach, such as GPTQ [2] and OmniQuant [28], to broaden the evaluation scope.

4.2 Analysis on the Number of Scaling Experts

To determine the optimal number of scaling experts for BinaryMoS, which effectively maintains the accuracy of binarized LLMs while minimizing memory usage, we conduct evaluations with

(a)

GatingScore

0.0

0.2

0.4

0.6

0.8

1.0

Scale0

Scale1

Scale3

Token0

Sequence

→TokenN

(b)

ScalingFactor(#ofExpert=1)

ScalingFactorforOutputDimension

Token-Adaptive Scaling Factor(# of Expert=4)

0.136

0.130

Values

0.124

0.118

Factor

0

20

40

60

80

100

ScalingF

ScalingFactorforInputDimension

0.152

0.144

0.136

0.128

0

20

40

60

80

100

Index of Input(Bottom)and Output(Top)Dimension

LLaMA-1-7B using varying numbers of scaling experts. This evaluation is conducted using only one-third of the training data for quick assessment. As shown in Table 2, performance metrics, including perplexity and accuracy, generally improve as the number of experts increases from 1 to 4. However, a further increase to 8 experts leads to a decline in model performance. This decline arises from the challenge of training routers to appropriately assign scales to tokens as the number of scales increases. Based on these observations, we choose to employ 4 experts in the BinaryMoS approach.

4.3 Analysis on the Token-Adaptive Scaling Factors

In this section, we explore the effectiveness of the proposed mixture of scale approach in generating token-adaptive scaling factors. To accomplish this, we analyze the gating scores for scaling experts and the scaling factors derived from these scores. For this analysis, we utilize the LLaMA-1-7B model and input sequences sampled from the C4 dataset.

Figure 3 showcases the gating scores and resulting token-adaptive scaling factors for out projection of the 18th layer across tokens of the input sequence. The experimental results reveal substantial variation in the gating scores for each expert across tokens. As depicted in Figure 3(b), while conventional binarization methods with static scaling factors, akin to having a single expert, offer a fixed scaling factor, the scaling experts of BinaryMoS successfully generate a diverse range of scaling factors. This highlights the efficacy of the mixture of scale approach, which adaptively determines the

scaling factor for each token, leading to a wider representation range. Consequently, we can expect that BinaryMoS effectively enhances the capacity of binarized models and improves model accuracy.

4.4 Perplexity and Accuracy Results of Binarized Models

The perplexity and zero-shot accuracy results of previous binarization methods and the proposed BinaryMoS are presented in Table 3. BinaryMoS consistently outperforms earlier binarization techniques across all metrics, effectively narrowing the performance disparity with their Float16 counterparts.

In particular, smaller LLMs such as OPT-125M and OPT-1.3B typically face challenges in maintaining linguistic capabilities under model compression. Previous methods like PB-LLM and BiLLM result in significant increases in perplexity, often exceeding 1000 for the OPT-125M model. While OneBit made substantial improvements, perplexity increases remained above 10. BinaryMoS, however, significantly enhances these outcomes by keeping the increase in perplexity below 10. Moreover, it boosts the accuracy of binarized models and diminishes the zero-shot accuracy gap to within 0.3% compared to Float16 models. The distinct advantage of BinaryMoS over previous approaches, especially OneBit, lies in its use of scaling experts. This evaluation underlines the efficacy of the BinaryMoS with mixture of scales approach.

4.5 Comparison between BinaryMoS and 2-bit Quantization

Since BinaryMoS consistently outperforms other binarization methods, we proceed to compare it with conventional 2-bit quantization techniques, GPTQ and OmniQuant. While these two approaches entail lower calibration overhead for quantization due to their use of the PTQ approach, they differ in their quantization methods. GPTQ and OmniQuant utilize a group-wise quantization approach, employing groups of 128 weights to finely quantize parameters and minimize quantization errors. Consequently, the memory demand during inference for these methods is more than double that of BinaryMoS. The comparison results, presented in Table 4, reveal that BinaryMoS even outperforms these 2-bit quantization methods, despite its lower memory requirement during inference. This once again underscores the effectiveness of integrating scaling experts.

5 Discussion and Future Work

BinaryMoS significantly improves the accuracy of binarized LLMs by increasing their representational capability with mixture of scales. This MoS approach holds promise for extension to multi-bit quantization, as multi-bit quantization techniques also involve scaling factors for regulating quantization step size. However, in this paper, our study does not delve into the effectiveness of the mixture of scales on multi-bit quanization schemes, leaving this avenue for future exploration.

Though BinaryMoS adopts the concept of MoE, it does not fully leverage advanced training techniques established in the field of MoE [26, 27, 34]. These advanced methods optimize routing functions and balance token assignments among experts, thereby enhancing MoE model accuracy. Thus, investigating these training techniques is another topic for future research.

6 Conclusion

This paper introduces BinaryMoS, a novel binarization technique designed to enhance the representation capability of binarized LLMs while preserving the fundamental advantage of binarization-low memory usage. BinaryMoS adopts the mixture of scale approach to dynamically adjust the scaling factors of binary weight values in a token-adaptive manner. Given that scaling factors play a crucial role in reducing binarization error and occupy a small portion of binarized models, this approach effectively mitigates information loss associated with binarization with minimal memory overhead. Our experimental findings demonstrate that BinaryMoS surpasses existing binarization approaches and even outperforms 2-bit quantization methods in both perplexity and zero-shot tasks.

References

Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang, 'Evaluating Quantized Large Language Models', arXiv preprint arXiv:2402.18158 , 2024.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 'GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers', International Conference on Learning Representations (ICLR) , 2023.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han, 'AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration', arXiv preprint arXiv:2306.00978 , 2023.

Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, Eunhyeok Park, 'OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models', Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2024.

Yuzhang Shang, Zhihang Yuan, Qiang Wu, Zhen Dong, 'PB-LLM: Partially Binarized Large Language Models', International Conference on Learning Representations (ICLR) , 2024.

Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, Xiaojuan Qi, 'BiLLM: Pushing the Limit of Post-Training Quantization for LLMs', International Conference on Machine Learning (ICML) , 2024.

Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, Wanxiang Che, 'OneBit: Towards Extremely Low-bit Large Language Models', arXiv preprint arXiv:2402.11295 , 2024.

Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang, ' A survey on model compression for large language models', arXiv preprint arXiv:2308.07633 , 2023.

Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe, 'Binary neural networks: A survey', arXiv preprint arXiv:2004.03333 , 2020.

Zechun Liu, Wenhan Luo, Baoyuan Wu, Xin Yang, Wei Liu, and Kwang-Ting Cheng, 'BiReal Net: Binarizing Deep Network Towards Real-Network Performance', Proceedings of the European Conference on Computer Vision (ECCV) , 2018.

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi, 'XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks', Proceedings of the European Conference on Computer Vision (ECCV) , 2016.

Zechun Liu, Zhiqiang Shen, Marios Savvides, Kwang-Ting Cheng, 'ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions', Proceedings of the European Conference on Computer Vision (ECCV) , 2020.

Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra, 'LLM-QAT: Data-Free Quantization Aware Training for Large Language Models', arXiv preprint arXiv:2305.17888 , 2023.

Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, Qun Liu, 'TernaryBERT: Distillation-aware Ultra-low Bit BERT', arXiv preprint arXiv:2009.12812 , 2020/

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, Luke Zettlemoyer, 'OPT: Open Pre-trained Transformer Language Models', arXiv preprint arXiv:2205.01068 , 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample, 'LLaMA: Open and Efficient Foundation Language Models', arXiv preprint arXiv:2302.13971 , 2023.

Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others, 'Llama 2: Open Foundation and Fine-Tuned Chat Models', arXiv preprint arXiv:2307.09288 , 2023.

Ilya Loshchilov, Frank Hutter, 'Decoupled Weight Decay Regularization', International Conference on Learning Representations (ICLR) , 2019.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova, 'BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions', arXiv preprint arXiv:1905.10044 , 2019.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi, 'PIQA: Reasoning about Physical Commonsense in Natural Language', Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2020.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi, 'HellaSwag: Can a Machine Really Finish Your Sentence?', arXiv preprint arXiv:1905.07830 , 2019.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi, 'WinoGrande: An Adversarial Winograd Schema Challenge at Scale', arXiv preprint arXiv:1907.10641 , 2019.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, 'Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge', arXiv preprint arXiv:1803.05457 , 2018.

Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher, 'Pointer Sentinel Mixture Models', arXiv preprint arXiv:1609.07843 , 2016.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu, 'Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer', arXiv preprint arXiv:1910.10683 , 2019.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean, 'Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer', arXiv preprint arXiv:1701.06538 , 2017.

William Fedus, Barret Zoph, Noam Shazeer, 'Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity', arXiv preprint arXiv:2101.03961 , 2021.

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, Ping Luo, 'OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models', International Conference on Learning Representations (ICLR) , 2024.

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei, 'BitNet: Scaling 1-bit Transformers for Large Language Models', arXiv preprint arXiv:2310.11453 , 2023.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer, 'QLoRA: Efficient Finetuning of Quantized LLMs', Advances in Neural Information Processing Systems, (NeurIPS) , 2023.

Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, Dan Alistarh, 'SpQR: ASparse-Quantized Representation for Near-Lossless LLM Weight Compression', arXiv preprint arXiv:2306.03078 , 2023.

Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer, 'SqueezeLLM: Dense-and-Sparse Quantization', International Conference on Machine Learning (ICML) , 2024.

Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, Christopher De Sa, 'QuIP: 2-Bit Quantization of Large Language Models With Guarantees', Advances in Neural Information Processing Systems, (NeurIPS) , 2023.

Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, William Fedus, 'ST-MoE: Designing Stable and Transferable Sparse Expert Models', arXiv preprint arXiv:2202.08906 , 2022.

Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christopher Meek, Ahmed Hassan Awadallah, Jianfeng Gao, 'Few-Shot Learning Evaluation in Natural Language Understanding', Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). , 2021.

Taesu Kim, Jongho Lee, Daehyun Ahn, Sarang Kim, Jiwoong Choi, Minkyu Kim, Hyungjun Kim, 'QUICK: Quantization-aware Interleaving and Conflict-free Kernel for Efficient LLM Inference', arXiv preprint arXiv:2402.10076 , 2024.

A Appendix

A.1 Ablation Study on Datasets

To determine the optimal dataset for training binarized models, we conduct a comparative analysis using various training datasets, as summarized in Table 5. The results indicate that models trained solely on the WikiText2 dataset, due to its relatively small dataset size, tend to exhibit overfitting tendencies and struggle to generalize to other datasets. While these models demonstrate considerable perplexity improvement on the WikiText2 evaluation, their perplexity on the C4 dataset and zero-shot accuracy is notably poor. Conversely, models trained exclusively on the C4 dataset perform well across a wide range of tasks, except for the evaluation on WikiText2. Following the approach of previous research [13], we also experiment with a generated dataset synthesized using the LLaMA-17B model. Although this dataset generally performs satisfactorily across various language modeling tasks, its performance lags behind that of the C4 dataset. Therefore, to enhance overall model performance, we opt to train the models on a mixed dataset comprising both C4 and WikiText2. Moreover, the accessibility of both C4 and WikiText2 as open-source datasets further facilitates their adoption for training purposes.

A.2 Latency Measurement

To assess the latency of GEMV operation for our BinaryMoS, we have evaluated the latency of previous binarized models and the BinaryMoS by developing appropriate CUDA kernels for 1bit matrix multiplication, modifying the CUDA kernel for multi-bit matrix multiplication [36]. Additionally, we further customize the CUDA kernel of BinaryMoS to fuse scaling experts and routing operations on top of the 1-bit matrix multiplication CUDA kernel. We measure the latency of the linear layers in LLaMA-7B and LLaMA-13B with batch size of 1 and results are presented in Table 6. All experiments are conducted on NVIDIA A6000 GPUs.

Previous methods like PB-LLM and BiLLM require extra matrix multiplications, making them very slow. OneBit, which employs the simplest binarization scheme, achieves significant improvement over the original Float16 model and shows the minimum latency. Meanwhile, our BinaryMoS introduces additional operations for processing scaling experts, which require far fewer operations compared to matrix multiplication. Consequently, BinaryMoS also shows similar latency results to OneBit. This demonstrates that the multi-scaling factor module in BinaryMoS improves performance in terms of perplexity and zero-shot accuracy with minimal overhead to latency.

A.3 Experimental Results for LLaMA-1-30B

Table 7 provides further experimental results on the LLaMA-1-30B model. In line with the trends observed in Table 3, BinaryMoS consistently surpasses other binarization approaches for this 30B model. This assessment highlights the effectiveness of BinaryMoS for large-scale LLMs.

A.4 Generation Quality

Figure 4 compares the generation quality between BinaryMoS and OneBit on LLaMA-1-13B model. BinaryMoS can generate contextually proper answers, whereas OneBit fails to generate correct answers. These results demonstrate that the BinaryMoS processes each token with token-adaptive scaling factors which contain contextual information and the improvement of BinaryMoS over OneBit is substantial enough to make binarized LLMs more applicable in practice.

Figure 4: Comparison of generation quality on the LLaMA-1-13B models with BinaryMoS and OneBit.

A.5 Limitations

While BinaryMoS effectively enhances the linguistic capabilities of binarized LLMs, its application to extremely large models such as LLaMA-2-70B poses challenging due to the high training cost inherent to QAT-based strategies. Hence, to facilitate the use of BinaryMoS in such large-scale models, it may be necessary to adapt it into a PTQ-based approach or to integrate it with parameterefficient training strategies in future work. Additionally, it is important to note that, despite the advancements brought about by by BinaryMoS, the reduction in linguistic performance of binarized LLMs coampred to their Float16 counterparts remains substantial. Consequently, to make LLM binarization practical for real-world applications, further advancements in binarization techniques are required.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses limitations in the "Discussion and Future Work" and "Limitations" section, including challenges and areas for future exploration.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results that require formal proofs. The focus is on empirical evaluation of the proposed method.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides comprehensive details on the experimental settings, including models used, datasets, evaluation metrics, and training processes. It ensures that the steps to reproduce the results are clear.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provide open access to the data and code to faithfully reproduce the main experimental results, as described in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all necessary training details, ensuring that the experimental results can be understood and replicated.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not report error bars or statistical significance for the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides sufficient information about the compute resources used for the experiments, mentioning the use of NVIDIA A100 GPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms to the NeurIPS Code of Ethics, focusing on improving model efficiency and performance in a responsible manner without evident ethical concerns.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses societal impacts, highlighting the potential for efficient deployment of LLMs on edge devices and acknowledging the importance of responsible use.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not pose high risks for misuse that would require specific safeguards.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper properly credits existing assets, such as datasets and models, and follows appropriate licenses and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce new assets. It focuses on methodological improvements and uses existing datasets and models for evaluation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve human subjects research, thus IRB approvals are not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies the problem of post-training binarization of large language models. Built upon OneBit, this work propose BinaryMoS, to use a mixture of scaling weights for the linear layer of binarized LLMs. Instead of static scales in OneBit, BinaryMoS employs a set of scaling weight experts and adaptively combines them given the input representation of current tokens. As a result, BinaryMoS offers stronger representation capabilities while incurring minimal computation and memory overhead. Experimental results on various LLM benchmarks show the proposed technique compares favorably to other binarization methods for LLMs.

STRENGTHS:
1. BinaryMoS has minimal computational and memory overhead, and keeps the architecture simple without introducing sparse weight matrices.
2. Mixture-of-expert technique could potentially applied to other quantization method.
3. The method works for a wide range of LLMs.
3. The proposed method out-performs other binarization techniques on several LLM benchmarks.

WEAKNESSES:
1. The paper lacks discussion of the cost of post-training adaptations. The training cost seems comparably high as it requires three epochs over the selected dataset.
2. The improvement over OneBit is small, and the improved binarization still perform significantly lower than other less aggressively quantized method.

QUESTIONS:
1. While post-training binarization performs poorly compared to the original base model, methods like BitNet seem to bridging the gap by training low-bit models from scratch. Can your method extend to the scenario where the model is trained from scratch?

LIMITATIONS:
The authors have adequately addressed the limitations of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

comments: SUMMARY:
This paper proposes using a mixture of scales for binarizing continuous latent weights. It includes a router implemented by a linear layer, outputting a softmax score which is then used as the combining matrix of the scale basis (called scale experts). The method uses two score basses, one for input $S_{in}$ and one for output $S_{out}$, each includes a number of scale vectors. The author conducted an experimental analysis of the number of experts to be used, and evaluated the proposed method on different language models with results showing performance gains by mixture of scales.

STRENGTHS:
The proposed idea is pretty nice, simple, and shows performance gains.

WEAKNESSES:
Weak significance: Although the idea is nice and valid, its scientific and technological significance as well as its originality is low. The paper could be more suitable for a workshop than a major venue like NeurIPS. The experimental analysis of the number of scaling experts shows that the performance is not always improved with the number of experts. This also strengthens the above opinion of low significance. 

Several minor typos to be corrected, such as to use a unique term for MoS.

QUESTIONS:
Related to (4) where the utilized scales are obtained by combining the scale basis using $G$, scale experts $S_{in}$ and $S_{out}$ are of primary importance in the method. How are these scale experts are constructed and/or learnt?

If the scale experts are learnt, what can be the benefits of learning $S_{in, out}$ then compute $\hat{S}_{in, out}$ instead of learning these scales directly?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a binarization technique for LLMs, inspired by the mixture-of-expert (MOE) model. In the proposed approach, multiple scaling factors for the binarized matrices are available, each treated as an expert just like in MOE. The model infers a weight combination of these scaling factors adaptively at each time step, hence the binarization strategy is different for each input token. The paper shows that this adaptive binarization technique does not increase much memory overhead as the traditional MOE does, but can effectively improve model quality, as compared with existing 1-bit or 2-bit quantization baselines.\n\nSTRENGTHS:\nThe proposed approach effectively improved the quality of binarized LLMs, which has always been a challenging task. The introduction of adaptive binarization makes a lot of sense and deserves broader attention and experimentation.\n\nWEAKNESSES:\nMore detailed analysis in terms of efficiency, scaling behaviors and ablation studies could have been given.\n\nQUESTIONS:\n1. The performance of BinaryMoS depends critically on the choice of static experts S_in, S_out. How robust is the proposed approach is to random initialization of these experts? Does the magnitude of these scaling experts matter a lot?\n\n2. Are there any interesting patterns learned for expert weights? For examples, whenever one expert gets assigned larger weights, do the input sequences contain specific n-grams or syntactic structure? It would be great to find some insights into how and why the model learned to make expert choices, for better interpretability.\n\n3. Related to question 2, if there is no clear interpretable expert assignment patterns that can be found, what if you just determine expert weights randomly without training? This should be a baseline to validate the effectiveness of the learning procedure.\n\n4. How important is the knowledge distillation loss? It seems there is no analysis to the relative importance of distillation.\n\n5. How does the quantization performance scale with model size? If there is a clear scaling law of BinaryMoS which appears to be more efficient than full-precision training, then this approach should become a standard for very large model training and inference.\n\nLIMITATIONS:\nThe paper addressed limitations in the \"Discussion and future work\" section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to compress large language models (LLMs) by quantizing weight values to 1-bit. The goal is to mitigate performance degradation seen in previous quantization methods applied to LLMs. They utilize ideas from Mixture-of-Experts and introduce token-adaptive scaling factors that help control the effective values of binarized weights. Their approach results in high compression ratios compared to earlier binarization methods, maintaining memory efficiency.\n\nSTRENGTHS:\n- Quantization helps lower the barriers to deploying large language models in compute-constrained environments. While previous approaches have been able to achieve this, it has come at the expense of linguistic utility.  BinaryMoS in an attempt to lower these barriers without these costs. \n\n- Through experiments on various benchmarks, they demonstrate the effectiveness of BinaryMoS in memory-efficient quantization \n\n- The paper offers a clear explanation of existing research work on binarization and this helps with clarity and readability.\n\nWEAKNESSES:\n- The motivation for extending the ideas from MoE to One-Bit is not entirely clear. Is the paper focused on applying binarization to MoE-style models? Please clarify this aspect. \n\n- The benefits of using 4 experts compared to 2 experts do not seem significant across tasks. Given the additional memory overhead introduced by these experts, wouldn't it be better to advocate for using two experts instead?\n\n- The analysis section on token-adaptive scaling factors does not adequately explain why BinaryMoS is preferable to One-Bit. While the variation in gating scores for experts across tokens is shown, there is no context provided about the sentences being analyzed. It seems possible that BinaryMoS is only useful and sensitive to certain domains and tasks and may not apply to every task.\n\n- The omission of latency measurements compared to other binarization methods raises questions about the robustness and efficiency of BinaryMoS. Since latency measurements are critical for real-world deployments of large language models, a discussion on this topic would be great\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nThe authors claim to address the limitations of their work in the \"Discussion and Future Work\" section, but this discussion is incomplete. They do not mention latency requirements and potential failure cases of BinaryMoS are also omitted and not listed as limitations. Additionally, they should acknowledge that performance loss is still expectedly incurred compared to FP16.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}