Bi-Mamba: Towards Accurate 1-Bit State Space Model
Authors: Shengkun Tang
Liqun Ma
Haonan Li
Mingjie Sun
Zhiqiang Shen
Venue: Accepted by TMLR ()

Abstract: The typical selective state-space model (SSM) of Mamba addresses several limitations of Transformers, such as quadratic computational complexity with sequence length and significant inference-time memory requirements due to the key-value cache. However, the growing size of Mamba models continues to pose training and deployment challenges and raises concerns due to considerable training and inference compute consumption. In this work, we introduce Bi-Mamba, a scalable and powerful 1-bit Mamba architecture designed for more efficient large language models with multiple sizes across 780M, 1.3B, and 2.7B. Bi-Mamba models are trained from scratch on data volume as regular LLM pertaining using an autoregressive distillation loss. Extensive experimental results on language modeling demonstrate that Bi-Mamba achieves performance comparable to its full-precision counterparts (e.g., FP16 or BF16) and much better accuracy than post-training-binarization (PTB) Mamba and binarization-aware training (BAT) Transformer baselines, while significantly reducing memory footprint and computational consumption compared to the original Mamba model. Our study pioneers a new linear computational complexity LLM framework under low-bit representation and facilitates future design of specialized hardware tailored for efficient 1-bit Mamba-based LLMs. Our code is provided in Supplementary Material and the pre-trained weights are available anonymously at https://drive.google.com/drive/folders/1jfk_TlDzFbER84ITvU2hOX2VyPC9H4MA?usp=sharing

Published in Transactions on Machine Learning Research (09/2025)
Bi-Mamba: Towards Accurate 1-Bit State Space Models
Shengkun Tang"
Department of Machine Learning Mohamed bin Zoyed University of Artificial Intelligence
Liqun Ma¹
liqun. ma www.maammmaaa.c..a aw..aaaiiiac.a..om
Department of Machine Learning Mohamed bin Zayed University of Artificial Intelligence
Haonan Li
haonan. Maonan.li@mbaaun www.limmmbuai.....
Department of Natural Language Processing Mohamed bin Zoyed. Universily of Artificial Intelligence
Mingjic Sun
mingjics@andreee cma edu
Department of Computer Science Carnegie Mellon University
Zhiqiang Shen'
senggiinn..ssee@@nnnnne ac
Department of Machine Learning Mohamed bin Zuzed University of Artificial Intelligence
Reviewed on OpenReview: https: // oppnneeveee. net/ form/ id- CKQ4AgoROm
Abstract
The typical Selective State-Space Model (SSM) used in Mamba addresses several limitations of Transformers, such as the quadratic computational complexity with respect to sequence length and the significant memory requirements during inference due to the key-value (KV) cache. However, the increasing size of Mamba models continues to pose challenges for train- ing and deployment, particularly due their substantial computational demands during both training and inference. In this work, we introduce Bi-Mamba, a scalable and powerful 1-bit Mamba architecture designed to enable more efficient large language models (LLMs), with model sizes of 780M, 1.3B, and 2.7B parameters. Bi-Mamba models are trained from scratch on standard LLM-scale dataset using an autoregressive distillation loss. Extensive experiments on language modeling benchmarks demonstrate that Bi-Mamba achieves perfor- mance comparable to its full-precision (FP16 or BF16) counterparts, while outperforiigg post-training binarization (PTB) Mamba and binarization-aware training (BAT) Trans- former baselines. Moreover, Bi-Mamba drastically reduces memory usage and computational cost compared to the original Mamba. Our work pioneers new line of linear-complextty LLMs under low-bit representation and paves the way for the design of specialized hardware optimized for elficient 1-bit Mamba-based models. Code and the pre-trained weights are available at http:///jttuubccommm//aaggggg//p Twngs/enggggggnnganeaaaaa Mi-Mamba
1 Introduction
The Selective State-Space Model (SSM) GG et al. 2021b 2022 has recently emerged as a powerful al- ternative to Transformers (Vaswani et al. 2017) in language modeling, achieving comparable or superior performance at small to moderate scales. SSMs scale lincarly with sequence length during training and maintain constant state size during generation, which significantly reduces computational and memory overhead, resulting in greater speed and efficiency.

---

Published in Transactions on Machine Learning Research (09/2025)
210080
220000
180000
GPTQ
10
GPTO
GPTQ
8707
1700p
13000
Bi-Mamba (Ours)
Bi-Mamba (Durs)
Bi-Mamba (Ours)
Perplesly
Perploxity
Perplexlt
20
10
21.2
16.3
21.8
14.0
14.5
113
Quantization Bit-width
Quantication Dit-width
Quantization Bit-width
(a) Wiki-2
(b) PTB
(c) C4
Figure I: Perplexity comparison of Bi Bi-Mamba, GPTQ and Bi-LLM on Wiki2, PTB and C4 datasets. GPTQ and Bi-LLM show significant performance degradation when the bit is low. Bi Bi-Mamba demonstrates low perplexity in bit and shows similar performance as GPTQ-8bit.
The representative SSM model, Mamba (Gu & Dao 2024 Dao & Gu 2024), has clear advantage in handling long-context sequences due to its linear complexity. In contrast, conventional Transformers cxhibit quadratic complexity as sequence length increases. This makes Mamba substantially more elficient for lasks involving large inputs or extended contexts, with memory and compute requirements that grow only linearly. In practice, this enables faster processing of long sequences with lower resource consumption, making Mamba well-suited for applications such long-document processing, conversational agents, and other scenarios requiring large-context management. On the other hand, Transformers require polynomially more resources as context length grows, which quickly becomes a bottleneck in long-context tasks.
Prior work [Devlin et al. 2019 Radford et al. 2019 Touvron et al. 2023 Achiam et al. 2023 on Transform- ers has been extensive, with numerous methods proposed to reduce their high computational and storage costs, such as pruning (Ma et al. 2023), model quantization (Frantar al. 2023), and KV cache optimiza-
tions PPope et al., 2023 Kwon al. 2023). Among these, quantization has proven highly effective (Deltmers et al. 2022b),, enabling reductions from 16-bit to 8-bit, 4-bit, and even 1-bit representations with minimal
performance loss (Frantar et al. 2023 Ma et al. 2024b
However, little is known about how SSM models (Mamba in particular) perform under low-bit quantization or binarization. this paper, we present Bi-Mamba, a novel approach that applies extreme quantization to SSM models by reducing weights to a binary sctting. We show that SSM models can be effectively binarizod for both training and inference, retaining high performance while dramatically reducing memory footprint and energy consumption.
Our work pioncers new framework for 1-bit representation in lincar computational complexity models, potentially facilitating the development of specialized hardware tailored for fully low-bit SSM-based language models. We provide comprehensive experiments showing that Bi-Mamba achieves competitive performance compared to its full-precision counterpart large language models (LLMs), thereby establishing the feasibility and benefits of binarizing SSM models. To further understand the behaviors of binarization on Mambas, conducted extensive empirical studies to analyze the distribution of pre-trained and binarized weights in Mambas. The results of this study detailed in Scction and Appendix A leading to two key observations:
As shown in Figure post-trrininng-inariiatttt PB-LLM (Shang et al. 2023) typically tend to shift the distribution of weights on Mamba after binarization, resulting in misaligned distribution to the optimal binary weights. Our binarization- aware training, however, ensures that the binarized weights remain close to the original weight distribution, preserving the largest capability in weight representation binarization.
methods like BiLLM (Huang et al. 2024) and
Based on empirical experiments, applying existing LLM post-binarization methods (Frantar et al. 2023 Huang et al. 2024). even when retaining salient weights, often severely degrades the per- formance of the Mamba model. Without accounting for salient weights, binarization- -wware training appears to be the only feasible solution for effectively binarizing models like Mamba while preserving competitive performance.
Equal contribution. 1 *Coresponding author.
2

---

Published in Transactions on Machine Learning Research (09/2025)
B1 Bi-Mmbaa Likear
Bl-Eiscor
Full-precision:
Post-binarizatinn
Unooo
Norm
16-bi
P
16 but
Corn
Column Wase
34
64
Lingor
Binarization-aware Training:
(1,41
W
Bl-Linecr
Mamla-22 Flock
Mamba-2 780M
Enteedinn
Mamba-2
Mamba-2 2.78
Enrbekiing
IN
Embestding
Betry bas
IM
Deta
Proj
Crndl
Crmvls
Pr
Conoid
in Proj
in Proj
Figure 2: Illustration of the -Mamba framework. Our Bi-Mamba binarizes both input and output projection matrices. Compared with the post-binarization method (Bi-LLM), our binarization-aware training method (Bi-Mamba) generates more similar weight distribution (after scaling) each part.
Our contributions in this paper are as follows: (1) This the first work to successfully binarize the Mamba architecture to 1-bit from scratch while maintaining strong performance. (2) We explore the potential parameter space for binarization in Mamba, offering valuable insights for future research.
(3) The Bi-Mamba model pretrained by our method can serve as robust base model for downstream tasks in resource-limited scenarios and can be casily adapted for other NLP applications
2
Related Work
Post- Teatnirainng Quantization. Due to their large parameter count, LLMs (Brown et al. 2020
OpenAl
2023
Touvron et al. 2023) are resource intensive to run. Therefore, they are often quantized to low-bit
representations during inference. This type of quantization typically called post-training quantization
(PTQ) (Dettmers al., 2022a Xiao et al. 2023 Wei et al., 2022 Frantar et al.) 2023 Yao et al. 2022: Xiao et al. 2023 Lin et al. 2024], where the quantization operation is applied to the pretrained models. Post-training quantization methods of LLMs are typically based on the empirical observations that small set of salient features in pretrained Transformers have significantly large magniludes, and quantization of
these features needs to be extra careful (Xiao et al.) 2023 Lin et al. 2024). PTQ methods have achieved
nearly lossless accuracy for INT8 weighlts-and-cccivvation quantization and INT4 weight-only quantization.
While many works (Huang et al. 2024 Egiazarian et al. 2024 Shao et al. 2023 Chee et al. 2023 Tseng
et al. 2024) have focused on pushing post-training weight quantization below 4 bits, they often lead a drastic drop in model performance.
Quantization-Awar Training. Different from post-training quantization, quantization-awan training (QAT) aims to learn the quantized models during training. In the LLM era, QAT is less investigated compared to PTQ because LLMs typically require huge amounts of compute resources to train. Li et al. 2023bb propose method that adopts LoRA fine-tuning during QAT, making the quantization procedure more resource-efficient However, these methods still require the pretrained weights to initialize the student LLMs. Wortsman et al. (2023) propose a method stabilize the training of low-bit large vision-language models from random initializations. Most recently, Ma al. (2024b) trained LLMs with ternary weights from random initializations, achieving non-trivial performance. Ma et. al. (022aa) further proposed a distillation based method to pretrain binarized LLMs from scratch.
We will make all our models, code, and training datasets fully availahle, we support further research this direction and encourage the exploration of binarized SSM models for more efficient, large language models.
3

---

Published in Transactions on Machine Learning Research (09/2025)
Model Size
Embedding
IN
Apias
A
Convid
Proj.
Out Proj.
Mamba-2 780M
9.901
0.029
0.0003
0.0003
0.0003
0.102
60.936
29.031
Mamba-2 1.3B
7.664
0.022
0.0002
0.0002
0.0002
0.078
62.270
29.964
Mamba-2 2.7B
4.763
0.018
0.0002
0.0002
0.0002
0.064
64.115
31.039
Table 1: Proportional distribution of parameters across different modules in Mamba-2. Input and output matrices take up the majority of the parameters in Mamba-2 models, around 90% and more.
State-Space Models. While LLMs often built with Transformers (Vaswani et al. 2017). their self- attention operations suffer from quadratic time complexity. This makes Transformers inelficient to on long sequences. Therefore there have been many research efforts aiming at addressing the efficiency issue (Peng et al. 2023 Beck et al. 2024 Katharopoulos et al. 2020 De et al. 2024). Among these efforts,
SSMs (Gu & Dao 2024 Dao & Gu 2024 Gu ct al. 2021a) are a type of recurrent neural networks. The
latest advanced SSMs like Mamba (Gu Dao (024)) and Mamnb-22 (Dao Gu 2024) have demonstrated
comparable performance Transformers, while having linear complexity with respect to the sequence length, Despite their promising capabilities, how to effoctively quantize this architecture has rarely been invostigated.
Concurrent to our work, Pierro & Abreu 2024 have looked at post-training quantization of Mamba.
3 Approach
3.1
Preliminary: Mamba Series
Mamba belongs to class of models known State Space Models (SSMs), which is able to offer performance and scaling laws comparable to the Transformer while remaining practical at extremely long sequence lengths (e.g., one million tokens). It achieves this extended context by eliminating "quadratic bottleneck" present in the attention mechanism. The vanilla structured state space sequence models (S4) transform a 1-dimensional sequence x E R² into another sequence y E Rº by utilizing a latent state h E RT.N as follows:
h Ah, + Ba,
(1)
= Che + Dr
where E RNN,, E RN1,, Ce RINN, E R. h is the hidden state, Te is the input, the observation that the model gets each time. h2 then represents the derivative of the hidden state, i.c. how the state is evolving.
Since the parameters in Equation I are constant through time, this model is linear time-invariant (LTI). The LTT model gives equal attention to all elements when processing sequences, which prevents the model from effectively understanding natural language. To address this, Mamba improved it to the Selective State Space Model, where B and C are obtained through a lincar projection of z4. Mcanwhile, the above equation applies to dynamic systems with continuous input and output signals. So, il needs to do discretization when dealing with text sequences:
Ā = exp( AA)
(2)
B = exp( A (exp(AAAAAA)) 1) AB
where A depends on xa and learnable paramcter Abias. Then, the calculating process of selective SSM is as follows:
he = Ahn-1 + Bxi
(3)
yt Ch++ + Daa
Centered around selective SSM, combined with linear projection for input and output along with layer normalization, this forms the basic block for Mamba.
Comparod to Mamba, Mamba-2 further replaces sclective SSM with the state space duality (SSD). In SSD, A is simplified as scalar-times-identie structure, and SSD uses larger head dinension, which is 1 in Mamba. Meanwhile, Mamba-2 block also introduces simplifications, such as removing sequential linear projections in the Mamba block, to facilitate parallel training. The basic structure of Mamba-2 is shown in Figure
4

---

Published in Transactions on Machine Learning Research (09/2025)
3.2 Bi-Mamba
Binarization Space in Mamba. To begin with, we need to identify what weight matrices can be binarized in the Mamba architecture. We use the latest Mamba-2 (Dao & Gu (0224) as our basc architecture. According to the description above, we can interpret the SSD matrices of A, B, C, D, and Atias more intuitively, as well as other layers including embedding, loyer normalization (LNN,, Conu-1d, inner linear projection, out linear projection in Mamba-2 to better understand the effect after binarization, so that determine the binarization space. Briefly, A is the transition state matrix, representing how the current state transitions to the next state. It answers how should the model gradually forget the less relevaml parts of the slate over time? B maps the new input to the state, addressing the point of which parts of the new imput should the model retain? C' maps the state to the output of the SSM, asking how can the model ulilize the current state to make an accurate next prediction? D shows how the new input directly influences the output, acting as a modified skip connection and asking how can the model incorporate the new input into the prediction!
Considering our base architecture Mamba-2, we present the specific parameters proportion for different layers across different sizes shown in Figure 2 and Table | It is observed that, in Mamba-2, the vast majority of the model's parameters are in the linear modules, excluding the causal head. For instance, in Mamba-2-2.7B, these parameters account for 95.2% of the entire model. Additionally, the embedding module shares parameters with the causal head. Binarizing the embedding significantly diminishes its capability to represent token semantics, thereby reducing model performance. Therefore, in our Bi-Mamba, we only binarize the parameters in the lincar modules (excluding the causal head). This strategy aims to maintain high compression ratio of 90% while still allowing the binarized model to perform effectively.
Simple Learnable Scaling Factors for Better Capability. To binarize Mamba, replace the original linear modules with the FBI-Linear module introduced by FBI-LLM (Ma et al.) 2024ap. We note that our primary goal is to demonstrate the feasibility of binarizing the Mamba architecture and show that it can achieve strong performance, not to claim that FBI-LLM is the optimal scheme here. While leveraging this scheme, our core motivation is the successful adaptation of binarization-aware training to the distinct SSM architecture of Mamba. Specifically, the FBI-Linear module primarily consists of two parts: a matrix W E Rmxn made up solely of [1,-1) and high-precision scale factors R" and E R The inference process of FBI-Linear is as follows:
y W
(4)
where Wd is derived by performing column-wiss multiplication with and addition with B respectively:
W = ax W + 8g
where a and B, are the learnable scaling and shifting factors at i-th layer.
Objective of Training. Our training objective is a cross-entropy loss between the outputs of the target student model and the pretrained teacher model at each step of the autoregressive scheme for next-token prediction. This can be expressed as:
1 --1 n
n ΣE>> T (o4t1), log log (2)t1)
LEI-Nanba
p"
(6)
where n is the number of input tokens. Here, pT (1*+1) represents the token distribution over the vocabulary at the k-th step predicted by the teacher model, and p (I++1) is the corresponding predicted distribution by the student model.
Overall Design of Bi-Mamba. During binarization-wware training of autoregressive distillation Ma et al. 2024a), we compute the cross-entropy between the output probability distributions of a high-precision pre- trained model and our target Bi-Mamba model. In this process, a and 3 are learnable parameters, while b is derived using the sign(-) function from a learnable high-precision matrix W E gmxa, Since the sign(-) function is non-diffeeentaable,
use the Straight Through Estimator (STE) [Bengio et al. 2013
prior
Siine B and C are derived from the linear projection of each layer's inputs, and the language model head is tied to the embedding layer, these modules are ignored in our design.
5

---

Published in Transactions on Machine Learning Research (09/2025)
studies (Rastegari et al., 2016 Alizadeh et al. 2019 in BNNs to approximate the gradients of the imput variables, enabling the continuation ion of backward propagation.
Complexity Analysis. Let L denote the sequence length, the model embedding width, and r x the hidden width of the MLP (usually r = 4), the Transformer cost O(L2d) for self-attention, ((L)) to form Q, K, V plus ((T2d) for the QKT product, and O x d8) for MLP. For Mamba, because each token is processed by single state-space update, the per-layer complexity is OLL d) and each token is one SSM step. For the binary variants, Binary Transformer has the same algebraic graph but multiply--ccummultt is replaced by single binary operator like XNOR and popcount on 32- 64-bit elements, yielding = 32x fewer loaded bits in weight bandwidth. Our Bi-Mamba also has OL x d) bit-ops (inheriting Mamba's linear scaling) and achieves a comparable R 32x reduction in weight bits.
O(L x d) bit-ops,
4 Experiments
In our experiments, we solely binarize the parameters in the most linear modules, while keeping other model parameters and activations at their original precision. Notably, we do not strictly represent the binarized parameters with 1-bit, instead, we use high-precision values to simulate the binarized parameters. We train Bi-Mamba on different scales and evaluate their performance across multiple tasks.
4.1 Setup
Training Dataset. Following FBI-LLM, we train Bi-Mamba with the Amber dataset (Liu et al. 2023 which
contains a total 1.26 Trillion tokens from Refined Web (Penedo et al. 2023), StarCoder (Li et al. 2023a 0 and
RedPajama-vi (Computer 2023). The data is partitioned into 360 chunks, each comprising approximately 3.5B tokens on average.
Training details. We train Bi-Mamba different scales with Mamba-2 architecture. Specifically, we binarize input projection and output projection matrices in 780M, 1.3B and 2.7B Mamba-2 models. We use LLaMA2-7B as the teacher model for all Bi Bi-Mambas to calculate autoregressive distillation loss. Therefore, all Bi-Mambas we trained have the same vocabulary and tokenizer as LLaMA2. We train models until convergence with 32x NVIDIA A100 GPUs in total and maintain BF16 precision while training. For configuring different sizes of Bi Bi-Mamba, the details can be found Table We follow the same architectures as the original Mamba-2 models and apply binarization on both input and output projection matrices. The training process uses the Adam optimizer with parameters B1 0.9 and B2 0.95. The initial learning rate is set at 2 and follows a cosine schedule, decreasing to 2.5e over 2,000 warm-up steps. Gradient clipping is set at 1.0. We train Bi -aaba 780M, 1.3B, 2.7B with 30 data chunks, which are 105B tokens.
Bi -amba 780M Bi-Mamba 1.3B
Bi Bi-Mamba 2.7B
model
1536
2,048
2,560
layer vocabulary size
48
48
64
32,000
32,000
32,000
learning rate batch size (token) teacher model
2.5e-4
2.5e
2.5e-4
0.5M
0.5M
0.5M
LLaMA2-7B
LLaMA2-7B
LLaMA2-7B
Table 2: The configuration and training details for Bi-Mamba.
Evaluation Metrics. We evaluate the models based on their zero-shot performance downstream tasks, including BoolQ CClrr et al., 2019), PIQA (Bisk et al. 2020), HellaSwag (Zellers et al. 2019), WinoGrande (Sakaguchi et al. 20211, ARC (Clark et al. 20018), and OpenbookQA (Mihaylov et al. 2018). We also evaluate Bi-Mamba and other baselines HumanEval and GSM8K datasets. All downstream evaluations are done with Im- tat--eaatation-haaaeee
package (Gao et al. 2024). We also use perplexity on Wikitext2 (Merity et al. 2016), PTB (Marcus et al.) 1993). C4 (Raffel et al. 2020) dataset as the evaluation metric. Perplexity measures how well a probability model predicts a. Loken, quantitatively measuring the model's generation power.
Baselines. We compare our work with quantization and binarization methods, namely GPTQ (Frantar et al. 2023 SqueezeLLL (Kim et. al. 2023). AQLM Egiazarian et al.] 2024), and Bi-LLM (Huang et al.

---

Published in Transactions on Machine Learning Research (09/2025)
Method
Model
Stae
3k.00
PIQA
385
Zero-hot Accurocy
WG
ARCe
ABCC
OBRQA
Arg.
W3k
Perplasity
PTB
C4
16.5
75004
646
02:0
40.3
53.3
40.6
26.4
108.5
192.
156.0
Sppeoee M.---M
75054
61.4
68.7
50.6
51.3
27.6
30.8
48.6
15.3
21.3
75054
92:4
50.3
75.7
51.3
15.6
15.1
36.2
1/0ee3
1.3re5
73
Sypweor M--2
M
75004
903
58.3
13.6
51.5
36.0
21.7
37.0
141.7
2163
325-4
36
75051
05.8
07.3
30.0
50.1
35.1
22.4
26.4
36.7
240.2
812.4
311.0
BLLLM
34
75051
04.1
72.9
26.0
50.6
25.5
26.5
27.5
15.3
1.6e-4
2.4e14
1.c1d
70051
05.2
08:1
35.1
50.3
51.8
21.4
20.0
41.3
Ozsdit"
56
75484
02.2
103.3
35.6
64.4
44.6
24.4
28.8
46.3
26.3
30.6
27.4
34
75084
38.5:10000)
08 08.0550.471 471
41
41,85000774
12
42
24.8241106
45-3:31.302
13.4410124
32,
32.4822/0000
14.8c50.110
Thoyl LaMA Mivaramaa
T
1.311
57.8
T3.3
38.8
55.3
30.1
36.0
53.0
7.5
30.5
8.9
DPT
Rhane cum anodd
7
1.391
57.8
19.5
13.7
54.5
51.0
78.5
33.4
31.1
14.5
30.3
16.1
Mambbr (ANAMOo
3091
M
1.333
61.3
7377
39.9
61.0
00.4
35.1
37.6
35.5
10.1
17.7
14.8
GPTQ.St
69.8
66.4
91.3
Squared LL33--000
1.33
020
72.2
18.0
25.0
12.5
13.5
16.7
0130
000
26.4
132:
10010
133:10
Squacee LLM
02.1
52.0
26.2
136.4
47.1
51.3
238.7
BILEM
BatNe-
65.7
OzudBii
01.8
21.4
FRI-LLM"
1.83
03.1
64.00
48.6
26.8
25 Manba
34
1.583
00.0.10125
08
4T.5100021
45.0.0.000
265100.10
22.210.114
48.4.0.167
32.7.00111 111
28.9100118
12.0...116
Marboo22
13.3
GPTQ Sti
36
2.73
64.8
10.9
$4.0
66.0
51.6
15.5
12.8
50.5
21.2
30.0
21.3
Squaseo Skil
36
2.73
65.1
76.0
65.0
62.9
61.9
14.5
36.2
52.7
10.5
15.2
15.4
GPT
31
2.733
01.4
20:3
25.5
12.00
25.5
25.5
20.0
16.
2:3016
13:T0
Squecee LLM Stil
34
2.733
47.6
40.0
26.0
68.4
26.2
24.5
26.6
35.5
1.
3.2-14
L7c16
AQUUSEi2 296
34
2.731
07.1
68.7
12.6
13.4
15.2
25.7
27.6
45.2
31.3
55.4
45.8
BELLM
36
2.78
62.3
18.8
27.7
15.00
25.1
25.1
28.2
38.5
1.7644
1-3644
OcdBi"
1 6.73
01.1
60.7
12.5
68.1
41.6
25.5
34.0
48.5
10.2
18.2
11.4
BaNss-1.5oa D.b0a!
8.08
01.4
71.
42.9
60.3
61.4
25.5
26.6
50.2
Orelie"
2.733
09:3
18.91
30.5
01.5
49.5
36.0
35.0
30.5
16.5
21.7
19.8
24 Facba
2.1
5801000111
12518.448
54.5100035
56 M0.100088
51,0100101
20.110:112
528100.95
50.810.171
10.0.10.00
21.010.106
11.310.121
Table 3: Performance comparison with baselines on downstream tasks and perplexity. Here, Model represents the architecture of the quantized model. We divide the table into three blocks based on model size. Our Bi-Mamba achieves lower perplexity than Bi-LLM and GPTQ on Wiki2, PTB and C4 datasets, as well as the best average performance on downstream tasks compared with GPTQ-2bit and Bi-LLM. t refers to the results are from the original paper while means the results are produced by the original implementation and weights. All other methods are implemented by us and applied on Mamba architecture. The results of Bi-Mamba are reported with standard deviation over 10 runs, each using different subsets of the test data.
2024). GPTQ, SqueezeLL.M, and AQLM are post-training quantization methods, whereas Bi-LLM is a post-training binarization method. We apply the official implementation of GPTQ and SqueezeLLM and quantize the Mamba-2 models into and 2 bits, respectively. For AQLM, we use their official implementation to quantize the 780M, 1.3B and 2.7B models into 2.04bit, 1.92bit and 2.09bit. For Bi-LLM, we also utilize their official implementation and binarize Mamba-2 models. Morcover, we add quantization-aware training methods, OneBit (Xu et al., 2024) and BitNet-1.58bit (Ma et al. 2024b) for comparison. For BitNet, we report the results from the original paper for comparison, whereas for OneBit, since only the official 7B weights were relcased, we include results only for the 7B models. We also add the comparison with FBI- LLM (Ma al., 2024a), which is transformer-based binary model. We report the results of FBI-LLM 1.3B model from their paper. Furthermore, we include results from open-source full-preeision transformer- based models of various sizes, such as OPT (Zhang et al. 2022], and TinyLLaMA (Zhang et al. 2024), references.
4.2 Main Results
Table presents the comparison of our Bi-Mamba model against various baselines on downstream tasks and perplexity on Wiki2, PTB, and C4 datasets. These evaluations provide insight into model generalization capabilitics without further task-spocific fine- finettning. The visualization of performance comparison is shown in Figure The performance on HumanEval and GSM8K is presented in Table
For the 780M Mamba-2 model, Bi-Mamba demonstrates an average downstream performance of 45.3, out- performing GPTQ-3bit and Bi-LLM, which achieve 42.6 and 38.1, respectively. In perplexity assessments, Bi-Mamba reports scores of 13.4, 32.4, and 14.5 on Wiki2, PTB, and C4, respectively, with the baseline models exhibiting up to 10x higher perplexity. SqueezeLLM and AQLM achieve better performance than
In the table, "M" indicating Mamba-2 (Dao & Gu 2024; and "T" indicating Transformer Vaswani and OBQA are abbreviations for HellaSwag, WinoGrande, and OpenhookQA, respectively.
2017). HS, WG.
7

---

Published in Transactions on Machine Learning Research (09/2025)
GPTQ on both downstream tasks and perplexity, yet they are still outperformed by the Bi-Mamba 780M model. Moreover, Bi-Mamba 780M surpasses the binarization-aware training method, BitNet in the zero-shot performance on downstream tasks. Bit obtains 44.3 scores on average while Bi-Mamba 780M achieves 45.3 scores.
For the 1.3B model, Bi-Mamba achieves a notable downstream accuracy of 48.4 on average, surpassing GPTQ-2bit's 35.3, SqueezeLLM-2bit's 46.5, AQLM's 39.6 and Bi-LLM's 36.2. This performance indicates Bi Bi-Mamba's 's enhanced generalization across wider range of tasks at this model size. Additionally, Bi-Mamba demonstrates substantially improved perplexity, registering scores of 11.7, 29.9, and 12.9 on Wiki2, PTB, and C4 datasots, respectively. In comparison, GPTQ-2bit and Bi-LLM present considerably higher perplexity values, underscoring the efficiency of Bi-Mamba's 's binarization in maintaining linguistic coherence. Moreover, Bi-Mamba 1.3B model beats the training- based method, BitNet and FBI-LLM, which obtains 45.4, 46.3 scores on average on the downstream tasks.
For the 2.7B model, Bi-Mamba further extends its lead, achieving an average rownstream accuracy of 50.6, compared to 35.8 for GPTQ-2bit, 35.5 for SqusezeLLM-2bit, 45.2 for AQLM-2.09 2.09bit, and 38.5 for Bi-LLM. Notably, Bi-Mamba maintains low perplexity across all datasets, with scores of 10.0, 21.9, and 11.3 on Wiki2, PTB, and C4, respectively. These results highlight Bi-Mamba's ability to retain high-level performance in both lask accuracy and linguistic fluency as model complexity scales up.
While BitNet-1.58bit does not provide each perplexity respectively, the average perplexity of the 3B model is 9.91 on C4 and Wikitext2, which is reported in their original paper with 8-bit activation and 1.58-bit weights. In contrast, Bi-Mamba keeps 16-bit activation and binarizes weights, obtaining 10.65 on average. BitNet-L.58b shows slightly better perplexity, which we attribute to its larger sizc-3B parameters compared to Bi-Mamba's s 2.7B and its higher precision at 1.58 bits. Moreover, OneBit on the transformer model only obtains 49.5 scores on average with 6.7B parameters, and BitNet gains 50.2 scores with 3.0B parameters. On the Mamba 780M with the 1.3B model, Bi-Mamba achieves better performance in both perplexity and downstream tasks; however, on the 2.7B model, Bi-Mamba shows improved perplexity but obtains comparable downstream task performance to Onebit. Bi-Mamba's lower perplexity shows stronger language modeling, but this does not always yield higher task accuracy especially when datasets' knowledge coverage limits model capability.
In summary, Bi-Mamba consistently demonstrates superior zero-shot performance and perplexity reduction across model sizes, substantiating its robustness and versatility compared to GPTQ and Bi-LLM, particularly in larger, more complex models.
For results on HumanFval and GSM8K, we notice that the full-precision models across all sizes perform poorly, producing results close to random guessing. Other quantization methods further degrade performance compared to their full-precision counterparts. In contrast, our proposed Bi-Mamba consistently achieves superior performance, outperforming all other methods, including the full-precision model. We believe the reason is the higher quality of pretraining data. Bi-Mamba applied Amber dataset, which is a high-quality dataset compared with Pile dataset used in the original Mamba model. We further conduct the instruction tuning on our base Bi-Mamba model with OpenOrca (Lian et al., 2023) dataset. We trained it for 3.3B tokens. After instruction- tuning the performance is further improved, demonstrating that Bi-Mamba can serve as strong base model. Overall, the results of the Mamba models and their quantizod versions are not highly competitive. The primary reasons for this are: 1) Limited Training Data. The Mamba-2-2.77 pre- trained model is trained with only 300B tokens the Pile dataset. This amount of training data is relatively small compared to saato-of-tho-rr models, which are often trained on significantly larger corpora (more than 10T tokens). As a result, the pre-trained model may not have fully developed strong reasoning and problem- solving capabilities. After the quantization or binarization, the reasoning ability collapses completely. 2) Lack of Instruction Tuning. The models in this evaluation have not been instruction-tuneld which is critical for performance on complex reasoning and coding benchmarks such as GSM8K and HumanEval. Instruction tuning enhances a model's 's ability to follow prompts effectively and generalize across tasks. Quantizing an instructed model could yield better performance on complex reasoning tasks.
8

---

Published in Transactions on Machine Learning Research (09/2025)
GPTQ -bit
GPTQ-2bit
GPTQ 2bit
HellaSwag.
BILLM
HellaSwag.
BILLM
HellaSwag.
BILLM
Bi-Mamba
Bi-Mamba
Bi-Mamba
WinoGrande
WinoGrande
WinoGrande
BoaiQ
BoolQ
BoalQ
ARC-Easy
ARC-Easy
ARC-Basy
OBQA
OBQA
OBQA
ARC-Challengé
ARC-Chatlenge
ARC-Chatlenge
(a) 2.7B
(b) 133B
(c) 780M
Figure 3: Visualization of results comparison Mamba-2 in the scales of 2.7B, 1.3B and 780M.
Model
Model Param.
HumanEval
GSM8K
Mamba-2-100 1-bit
780M
0.81
0.76
GPTQ-2bit
780M
0.00
0.68
AQLM-2.04bit
780M
0.00
0.61
SqueczeLLM-226
780M
0.00
0,00
BiLLM-2bit
780M
0.00
0.07
B1 Mamba
780M
0.85
1.29
B1 3i-Mamba (Instruct)
780M
1.22
1.97
Mamba-2-16ble
13B
1.83
0.76
GPTQ-2bit
1:3B
0.23
0.37
AQLM-1.92bit
1.3B
0.20
0.38
SqueczeLLM-2bit
1:3B
0.20
0.15
BiL.LM-bbtt
1.3
0.18
0.15
Bi-Mamba
L3B
0.71
1.95
Bi-Hamba (Instruct)
1:3B
3.66
2.88
Mamba-2-16bit
2.7B
1.07
0.91
GPTQ-2bit
AQLM-2.09hb
SqueeseLLMM2226
2.7B
0.23
BiI.LM-2bi
0.30
B1-Namba
2.71
0.91
1.29
Bi-Mamba (Instruct)
6.10
Table 4: Porformance comparison of different mothods on HumanExal and GSMSK evaluation set.
5
Analysis
5.1
Training Result Dynamics
In this section, we discuss the performance of Bi-Mamba as the training progresses with different training costs/budgets. The main results are shown in Figure 4 We present the downstream performance and perplexity curves across different training costs for the Mamba-2-780M model. More results on 2.7B and 1.3B Mamba-2 models can be found in the Appendix A.1 From the figure, we can observe that the perplexity decreases quickly at the beginning of training and gradually converges to the full-precision perplexity. The perplexity of Bi Bi-Mamba the wiki2 and C4 datasets is more stable than the perplexity on the PTB dataset. Notably, on the C4 dataset, the final perplexity of Bi-Mamba is even lower than the full-precision model, highlighting the superior performance of Bi -Mmba. Interestingly, early in training, our Bi-Mamba model surpasses GPTQ-3bit on the perplexity, demonstrating the effectiveness binarization-avare training. Since the perplexity of GPTQ-2bit and BiLLM is extremely high on all datascts, we omit them in the figure and refer Table 3 for detailed results of GPTQ-2bit and BiLLM. Moving the downstream task evaluation, first observe the catastrophic performance degradation of the binarized models, whose performance is

---

Published in Transactions on Machine Learning Research (09/2023)
Perplexity
102
GPTQ-bbit Bi Mamba
Perp lexitty
10a
GPTD-bbit
Bi-Mamba
Perplexd pelilylxtyy
102
GPTO-bbit
Bi-Mamba
10
12
2
4
6
8
10
12
2
4
6
8
10
12
Data Chunks
Data Chunks
Data Chunks
(a) Wiki
(b) PTB
(c) C4
80
80
Prec-Proi
Precsinciioonn
GPT
GPTD-2b4t
GPT0-2bt
GPTQ-224t
BHLUM
60
BHLLM
60
BHLLM
30
Mamba
Acc.
Bombo
Acc.
Mambo
40
40
60
30
30
40
20
20
2
4
6
10
12
8
10
12
4
6
8
10
12
2
4
6
8
10
12
Data Chunks
Data Chunks
Data Chunks
Data Chunks
(d) ARC-C
(e) ARC-E
(f) HS
(g) BoolQ
70
Rull-Pracision 2bit
Full-Precision
Full-Precision
GPTO 2bit
2bit
GPTO 2bit
80
BALLM
Acc.
60
RalLM.
BAILM
BHLLM
40
60
30
20
2
10 12
8
10
12
4
6
8
10
12
2
4
6
8
10
12
Data Chunks
Data Chunks
Data Chunks
Data Chunks
(h) PIQA
(i) WG
OBQA
(k) Avg.
Figure 4: The training result dynamics of Downstream performance and perplexity of Bi -Mama.
Zero-shot Aec.
Perplexity PTB
Model
BoolQ
PIQA
HS
WG
ARC-e
ARC-e
OBQA
Avg.
Wiki2
C4
Mamba-2 2- 780M
61.5
71.8
60.2
54.3
28.5
52.5
11.8
20.0
16.5
Bi-Mamba (In. Proj]
68.3
53.7
42.6
24.3
45.5
14.4
Bi-Namba (In Proj + Out Proj)
58.5
68.0
41.6
52.0
42.4
24.3
30.6
45.3
13.4
32.4
14.5
Mamba-2-1e 2-L.3B
64.3
73.7
59.9
61.0
60.4
33.1
37.8
55.8
10.4
17.7
14.8
Bi-Mamba (In. Proj)
62.1
71.7
50.4
53.8
49.5
26.8
33.0
49.6
10.7
26.0
12.0
B1-Namba (Ia Proj + Out Proj)
60.0
68.8
47.3
55.9
48.0
26.3
32.2
48.4
11.7
29.9
12.9
Hlamba-2-2775 Bi MamMamba (In Proj)
70.7
76.3
66.6
63.9
64.8
36.3
38.8
59.6
9.1
15.3
13.3
62.4
74.6
58.9
57.1
54.0
29.4
35.4
53.1
9.1
19.5
10.5
Bi-Mamba (In Proj + Out Proj)
58.0
72.5
54.3
56.1
51.4
29.1
32.6
50.6
10.0
21.9
11.3
Table 5: Performance comparison of bartiaal-inarization and binarization of In Proj and Out Proj on perplexity and downstream tasks. The results show that the performance gap between partial-binarivation and binarization of In Proj and Out Proj is not significant, indicating that the In Proj and Out Proj binarized model can maintain competitive performance with the partial binarization model.
even lower than the random results on many benchmarks, such as the results on ARC-E and ARC-C. This indicates that directly applying the naive binarization method destroys the ability the full-precision model. However, after binerization-aware training, the model recovers the performance on all benchmarks and finally outperforms all baselines, including GPTQQ-bit, GPT-2bit, and Bi-LLM. This underscores the importance of binarization-aware training in achieving competitive results.
5.2 Binarization Space
In this section, we explore the binarization space of Mamba models and discuss the effect of binarizing each part. We conduct experiments that binarize the In Proj and binarize both In Proj and Ouo__rooj. The binarized models are trained with the same data. The results are shown in Table 5 Partial and full binarization are compared with the full-precision Mamba model on perplexity and downstream tasks. First, the results in the table show that partial binarization generally retains higher zero-shot accuracy compared
10

---

Published in Transactions on Machine Learning Research (09/2025)
Zero-shot Acc. t
Perplexity 4
Model
Tokens
GPU Hours
BoolQ
PIQA
HS
WG
ARC-
ARC-c
OBQA
Avg.
Wiki2
PTB
C44
Mamiss-2 (130M, 16:ba)
300B
1180
55.1
64.0
35.3
52.6
47.4
24.1
30.6
44.2
20.0
35.1
25.2
Mamba-2 (370M, 16-bit)
300B
2360
54.0
46.9
55.4
26.7
32.4
19.0
Bi-Mamba (780M, 1-bit)
105B
4640
58.5
68.0
41.6
52.0
24.3
30.6
45.3
32.4
14.5
Bi-Mamba (1.31, 1-bit)
105B
5780
60.0
68.8
47.3
55.9
48.0
26.3
32.2
48.4
11.7
29.9
12.9
B1-Namba (2.78, 1-bit)
105B
7822
58.0
72.5
54.3
56.1
51.4
29.1
32.6
50.6
10.0
21.9
11.3
Table 6: Performance comparison of Bi-Mamba and full- full-precision pretrained Mamba-2 models in small sizes, All our Bi Bi-Mamba models are better than Mamba-2-1300 16-bit model, which is equivalent to a 2.0B model in 1-bit. Moreover, Bi-Mamba 1.3B and 2.7B models achieve higher performance than Mamba-370M 16-bit model, which is equivalent to a 5.9B model in 1-bit, demonstrating the effectiveness of quantization-awar training instead training a small model directly. The GPU hours are evaluated on A100 S0G.
to full binarization. However, the performance gap between the partial and full binarized models is not significant. For instance, in the Hamba-2-2.77B model, partial binarization achieves an average accuracy of 53.1, while full binarization reduces this to 50.4. Across all model sizes, partial-binarized Bi-Mamba consistently outperforms full- binariia Bi -i-Mamba on most benchmarks, though shows minor performance degradation compared with full-precision models. It also suggests that fully binarization remains highly competitive and does not substantially lag. In terms of perplexity, the fully binarized model also performs comparably to the partial model. For example, the Mamba-2-780n model, the C4 datasel perplexity for full- binarized Bi-Mamba (Fully) is 15.0, compared to 14.4 for hartial-innarieed Bi -i-ma, demonstrating that full binarization does not impose significant perplexity incrcase. These findings highlight that the fully binarized model can maintain competitive performance with the partial binarization model, particularly in terms of perplexity, while still benefiting from greater storage and computational efficiency.
5.3 Comparison with Full Precision Small Models
Instead of binarization, one can train small models with full precision from scratch. We add the performance comparison of Bi-Mamba and small models pretrained with full precision, as shown in Table G We utilize the official pretrained weight from Mamba2, including models with 130M and 370M parameters pretrained with 300B tokens. 130M and 370M models in 16 bits are equivalent to 2.0B and 5.9B models in 1 bit, respectively. From the table, all Bi-Mamba models including the 780M, 1.3B, and 2.7B variants trained on only 105B tokens caakens-outprrorm the 130M full-precision models in average performance. Specifically, Mamba- 2 130M obtains 44.2 accuracy on downstream lasks, with perplexities of 20.0, 35.1, and 25.2 on the Wiki, PTB, and C4 datasets, respectively. In comparison, the smallest Bi-Mamba model, with 780M parameters, achieves 45.3 accuracy on downstream tasks and perplexities of 13.4, 32.4, and 14.5 on Wiki, PTB, and C4, respectively. Moreover, Bi-Mamba 1.3B and 2.7B models achieve higher average performance than the full- precision 370M Mamba-2 model. Full-precision Mamba-2 370M gains 48.4 on average on downstream tasks. The Bi-Mamba 1.3B model outperforms the 370M Mamba-2 model, achieving 48.4 accuracy on downstream tasks. The results indicate that binarization with post-training is better than training a full-precision small model from scratch. Although Bi-Mamba includes extra training efforts due to the distillation compared with the small model, the overall GPU hours are comparable with the model with similar parameter size trained from scratch because of the lower training token requirement. Furthermore, our work is centered on the deployment phase, where binarized models can yield substantial benefits in terms of memory footprint, storage size, and inference officiency-especially in CPU- or edge-constrained environments. As shown in Table g the fully binarized Bi-Mamba-2.7B model achieves 5x memory reduction, 3x faster thronghput, and 3x lower energy consumption. Training-time cost. is higher, but this a one-time investment, amortized over repeated inference runs particularly in latency or resource-constrainee deployment scenarios.
5.4 Storage Efficiency
Bi-Mamba trained with full precision but can be saved as binary values in storage and inference. During training, we use Sign(-) function to obtain the binarized weights. Model binarization can significantly reduce the storage requirement on disk. Following Bi-LLM (Huang et al. 2024), we provide the theoreti-
11

---

Published in Transactions on Machine Learning Research (09/2025)
Model
Model Param.
Storage Size
Compress Ratio
Mamba-2
780M
1.45GB
Bi-Mamba (InProj)
780M
0.63GB
56.5%
Bi Bi-Mamba (Full)
780M
0.22GBB
84.8%
Mamba-2
1.3B
2.50GB
Bi-Mamba (InProj)
1.3B
1.0GGB
59.6%
Bi Bi-Mamba (Full)
1.3B
0.33GB
86.8%
Mamba-2
2.7B
5.03GB
Bi Bi-Mamba (InProj)
2.7B
2.01GB
60.0%
Bi Bi-Mamba (Full)
2.7B
0.55GB
89.0%
Table 7: Storage elficiency Bi-Mamba. Compared with partial binarization, full binarization can reduce the storage size significantly in all scales.
Model
Model Param.
Memory
Tokens/s
Energy/122 Tokens
Mamba-2-16bit
2.7B
5.03GB
24.59 1: 2.26
148.4J
Bi-Nanba
2.7B
1.0GBB
77.81 +: 1.81
44.9J
Table 8: Resource consumption analysis of Bi-Mamba including memory, throughput and energy consumption Mac M4 Pro CPU platform. Bi Bi-Mamba reduces memory consumption significantly by more than 5x and the energy consumption by more than 3 times. The throughput of inference is also increased to 3.16x, from 24.59 to 77.81.
cal storage requirement for our Bi-Mamba in different model sizes compared with full-precision models, as shown in Table 7 For cach parameter size, the storage requirements for the original full- cill-precision Mamba-2 model are substantially larger than those for the binarized Bi-Mamba, including partial and full binarization. Specifically, binaized Bi-Mamba demonstrates the highest compression ratio, achieving reductions of more than 80%. In contrast, partial-binarized Bi-Mamba provides relatively moderate compression, ranging from 55% to 60%. This analysis highlights the efficiency of full binarization in significantly reducing storage requirements while maintaining the model parameter count, making it a highly storage-efficient alternative for large models.
5.5 Resource Consumption Analysis
We present the resource consumption analysis of Bi-Mamba, as shown in Table S We deploy our Bi Bi-Mamba on the CPU of Mac M4 Pro Chip by implementing Bi Bi-Mamba with 1lama. cpp framework The throughput is measured on text generation with 128 tokens generated. The computation over 5 runs with 8 CPU threads. For the energy computation, we utilize the asitor tool record the average CPU power consumption on Mac system, well as the overall computation clock time. The final energy is computed the product of the average power consumption and the program's running time, The memory requirements for each model are evaluated with 2,048 tokens and batch size of 128. Notably, Bi-Mamba reduces memory usage by more than 5x and energy consumption by 3x, while increasing inference throughput to 3.16x ffom 24.59 to 77.81. Moreover, energy consumption reduced by more than 3x, from 148.4 J to 44.9 for 128-token generation. These improvements in resource efficiency highlight Bi-Mamba's 's potential to significantly reduce computational costs in practice.
repeated and calculated
5.6 Results on Multilingual Benchmark
To show the performance on multilingual benchmark, we provide the results of Bi-Mamba on xwinograd in the table below prove the generalization capability of our model, which is multilingual benchmark including English, Chinese, Russian, Japanese, French and on, shown in Table We evaluate all models with 2.7B parameters, except for OneBit, which has 7B parameters. Compared with both PTQ and
https://github... http:/sthw...cmm/mmmr/////a/aaaa https://github.. wow/t/tt//sstttppppppppp
12

---

Published in Transactions on Machine Learning Research (09/2025)
Model
Param.
Xwinograd
Mamba-2- 2----22-0000 2-16bit
2.7B
75.2
GPTQ-3bit
2.7B
64.8
GPTQ Q-iit
2.7B
49.2
OneBit
2.7B
50.4
Bi-Mamba
2.7B
68.0
Table 9: Performance comparison Xwinograd, which is multilingual benchmark.
Zero-shot Acc.
Perplexity
Model
BoolQ
PIQA
WG
ARC-e
ARC-e
OBQA
Avg.
Wiki2
PTB
2-700M
61.5
71.8
54.9
60.2
54.3
28.5
36.2
52.5
11.8
20.0
16.5
Bi Bi-Mamba (No KD)
50.5
65.8
37.8
50.9
39.7
23.8
30.4
42.7
14.9
30.9
15.6
Bi Bi-Mamba (KI-Div)
56.8
66.5
38.1
51.6
39.8
22.7
28.2
43.3
15.0
27.3
15.6
BiiMamba (Phi-3.5)
49.6
66.8
39.0
53.2
40.6
23.7
30.8
43.4
14.5
27.3
16.6
Bi -Mamba
58.5
68.0
52.0
42.4
24.3
30.6
45.3
13.4
32.4
Table 10: Ablation study of Bi-Mamba. This table includes the performance comparison of different teachers and knowledge distillation strategies. Our autoregressive knowledge distillation brings improvement to the binarization-aware training regardless of the choice of teachers.
QAT methods, Bi-Mamba achieves the best performance the multilingual benchmark, even surpassing the GPTQQ3bit model.
5.7 Ablation Study
In this section, we provide the ablation study of Bi-Mamba. As shown in Table 10 we conduct various ablation studies in the 780M model including the model without knowledge distillation (w/o KD in the table), the model utilizing the KL divergence loss as distillation loss (KL Div in the table) and the model using a different teacher model importance of each component, as removing knowledge distillation (KD) or using alternate loss functions (KL Div and Phi-3.5) significantly reduces performance compared to the full Bi-Mamba model, which achieves the best results across all metrics except PTB. Moreover, the stronger performance with Llama2-7B teacher compared with Phi-3.5 a teacher indicates that the better the teacher is, the higher performance the student will achieve. Specifically, Bi-mamba trained with the original autoregressive loss obtains the lowest performance compared with other models trained with KD. The average accuracy on downstream tasks 42.7, which surpassed by the model trained with KL-Div loss, namely 43.3. With a different teacher, Phi-3.5, Bi B1-Mamba achieves similar performance the model trained with Llama-2-7B the teacher, demonstrating the effectiveness of our proposed autoregressive knowledge distillation.
(Abdin et al., (0244). The results demonstrate the
6 Conclusion
We introduce Bi-Mamba, scalable and efficient 1-bit Mamba architecture designed for large language models in multiple sizes: 780M, 1.3B, and 2.7B parameters. We begin by identifying the binarization space within the Mamba architecture. Then, Bi-Mamba models are trained from scratch on large datasets, similar to standard LLM pretraining, using an autoregressive distillation loss. Extensive language modcling experiments show that Bi-Mamba achieves competitive performance, only slightly lower than its full-precision counterparts (e.g., FP16 or BF16), while substantially reducing memory usage and computational cost compared to the original-precision Mamba. This study provided novel, first accessible and low-bit framework with lincar computational complexity, laying the foundation for developing specialized hardware optimized for efficient 1-bit Mamba-based LLMs.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Limitations and Ethical Statements
While Bi-Mamba achieves competitive performance to full-precision models, there may still be trade-offs in accuracy, particularly in complex tasks that rely heavily on nuanced language understanding. Also, full deployment may require specialized hardware to maximize efficiency gains, limiting accessibility on standard hardware setups. On ethical part, reducing model precision could risk oversimplifying nuanced patterns in data, potentially amplifying biases present the training data. Morcover, while Bi-Mamba reduces energy consumption during inference, training binary models from scratch can still be computationally intensive.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahrec, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint 2xix:2444.14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.000 08774, 2023.
Milad Alizadeh, Javier Fernández- Marmandez-Marqués, Nicholas D. Lane, and Yarin Gal. A systematic study of binary neural networks' optimisation. In International Conference Learning Representations, 2019. URL
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael
Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. Advances in Neural Information Processing Systems, 2024.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint urXiv:1308.3432 2013.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 7432 7439, 2020.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint ar 1xXt::205514465, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M. De Sa. Quip: 2-bit quantization of large language models with guarantees. In NearIPS, 2023.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. ArXiu, /tt///900..0040
2019. URL https://app...... wwwppppp.a.amttcccc.ooooooooooee
1655166367
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiu, abs/1803.054451 2018. URL http::::/app.seeaattccssooee htt.aaarrrrrrrroroossD3:322222
Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL
httpp:://gittbb.cmm/// http://www..ggggeeecrcoo//////aa
Tri Dao and Albert Gu. Transformers ssms: Generalized models and efficient algorithms through struc- tured state space duality, 2024. URL https::/arxxvv.rr//bsss 21060
Soham De, Samuel L. Smith, Anushan Fernando, Alcksandar Botev, and et al. Griffin: Mixing gated lincar recurrences with local attention for efficient language models, 2024. UtL/sttps:// http://axxx..og/aa///////// org/abs/2402, 19427
14

---

Published in Transactions on Machine Learning Research (09/2025)
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int():: 8-bit matrix multiplication for transformers at scale. In NewrIPS, 2022a.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Lnt. 8-bit matrix multiplication for transformers at scale, 2022b. URL https://arxix> wttp.a///20..000//////
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. BERT: pro-training of deep bidi- rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (ods.); Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Lánguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-4186 Association for Computational Linguistics, 2019. doi: 10.18653/V1 /V/11/1433 URL https://doi. wtweddi....o/oooeoeeeoeeoeoooeee org/ 10.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024. URL https://aara
org/ 0ts/ab//2400.00611
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate tost-training quantization for generative pre-trained transformers, 2023. URL
17322
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas MuennighofT, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
URL https://zenoo:
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL petps://arxiv..
abs/ abtpa///./.005555
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modcling long sequences with structured state spaces, 2021a. URL http:rr/aarriv.ooggaaa//111..... 00396
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Advances in Neural Information Processing Systems 94: Annual Conference
Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 572 585, 2021b. URL http::/tp///p wewpeeeddiinnnnnniis...... cc/paper/ 00t00////00000000000000000000000
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for llms, 2024. URL https://aaat org/abs/2402.0425
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers rnns: Fast autoregressive transformers with linear attention, 2020. URL
16236
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. ar preprint Nii::2306.0762) 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonza- lez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pp. 611-626. ACM, 2023. doi: 10.1145/3600006.313333 URL http:///abioor/11.111/////0//000
15

---

Published in Transactions on Machine Learning Research (09/2025)
Raymond Li, Loubna. Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint vrXic:2305.06166 2023a.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: hora-fine-luning-wwaar quantization for large language models. arviv prepriul or #Xix:23100006552 2023b.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium's Openorca: An open dataset of gpt augmented flan reasoning traces. https: https://huggingfacee co/ co/daattasetssppeeaOrraaa OpenOrcal 2023.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source llms. arXiv preprinl Xrxi:2331.. 06550, 2023.
Liqun Ma, Mingjie Sun, and Zhiqiang Shen. Fbi-llm: Scaling up fully binarized 1lms from scratch via autoregressive distillation, 2024a. URL https: http:p/://axvv..o////bbass// arggabs/24000 .70093
Shuming Ma, Hongyu Wang, Lingxiao Ma, Tei Wang, Wenhui Wang, Shaohan Huang, Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprint 2xXt:2442.17766 2024b.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models, 2023. URL httt::://xxxi..or/abbb//////// htg//bbs/2305 11627
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated cor- pus of English: The Penn Treebank. Computational Linguistics, 19(2):313 330, 1993. URL https:
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint ar 1609. 1101666899...... 07848, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, 2018. CRL https://api. htap::://ap..smmaniccchoooo wwwp:/opooooooooooooo...oo 52183777
OpenAL. Gpt-4 technical report. arXiv preprint arXi::2303.08774; 2023.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobcidli, Baptiste Pannier, Ebtesam Almazrouci, and Julien Launay. The refinedweb datasct for falcon Ilm: outperforming curated corpora with web data, and web data only. ar preprint 0111:2200.001166, 2023.
Bo Peng, Fric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, and et al. Rwkv: Reinventing rmns for the transformer era, 2023. URL https://arxiv..rgg
13048
Alessandro Pierro and Steven Abreu. Mamba-ptq: Outlier channels in recurrent large language models, 2024. URL http::////axxi..g//aba////////// .1399vv1
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In Proceed- ings of the Sixth Conference on Machine Learning and Systems, ML.Sys 2023, Miami, FL, USA, June 4-8, 2023. misys.org, 2023. URL https::/ .wwpiiinnnssss.............mmmme mlsys. hallllpppeeaaappalllllllpaaaelle fwwes//pppr//222222 ha/oh/ html
16

---

Published in Transactions on Machine Learning Research (09/2025)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAT T blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140))11-67 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. Tn European conference on computer vision, pp- 525-542 Springer, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):90-106, 2021.
Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large language models. arxiv preprint or 0rXiv:2310.0000 00.0000, 2023.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Ommiquant: Ommidirectionally calibrated quantization for large language models, 2023. URL https://aaxii: www.:org/bbbb//sse./ 13137
htttp:://axx...orr/a////333
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL
http:://araav.orrr/ab///222. 13971
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codebooks, 2024. URL https://arxiv orge /bs// 2402. 04396
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL Abstract.
cc cctppppr//111/
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. In NewrIPS, 2022.
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-procision training for largo-scale vision-language models, 2023. URL attps://arxivv.rr//ee 2304. 13013
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38088 PMLR, 2023.
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. arXiv preprini arXiv:2402.1 11295, 2024.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, 2019. URL https://api. htt.://iaeecccaaae.oooo///paaaaa
17

---

Published in Transactions on Machine Learning Research (09/2025)
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprinl arxiv:240 02235. 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint Xrx 01068, 2022.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Appendix
A More Experimental Results
A.1 Full Results on Downstream Tasks
We present the performance dynamics of the 2.7B and 1.3B Bi-Mamba models during training in terms of perplexity and multiple downstream tasks, as shown in Figure 5 and Each configuration demonstrates different trade-offs between model performance and computational efficiency of Bi-Mamba across various datasets.
We can observe that both the 2.7B and 1.3B Bi-Mamba models consistently outperform GPTQ-2bit and BiLLM after reaching certain training stage. Specifically, Bi Bi-Mamba generally exhibits gradual decrease in perplexity, indicating the effectiveness of the training. In some cases such as 3i-Mamba-2.77B on C4 dataset, the perplexity is even better than in the full-precision Mamba models. In downstream tasks such as ARC-C, ARC-E, HS, and PIQA, Bi -amaa consistently outperforms Bi-LLM, indicating that it is more effective low-bit quantization approach for maintaining accuracy across various data sizes.
Additionally, the performance on average increases steadily with more training costs. The overall trend demonstrates that Bi-Mamba provides robust alternative, balancing computational efficiency with compet- itive performance. This makes Bi-Mamba particularly valuable in resourc-constrrineed environments where some trade-off in precision is acceptable. At the current training stage, the models have not yet. fully converged, indicating potential for further performance gains.
30
Full Precision
Full Precision GPTO-3bit Bi Mamba
Full Procision
Perplexity
GPTO-3bit
Perplexity
40
Perplexity
30
GPTO-3Nt
20
Bi Mamba
Bi Mamba
30
20
10
20
10
10
4
6
10
6
8
10
2
6
8
10
Data Chunks
Data Chunks
Data Chunks
(a) Wiki
(b) PTB
(c) C4
40
70
80
90
Full-Precision
Full-Precision
Full-Precision
Full-Precision
GPTQ 2bit
60
GPTQ 2bit
2bit
GP BPTQ-2bbt
Bi-LLM
BI-LMM
Bi-LM
Bi-LLM
Acc.
30
Bi-Mamba
Acc.
Bi.Mamba
Acc.
Bi-Mamba
40
60
40
30
30
20
20
20
40
4
10
4
6
10
6
8
10
2
8
10
Data Chunks
Data Chunks
Data Chunks
Data Chunks
(d) ARC-C
(c) ARC-E
(f) HS
(g) BoolQ
70
60
80
Full-Precision
Sall-Precision
Pull--rrcision
100
GPTQ 2bit
GPTO-2bitt
GPTO-2bit
GPTO-2bit
BI-LUM
Bi-LIM
Bi-LLM
BI-
Acc.
80
Acc.
Bi-Mamba
Acc.
40
Bi-Mamba
Acc.
40
20
30
8
10
2
8
10
2
10
4
Data Chunks
Data Chunks
Data Chunks
Data Chunks
(h) PIQA
(i) WG
(i) OBQA
(k) Avg.
Figure 5: The downstream performance and perplexity curve of Bi -amba- 2.7B with different training costs.
19

---

Published in Transactions on Machine Learning Research (09/2025)
40
60
Full-Precision
Full-rreiiioo
Pull-Prectsion
Perplexity
30
G
Perplexity
GPTQ-3bit
Perplexity
40
GPT
Bi-Mamba
Bi-Mamba
Bi-Mamaa
40
30
20
30
20
10
20
10
1
5
2
6
3
Data Chunks
Data Chunks
Data Chunks
(a) Wiki
(b) PTB
(c) C4
40
80
80
100
Full Precision
Full Precisinn
Full Precision
Full-Pracision
GPTO 2bit
60
GPTO-2bit
60
GPTQ-2bit
80
GPTO-2bit
Acc.
30
Bi-Mamba
Bi-Mamba
Acc.
Bi-Mamba
Bi-Mabb
Bi-LLM
60
40
40
30
30
40
20
20
30
1
2
3
Data Chunks
Data Chunks
Data Chunks
Data Chunks
(d) ARC-C
(e) ARC-E
(f) HS
(g) BoolQ
100
70
Full-Precision
Full-Precision GPTQ-2bit
Full-Precision GPTQ-2bit
60
Full-Precision
GPTQ-206 2bit
GPTQ:bbit
80
40
Bi-LLM
Acc.
Mamba
Acc
Bi Mamba
Acc.
BI Mamba
ACC.
Bi-Mamba
60
30
40
40
20
30
1
2
3
6
3
4
5
1
4
5
1
2
4
5
6
Data Chunks
Data Chunks
Data Chunks
Data Chunks
(h) PIQA
(i) WG
(i) OBQA
(k) Avg.
Figure 6: The downstream performance and perplexity curve of Bi Bi-Mamba-la 1.3B with different training costs.
We provide the full results the ablation study on downstream tasks, as shown Table 10 Without knowledge distillation, the model obtains the lowest average zero-shot accuracy on downstream tasks. With knowledge distillation, Bi-Mamba achieves the highest accuracy downstream tasks, demonstrating the effectiveness of our Bi-Mamba.
A.2 Weight Distribution
We visualized the weight distributions of different modules in Mamba-2 (Orange histograms) and Bi-Mamba (Blue histograms), shown in Figure 8 g and 10 We visualize the weight parameter distributions of different modules in the first (1st), mid (24th) and final (48th) layers of the corresponding 780M models. The input and output projection matrices are the values after re-scaling. Each pair of histograms compares how Bi-Mamba modifies the distribution of weights in different modules, no matter whether the module is binarized or not, illustrating the impact Bi-Mamba on each module.
Specifically, in the first layer, the weight distribution of the original Mamba-2 such as Convid. Convid.weigtt Convid.ias and D are tightly concentrated, indicating the strong focus on specific values. In contrast, the weight distribution in Bi Bi-Mamba in the first layer is much divergent with additional peaks in the his- tograms such as in A-log, Convid, bias and D. The divergent weight distributions in Bi-Mamba suggest that Bi-Mamba intentionally captures broader values in the initial layers to retain sufficient information for bina- rized modules.
With more variability the initial layers, Bi-Mamba can process the diverse initial features even low- bit precision. In the mid-depth layers such as the 24th layer, the weight distribution of both the original Mamba and Bi Bi-Mamba show similar patterns in the first layer. However, the divergence is moderate
20

---

Published in Transactions on Machine Learning Research (09/2025)
2.0
1.2
0.7
20
(a) A-log-FPH FP16
(b) weight.weight-FF FP16
(c) D-FPI6
(d) In-proj-FPP FP16
12-
+803
644
(e) A-log vanilla loss
(f) Convld.ae weight vanilla loss
(g) vanilla loss
(h) In-proj vanilla loss
14-
0605
68:
64
04
-20
-25
-38
-08
100
35
1309
0250 8355
032
3350
E315 odde
6:00
0.02
0.6
(1) A-log KT.loss
(i) wd.igh.weightt KT=loss
(k) KT.lloss
(I) In-proj KT=loss
36
(m)
A-log in -aabb
(n) Convld weight in B1 Mamba
(o) B1-Mamba
(p) In-proj B1-Mamba
Figure 7: Distribution comparison of precision FP16 and different training objectives. It shows that different
training objectives generate similar weight distributions.
compared with the divergence in the first layer. This suggests that in the intermediate layers, Bi-Mamba can refine the intermodiate representation with generalization with binarized weights. Finally, in the last layer, the divergence patterns also remain while the distribution much narrower compared with previous layers, reflecting more concentrated range of values.
We also provide the weight distribution of different training objectives, shown in Figure 7 With bina- rization, different training objectives including vanilla loss, KL-Divergence loss and our autoregressive loss, generate similar weight distribution after training.
The focused distribution helps to model to generate stable and reliable final representation. In all, our Bi-Mamba includes much wider distribution capture more information at the beginning stages while the distribution tends to be more centralized progressively to output stable final results.
21

---

Published in Transactions on Machine Learning Research (09/2025)
(a) A-log
(b) Convld.weight
(c) Convld.Bia
(d) D
100
-24 424
(c) A-log Bi Bi-Mamba
(f) Conv Id. weight Bi-Mamba
(g) Couvld. Bias Bi-Mamba
(h) in Bi-Namba
1.0
1.15
1.45
839
035
825
01
33
15
20
80
16
540
-55
-18
03
10
ii) Abias
(j) Norm
(k) In-proj
(l) Out-proj
540000
14000
(m) Abias in B1-Mamba
(n) Norm Bi-Namba
(o) In-proj in Bi-Mamba
(p) Out-proj in Bi-Mamba
Figure 8: Distribution Comparison of each weight in Mamba-2 and Bi-Mamba modules at the 1st layer
B Generation Case
We provide generation cases from our models in different scales including 2.7B, 1.3B and 780M, and other baseline models including GPTQQ-bit, GPTQ-2bit and Bi-LLM as shown in Figure II 12 and 13
It observed that Mamba-2 consistently produces coherent answers with meaningful semantic information but often repeats the content excessively. Bi-Mamba, while also retaining coherence and context after bi- narization, also shows repetition, especially in phrases. Nevertheless, Bi-Mamba is more robust than other baseline methods in preserving relevant information.
Post-training quantization methods, particularly at lower bit levels (e.g., GPTQ-2bit and 1bit BiLLM), tend to produce meaningless or garbled content. Specifically, GPTQ-3bit occasionally provides coherent starts but quickly devolves into repetitive nonsensical text, indicating limited content understanding and generation ability after quantization. Other low-bit. settings such GPTQ-2bir and Bi-LLM generally fail to maintain coherence for generation, resulting in meaningless symbols generation, especially in smaller models.
22

---

Published in Transactions on Machine Learning Research (09/202)
3400
(a) A-log
Congld.weight
(c) Convld.Bias
(d)
500
000
-0.9
-2.0
-2.9
-27
000
03
(e) A-log in B1-Mamba
(f) Conv weight in B1-Mamba
(g) Conv 1d. Bias in B1 91-Mamba
(h) D in B1 1i-Mamba
245
355
125
1.00
0.80
611
819
825
(i) Abias
(j) Norm
(k) In-proj
(l) Out-proj
04
10000
(m) Abias
Bi-Mamba
(1) Norm Bi-Namba
(o) In-proj in Bi-Mamba
(p) Out-proj in Bi-Mamba
Figure 9: Distribution Comparison of each weight in Mamba-2 and Bi Bi-Mamba modules at the 24th layer.
23

---

Published in Transactions on Machine Learning Research (09/2025)
0907
A-log
Convld.wr weight
(c) Conv Ld.vld. Bias
(d)
3.
30-
255
25-
20-
13-
280
150
10-
(0) A-log in Bi 31-Mamba
(f) weight in Bi-Mamba
(g) Convld Bias in B1-Mamba
(h) D in Bi Bi-Kamba
1.0
L00
219
i1) A bias
(j) Norm
(k) In-proj
Out-proj
200
04-
03
-2.8
-1.0
+2.6
6.35
6.0
0.8
6.6
-0.05
610
(m) Asias in B1-Mamba
(n) Norm in Bi-Namba
(0) In-proj in Bi Bi-Mamba
(p) Out-proj in Bi Bi-Mamba
Figure 10: Distribution Comparison of each weight in Mamba-2 and Bi-Mamba modules at the 48th layer.
Prorpt: world's
painter?
Marba-2
VnLoonardo da Vinci. InLeonardo da Vinci was born in 1452 in the Italian city of Vinck. Antle was the son of notary, Inke brilliant artist and scientist. trtle was a grest inventor and a great engineer. Vitte was great musician and a great poet. innle was great writer great philasopher.
great sathesaticlan
great scientist. great susician and great Poet. was great writer and great
great
great
ongineer, Vntte was great painton and great sculptor. AnHe philosopher, Viite was great musician and grest poet. (REPEAT)
Leonardo Vinciin2.. Leonardo Vinctiin.. Leonardo Vinciins.. (REPEAT)
Vinciie.. Leonardo Vinciios, Leonardo Vinciiin6. Leonardo Vinciv:..
(IsProj)
Bi-Manba (Fully)
1. Leonardo da Vinciin2. Leonardo da Vincilina, Leonardo da Vinciln4. Leonardo da Vincilins, Leonardo da Vincilnn.. Leonardo da Vinciln7. da vincive8. (REPEAT)
GPTQ
The nost fasous painter in the world is vincent Van Gogh, and the nost fasous artistt in the world is Vincent Van AnThe nost fanous artist in the world is Vincent Van 6agh, and the rost fanous artist in ktuskaa' is world is Vincent van Gogh, In rost fanous kttssk
(3-b1t)
own painting, nost Fasous actist in kruska kruskaas world is ktuska's own paintingg ktuska's painting is kituska's is own painting, (REPEAT)
artist
own painting,
GPTQ
/bb, fitting C- (unC of price in pil dstin, WHE, IntovnaccordMMerrnn ( andy... 3t (in and-sox2 on iddaa An
(2-bit)
BILLM
Ve OufffdER is answors int Vndeerin jowssurrRoterraaaee (NEANINGLESS)
Figure 11: The generation cases for 2.7B models.
24

---

Published in Transactions on Machine Learning Research (09/202))
Pronpt: How did Julius Caesar die//nknswer:
Masba-2 2
lnke staboed Rosan soldier named Sextus inintt stabbed Ronan soldier naned Sextus Porpey, AniiThh
is: iniiitt was stabbed by Roman soldier named Sextus Pospey, ininthe word "stabbing" is verb, and ite verb "stabbing" is noun. Aonnnnn was stabhed by Rosan soldier nased Sextus intaThe "stabhinn" varb,
verb "stabiinng
Veon stabbed Rosan soldien
Sextus Porpey.
Bi-Manba (InPro5)
1. He was killed by Rowan soldter. 1n2. He was killed by Ronan soldien, \na. He was killed by a Roman soldien, 1n4, He was killed by soldier, kns.. (REPEAT)
years ago iotesser:::: 188 years ago 160 years ago agtiuwss years. agointnswers 180 years agoi (nlnnsuee 100 years years ago'ntnsser:: years ago (REPEAT)
(Fully)
Vnhow did Jullus Caesar die?: VrHow did Dulius Caesar die?: ViHow did Julius Coesar die?: (ntn Vnhow did Jullus Caesar die?: VnHow did Julius Caesan die?: VnHlww
(3-b15)
Caesan die?:
Julius Caesan die?: (REPEAT)
Inafterinfallever in MC1 N-in, I-inn,A (fffe (nothingyi] orioni] box ocks Vnnumbers <T detaluttaatrroheen
(2-b15)
portianlerttsorvvesssppooo (MEANINGLESS)
BiLIN
Inln"Juluccan", Inin"auctniv" "alnknn" (REANERGLESS)
Figure 12: The generation cases for 1.3B models.
Prorpt: Give three tips for staying healthy.\nnnsserr AnAnswer:
Marba-2
\nd. Eat healthy An2t Exancise regularly. (n3. Get enough sleop, Intnnnswer:: \nd. Eat a healthy diet. 1n2, Exercise regularly. Vra. enough sleep.Aninteewwe
healthy diet. 1n2, Exercise regularly. enough sleep. (REPEAT)
81 B1-Parba [InPro3)
1. Get enough k1e,p. fet enough exercise. (n3. Get enough rest, Intnswrr 1. det enough sleep. 2. Get enough exercise. 5, Get enough rest. Antnswrr:: Get enough sleep. Get enough rest. 3. enough exercise. in (REPEAT)
B1 Ba-barba (Fully)
1. healthy An2,
healthy diet.
healthy diet. healthy diet. An5. Fat healthy diet.
healthy A177.An7. Eat healthy diet. ina. (REPEAT)
GPTQ
vnd. vogetarian while,
fattie, little
(3-b1t)
(NEANINGLESS)
all balcon Uphabetf]s,>,, tos- confliscame-nnn(.-. -V- andinly and alleruckland e/ and, at E' E'ipel J/rains Gesch- bath/ forin- aceutics. arrangatace (HEANINGLESS)
(2-bit)
An.An.nn An]ekncu An... Jan...... (MEANINGLESS)
Figure 13: The generation cases for 780M models.
25


Reviews:
[{"review_id": "kLdU0TYdY3", "reviewer": "Reviewer_zviP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* **Novelty**: This is the first work to successfully train 1-bit binarized Mamba models from scratch using binarization-aware training, filling a clear gap in the current literature on state space model quantization.\n\n* **Efficiency**: Bi-Mamba demonstrates significant improvements in storage (up to 89% compression), memory usage, inference throughput, and energy consumption while maintaining competitive performance.\n\n* **Hardware Relevance**: The work lays a foundation for low-bit model deployment on edge devices or CPUs, and promotes future hardware-software co-design efforts.\n\n**Weaknesses**\n\n* **Instruction-Tuning Absence**: The models are not instruction-tuned, which limits their competitiveness on reasoning and coding benchmarks like GSM8K and HumanEval. This could underrepresent the potential of Bi-Mamba in real-world applications.\n\n* **High Training Cost**: Despite inference efficiency, training from scratch requires substantial resources (e.g., thousands of A100 GPU hours), which may hinder reproducibility and accessibility.\n\n* **Limited Generalization Claims**: While the model performs well on standard benchmarks, it's unclear how robust Bi-Mamba is in more diverse real-world settings or multilingual contexts.\n\n* **Dependency on Teacher Model**: Performance is heavily reliant on the quality of the full-precision teacher model (e.g., LLaMA2-7B), and the impact of teacher quality on final model performance is not deeply explored.\n\n* **Use of Simulated Binarization During Training**: During training, full-precision representations are still used for backpropagation. While this is standard in binarization-aware training, it may overestimate efficiency during training compared to full binary pipelines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Bi-Mamba introduces the first 1-bit binarized Mamba architecture for efficient large language models. It selectively binarizes the linear input and output projection layers in Mamba-2, which account for over 90% of model parameters, while preserving embeddings to retain performance. Using binarization-aware training with a high-precision teacher model and FBI-Linear modules, Bi-Mamba achieves strong results across 780M, 1.3B, and 2.7B scales. It outperforms existing 2-bit and 3-bit baselines like GPTQ and Bi-LLM in both downstream accuracy and perplexity, with minimal performance loss compared to full-precision models. Bi-Mamba reduces model size by up to 89%, lowers energy and memory consumption by over 3× and 5× respectively, and improves inference speed by over 3×, demonstrating its potential for deployment in resource-constrained settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical Changes**\n\n1. **Instruction-Tuning Evaluation**\n   Add results or discussion on instruction-tuned Bi-Mamba models (or clarify future plans for this). Many real-world applications rely on instruction-following capabilities, and current results on GSM8K and HumanEval are uncompetitive partly due to this omission.\n\n2. **Clarify Training Cost vs. Efficiency Trade-off**\n   The paper emphasizes inference efficiency but requires heavy training resources (e.g., >7000 GPU-hours). A more transparent discussion of the cost-performance trade-off is needed to justify the practical value of training binary models from scratch.\n\n\n**Recommended (Non-Critical) Changes**\n\n3. **Expand Discussion on Teacher Model Quality**\n   Since Bi-Mamba relies on knowledge distillation from LLaMA2-7B, it would be helpful to analyze or discuss how the choice or quality of the teacher affects the final performance.\n\n4. **Include Real Binary Deployment Benchmarks**\n   Although the paper simulates binarized weights during training, a more detailed account of actual 1-bit inference (e.g., with hardware-efficient kernels) would strengthen the claim of practical applicability.\n\n5. **Improve Clarity of Figures**\n   Some figures (e.g., performance trends in Figure 4) can benefit from clearer legends, axis labels, or simplified layouts for readability.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jDCZn6b10y", "reviewer": "Reviewer_hjFX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have described the strengths of the paper under the SUMMARY; here I list weaknesses.\n\n\nMAJOR: What exactly have we learned from this study?  The field as a whole has moved toward QAT and away from PQT (whenever the cost of training is not a factor); this study simply applies that lesson to Mamba.\n\nPerhaps the punchline is supposed to be the precise technique used for binarization, FBI-LLM.  But there are obstacles to this.  First, the scheme is not new to this paper (it was introduced in another paper last year, but applied to transformers).  Second, although the technique differs on some points from other QAT schemes (e.g., FBI-LLM learns real-valued scaling and shift parameters *severally* for each column of the matrix), we don't get much intuition about which aspects of the technique are important.  In particular, no comparison is made to any other QAT technique applied to Mamba models.\n\nThe authors also argue that their Bi-Mamba outperforms quantized transformer models, which would certainly be interesting.  But the results reported on this question have problems:\n\nFirst of all, FBI-LLM is evidently (slightly) more effective when applied to Mamba than to when applied to transformers, it's original application.  This is potentially interesting---is there an explanation?\n\nSecond, in comparing to BitNet, the authors use the reported results from that paper [Ma et alia 2024].  But that version of BitNet quantizes the activations to 8 bits (i.e., in addition to quantizing the weights), in contrast to the full-precision activations in Bi-Mamba; and was trained only on 100B tokens of the RedPajama dataset, in contrast to the 1.2T tokens used to train the Bi-Mamba models.  As it stands, the BitNet models have very similar accuracies to Bi-Mamba, so it seems quite plausible that on equal footing, BitNet would outperform Bi-Mamba.  Furthermore, perplexities are not reported in this MS for BitNet, ostensibly because they were not reported in the original paper---but they are, in Table 1, and indeed are lower than those achieved by Bi-Mamba.\n\nThe other point of comparison for QAT transformers is OneBit, but perplexity isn't reported for it.  Why not?  (The authors indicate that they have access to the weights from that model.)\n\n\nMINOR:\nI realize that this area of the literature does not much make use of statistical tests, but it's never too late to start.  Surely there is some variation in performance across multiple re-trainings (or even just on different subsets of the test data), and ideally the authors would produce *distributions* of accuracies (and perplexities), and then compare the distributions for different training schemes with a t-test or the like.\n\nCan the authors confirm that when applying the PQT schemes (GPTQ, SqueezeLLM, etc.) to Mamba, they only quantized the input and output projections, as in Bi-Mamba? \n\nWhat is the bold font in Table 3 supposed to indicate?  Bi-Mamba's zero-shot accuracy is bolded even when it is not the highest.\n\nRows in the table that are copied from published papers (as for BitNet, e.g.), as opposed to reproduced by the authors, should be marked as such in the table.\n\nSection 5.2: I'm not sure we learn much from this section.  Table 5 mostly just repeats the data from Table 3.  The only addition is a Bi-Mamba in which only the in-projection has been binarized.  Is the point simply that binarizing even less, namely not binarizing the output projection, doesn't really improve performance?  This is a somewhat arbitrary datum.  (More useful might be to see a plot of, say, performance vs. compression rato---not just for these two point, but also for binarizing *more* of the network.  Then we would presumably see clearly that the authors have chosen the right fraction of the network to binarize.)  It's also confusing that the Bi-Mamba model in which (only) the input and output projections are binarized is here called \"fully binarized.\"\n\nSection 5.3: The Bi-Mamba models achieve slightly higher accuracies than the comparable small Mambas, but require far more training time.  Is the energy trade-off worthwhile here?\n\n\nClarity: Some text is sometimes repetitive (the quadratic computation cost of transformer is repeated several times) and could be much reduced, and there are some confusing sentences (I note some below).  But the bigger issue is the grammatical errors, which can be found in roughly half of the sentences.  I think it would be unfortunate if this manuscript (whether it is published in TMLR or elsewhere) didn't receive proper attention because it is hard to read.  I strongly encourage the authors to have the MS read and edited by a native speaker of English---or, if that is impossible, at least an LLM.\n\n\"...on data volume as regular LLM pertaining using an autoregressive distillation loss\"\nI don't understand this clause.\n\n\"Prior works....\"  In this context, \"work\" is a mass noun and shouldn't be pluralized.\n\n\"tenary weights\" -> \"ternary weights\"?\n\n\"h_t then represents the derivative of the hidden state...\"\nWhy write it as h_t?  If Eq. 1 is supposed to describe the continuous-time dynamics, it should just be written $$\\dot h_t = Ah_t + B x_t$$ (etc.).\n\n\"Since B and C are derived from the linear projection of each layer’s inputs, and the language model head is tied to the embedding layer, these modules are ignored in our design.\"\nI don't understand this sentence.\n\netc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors convincingly show that, at least for simple multiple-choice data sets, quantization of Mamba-class LLMs should be carried out with quantization-aware training (QAT) rather than post-training quantization (PTQ).  In particular, they show that a recently introduced QAT binarization scheme called FBI-LLM yields better zero-shot accuracies and perplexities than those achieved under various PTQ schemes.  The authors also support their argument in favor of this scheme with several ancillary analyses, showing (e.g.) that comparably sized (on disk), 16-bit, Mamba models perform worse than binarized Mambas.  The model-compression ratios are on the order of 85--90%, and the binarization scheme allows (in theory) for the replacement of many multiplications with binary operations, so the results have important implications for low-energy, storage-efficient LLMs.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should (critical)\n\n- clarify what we learn from the success of FBI-LLM applied to Mamba models, including comparing it to other quantization-aware schemes applied to Mamba\n- make an apples-to-apples comparison for BitNet (or as close as possible, since BitNet uses 1.58 bits rather than just 1)\n- report the perplexities for OneBit\n\nIdeally, they would also (strengthen)\n- use statistical tests when comparing model performance\n- address my other minor concerns", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oDM1r4q5v2", "reviewer": "Reviewer_VSpb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n + extensive experimental section on various parameter sizes (760M, 1.3B, 2.7B), many performance data sets, several quantized comparisons, including quantized GPT models.   \n\n + clearly better accuracy (especially as factor of space saving) compared to other models.  \n\n + nice discussion and empirical results on space and energy savings.  \n\n + Seems to be important step towards effective highly quantized models.  If this is a direction the field will go to (for space, energy savings) then this paper will be important.  \n\n + Does a nice job trying to explain the Mamba architecture to make the paper more self-contained.  \n\n\nWeaknesses:\n - Although, this appears the best performing of the quantized methods, it appears to consistently do worse (and significantly worse) than the non-quantized models.  \n\n - Table 4 seems interesting, since Bi-Mamba sometimes outperforms Mamba-2-16bit (does this means it improves upon the non-quantized model!?!).  But while referenced twice in the text, this Table is not clearly explained.  \n\n - Some of the visualizations felt pointless.  In particular the lifted pie charts in Figure 2, as well as the distribution plots in that figure.  And the radio plots in Figure 3.  \n\n\n\nNotes:  \n - The second paragraph (page 1) uses the term \"exponentially\" in a way I think does not describe an exponential phenomenon-- maybe polynomial?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies how to compress parameter variables to 1 bit within the State-Storage Models, specifically adapting the Mamba architecture.  The paper demonstrates improved performance over other 1 or 2/4 bit compression schemes for comparable models, but a drop in performance compared to the non-compressed models. \n \nThe Mamba architecture this paper studies is from a couple years ago, and while much of the work is un-published, it has 1000s of citations.  It is based on a Selective State Space Model that is a potential replacement for transformers that has linear (as opposed to quadratic) dependence on the context window.  \nThis paper shows that the architectures are also more amenable to quantization than other models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please explain Table 4 better, improve or remove un-needed visualizations, and rethink the work \"exponentially\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)