SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba
Authors: Yulong Huang
Jianxiong Tang
Chao Wang
Ziyi Wang
Jianguo Zhang
Zhichao Lu
Bojun Cheng
Luziwei Leng
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have achieved remarkable performance across tasks but remain energy-intensive due to dense matrix operations. Spiking neural networks (SNNs) improve energy efficiency by replacing dense matrix multiplications with sparse accumulations. Their sparse spike activity enables efficient LLMs deployment on edge devices. However, prior SNN-based LLMs often sacrifice performance for efficiency, and recovering accuracy typically requires full pretraining, which is costly and impractical. To address this, we propose SpikingMamba, an energy-efficient SNN-based LLMs distilled from Mamba that improves energy efficiency with minimal accuracy sacrifice. SpikingMamba integrates two key components: (a) SI-LIF, a signed-integer spiking neuron that preserves semantic polarity through signed multi-level spike representations. (b) A training-exclusive Smoothed Gradient Compensation (SGC) path mitigating quantization loss while preserving spike-driven efficiency. We employ a single-stage distillation strategy to transfer the zero-shot ability of pretrained Mamba and further enhance it via reinforcement learning (RL). Experiments show that SpikingMamba-1.3B achieves a 4.76$\times$ energy benefit, with only a 4.78\% zero-shot accuracy gap compared to the original Mamba. The model achieves a further 2.55\% accuracy improvement after RL, narrowing the performance gap from 4.78\% to 2.23\%.

Chao Wang" Published in Transactions on Machine Learning Research (01/2026)
Spiking Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba
Yulong Huang"
The Hong Kong University of Science and Technology (Guangzhou)
Jianxiong Tang'
jintang@ecttn.eedu.c
Deportment of Computer Science, City University of Hong Kong
Monnge2023@mailssuteer edu.cn
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen
Ziyi Wang
51265901 www.:11.0..0....u..b......
School of Computer Science and Technology, East China Normal University, Shanghai
Jianguo Zhang
zhangig Bestiighsssshhheee edu.cn
Department of Compuler Science and Engineering, Southern University of Science and Technology, Shenzhen
Zhichao Lu
zhichao caachaoo.u@@eity.eede..
Department of Computer Science, City University Hong Kong
Bojun Cheng
bocheng@hbust-gr.comm www.ee.eee..ooo
The Hong Kong University of Science and Technology (Guangzhou)
Luziwei Leng
lenglusturrsShuaues.ccom
ACSTab, Huawei Technologies Co., Ltd., Shenshen
'Theee authors contributed equally. "Corresponding authore.
YL Huang did this work during internship at Huawei ACSLab.
Reviewed on OpenReview:
Abstract
Large Language Models (LLMs) have achieved remarkable performance across tasks but remain energy-intensive due to dense matrix operations. Spiking neural networks (SNNs) improve energy efficiency by replacing dense matrix multiplications with sparse accumu- lations. Their sparse spike activity enables efficient LLMS deployment edge devices. However, prior SNN-based LLMs often sacrifice performance for efficiency, and recovering accuracy typically requires full pretraining, which is costly and impractical. To address this, we propose SpikingMamba, an energy-efficient SNN-based LLMs distilled from Mamba that improves energy efficiency with minimal accuracy sacrifice. SpikingMamba integrates two key components: (a) SI-LIF, signed-integer spiking neuron that preserves semantic polarity through signed multi-level spike representations. (b) A training-exclusive Smoothed Gra- dient Compensation (SGC) path mitigating quantization loss while preserving spike-driven efficiency. We employ a single-stage distillation strategy to transfer the zero-shot ability of pretrained Mamba and further enhance it via reinforcement learning (RT.). Experiments show that SpikingMamba-LL33t achieves a 4.76x energy benefit, with only a 4.78% zero- shot accuracy gap compared the original Mamba. The model achieves a further 2.55% accuracy improvement after RL, narrowing the performance gap from 4.78% to 2.23%.

---

Published in Transactions on Machine Learning Research (01/2026)
1 Introduction
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and domains (Li et al., 2025; Jiang et al., 2023; Liu al., 2024; Team al., 2023; Touvron et al., 2023; Chen al., 2024a; Geiping al., 2025; Jimenez et al., 2023; Wang et al., 2024b). However, their computational and energy requirements grow rapidly with scale, limiting their deployment in latency and energy constrained environments (Argerich & Patiñoo-Mai Martínez, 2024).
These limitations stem largely from the fact that most modern LLMs are built. on decoder-only Trans- former architectures, where the self-attention mechanism incurs quadratic O(L2) complexity during infer- ence. Transformer-basee LLMs exccute inference with a prefill stage followed by an autoregressive decoding stage (Touvron et al., 2023). In both stages, the self-attention mechanism suffers from quadratic O(L?) complexity with respect to sequence length L (Yang et al., 2023). Additionally, the autoregressive decoding requires maintaining growing Key-Value (KV) cache during inference, leading O(L) memory overhead. These limitations lead to inefficient computation, especially for long-conted context inference.
To address the quadratic cost of attention, the recently proposed Mamba architecture (Gu & Dao, 2023; Dao & Gu, 2024) replaces altention with a selective state space model (SSM). This design achieves linear- time OL) sequence modeling while requiring only O(1) memory during autoregressive generation (Wang et al., 2024a), eliminating KV caching and improving efficiency for long-context and edge scenarios. However, despite these architectural improvements, Mamba still relics heavily dense matrix multiplications, making energy consumption critical bottleneck, especially on battery-powered embedded devices (Xing et al., 2025).
To address this challenge, we propose SpikingMamba, a spiking large language model that combines the architectural efficiency Mamba with the energy-saving advantages of SNNs. Rather than training spiking model from scratch, we distill SpikingMamba from a pretrained Mamba2 using single-stage self-distillation strategy. However, naive LIF neurons struggle to preserve semantic information during distillation, leading to degraded feature representations and suboptimal performance. To overcome this, we design SI-LIF neuron and a Smoothed Gradient Compensation path for SpikingMamba modeling. Together, these components improve distillation quality while maintaining the sparse, low-power nature of SNNs. Our contributions are summarized:
We propose SpikingMamba, a recurrent spiking LLMs that integrates a novel Signed Integer Integrate-and-Fire (SI-LIF) neuron. This design enhances feature representation for distillation by introducing negative activations, forming a ternary spiking scheme that better captures the mag- nitude and polarity of semantic representations.
We introduce the Smoothed Gradient Compensation (SGC) path to mitigate quantization-induce representational fidelity loss from spikes. This auxiliary path operates exclusively during training, preserving spike-driven inference advantages. When applied to just 3 layers, SGC path improves 1.3B SpikingMamba's zero-shot accuracy by 0.8% on average.
We propose Singlo-Stage Sclf- strategy, combining KL divergence on logits with hidden- state aligument loss, effectively transferring zero-shot ability from Mamba without full pretraining. Additionally, SpikingMamma is compatible with RL methods such as DPO (Rafailov et al., 2024) and KTO (Ethayarajh al., 2024), which further enhance performance with tiny cost.
In experiments on a -.BB-parameter model, SpikingMamba achieves a ~4.76 energy benefit over Mamba2, while maintaining commonsense zero-shot accuracy within 4.78% degradation margin after distillation. With reinforcement learning, SpikingMamba achieves a further 2.53% accuracy improvement, using fewer than 10 GPU-hours.
2

---

Published in Transactions on Machine Learning Research (01/2026)
2 Related Work
2.1 Mamba and Distillation.
The Mamba (Gu & Dao, 2023) architocture and its successor Mamba2 (Dao & Gu, 2024) have been proposed as efficient recurrent alternatives to Transformers for sequence modeling. Leveraging structured state space models (SSMs), they eliminate ( quadratic attention and O(T) KV caching, while achieving performance par with or better than Transformers of similar size (Wang et al., 2024a). Their recurrent structure and constant memory make hamba-based LLMs well-suited for edge deployment. Therefore, recent studies such as M.LMA (Wang et al., 2024a), LoLCats (Zhang et al., 2024) and Llamba (Bick et al., 2025) explore distilling Transformer-base LLMs into Mamba-base based models with minimal training cost.
2.2 SNNs for LLMs and Mamba
Improving LLM efficiency remains fundamental challenge for edge deployment (Qu et al., 2025; Cai el al., 2024). Quantization reduces model size and compute, yet activation quantization is difficult due to high- magnitude outliers in LLMs, often inducing severe accuracy loss under low-bit settings (Lin et al., 2024; Yu et al., 2025). Additionally, quantization does not alleviate the inherent energy cost of dense matrix multiplications. In contrast, spiking neural networks (SNNs) utilize binary, event-driven activations that exhibit both spatial and temporal sparsity. For example, prior work reports spike rates as low as 20% (Tang et al., 2024a), this sparsity translates to significant savings in computation and I/O bandwidth, replacing dense MAC operations with sparse Accumulations (AC). These properties make SNNs promising direction for energy-efficient inference.
Recent work on spiking language models follows two main paradigns: converting pretrained ANNs or train- ing SNNs from scratch. Conversion-basee methods, such as SpikingBERT (Bal & Sengupta, 2024), Spike- BERT (Lv et al., 2023), and SpikeLLM (Xing et al., 2025), reuse pretrained weights but require many inference timesteps for spike accumulation, increasing lateney and energy cost. Alternatively, models like SpikeGPT (Zhu et al., 2023), SpikingSSMs (Shen et al., 2025), and SpikeLM (Xing et al., 2024b) are trained from scratch and support sparse computation, but incur high training costs to scale larger models. Table I compares these models with different training strategies, neuron step, inference complexity and zero-shot ability.
Recent efforts have explored combining SNNs with Mamba, primarily in vision tasks such as video under- standing (Li et al., 2024), event-based classification (Qin & Liu, 2024), action recognition (Chen et al., 2024b), and point cloud processing (Wu et al., 2025). These methods vision-centric and typically replace the activation function with spiking neurons or stack Mamba blocks alongside SNN modules. However, Mamba was originally designed for language modeling, and its integration with SNNs for energy-efficient LLMs remains unexplored.
Table 1: Comparison of SNN-based language models. SpikingMamba is the first linear-time SNN L.LMM at the pillion-parameter scale supporting distillation, low-latency inference, and reinforcement lcarning. T denotes the number of repeated computations per token and D denotes an integer range. A larger T leads to higher inference latency.
Model
Size
Training
Step (T x D)
Infer. Comp. Zero-Shot
SpikcBERT (Lv et al., 2023)
M
2-stage distill
4 1x1
O(4I²)
SpikingBETT (Bal & Sengupta, 2024)
50 M
3-stage distill
125 x 1
O(125
SpikeLM (Xing et al., 2024b)
194 M
2-stage distill
4 #1
O(4L²)
SPikeGPT (Zhu al., 2023)
216 M
Direct Finetune
1x1
O(L)
SpikingSMMs (Shen al., 2025)
M
Direct
1x1
O(L)
SpikeSSMs (Zhong et al., 2024)
75 M
Direct
1x1
O(L)
X
SpikeLLM (Xing et al., 2024a)
7 B 70 B
Quantization
2
O(32L²)
SpikingMamba (Ours)
1.3 B
1-stage distill RL
1x
O(4L)
3

---

Published in Transactions on Machine Learning Research (01/2026)
3 Preliminary and Motivation
Notation. Throughout the paper and Appendix, vectors and matrices are denoted by bold italic lowercase and bold capital letters, respectively (e.g,, æ and W). For a fully connected layer, y = W E Rdamt where We E Rdom xdu and æ E R The symbol W., denotes the i-th column of matrix, and (E (ilcondition) W. indicates the column-wise sum over indices satisfying the condition. The subscript 2; refers to the variable æ at the t-th token, while the brackct æ[t] denotes the f-th micro-timestep within neuron dynamics per token.
3.1 Spiking Neurons
The Leaky Integrate-and--Free (LIF) is one of the most widely adopted neuron models for modeling spiking behaviors in SNNs (Huang et al., 2024b; Meng et al., 2023). At each timestep t, the neuron at layer integrates its postsynaptic current e'(t] with its membrane potential from the previous timestep u' [t 1]
(Eq 1):
u' 1' BB ( 1] VI 1J) + dM
(1)
s' [t] = [t] Via);
(2)
where 3 E (0, 1) the decay factor mimicking the leaky mechanism. The postsynaptic current c'It] is computed by the synaptic operation * (cither fully connected or convolutional) between the weight matrix W' and the presynaptic spikes [1] (i.e., cd(4] = W * (()) When the membrane potential u u'1[] exceeds threshold Vin, the neuron emits binary spike sf [t] {0, 1} according to the Heaviside step function O(x), which outputs 1 for 2 0 and 0 otherwise, as shown in Eq 2.
To address the quanlization errors inherent in standard LIF neurons, Luo et al. (2024) proposed the Integer Leaky Integrate-and-Fire (I-IF)) neuron. I-LIF emits integer-valued outputs during training and converts them into binary (0/1) spikes during inference, thereby enhancing training dynamics while preserving spike- driven inference. Specifically, the spiking function in Eq modified as:
s' {11] Clip (Round (,),
where Round(.) denotes the rounding function, Clip(æ, min, max) restricts to the range [min, max], and D is a hyperparameter indicating the maximum integer output. By producing discrete integer outputs during training (Yao et al., 2025), I-LIF enhances optimization performance while maintaining efficient 0/1 spike- driven inference at test time, as demonstrated by Tuo et al. (2024). The concurrent work SpikingBrain (Pan et al., 2025) produces integer-valued activations for efficient forward inference. However, this work is designed exclusively for quantized inference and does not verify end-to-end training.
3.2 Motivation
In Mamba2, over 90% of parameters (c.g., 92.3% in 1.3B) are in the input/ Linear layer, corresponding to Wi. and out respectively (Further details are provided in the Appendix A and B). The embedding layer shares parameters with the causal head, but its relative size shrinks as the model scales. Binarizing embeddings harms token representation and leads to performance loss. Therefore, only focus the Linear layer due to the computations ional dominance of the Linear layer in Multiply-Accumulate (MAC) operations. To reduce this computational cost, we insert spiking neurons upstream of input and output projections, converting dense MACs into sparse Accumulation (AC) operations. The Sparse AC operations refer to accumulation operations without the multiplication step. This occurs in event-driven spiking computation, where additions are performed only when spikes are present, therefore leading to sparsity the operation count. This design enhances energy efficiency while preserving semantic fidelity.
4 Method
We propose SpikingMamba, an mergy-efficien extension of Mamba2 designed to maintain performance with reduced computational cost. It introduces three novel components: (i) signed spiking neuron (SI-LIF) that
4

---

Published in Transactions on Machine Learning Research (01/2026)
(a) Training Pipeline and Architecture
(b) SpikingMamba Block
(c) Smoothed Gradient Compensation Path
Li
(I(rrllps)
Inference Forward
Training Forward
Gradients Flow
Compensation
Original
X2 Rd
Linear Projection
Nith Decoder
N-th Decoder
Output Projection
Ye E Rd
W R 2d2x22
N layers
Mamha2
Mamba
Spiking
SSM
SI-LIF
{st E z
D DES, D) Liiidden
Pre-Norm
Pre-Norm
A B C 1x
ConviD xBC
1-th Decoder
1-th Decoder
Linear Projection
Y RR
0-th Decoder
0-th Decoder
Input Projection
W' E Rd,xd=
Fmheeding
Embedding
Teacher Model
Student Model
SpikingMambr
Mímic Function: f(x)) = tanh(x,)
(a) Overview of the training architecture. (b) Illustration of the SpikingMamba block. (c) Illustration of the Smoothed Gradient Compensation path.
Figure 1:
captures negative polarity and preserves magnitude during training; (ii) lightweight Smoothed Gradient Compensation path that restores essential features lost in the spiking conversion, and (iii) a hybrid training strategy combining knowledge distillation and to transfer teacher model ability while reinforcement Icarning for improving model performance in further. An overview of the full framework is shown in Figure 1(a).
4.1 SpikingMamba Block
To reduce energy consumption in the most computationally expensive parts of the model, SpikingMamba introduces neurons before both the input and output projection layers. These projections dominate memory and compute overhead, making them ideal targets for spike-driven optimization.
At each timestep t, the imput æft] E R is passed through spiking neuron function fsN(-), yielding sparse binary activation s(t] (s(()) E {0,1) Rather than performing dense matrix-vector multiplication y(t] = Wx(t), where W E RRo ddn is the projection weight, we compute sparse row-wise accumulation over firing indices
y/t]
Σ
W
(4)
where the conditional summation is determined by the sparse binary activation s[1] = SSN(2= (x((4), which is produced by the spiking neuron and thus remains implicitly dependent on the input s[t). This converts costly MAC operations into efficient AC operations. The sparse spiking pattern further reduces memory traffic, improving overall compute and I/O efficiency.
4.1.1 SI-LIF Neuron
Existing integer spiking neurons remain mechanistically incomplete for large language models: I-LIF's non-negative [0, D] range erases semantic polarity (Luo et al., 2024), whereas ternary neurons (Guo et al., 2024; Xing et. al., 2024b) with ({-1,0,1)) impose coarse magnitude discretizations, jointly provoking a distribution shift that permeates Mamba's recurrent state.
To rectify this dual defect, we propose the Signed Integer titiooy-nntgrat--n---eee (SI-LIF) neuron, a bounded signed-integer spike neuron that supports both integer-valued training and spike-driven inference.
5

---

Published in Transactions on Machine Learning Research (01/2026)
4.1.2 Integer- Valued Training
At each step, the membrane potential is rounded and clipped into a tunable symmetric range:
81 Clip (liipRRond(<z))) -D, D),
(5)
here, D explicitly determines the trade-off in quantization accuracy, while the gradient of 8 with respect to æt is approximated using a rectangular surrogate function, which yields zero derivatives outside the interval:
Ust dx.
{ a, - otherwise,
(6)
where the scaling factor denotes the slope of the surrogate gradient. In this work, adopt by default. Compared to I-LIF and ternary neurons, SI-LIF encodes information with both polarity and amplitude, thereby enhancing the representational capacity of SNNs during training.
4.1.3 Spike- Spike-Driven Inference
This paragraph details how the trained SI-LIF neuron performs event-driven inference while maintaining exact equivalence to the dense transformation y: Ws., where 8, SI-LIF(x;,) E []., To enable the event-driven inference, we reconstruct the integer E [-D, DJ as binary s(i] E {0,1} through the neuronal dynamics of SI-LIF within the micro-timestep i = of an internal temporal window of length D. For == -,...., D, the neuronal dynamics strictly obey the discrete-time LIF equations:
v|i [v[ - 1| Va s/i - 11) + x(1].
(7)
s|1] (v - Vv.),
(8)
where B 1 for exact reconstruction and s[1] 1) denotes the binary spike vector at micro-step i.
Specifically, the x[1] æ at first micro-timestep while x[i] 0 for other i, where the |- denotes the absolute value taken. Consequently, the summation of the spiking sequence equals the integer activation 8t produced by SI-LIF during training. The layer output is then recovered exactly by sparse weighted summation over the spike train:
D y: = W . (xx)) = Σ w. (sgn(z)))s(1))
D dm
=
-Σ> i= j=1
sgn( W..)
ss1l, =
Σ G| =
sgn W.s
(9)
For Training
For Spike-Driven Driven Inference
where the binary spike s(i] is obtained from the dynamic Eq 7-8, then s(i] serves as a selection signal to choose the corresponding j-th row the W.j to summarize together. Further, the sgn( is the single-bit sign flag. This sign retrieval incurs zero additional cost: the polarity bit is obtained hardware by a single XOR with the sign bit of the input, enabling instantaneous sign inversion via bit-flip without additional arithmetic operation.
Each inner summation corresponds lo row-wise addition triggered exclusively by sparsity active spikes, thereby replacing the dense MAC (Wæ,) with at most D times event-driven accumulations while maintaining cxact equivalence. These spike-based computations enable fine-grained signed encoding while maintaining neuromorphic sparsity.
4.2 Smoothed Gradient Compensation Path
Although SI-LIF neurons capture information from negative activation domains, their quantized charac- teristics can introduce gradient approximation errors during backpropagation. To alleviate these issues, we introduce a Smoothed Gradient Compensation (SGC) path (denoted by dashed lines in Figure 1(c)), which approximates the output of the primary spiking pathway and facilitates model training through its differentiable nature.

---

Published in Transactions on Machine Learning Research (01/2026)
Specifically, we apply distributional alignment constraint to enforce semantic consistency. This means that ensuring both pathways produce similar predictive distributions despite the quantized nature of the SI-LIF outputs:
T
Hidden
Σ
[softmax(y)) - softmax(y)115
(10)
27
where y = W (z+) is the output from the spiking pathway, and y W (m())) is the output from the SGC path. The projection matrix W Rdixdd initialized based on W Ruh xd2 and subsequently updated through training.
To ensure scale consistency between fa( and the SI-LIF output fax(2,), range-preserving mimic func- tion is introduced:
(m,))) D x tanh(xx).
(11)
where D is the spike amplitude heprpaarameter of SI-LIF neurons. This formulation ensures strict alignment in dynamic range [-D, D] with the spiking outputs, while enabling smooth gradients HCHidden/ / /0x, to support parameter optimization. Upon completion of training, only the spiking pathway is retained for event-driven inference.
4.3 Training Framework
To enable stable and effective training of Spiking.Mamba without full-scale pretraining, we adopt a learning framework: (1) knowledge distillation (KD) from a pretrained Mamba2 teacher, and (2) aligmment via reinforcement learning (RL). This combination transfers both logit-level behavior and internal representations the spiking model while addressing its representation shift problem.
4.3.1 Distillation Stage
We distillate SpikingMamba via supervised fine-tuning using objective includes both output-level and hidden-level alignment:
pseudo-labels. The training
L = LKL + LHiddeu:
(12)
The first term minimizes the KL divergence between the output distributions of teacher and student:
1 CKL
Σ T
KL 10a, 0r))0p011111 x;, (s))).
(13)
where Or and 0s are the leacher and student parameters, respectively. The second term Liadeen corresponds to the hidden state aligmment defined in Eq 10, ensuring fidelity of intermediate representations, In practice in this work, we only incorporate the SGC path on 3 layers with the first layer, middle layer and last layer, which could balance the training efficiency and performance. Specifically, these correspond to the 1st, 12th, and 24th layers in the 130M model, and the 1st, 24th, and 48th layers in the 1.3B model, which contain 24 and 48 layers in total, respectively.
4.3.2 Alignment Stage
To further enhance zero-shot reasoning and preference alignment, we apply reinforcement learning using Direct Preference Optimization (DPO) (Rafailov et al., 2023) and its stabilized variant KTO (Ethayarajh et al., 2024). Given a prompt x with preferred response yo and dispreferred response yı, DPO maximizes:
mg = max pe log a (B (Sw - -()),
(14)
pla 1 is the sigmoid function and 07 is the reference policy, While DPO is effective, is unstable due local gradient noise and harsh penalization.
p(uml==00) where fw = log ply.. [.;7], fi = log xxx1==0)

---

Published in Transactions on Machine Learning Research (01/2026)
KTO addresses these issues by replacing the pairwise structure with global reward bascline. For single labeled pair (a, y), the reward is r(a, v) = 3 log p(u]=;07)) The final KTO loss becomes:
p(yl=)) which is compared a dataset-level baseline 2ref-
LKTO = ((aD)D (w(y)( (1 - (sy (r(a, v) ==
(15)
where Sy E {+1,-11 -1} denotes preference labels, and w(y) is sample weighting function. This allows for efficient use of both paired and unpaired data while improving training stability.
5
Experiments
To evaluate the effectiveness, efficiency, and generalization capabilities of SpikingMamba, we conduct exper- iments across multiple settings (The detailed experimental setup is provided in the section 5.1). We first assess zero-shot accuracy on general reasoning tasks and measure generative performance using perplexity on standard language modeling benchmarks. Next, we analyze the energy efficiency of SpikingMambaa under various configurations. Finally, ablation studies further validate the contribution of each component and the distillation pipeline.
5.1 Experimental Setup
Implementation Details. During distillation, we perform supervised fine-tuning on the GenQA (Chen et al., 2024c), InfinityInstruct (BAAI, 2024), and OpenHermes 2.5 (Teknium, 2023) datascts, following the single-epoch strategy used in MLLM (Wang et. al., 2024a). The teacher is pretrained Mamba2 of the same size, and SpikingMamba is initialized with the corresponding Mamba2 weights. For reinforce- ment learning (RL), apply Direct Preference Optimization (DPO) (Rafailov et al., 2024) or its variant KTO (Ethayarajh et al., 2024) using the UltraFeedback (Cui et al., 2023). All models are trained with the AdamW optimizer (3 = (0.9, 0.98) and a global batch size of 32, using linear warm-up for the first 1% of steps followed by cosine annealing. The sequence length fixed at 2048 tokens, and the embedding layer remains frozen during training. All experiments are conducted using NVIDIA A100 GPUs with BF16 precision. For the 1.3B model, distillation takes around 42 hours, and RL takes around 1 hour, We use a fixed learning rate of 1e-3 for distillation and 5c-6 for RI.
Evaluation Datasets. We utilize the open-source LM Evaluation Harness library (Gao et al., 2023) (from the big-refactor branch) to evaluate six standard tasks: BoolQ accuracy (Clark et al., 2019), PIQA accuracy (Bisk et al., 2020), HellaSwag (HS) normalized accuracy (Zellers et al., 2019), WinoGrannee (WG) accuracy (Sakaguchi et al., 2021), ARC-Easy and A ARC-Challenge (AE and AC) accuracy and normalized accuracy (Clark et al., 2018). Each task evaluated by analyzing the probability assigned by the model to each potential answer choice. We also report perplexity on the WikiText-2 (Merity et al., 2016), C4 (Raffel et al., 2020), PTB (Marcus al., 1993) dataset as an additional metric. Perplexity measures how well a probability model predicts token, quantitatively measuring the model's generation power.
Baseline. We primarily compare SpikingMamba against the pretrained Mamba2 baseline. We further include spike-based language models such as SpikeLLM (Xing et al., 2024a), SpikingSSMs (Shen et al., 2025) and SpikeGPT (Zhu et al., 2023) for a comprehensive comparison. Although SNNs and quantization are orthogonal methods, spiking out can be viewed as a 1-bit activation quantization. Thus, we additionally compare against the 1-bit quantized Mamba (Tang et al., 2024b).
5.2
Main Results
5.2.1
Effectiveness of SpikingMamba
As shown in Table 2, both SI-LIF and the SGC path are critical to retaining Mamba performance in the spiking setting. Specifically, using the SGC path consistently improves performance compared to variants without it, indicating its effectiveness in recovering information loss from the SNN path. Furthermore,
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Comparison with 1-bit Mamba quantization and SNN-based LLMs. We adopt a unified "step" notation T D from Yao et al. (2025), where T is the number of timesteps and D the maximum integer activation (*D 1" is LIF, "D > 1 is I-LIF (Yao et al., 2025), "+D" is SI-LIF). T 1 implies extra T times repeated computation needed for per token. "P." denotes parameter count, "T. n. indicates training tokens, and "PT" marks pretrained models. "Diff." shows average accuracy gap (Red: Performance drop compared to the ANN model). The SGC denotes whether to use the Smoothed Gradient Compensation path. While SpikingMamba, denotes our fully SNN model, we highlight with Orange background color.
Zero-shot Accuracy (%) (t)
Model
P.
T.
SNN
SGC
Step
BoolQ
PIQA
118
AE
ACNiem
Avg.
Diff.
Mambal
130M
PT
56.63
64.85
35.13
52.17
47.31
24.40
46.58
SpikinMMamb,,
130M
B
1x 4:4
51.01
61.10
31.44
51.62
46.39
22.75
44.09
-2.49
SpilingMfamhe,
130M
7 B
+-4
53.27
62.19
31.32
51.78
45.96
22.27
44.47
-2.12
Mamba2
1.30
PT
64.54
73.72
30.94
61.09
64.18
33.1 11
59.40
Bi-LLM (Mannas) (Hoong 2022is]
1.3B
1260
30.60
38.03
Bi- Mamba (Ting et al. 20216)
1.33
1260 B
X
62.00
09:20
43.10
53.70
13.90
21. 60)
49.38
-10.01
SpikingMamblss,
1.38
B
61.83
58.21
-6.19
SpikingMambe,
1.33
7 B
x
1 1:1
58.78
09.53
18.85
51.22
62.58
29.18
53.80
-5.54
SpikingMombe,
1.30
x
48.66
63.44
.4.78
LLAMA1-7B
7B
PT
73.08
77:47
73.00
67.07
52.48
41.446
64.08
SpixeLLM (Ning et al., 202ta IOLR
7B
-
2 x 16
65.45
41.67
32.51
64.37
58.39
54.3
52.48
-11.57
Mamba2 + DPO
1.3B
x
63.64
73.23
61.22
60.69
64.02
37.03
59.97
SpikingMambe,, DPO
50.15
63.26
-1.9
SpiaingMambe, DPO
1.30
/
x 4:4
62.23
70.73
52.39
57.14
63.64
35.32
56.91
-2.49
Mamba2 KTO
1.38
60.55
73.78
61.67
61.48
66.12
35.41
30.84
SpidtngManhe, KTO
1.311
x
1x
+44
63.61
70:46
32:12
56.59
65.28
32.34
M6.73
-3.11
SpikingManles, + KTO
1.3B
x
4:4
02.72
71.33
32.20
06.27
66.29
34.22
57.17
-2.23
the proposed SI-LIF neuron, which supports both positive and negative spike amplitudes, outperforms the I-LIF' variant that only produces positive spikes, demonstrating the benefit of richer activation dynamics. Compared to 7B SLikeI.LM,, 1.3B SpikingMamba exhibits 4.78% performance drop in zero-shot accuracy while SpikeLLM-7B drops 11.57%. Importantly, SpikingMamba--L.55 achieves 54.62% zero-shot performance with just 7B tokens for distillation, while Bi-Mamba-1.3B achieves 49.38% in contrast to the 1280B tokens needed. Under the same training conditions, the SpikingMamba from 54.62% improves to 56.91% and 57.17% after DPO and KTO reinforcement learning, achieving 2.29% and 2.55% improvement, while Mamba2 improves only 0.57% and 0.44%, which indicates the SNN-based LLM could potentially benefit more from reinforcement learning than the ANN-based LLM.
5.2.2 Comparison to SNNs Training from Scratch
Table further evaluates the zero-shot generalization ability of SpikingMamba against. SNN-basod language models trained from scratch. Despite not being specifically tuned on WikiText datasets, SpikingMamba achieves (T D = =4) lower perplexity than fully-trained SNN language models such SpikeGPT and SpikingSSMs. This shows that by leveraging a pretrained teacher and an efficient distillation process, SpikingMamba can preserve zero-shot performance at significantly lower training cost.
5.2.3 Perplexity
We analyze the perplexity (The lower is better) across datasets, including WikiText-2 (Merity al., 2016), C4 (Raffel et al., 2020), PTB (Marcus et al., 1993) as done in Bi-Mamba (Tang et al., 2024b). We observe consistent performance hicrarchy across configurations (SI-LIF > I-LIF (Yao et al., 2025) > LIF) as shown in Table 4. This trend highlights the importance of ternary spikes for SNN-based LLMs modeling. The same trend also persists at the 130M scale (Further details are provided in the Supplementary Materials). While Bi-Mamba requires 1260B tokens and a larger teacher model, our method achieves comparable or better perplexity (c.g., 25.11 vs. 28.9 on PTB) using only 7B tokens and the same model size, demonstrating the efficiency of our distillation approach.
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Comparison with spiking language models on WikiText-108, measured in token-level perplexity (PPL) (L). SpikeGPT (Zhu al., 2023) and SpikingSSMs (Shen et al., 2025) are trained on the WikiText datasets, while SaikingMamba reports zero-shot results without any additional training or finetuning on these datasets.
Model
SNN
Parameters
WikiText-103 PPL (4)
GPT2 Small (Radfford et al., 2019)
K
124M
29.41
GPT2 Medium (Radford et al., 2019)
346M
26.37
Mamba2 [Dau & Gu, 2024)
130M
19.56
SpicGPT (Zhu al., 2023)
216M
39.75
SpikingSSMs (Shen al,, 2025)
75M
33.94
SpikeSSMs (Zhong et al,, 2021)
V
75M
33.18
SpikingMamba (Ours)
130M
26.32
Table 4: The perplexity under the different configurations. "P." means the model paramcters. "SGC" denotes using the Smoothed Gradient Compensation path. "Step" means a different neuron (*D > 1 is I-LIF (Yao et al., 2025), "+D" is SI-LIF).
Parameters
SCC
Step
Wikk2 (L)
C4 (1)
PTB (1)
Mambba2
1.3
10.12
14.78
17.72
GPTQ-3tA 3tit (Faantar et al. 2022)
1.38
29.3
37.3
56.5
GPTQ-2bit (Praatar
2022)
1.2++6
10c+66
BLLM (Huang el al., 2024aa
1.38
1
19433.2
4013.6
35/0..8
Bi-Mamba (Tang al., 2024b)
SpikingManba
1.3
X
1x1
31.71
36.87
52.25
SpikingMamba
SpikingManba
1.3
1x
15.79
21.01
26.20
SpikingManba
1x
15.17
5.3 Energy Analysis
Spiking.Mamba offers significant energy efficiency advantages over Mamba during inference. To quantify these benefits, we compare the energy efficiency ratio (Ex/Es) between Mamba2 and SpikingMamba across different model sizes and neuron types (Additional details are provided in the Appendix C). As shown in Figure 2, larger models benefit more from spike-driven computation due the proportion of linear projection increasing the model scale. Across neuron types, LIF yields the highest efficiency due to its strict binarization, followed by I-LIF and SI-LIF. While SI-LIF slightly sacrifices efficiency, it enables significantly higher expressiveness and lower accuracy loss, highlighting practical trade-off.
Introducing SGC (striped bars) slightly increases computation but reduces spike rates in most cases, es- pecially in input projections, leading to an overall net energy gain. This ellect holds even with SI-LIF neurons, where semantic fidelity is better preserved. Reinforcement learning (green bars) via DPO or KTO further improves accuracy without affecting energy use, since the architecture remains unchanged. Together, these results show that SGC and RL can be combined with SI-LIF maintain high energy efficiency while narrowing the accuracy gap with the full Mamba2.
5.4 Ablation Study
We perform ablation studies systematically evaluate the impact key components in SpikingMamba, including the Neuron ablation, SGC integration, loss design and training framework. The results reveal which factors are critical for achieving competitive performancc.
SpikingMamba Architecture. Table 5 shows that both Smoothed Gradient Compensation path and SI- LIP contribute significantly to performance. This ablation study on the 1.3B SpikingMamba model SI-LIF with 1 x #4 and I-LIF 4). Removing the SGC path with a 0.76% performance drop. But replacing the SI-LIF with the I-LIF neuron results consistent drop in accuracy with 1.41%. Furthermore, SI-LIF best matches the Mamba2 block's output distribution, while LIF and I-LIF's positive-only spiking cause polarized outputs (Further details provided in the Appendix D and E).
10

---

Published in Transactions on Machine Learning Research (01/2026)
Energy Efficiency Ratio Ea/E5 with Neuron Type and Smoothed Gradient Compensation (SGC) Usage
10
130M
SGC used
14,FF
Ea/Es (1)
1.38
Protnined Man (E,
0
lxt
Land
SGc
lxad
L.0
l2l
1.30
1xd
1.16
Isad
1.3
soc
pPo
tTTo
ixsd
Figure 2: The inference energy efficiency ratio (Ex/Es, of SpikingMamba under various configurations (Ex:
energy of Mamba2, Esg energy of SpikingMambba). Colors denote model size, striped bars indicate SGC path usage, and marker shapes indicate neuron types. Detailed data and fire rate are provided in Appendix C.
Table Neuron ablation study on 1.3B Model.
Table 6: Ablation study on the distillation loss.
Method
SGC
Neuron
Avg. (%)
Diff.
Method
LKa
SGC
Avg.
Diff.
SpikingMamba- 1.3
SI-LIF
54.62
SpikingMamba 130M
44.47
SpikingMamba- 1.3B
x
SI-LIF
53.86
-0.76
SpikingMambb- 130M w/o SGC
44.09
-0.38
SpikingMamba--13B
X
I-LIF
53.21
-1.41
SpikingMamba- 1.3B
54.62
SpikingMamba-L.22
x
LIF
48.35
-6.27
SpikingMamha--1.55 w/o SGC
53.86
-0.76
Energy Accuracy Tradcoff
Distillation Loss. Table presents an ablation study on the distillation losses, highlighting that both components are es- sential. The combination of LKL and SGC (CHidden) yiclds the best results, while omitting either loss harms performance. For the 130M and 1.3B models, omitting SGC (Lmaden) resulted in accuracy degradation of 0.38% and 0.76%, respectively. This scaling trend suggests that larger models benefit more signifi- cantly from hidden-state aligument. loss.
54
1x18
1x44
52-
Accuracy (%)
50
x
48
46
1x+4
130M
44
[x4
1.3B
Energy Accuracy Tradeoff. Figure 3 illustrates how vary- ing the integer spike range affects the energy accurayyy balance in both the 130M and 1.3B model configurations. To ensure fair comparison, all results in Figure exclude the SGC module, which is specifically designed for SI-LIF and would otherwise confer an inherent advantage. Expanding the neuron range D, either by increasing its magnitude or by introducing symmet- ric negative values, consistently leads to higher accuracy while siruultaneously reducing the energy ratio (EANN/EsNN). The trend is even more pronounced in the 1.3B model, indicating that broader integer spike ranges enhance representational capacity without compromising energy efficiency. Note that incrcasing D improves accuracy, but it also introduces additional latency during inference.
42
4
8 10
Energy Ratio (t)
Figure 3: The Enorgy-accuracy tradeoff for 130M and 1.3B models under different spike neurons. Marker size indicates model scale,
Training Framework w/o SGC Path. Notably, the Smoothed Gradient Compensation (SGC) path delivers substantial performance gains despite minimal implementation, introduced in mercly three layers. While the framework maintains functionality without SGC, evidenced by a moderate accuracy decline from 54.62% to 53.86%, its integration reveals significant synergistic effects with reinforcement learning. Crucially, KTO preference optimization achieves 57.17% accuracy with SGC versus 56. 56.73% without, demonstrating a 0.44% absolute improvement. This confirms SGC's role gradient compensation mechanism that enhances reward-driven adaptation during alignment tuning.
11

---

Published in Transactions on Machine Learning Research (01/2026)
6 Conclusion
We propose SpikingMamba for enabling energy-effiiient inference of LLMs. By replacing dense projections with sparse spiking operations and compensating with a lightweight Smoothed Gradient Compensation path, SpikingMamba significantly reduces computation. A simple distillation and alignment strategy al- lows adaptation from a pretrained model without full retraining. Extensive experiments demonstrate that SpikingMamba achieves substantial energy efficiency improvements with minimal performance degradation across common language reasoning benchmarks.
Acknowledgments
This work supported in part by the Science and Technology Innovation 2030-Major Project (Brain Sci- ence and Brain-Like Intelligence Technology) under Grant 20222D0205700, the National Key Research and Development Program of China 1021FF11000000 the National Natural Science Foundation of China (Grant No.62276121, 12326604), Young Scientists Fund of the National Natural Science Foundation of China (Grant 62305278), The Guangdong Basic and Applied Basic Research Foundation and Youth S&T Talent Support Programme of Guangdong Provincial Association for Science and Technology (SKXLCC025460))
(515511155)1758)
References
Mauricio Fadel Argerich and Marta Patiño-Martínez. Mcasuring and improving the energy efficiency of large language models inference. IEEE Access, 2024.
BAAI. Infinity instruct. arXiv preprint ar 4XXiv:2406.XXXX, 2024.
Malyaban Bal and Abhronil Sengupta. Spikingbert: Distilling bort to train spiking language modols using implicit differentiation. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 10998-11006, 2024.
Aviv Bick, Tobias Katsch, Nimit Sohoni, Arjun Desai, and Albert Gu. Llamba: Scaling distilled recurrent models for efficient language processing. arXiv preprint or Xiv:2500. 14458, 2025.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about Physical Common- sense in Natural Language. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 7432-7439, 2020.
Fenglong Cai, Dong Yuan, Zhe Yang, and Lizhen Cui. Edge-Ilm: A collaborative framework for large language model serving in edge computing. In 2024 TEEE International Conference on Web Services (ICWS), pp. 799 90-8009.4 JEÉE, 2024.
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathe- matical reasoning. arXiv preprint arx::22406. 10858, 2024a.
Jiaqi Chen, Yan Yang, Shizhuo Deng, Da Teng, and Liyuan Pan. Spikmamba: When smn meets mamba in bvent-based human action rocognition. In Proceedings of the 6th ACM International Conference on Multimedia in Asia, pp. 1-8, 2024b.
Jiuhai Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi Zhou, and Tom Goldstein. Genqa: Generating millions of instructions from handful of prompts. arXiv preprint arXiv:2406. 10323, 2024c.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint Xiv: 05457, 2018.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrafeedbbcc:: Boosting language models with high-quality feedback. arv preprinl arXiv:2310001377, 2023.
Tri Dao and Albert Gu. Transformers sams: Generalized models and efficient algorithms through strue- tured state space duality. arXiv preprint arXiv:2405.2100 200000 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01300, 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv22210. 17323, 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Mucnnighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.
URL https: //ze://////oo.ooo arg/recordss10256633
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. Xiv preprinl or Xiw:2502.00111 71, 2025.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprinl arXiv:2312.0000 00752, 2023.
Yufei Guo, Yuanpei Chen, Xiaode Liu, Weihang Peng, Yuhan Zhang, Xuhui Huang, and Zhe Ma. Ternary spike: Learning ternary spikes for spiking neural networks. In Proceedings of the AAAT conference on artificial intelligence, volume 38, pp. 12244 12252, 2024.
Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC), pp. 10-14. IEEE, 2014.
Wei Huang, Yangdong Luu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Läu, Michele Magno, and Xiao- juan Qi. Billm: Pushing the limit of post-training quantization for llms. arXie preprint orXiv:2402.04291 2024a.
Yulong Huang, Xiaopeng Lin, Hongwei Ren, Haotian Fu, Yue Zhou, Zunchang Liu, Biao Pan, and Bojun Cheng. Clif: Complementary leaky integrate-and-fire neuron for spiking neural networks. In International Conference on Machine Learning, pp. 19949-19972. PMT.R, 2024b.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //aavvv.//ii...e/aa org/abs /2300. 2tt.a//2............
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprinl arXiv:2310.06770 2023.
Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arxiv preprint arXiv:2501.08313 2025.
Wenrui Li, Xiaopeng Hong, Ruiqin Xiong, and Xiaopeng Fan. Spikemba: Multi-modal spiking saliency mamba for temporal video grounding. arXiv preprinl arXiv:240.00117, 2024.
Ji Lin, Jiaming Tang, Haotian Tang. Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Procerdings of Machine Learning and Systems, 6:87-100, 2024.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXie preprinl ar arx:2224 12. 19437, 2024.
Xinhao Luo, Man Yao, Yuhong Chou, Bo Xu, and Guoqi Li. Integer-valute valued training and spike-driven inference spiking neural network for high-performance and energy-efficient object detection. In European Conference Computer Vision, pp. 253 272. Springer, 2024.
Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Spikebert: A language spikformer learned from bert with knowledge distillation. arXiv preprint ar Xix:22308 15122, 2023.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313 -30, 1993.
Qingyan Meng, Mingging Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Towards memory- and time-efficient backpropagation for training spiking neural networks. In Proceedings of the IEEE/CVP international conference on computer vision, pp. 2176-6176, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1609 07843, 2016.
Yuqi Pan, Yupeng Feng, Jinghao Zhuang, Siyu Ding, Zehao Liu, Bohan Sun, Yuhong Chou, Han Xu, Xuerui Qiu, Anlin Deng, et al. Spikingbrain technical report: Spiking brain-inspired large models. arXiv preprint arXiv:2509.0522 2025.
Jiahao Qin and Feng Liu. Mamba-spike: Enhancing the mamba architecture with spiking front-end for efficient temporal data processing. In Computer Graphics International Conference, pp. 323-334. Springer, 2024.
Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, and Kaibin Huang. Mobile edge intelligence for large language models: A contemporary survey, IEEE Commumications Surveys & Tutorials, 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Πya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea. Finn. Direct prefereuce optimization: Your language model secretly a reward model. Advances in Neural Information Processing Systems, 36:53728 53741, 2023.
Rafael Rafailov, Archit. Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(10)):1667, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
Shnaijie Shen, Chao Wang, Renzhuo Huang, Yan Zhong, Qinghai Guo, Zhichao Lu, Jianguo Zhang, and Luziwei Leng. Spikingssms: Learning long sequences with sparse and parallel spiking state space models, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 20380-20388, 2025.
Jianxiong Tang, Jian-Huang Lai, Lingxiao Yang, and Xiaohua Xie. Spike-temporal latent representation for energy-efficient event-to-video reconstruction. In European Conference on Computer Vision, pp. 163-179. Springer, 2024a.
Shengkun Tang, Liqun Ma, Haonan Li, Mingjie Sun, and Zhiqiang Shen. Bi-mamba: Towards accurate 1-bit state space models. XrXio preprint arXim:2411.11 11.12/11111111.1 2024b.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Ban--atttstte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint (rXii:2312.118000 2023.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023. URL
http::::/huggnngface.com ww//atasets/teennttt.. http://ttkiimmmpppieee Oppn//ereeeeee.5555 aaaaa//ttttseeeaaa/c//aaerreeera
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arxi:22302. 13971, 2023.
Junxiong Wang, Daniele Paliotta, Avner May, Alexander Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems, 37:62432- 62457, 2024a.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingehen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2024b.
Peixi Wu, Bosong Chai, Menghua Zheng, Wei Li, Zhangehi Hu, Jie Chen, Zheyu Zhang, Hebei Li, and Xiaoyan Sun. Efficient spiking point mamba for point cloud analysis. arXiv preprint arXiv:2504. 14971, 2025.
Xingrun Xing, Boyan Gao, Zheng Zhang, David A Clifton, Shitao Xiao, Li Du, Guoqi Li, and Jiajun Zhang, Spikellm: Scaling up spiking neural network to large language models via saliency- spiking, arXiv preprint 2XXx:2407.04752 2024a.
Xingrun Xing, Zheng Zhang, Ziyi Ni, Shitao Xiao, Yiming Ju, Siqi Fan, Yequan Wang, Jiajun Zhang, and Guoqi Li. Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms. arXiv preprint arXi::2406. 08287, 2024b.
Xingrum Xing, Zheng Liu, Shitao Xiao, Boyan Gao, Yiming Liang, Wanpeng Zhang, Haokun Lin, Guoqi Li, and Jiajun Zhang. Efficientllm: Scalable pruning-awar aware pretraining for architecturo--guostii edge language models, 2025. URL https://arxi: wwwrexii.org//bbbee
org/abs//////eeeooooo...ooe. aab./b//2 06663.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention trans- formers with hardware-efficient training. arXiv preprinl arXic:2312. 2666:333.00665, 2023.
Man Yao, Xuerui Qiu, Tianxiang Hu, Jiakui Hu, Yuhong Chou, Keyu Tian, Jianxing Liao, Luziwei Leng, Bo Xu, and Guoqi Li. Scaling spike-driven transformer with efficient spike firing approximation training. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2025.
Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. Slender-mamba: Fully quantized mamba 1.58 bits from head to toe. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 4715-4724, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint ar Xiv::1900 0905000000 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Aduances in Neural Information Processing Systems, 32, 2019.
Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramosh, and Christopher Ré. Lolcats: On low-rank lincarizing of large language models. arXiv preprint 1rXiv:2410.10255 2024.
Yan Zhong, Ruoyu Zhao, Chao Wang, Qinghai Guo, Jianguo Zhang, Zhichao Lu, and Luziwei Leng. Spike- ssm: A sparse, precise, and efficient spiking state space model for long sequences learning. arXio preprint 1rX 17268 2024.
Rui-Jie Zhu, Qihang Zhao, Guoqi Li, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXix:2802.110 2023.
15

---

Published in Transactions on Machine Learning Research (01/2026)
A
Mamba2 Block
The Mamba2 (Dao & Gu, 2024) architecture consists of L stacked layers. At each layer, given the input E I al time step I, the processing begins with unified input projection:
u us Win E (2H
(16)
21, F, By C,C,C, Split(u)),
(17)
where Wn E (DX(2H-P22--HI is the input linear projection, D is the model dimension, and H, N are the number heads, dimension cach head and state dimonsion, respectively. The hyperparameter often keeps the relationship with H = 2D. The projected features ui are then split across heads and along channels. We reshape the activation to obtain input at(() E RP, and A/ E R for each head d = L,..., H, The inpetedependent variables for each head are computed as:
a = exp(-A, cxp(A(d)) E R,
c(d) a (Convld(C RRV
B (d) a (Convld(B))) RN,
(d) a (Convid((z)) (())) (()) R..
x
where A Softplus(A) (A + A (d)
blas) R. and af) the SiLU activation and Convld denotes short causal convolution. The hidden state update and per-head output computed as:
h (4) h 1
(d) (a 1) + (A B( at E ERRX,
(d)
(18)
o = c (dd)) h + (d)) @ 2 ERP
(19)
y1 = Norm (Concatto (1)
of
(H) o 0z ER"P.
(20)
y ye W. out R.D
(21)
where denotes the outer product, D(4) E R and A E R are trainable parameters, Norm(-) denotes RMS normalization (Zhang & Sennrich, 2019), and Wout E H-PD is the output projection matrix.
B Activation Statistics
To better understand the internal activation behavior of Mamba2, we visualize the distribution and structure of intermediate activations in Figurcs 4 and 5, and quantitatively evaluate their impact model performance in Table 7.
Figures 4 show the layer-wise distributions of u, and ye across the 24-layer Hambba2130M model. While u, values are relatively centered and stable across layers, Yr exhibits increasingly high-magnitude activation in deeper layers, with a wide dynamic range. Figure 5 further confirms this by visualizing the token-wisst and channel-wise activation values, where yt shows sparse but extreme peaks.
To simulate spike-based activations, we clamp the maximum activation value per channel to either or 1, mimicking a spiking output. Table 7 shows that this binarization leads to significant performance degradation across all tasks up to a 9.7% drop in average accuracy, demonstrating that high-magnitude activations play critical role in preserving semantic fidelity. Notably, both u, and contribute to this effect, though the impact of clamping y:. is more severe due to its broader variance.
These results highlight a key challenge deploying spike- LLMs: naive binarization of positive acti- vations leads substantial loss in representational capacity, motivating our design of ternary neurons and auxiliary paths to preserve high-value semantics.
16

---

Published in Transactions on Machine Learning Research (01/2020)
Mamba2-130m (winogrande task: Activation u)
Mamba2-130m (winogrande task: Activation y)
10
10
Activation Value
10
Activation Value
10
10"
-10"
-10
-10"
-10"
-10
1011 1213 14 18192021
11 14 22
Layer Index
Layer Index
uy distribution
(b) Ye distribution
Figure 4: Activation distributions in Mamba2: (a) input projection u, (Eq 16), (b) output projection y (Fq 21)
Memboc-2 reingrands Acthutton
Choad
Chamse
(a) u, statistics
(b) Yt statistics
Figure 5: Activation statistics across channels and tokens Mamba2: (a) input projection u, (Eq 16), (b)
output projection y/ (Eq 21).
C Energy Computation
C.1 Energy Details
To quantify the energy efficiency of SpikingMamba compared to the original Mamba2 model, we present detailed energy consumption statisties for all layers in both the 130M and 1.3B model variants. Table 8 shows the total energy cost across major components such input/outpit projection and SSM (state space model) for the 130M model. Similarly, Table 9 presents results for the 1.3B model, where we include settings with and without Smoothed Gradient Compensation (SGC) path and reinforcement learning (DPO/KTO). The detailed compulation ref from Section C.2.
The energy is computed using a breakdown of core components, with the total cost measured in arbitrary units that are consistent across model types. The Ex/Es ratio in the final column captures the relative energy efficiency, where EA represents the energy of the original Mamba2 model, and Es corresponds to SpikingMamba under each variant. Notably, SpikingMamba configurations consistently reduce energy by 2.9-4.7x in the 130M model and up to 4.7-9.8x in the 1.3B model.
Table 7: Zero-shot accuracy obtained by setting the maximum activation in each channel to 1 or 0.
HellaSwag
PiQA
Arc-E
Arc-C
BoolQ
WinoGrande Avg. (%)
Diff. (%)
Mamba2-1388 130m
35.22
61.25
47.31
21.06
54.62
52.25
46.29
Imax
27.93
53.16
31.44
21.15
40.18
51.54
38.07
-8.22
Fmax
25.84
53.37
27.48
23.72
37.83
52.80
36.84
-9.45
Umax =
26.18
50.60
26.22
25.34
40.24
50.83
36.57
-9.72
Umax 24.50
49.56
25.93
27.47
49.27
49.80
37.76
-8.53
17

---

Published in Transactions on Machine Learning Research (01/2026)
Table 8: The total energy consumption of the 130M model. The I-LIF is set as T'x D = 1xx4, 4, SI-LIF is set as T #4. "SGC" means whether to use a smoothed gradient compensation path during training.
SGC
Step
frim
frou
In Proj.
Out Proj-
SSM
Others
Total (L)
(t)
Mamba2
284.2067
130.2331
82.7476
1.4733
498.6607
1
Spiking.Mabaa
X
1 1x
0.3180
0.1583
17 7.6882
4.0259
82.7476
1.4733
105.9294
4.7075
SpikingMamba
X
1x4
0.3294
0.0509
73.1770
5.1878
82.7476
1.4733
5858
3.0671
SpikingMamba
1x
0.3498
0.1215
77 8034
12.3835
82.7476
1.4733
174.4078
2.8592
SpikingMamba V
1x #4
0.3476
0.1236
77.3141
12.5975
82.7476
1.4733
174.1325
2.8637
Table 9: The total energy consumption of the 1.3B model. The I-LIF is set T x D = 1 4, SI-LIF is set
as T x D = 1 +4. "SGC" means whether to use a smoothed gradient compensation path during training.
SGC
Step
frm
frous
In Proj.
Out Proj.
SSM
Others
Total (4)
Ea
(t)
Mamba2
3849. 1128
1852.2046
441.3204
7.4809
6150. 1180.1188
SpikingMamba
X
1x 1
0.1605
0.1483
120.8705
53.7421
441.3204
7.4809
623.4140
9.8652
SpikingMamba
0.2196
0.0156
661.5115
22.6130
441.3204
7.4809
1132.9263
5.4285
SpikingMamba
X
1x
14
0.2529
0.0612
761 8833
88.6691
441.3204
7. 7.4809
1299.3588
4.7332
SpibingMamba
1
14
0.2499
0.0619
752.7860
89.7272
441.3204
7.4809
1291.3147
4.7627
DPO
x 14
0,2533
0.0633
760.0159
93.0612
441.3204
7,4809
1301.8783
4.7240
KTO
x +4
0.2523
0.0624
763.0280
91.7566
441.3204
7.4809
1303.5860
4.7178
Table 10: Energy Evaluation for Mamba2 block.
Operation
# Operations x # Energy
In Proj.
[DD + 2N + H) D EMM
Out Proj.
D x 2D x EMM
dLB
H X ((P x 1) x (1 X N)) x EMM
C * 14 Convld
H x (P x N) X FMM
(2D 4 2N) X 4 EMM
Act
3 x (2D) x ΕEM
D,D
H x P x EEM
rdB
H x P x N x FEM
Adt
H x EEM
Ab,
Н P N X FEM
Norm
2D X EEM
y act(2)
2D X EEM
dt + disias
H X EADD
Ah; + xdB
H P x N x EADD
y + Dr
H P x EADD
C.2 Detail Computation
Tables 10 12 provide the exact computational formulations used for energy estimation, the operation mainly refers to Appendix A. These derived based the number of operations per module and their associated energy costs for different operation types. All operations assume 32-bit floating-poin implementation on 45mm technology, where matrix multiplication Exps = 4.6pJ, element-wise multiplication EEM = 3.7pJ and addition EADD = 0.9pJ (Horowitz, 2014).
Table 10 details the computation of energy consumption for Mamba2, listing core operations in each block such as projections, state transitions, and nonlinear components. Table 11 extends this SpikingMamba using LIIF neurons, where the energy is scaled by spike rates (fr), SGC path and spike-based neuron op-
18

---

Published in Transactions on Machine Learning Research (01/2026)
Table 11: Energy Evaluation for SpikingMamba with LIF.
Operation
# Operations x # Energy
In Proj.
I'm (4D + 2N + H) D x EADD
Out Proj.
frout D 2D EADD
dtB
H x ((P 1) x (1 x N)) x FMM
C h4
H X (P x N) x EлM
Convld
(2D + 2N) x 4 x EMM
Act
3 x (2D) x FEM
xD
H P X ΣEM
vdB
H P X N x ΕFM
Adt
H X ΕEM
Ahe
H X P X x EEM
Norm
2D X ERM
y * ac)(2)
2D X EEM
dt + dtbias
H X EADD
Ah, + adB
H X P x N x EADD
y + Dz
H P x EADD
Spiking Neuronl
(D D) ΕEM + (D+ D) X EADD
Spiking Neuron2
(2D + 2D) x EEM + (2D + 2D) x EADD
Table 12: Energy Evaluation for SpikingMamba with I-LIF and SI-LIF, where frim =
#Spike TxkxD frout
#Spike Txkx2D
Operation
# Operations # Energy
Proj.
k x frim x (4D + 2N + H) x D x FADD
Out Proj.
k x frout x x 2D x EADD
dtB
H ((P x 1) x (1 N)) x EMM
C * h4
H x (P x N) x FMM
Convid
(2D 2N) x EMM
Act
3 X (2D) x EEM
x
H ΣEM
xdB
H N ΕEM
Adt
H X EEM
Aht
H x P N EEM
Norm
2D X FEM
y act(2)
2D X ÉEM
dl + dlaias
H EADD
Ah, + rdB
H X P x N EADD
y Dx
H X P EADD
crations are accounted for explicitly. Table 12 generalizes further to SpikingMamba with I-LIF and SI-LIF neurons. Here, separate spike rates frin and frout are introduced to model the proportion of spike activations relative dense inputs/outputs, allowing precise modeling of spiking sparsity effects.
Together, these tables form the analytical backbone of the energy evaluations reported in the main text and demonstrate how sparsity and low-rank approximations contribute to overall energy reduction in Spiking- Mamba.
19

---

Published in Transactions on Machine Learning Research (01/2026)
D Experimental Details
Table 13 presents the detailed zero-shot accuracy values and perplexity corresponding to the results reported in the main lext.
Table 13: The energy consumption with different configurations. 'P." mcans the model parameters. Step means T D for Spiking Neuron. The standard LIF is 1 x when D 0 means using I-LIF, while = =D means using SI-LIF.
P.
r
Step
Total Energy (4)
Es (1)
Wiki2 (1)
C4 (4)
PTB (L)
Mamba2
130M
498.6601
20.04
25.23
35.11
SpikingMamba
130M
X
X
1
105.9294
4.7075
55.76
55.97
89.74
SpikingMamba
130M
X
X
162.5858
3.0671
32.12
36.70
54.36
SpikingMamba
130M
X
1 1 #4
172.5421
2.8901
28.55
33.62
48.26
Mamba2
1.3B
6150.1188
10.42
14.78
17.72
SpikingMamba
1.3B
1
623.4140
9.8652
31.71
36.87
52.25
SpikingMamba
1.3B
X 1x4
1132.9263
5.4285
17.90
23.45
30.19
SpikingMamba
1.3B
X 1x #4
1299.3538
4.7332
15.79
21.01
26.20
GPTQ-3bit
1.3B
N/A
N/A A
29.3
37.3
56.5
GPTQ-2bit
1.3B
N/A A
N/A
1.2e-46
1.3e-6
1.0e+-6
BILLM
1.3B
1
N/A A
N/A
4943.2
4013.6
3540.8
Bi-Mamba
1.3B
1
N/A
N/A
12.6
13.6
28.9
E Block Activation Distribution Analysis
To further analyze the impact of different spiking neuron configurations, visualize the activation distri- butions of hidden outputs from the final block across multiple 130M models. Figures 6-8 show histograms of activation values (x-axis) and their frequency (y-axis) for three model groups, each compared against the same pretrained baseline (Pretrained Mamba2).
In Figure the LIF model exhibits severe output polarization due to its binary firing nature. Figure 7 demon- strates that I-LIF reduces this binarization effect through continuous activations. However, its positive-only outputs still lead to skewed distribution. Finally, Figure 8 shows that SI-LIF, which supports both positive and ive outputs, resolves the polarization issue entirely.
2008
-00
4990
-200
245
1806
4499
-06
-1500
-15
-99
-26
(a) Pretrained Mamba2
(b) LIF
Figure 6: Activation distribution comparison for LIF-basod models.
20

---

Published in Transactions on Machine Learning Research (01/2026)
-20
-000
+000
-100
-00
(a) Pretrained Mamba2
(b) I-LIF x 4)
Figure 7: Activation distribution comparison for I-LIF (1 4) models.
-50
-00
300
(a) Pretrained Mamba2
(b) SI-LIF (1 +4)
Figure 8: Activation distribution comparison for SI-LIF (1 +4) models.
21


Reviews:
[{"review_id": "f6nhNICY5t", "reviewer": "Reviewer_y4Kx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the interest of implementing large language models more efficiently, there have been various recent proposals for using sparsely spiking neurons (i.e. those with quantized outputs) to construct models for which inference is faster and more efficient. This paper contributes a novel method for constructing a model which uses spiking neurons at key bottlenecks, based on the Mamba2 architecture [Dao & Gu, 2024]. Specifically, the paper's novel insights include a quantization scheme which allows spiking neurons to take a range of positive and negative values (\"Ternary Integer Leaky-Integrate-and-Fire\") as well as the use of an additional, fully-differentiable path at training time to reduce quantization issues while computing gradients (\"Smoothed Gradient Compensation Path\"). The method's effectiveness is evaluated by comparing its accuracy as a language model against the Mamba2 base model plus various other spiking models, plus its energy efficiency against Mamba2, and through an ablation study.\n\n\nTri Dao and Albert Gu. \"Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality.\" arXiv preprint arXiv:2405.21060, 2024", "claims_and_evidence_yn": "Yes", "claims_explanation": "I found the experimental evaluation to be reasonably comprehensive and convincing. One shortcoming is that the ablation study demonstrates that increasing the range of integer values for spikes improves performance; however, only two ranges are examined, so it would be good to explore this trend a little more thoroughly. It would also be useful to establish the energy/accuracy tradeoff a little more rigorously, perhaps as a scatterplot with energy and accuracy as axes.", "audience_yn": "Yes", "audience_explanation": "I expect this paper to be of some immediate interest to researchers working on efficient language models. It may also find some readers in the broader spiking neural networks community for whom the demonstrated performance advantage of TI-LIF spiking neurons at scale is relevant.", "requested_changes": "* Please keep the usage of the notation $T$ and $D$ consistent and define them both before they are used in Table 1.\n* The evaluation protocol should be defined more clearly in the main text. What exactly does zero-shot accuracy mean here? What data are used to estimate it?", "broader_impact_concerns": "It would be good to provide a comprehensive summary of resources used in generating the experimental data.", "additional_comments": "* It is stated that SGC is only used for three layers: the \"first\", \"last\", and \"middle\". I'm not sure which these are.\n* The spiking neural networks community has surely explored neurons similar to the TI-LIF neurons examined here, right? It would be useful to mention this literature or else comment upon the gap.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YbW1sIZa8h", "reviewer": "Reviewer_QixS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose to distill mamba2 in a partially spiking neural net architecture which is modeled closely to the original mamba2 architecture. From a high level view, they introduce spiking computations in the input projection layer and in the output projection layer of a mamba block. They are justifying this choice of partial spike-introduction. \nThey propose distillation as training mode which makes sense.\nAs for additional technical contributions, they propose to use during training a ternary integer-valued lif neuron, a temporary non-spiking block ensuring an additional gradient flow path called Smoothed Gradient Compensation. \n\nThey also evaluate the impact of DPO and KTO reinforcement learning for post-distillation alignment.\n\nIn terms of presentation, they show a comparison to other spiking architectures in relevant parameters in Table 1 which is informative and an overview graphic in Figure 1.\n\nThey measure zero-shot accuracy on LLM benchmark tasks.\nThey measure zero-shot perplexity on two textual tasks for comparison against other spiking models. \n\nAn energy analysis is performed with details given in the appendix.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are very well supported.\n They show a slight gap to the original Mamba2 with a gain in energy efficiency. They show experimental results for the gap, and bring a computation of the energy efficiency based on the observed operations. They claim two technical contributions and discuss them well in the paper. They show the impact of the proposed changes with an ablation study. They measure zero-shot perplexity on LLM tasks. The claims are well supported by the experiments.", "audience_yn": "Yes", "audience_explanation": "- The paper proposes a partially spiking neural net architecture which may hold promise for more energy-efficient inference. It does this for Mamba, which is a reasonable competitor of attention-based LLMs. \n\n- They provide it with improvements of the gradient flow for distillation training. \n\n- The model achieves an acceptable tradeoff between accuracy loss and energy-reduction .\n\n- The paper is also in many parts well readable. Current readability issues (see requested changes) is something that can be healed in a revision.\n\nOverall it is of high interest for parts of the community and is in the view of this reviewer a good paper.", "requested_changes": "The paragraph Spike-Driven Inference is hard to read:\n\n- The authors do not write at its start what they aim to do.  It seems that they want to compute $y_t = W x_t$ using a spiking architecture where typically $x_t \\in [-D,+D]$. please correct if something is half wrong in this claim. Whatever is correct, please write what you want to do in this paragraph.\n\n- With the preceding section one expect $s_t$ to be used in there. An additional sentence, that during training $x_t$ will be replaced by $s_t$ would be good (if that is not mistaken by me).\n\n- equation (9): there is something confusing about sign(x_t): Should it not be component-wise multiplied to s[i] (as it is input to the matrix multiplication with W) ? That is to put it inside the sums and as $ \\mathrm{sign}(\\mathbf{x}_t)_j s[i]_j  $ ?\n\n- Table 1 should name D as integer range\n\n- \"Upon completion of training, only **this** spiking pathway is retained for event-driven\ninference.\" \n\nchange to: Upon completion of training, only **the** spiking pathway is retained for event-driveninference.\n\n- It is not a must-change, however, ternary integer LIF is a bit misleading, though without doing harm: Ternary comes from three. With a general D this is gone. It would be better to name it signed-integer LIF (SI-LIF).", "broader_impact_concerns": "None", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}, {"review_id": "bshQiXzxuj", "reviewer": "Reviewer_xvRP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes SpikingMamba, an energy-efficient SNN-based LLMs distilled from Mamba2 that improves energy efficiency with minimal accuracy sacrifice. TI-LIF is a ternary-integer spiking neuron that extends the integer LIF neuron. Smoothed Gradient Compensation (SGC) is developed to alleviate the gradient approximation errors during back-propagation due to the quantized characteristics of TI-LIF neurons. Finally, self-distillation and reinforcement learning algorithms (DPO and KTO) are adopted to transfer the zero-shot ability from Mamba2 and to achieve preference alignment. Experiments demonstrate that SpikingMamba obtains energy efficiency improvements with minimal performance degradation across multiple language reasoning datasets, outshining previous spiking based language models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in the submission are supported by several experimental results.", "audience_yn": "Yes", "audience_explanation": "Spiking neural networks can provide an energy-efficient alternative to large language models such as Mamba2.", "requested_changes": "Pros: \n\n1.\tAlthough I am not working in spiking neural networks in recent years, it appears that the claims made in the submission are supported by convincing and clear evidence. \n\n\n2.\tThe main contributions and the technical details are clearly presented, despite some contents requiring more explanations (see minor points below). \n\n\nCons:\n\n1.\tThe three contributions summarized in the introduction sections appears to be straightforward and do not exhibit strong novelty. TI-LIF neurons simply incorporate -1 into the existing integer LIF neurons. Smoothed Gradient Compensation is a good idea to decrease the approximation errors, but the technical complexity is low. The third contribution, single-stage self-distillation and the use of reinforcement learning algorithms, seems to be weak and less convincing. \n\n\n2.\tThe presentation could be improved. For instance, the writing style (with bold headlines) in Introduction section is not common in academic papers. Also there are some claims that needs more explanations (see below).", "broader_impact_concerns": "I think there is no need to add a Broader Impact Statement.", "additional_comments": "Minor points: \n\n1.\tAbstract: “achieves a further 2.55% accuracy improvement after RL”. This sentence is somewhat misleading. At first glance I thought that using RL algorithms the performance can outperform the original Mamba2. The fact is that the results after RL are still inferior to Mamba2. \n\n\n2.\tA host of references listed in the introduction make the reader difficult to read the main idea of this paper. The authors should cite several references as needed after the sentence or some notable research lines, rather than give a long list of references. \n\n\n3.\tThe introduction section should focus on the main idea of this paper, and defer other irrelevant works into the Related Work section. For example, the discussion on KV-cache, quantization for LLMs, and ANN-to-SNN conversion, are irrelevant to the main work of this paper. \n\n\n4.\tPage 2, the full name of SSM should be offered before using the abbreviation. \n\n\n5.\tPage 2, bold fonts are misused too much in the contribution part. \n\n\n6.\tTable 1, the notation D should be explained in the table title. \n\n\n7.\tPage 4, eq. (1) and (2). Notations should be given before the LIF neuron model, otherwise the reader feels hard to understand. Also, I want to know the input and output of this model. I read the reference (Wu 2018) and found that the notations and formula are very different to eq. (1) and (2) in this paper. Hence it is strange to cite Wu 2018. Why not cite the original LIF paper or some significant survey paper in spiking networks? \n\n\n8.\tPage 4, what is the meaning of “sparse AC operations”? \n\n\n9.\tPage 4, eq. (4). The y[t] is only related to W, not with x. The original formula is y[t]=Wx[t]. \n\n\n10.\tPage 5, eq. (6). When x_t is in [-D, D], why is the derivative equal to alpha rather than 1? How can we handle the round operator when calculating the derivative? \n\n\n11.\tPage 5, eq. (7) and (8). I can not understand that beta=1 can ensure exact reconstruction. Why, and what to reconstruct? \n\n12.\tPage 6, eq. (10). What is the implication of “semantic consistency”? \n\n13.\tTable 2. The T times D notation follows the paper Yao 2025. Since the authors have read that paper, it is curious to ask “why the method proposed in Yao 2025 is not compared in Table 2 and following tables?” \n\n14.\tPage 8, table 3. SGC does not occur in this table, so there is no need to describe SGC. \n\n15.\tPage 8, Perplexity. Table 4 should be referred in the main text. \n\n16.\tPage 9, Energy Analysis. Is this part of results devoted to training or inference?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)