Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model
Authors: Kaiwen Tang
Zhanglu Yan
Weng-Fai Wong
Venue: ICML 2025 poster (2025)
Keywords: Spiking Neural Networks; Small Language Models; Energy Efficiency
Primary area: deep_learning->algorithms

Abstract: For reasons such as privacy, there are use cases for language models at the edge. This has given rise to small language models targeted for deployment in resource-constrained devices where energy efficiency is critical. Spiking neural networks (SNNs) offer a promising solution due to their energy efficiency, and there are already works on realizing transformer-based models on SNNs. However, key operations like softmax and layer normalization (LN) are difficult to implement on neuromorphic hardware, and many of these early works sidestepped them. To address these challenges, we introduce Sorbet, a transformer-based spiking language model that is more neuromorphic hardware-compatible. Sorbet incorporates a novel shifting-based softmax called PTsoftmax and a BitShifting-based PowerNorm (BSPN), both designed to replace the respective energy-intensive operations. By leveraging knowledge distillation and model quantization, Sorbet achieved a highly compressed binary weight model that maintains competitive performance while achieving $27.16\times$ energy savings compared to BERT. We validate Sorbet through extensive testing on the GLUE benchmark and a series of ablation studies, demonstrating its potential as an energy-efficient solution for language model inference.  Our code is publicly available at [https://github.com/Kaiwen-Tang/Sorbet](https://github.com/Kaiwen-Tang/Sorbet)

Sorbet: A Neuromorphic Hardware-Compatine Transformer-Based Spiking Language Model
Kaiwen Tang'
Zhanglu Yan
Weng-Fai Wong'
Abstract
et al., 2022) has prompted research into distilling small lan- guage models (SLMs) (Sun et al., 2020; Eldan & Li, 2023; Zhang et al., 2024) from LLMs that can run on the resource- constrained edge devices. Local inference on the device is critical in situations where data privacy essential or when connectivity powerful remote computing resources is un- feasible (Sun et al., 2020; Sanh, 2019). The high demand for energy efficiency drives research to simplify SLM infer- ence, making it better suited for devices with low energy consumption while maintaining adequate performance.
For reasons such as privacy, there are use cases for language models at the edge. This has given rise to small language models targeted for deployment in resource- connsreendd devices where energy effi- ciency is critical. Spiking neural networks (SNNs) offer promising solution due to their energy effi- ciency, and there are already works on realizing transformer-base models on SNNs. However, key operations like softmax and layer normal- ization (LN) are difficult to implement on neu- romorphic hardware, and many of these early works sidestepped them. To address these chal- lenges, introduce Sorbet, transformer-basee spiking language model that is more neuromor- phic hardware-compatiile.. Sorbet incorporates a novel shifting-based softmax called PTsoftmax and a Bit Shifting PowerNorm (BSPN), both de- signed to replace the respective energy-intensive operations. By leveraging knowledge distilla- tion and model quantization, Sorbet achieved a highly compressed binary weight model that main- tains competitive performance while achieving 27.16x energy savings compared to BERT. We validate Sorbet through extensive testing on the GLUE benchmark and a series of ablation studies, demonstrating its potential as an energy-efficien solution for language model inference. Our code is publicly available at https://gitubb..om/aaiii Tang/Sorbet
In parallel, spiking neural networks (SNNs) have gained sig- nificant attention due to their remarkable energy efficiency. SNNs closely mimic biological neural networks and are multiplication-free, resulting in much lower energy con- sumption compared to artificial neural networks (ANNs). Developing SNNs on neuromorphic hardware significantly improves energy efficiency while preserving impressive per- formance compared to advanced ANNs (Guo et al., 2023; Shi et al., 2024). For example, the SNN based on the ViT- base architecture (Dosovitskiy et al., 2020) achieves com- petitive accuracy levels (e.g., up 80.10% on ImageNet) to its ANN counterpart, but with only one-tenth of energy consumption (Zhou et al., 2024).
However, developing transformer (Vaswani et al., 2017)- based spiking language models is challenging, particularly in encoding spikes and handling the incompatible operations of neuromorphic hardware. Previous works have focused replacing matrix multiplications with encoding methods, like SpikFormer (Zhou et al., 2024), SpikeBERT (Lv et al., 2023), SpikeLM (Xing et al.), and SpikingBERT (Bal & Sengupta, 2024). However, the transformer architecture also includes softmax and layer normalization (L.N) operations, which are energy-intensive and incompatible with neuromor- phic hardware. Models like SpikeLM and SpikingBERT, which target natural language processing (NLP) tasks, still rely softmax and LN operations, limiting their compati- bility with neuromorphic hardware. SpikFormer addresses this issue by adopting features from convolutional networks and batch normalization, avoiding the use of these opera- tions. While effective for vision tasks, their applicability to language tasks, which rely more heavily on operations like LN, remains unproven. Thus, there no transformer-based
1. Introduction
The phenomenal success of large language models (LLMs) (Devlin et al., 2019; Brown et al., 2020: Raffel et al., 2020; Touvron et al., 2023; Lewis et al., 2020; Taylor
'School of Computing, National University of Singapore, Singapore, Singapore. Correspondence to: Zhanglu Yan <alyan@nus.edu.s.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sorbet: A Neuromorphic Homptabe-Compattibl Heassformer-Basee Spiking Language Model
spiking language model that is both hardware-compatible and performs well to date.
achieve full quantization. Sorbet is designed for neuro- morphic hardware, enabling energy-efficient inference with comparable performance to ANN counterparts.
Table 1. Comparison with other models. The '+' in the NLP row indicates the capability perform language tasks, while the '+' in the softmax row denotes the inclusion of softmax operations.
2. Related Work
BERT
Spikformer
SpikeLM
Sorbet
Transformer-based SNNs Transformed-basee SNNs leverage the energy efficiency of SNN architectures. Several transformer- based SNNs have been proposed for computer vision tasks, like Spikformer (Zhou et al.), Spikeformer (Li et al., 2024), Spike-driven Transformer (Yao et al., 2024) and STCA-SNN (Wu et al., 2023). To overcome the limi- tations of SNNs in handling complex operations, recent de- velopments like Spikformer and Spike- Transformer leverage task-specific characteristics and integrate convolu- tional layers into the architectures. Meanwhile, transformer- based SNNs for NLP tasks have progressed more slowly, The importance of LN in NLP tasks underscores the chal- lenges faced when adapting SNNs for such applications. Re- cently models include SpikeBERT (Lv et al., 2023), Spiking- BERT (Bal Sengupta, 2024) and SpikeGPT (Zhu et al.). Among them. SpikeBERT employed even more layer nor- malization than the original BERT (Devlin et al., 2019), while SpikeGPT and SpikingBERT used complicated opera- tions like exponential operation and softmax. In conclusion, none of these works can be practically deployed to neuro- morphic hardware.
NLP
+
+
+
softmax
+
+
Norm
LN
BN 32-bits
LN
BSPN 1-bit
Weight
32-bits
1-bit
In this work, we introduce an SNN-compatible normaliza- tion technique called Bit Shifting PowerNorm (BSPN) and a novel shifting-based softmax called Power-of-Two softmax (PTsoftmax). Based on these operations, we propose Sor- bet, a novel language model that operates without relying on complex operations. A comparison between Sorbet and previous works is shown in Table 1. To our knowledge, Sorbet is the first model designed for language tasks that overcomes the challenge of deploying transformer-base basse language models on neuromorphic hardware, which does not support complex operations like division and square roots. In the training process, we apply knowledge distil- lation techniques to constrain the Sorbet model to binary weights, significantly compressing its size and reducing in- ference energy consumption. As result, Sorbet is highly practical model for aooouurc--costraiing devices.
Our tests on the General Language Understanding Evalu- ation (GLUE) benchmark (Wang et al.) demonstrate that Sorbet maintains stable performance while achieving energy savings of 27. 16 compared to BERT and 3.16x compared to SpikeLM. Our contributions are summarized as follows:
Quantized BERT Model quantization reduces the preci- sion of model weights and activations, such as converting 32-bit floating-poin numbers to 8-bit or 4-bit integers. Mod- els like BinaryBERT (Bai et al., 2021) and BiT (Liu et al., 2022) have applied quantization to BERT, achieving notable improvements in model compression and energy efficiency. However, these quantization methods do not align with the requirements of SNNs, as they retain complex operations.
We are the first to explore the neuromorphic hardware- compatible operators in transformer-based models, identifying the challenge of transferring ANNs with transformer structures into SNNs, which lies in op- erations like softmax and LN. Solving this challenge would complete the last step in enabling SNNs for natural language processing.
Simplified Architecture Several approaches aim to sim- plify the transformer architecture, such as linear complex- ity attention mechanisms or hardware-efficient alternatives that offer potential pathways for adaptation (Han et al., 2023; Dao et al., 2022; Lu et al., 2021; Katharopoulos et al., 2020). Additionally, methods have been proposed to simplify computationally-inten
We propose PT'oottmx and BSPN, two operators that replace softmax and layer normalization. These opera- tors rely on bit-shifting instead of expensive operations, making them compatible with neuromorphic hardware and further reducing the model's computational cost.
operations within the transformer. For example, I-BERT (Kim et al., 2021) and I-ViT (Li & Gu, 2023) simplify activation functions, nor- malization functions, and softmax operations with approx- imation methods. However, these designs are difficult to realize with neuromorphic hardware that do not support multiplications and divisions.
We present Sorbet, a transformer-baeee binary spiking language model derived from BERT. In addition to using these two operators, Sorbet incorporates other design innovations and refined training process to
2

---

Sorbet: A Neuromorphic Horpwabe-Compatibbe Transformer-Bant Based Spiking Language Model
3. Preliminary
bit-shifting and addition, to ensure compatibility with neu- romorphic hardware. This requires substituting ANN trans- former layers with SNN-compatible counterparts that avoid complex operations but still maintain the functionality of the original architecture.
Spiking Neural Networks SNNs, inspired by biological neural systems, use discrete events (spikes) for communica- tion. Unlike traditional neural networks, SNNs emulate the spike-based communication of neurons, making them more biologically plausible. This mechanism allows efficient pro- cessing of temporal data and offers high energy efficiency, making SNNs ideal for applications like robotics, signal processing, and pattern recognition (Kasabov et al., 2013; Kim et al., 2018; Lobov et al., 2020). However, their non- differentiable nature makes training challenging. Current approaches typically involve surrogate gradients or ANN- to-SNN conversion after training an ANN with similar architecture. In either case, these methods leverage ad- vanced ANN structures to construct analogous SNN models. Surrogate gradients approximate the gradients during back- propagation, enabling SNN training despite their discrete nature. These gradients smooth out dindeffeerntiable spike events. ANN-to-SNN conversion involves transforming trained ANN model into an SNN by mimicking ANN neuron behavior with the spikes. In Sorbet, we use ANN-to-SNN conversion.
4. Methods
In this work, we design Sorbet, spiking language model ca- pable of handling NLP tasks and being practically deployed on neuromorphic hardware platforms. To achieve this, we base the model architecture on transformers due to their proven effectiveness in NLP. As previously discussed Sec- tion 3, neuromorphic hardware does not support standard ANN operations. So we first introduce BSPN (Section 4. and PTsoftmax (Section 4.2), which are fully compatible with neuromorphic hardware and provide high-quality re- sults for replacing traditional LN and softmax. We then describe the Sorbet architecture (Section 4.3) and the train- ing process (Section 4.4).
4.1. Bit Shifting PowerNorm
Transformer-based language models like BERT typically employ LN. LN calculates the mean and variance across all features for each data point in layer's input, normalizes the inputs, and applies learnable scale and shift. Due to hard- ware limitations discussed earlier, LN cannot be directly adopted in our model. On the other hand, batch normaliza- tion (BN) is favored in SNNs because its learnable param- eters can be merged into the weights during the inference However, stage, hence becomes SNN-hardware-frieene the relatively poor performance of BN makes it unsuitable for our model.
Spike Neuron Model The integrate and fire (IF) model is the most popular spike neuron model for generating spike trains (Bu et al.). It offers a simple representation of how SNN neurons accumulate membrane potentials and output spikes. In the IF model, the membrane potential V of neuron is treated as capacitor that accumulates the input currents over time, a process that the following differential equation can describe:
dV
di (y)(
Here, (t) represents the synaptic input current. When the membrane potential V exceeds a certain threshold 0, the neuron generates a spike. In this paper, we adopt an enhanced IF model known as average IF spike generation (ASG) (Yan et al., 2025). One advantage of ASG over the basic IF model is the reduction of memory access energy. Unlike the IF model, which accesses weights each time step, ASG accumulates input spikes and calculates mem- brane potential in a single pass, requiring only one access to the weights. The algorithm for ASG can be found in Ap- pendix A Algorithm
To improve BN's performance, PowerNorm (Shen al., 2020) introduced a relaxed zero mean BN. However, Power- Norm incorporates root-mean-squan layer normalization (RMSLN):
X
RMSLN(x))
22)
n +
This is too ssource-intensing and hence is not fully compat- ible with neuromorphic hardware.
To address the issue with PowerNorm, we define a novel normalization layer called Bü Shifting PowerNorm (BSPN), which replaces RMSLN and avoid complex square and square root operations. We begin by grouping the input. Within each group, denote the vector as x E IR", we calcu- late the L.1 norm.
Challenges of Adapting Transformers to SNNs While SNNs are recognized for their energy efficiency and ex- plainability, adapting transformer models to spike neurons presents a significant challenge. Spike neurons cannot di- rectly perform essential operations such as multiplication and division, which are fundamental to traditional ANN- based transformers. Consequently, transformer layers in SNNs must be replaced with simpler operations, such as
Then the L1 norm is approximated by the next power of two for hardware efficiency. This allows the "dividing by LI norm" step to be implemented via bit shifting. The
3

---

Sorbet: A Neuromorphic Horpaaie-Cooppatible Transformer-Ban Based Spiking Language Model
approximation can be achieved either by taking log2 and then exponentiating it back, more efficiently via lookup table. We then perform the relaxed zero-mean BN as in the PowerNorm. To optimize hardware efficiency, the scaling factor i can be further quantized to a power of two. The complete BSPN algorithm is outlined Algorithm where
loss Lpx under PowerNorm is bounded by constant, de- noted as C. We define the BSPN loss by
LBSPN (X)
LPx(9(X));
(4)
CBSNN also has a bounded gradient w.r.. X. specifically
y @ is denoted as [y,, X1 ..... Md: Xa :].
ÖLBSPN ax,
S C
for all X.
(5)
Algorithm Bit Shifting PowerNorm (BSPN)
1: Input: X R4x"; Number of attention heads h;
2: Output: Y;
The detailed proof is provided in Appendix B.1.
3: Step 1: Group Scaling
To ensure stable training and effective generalization, is important control the Lipschitz constant of the loss function. In (Shen et al., 2020). the author proved that ap- plying PowerNorm can lead to smaller Lipschitz constant of the loss compared to BN under mild assumption. BSPN exhibits similar behavior, as outlined below:
4: Group channels into h groups.
5: S + +Σ:n [X;]
6: k (S) {Can be done by using look-up table}
7: X <<X>>
8: Step 2: Normalization PowerNorm
9: For Training:
10: 2
E X
Lemma 4.3 (1-Lipschitz Property of ((X)). For any X, Y E R", under mild assumption, we have
11: & &
12: Y+
X
[1x(X)- -((Y)l S [|--Y]]
(6)
13: di + anl? + (1- (Exponential moving average
update for 2
14: For Inference:
The assumption and detailed proof are provided in the Ap- pendix B.I. For the loss function LBSPN for convenience in comparison, we use &B instead of v. Since ((X) is I-L-ipshitz, BSPN does not increase the overall Lipschitz constant of loss compared to PowerNorm.
15: 7 +
The two advantages of BSPN are that uses only SNN- compatible operations and significantly simplifies the com- putation process. Firstly, like PowerNorm, BSPN incorpo- rates the computation of runtime variance, which is then used during inference. Compared to IN, this eliminates the need for redundant calculations during inference.
Lemma 4.4 (Effect of BSPN on the Lipschitz Constant of the Loss). With lemma 4.3 and lemma 2 in (Shen et al., 2020), we further have:
@CBsPN OX.;;
|2 S
1LpN 0x
o (WB); 0x
0LaN
Secondly, BSPN avoids RMSLN by using the LI norm and approximating the divisor as a power of two. These features make BSPN a practical design for SNN models and significantly reduce energy consumption, as shown in Section 4.3 and Section 5.2.
S
(7)
From (Shen et al., 2020), the empirical result supports that (en). is typically smaller than 1. Therefore, the Lipschitz constant of CASPN is also smaller than that of LAN in prac- tice. These properties BSPN imply that it enhances train- ing stability by preventing gradient explosion or vanishing, ensuring controlled updates. This makes BSPN a robust and efficient alternative LN.
Tie
Next, we show that the BSPN is not only deployable on neu- romorphic hardware but also possesses desirable properties that make it a suitable replacement for LN in transformer models.
Definition 4.1. Let t: I= + R" be defined by
4.2. Power-of-Two Softmax
X
1 Σ
+(X)
where S(X)
X+I-
In transformer-based models, the softmax function plays a crucial role, especially in the attention mechanisms where it is used to caleulate the distribution of attention weights across different inputs. For vector z [21, 22, .... ),), softmax can be calculated as follows:
logz (s(x))
2
Since bounded gradients are necessary for convergence, we first analyze how the proposed bit-shifting-bar based step (.X) maintains PowerNorm's gradient boundedness.
softmax (2+)
(8)
Theorem 4.2 (BSPN Preserves Bounded Gradient). The
Σn 6

---

Sorbet: A Neuromorphic Horptabe-Compattibe Transformer-Ban Basedd Spiking Language Model
Due the complexity of the exponential and division opera- tions involved in the softmax function, it is too sophisticated for neuromorphic hardware, making direct utilization of soft- max in SNNs impractical. We propose softmax alternative aligned with SNN computational conventions, enabling a more streamlined attention mechanism suitable for SNNs.
used interchangeably. In partieular, P2(2))) is well suited representing probability distribution in (0, 1Jsuch that the probabilities sum to 1. We then examine the approximation between (2,, and PTsoftmax.
Lemma 4.5. For all i E n, we have 1 Pa(22) s PTsoffmax((;) S 2/2F)
242
To approximate the softmax function, we first replace the exponential operation with powers of two, thus defining base-2 softmax:
A detailed proof is given in Appendix B.2. Lemma 4.5 shows that the approximation error of PTsoftmax remains within a constant factor of the traditional softmax, con- firming its practical applicability. Unlike classic softmax, PTsoftmax does not strictly sum 1, but theoretical analy- sis and experimental results (Section 5.3) indicate that this discrepancy has minor impact on performance.
2
Base-2 softmax =
(9)
Σラ= 2
Since the base-2 softmax still involves division, we further approximate Σj=1 2 as the nearest power of two Z:
[(hess
4.3. Sorbet Architecture
Σ 2 2
7 2*
(10)
We now present Sorbet, our spiking language model. At high level, Sorbet resembles a standard transformer design but is customized for the SNN platform by replacing various layers with SNN-compatible ones, including BSPN (Sec- tion 4.1), ), PTsoftmax (Section 4.2), and ReLUU, equipped with additional spike neurons and avoiding multiplication with accumulation operations. Figure shows the archi- tecture of Sorbet along with the current ANN design (e.g..
Here, k is the integer logarithm base 2 of the sum, rounded up, ensuring Ž is the nearest power of two approximating the softmax denominator. To facilitate bit-shifting, we also round up 2, Our proposed purely power-of- softmax (PTsoftmax) can then be defined as:
2[=] ΣØ 2 R 2
BERT).
PTsoftmax(22)
(11)
Each activation is encoded using spike neurons to generate spike trains. Our spiking self-attention mechanism is then defined as:
Given z; and k the operation 2 can be efficiently com- puted via left shift operation. The complete Postmaax computation is detailed in Algorithm 2.
SpikingAutn(z)
(13)
SN (PTsoollta (
Algorithm 2 Power-of-two Softmax (PTsoftmax)
1: Input: Attention scores matrix S E Roxxe
2: Output: Attention probabilities matrix P E RRxxaxh
3: S' - -[S] - max([S1) [Normalize S}
where Q, K, V are derived from a quantized bingry linear function, and a is a constant that can be merged into the weights. SN denotes the spike neuron used to generate spike trains. In Sorbet, at the position of matrix multiplica- tion in the ANN, one of the multiplicands is encoded into spike train. Matrix multiplications involving SN are sim- plified to accumulations over the indices corresponding to spike events (1s in the spike train).
4: A < 28'
5: Z < Σ(2"") {Compute the sum of exponentials)
6: k < [log2( [
7: P < A > k (Use right shift to divide by Ž 2*
8: return P
To analyze the approximation error of PTsoftmax relative to the original softmax, we followed (Zhang et al., 2022),
In sub-layers originally defined as LayerNorm( + Sublayer(z)), we instead use BSPN(SN(2)) + BinaryLayer(SN(z))) To preserve performance af- ter quantization, also adjust the training process. Notably, existing neuromorphic chips such as Loihi, IBM TrueNorth, and NeuroGrid (Davies et al., 2018; Akopyan et al., 2015; Benjamin et al., 2014) already support spiking and bit-shifting operations, making them well suited for these innovations.
and used the generalized base-3 softmax function, F2(x:))
defined as:
Fę(x;; = g
(12)
E/= 8
Here, the traditional softmax corresponds to (2;)), and the base-2 softmax is F2 (x.). According to (Zhang et al., 2022), both (x:) and (2(2)) exhibit similar behavior and can be
5

---

Sorbet: A Neuromorphic Hardware-Compatibbe Seiastorme-BBaser Spiking Language Model
Embeediiggs
Binary Embeekiings
Linear
Linear K
Linear V
Linear Layer
Q
SN
SN
SN
SN
GeLU
BiFC
BIFC
BiFC
BiFC
Softmax
Lineer Layer
ReLU-SN
Layer Norn
BiFC
Linear Layer
Classificr
PTSoftma
BSPN
Layer Norm
a
Binary Weight
SN
BIPC
FC I ayer
Accumulation
SN
SN
Spike
Spike train
BIFC
Classifier
Binary
Matrix Multiplication
spike train (-00
BSPN
d
Figure 1. Comparison of the architecture of BERT and Sorbet. (a) The multi-head self-attention block of BERT; (b) The feed-forward network of BERT, (c) The spiking multi-head self-attention of Sorbet; (d) The spiking norwadd network of Sorbet. All the outputs of SN are spike trains ensure Sorbet is multiplication-fret The red-bordered box highlights our proposed operations.
4.4. Training Process
from the student and teacher models, respectively. The backpropagation can be calculated as:
Firstly, to enhance the energy efficiency of our model and enable the encoding of all activations into spike trains, we quantize all weights to bit and activations to 4 bits. For model quantization, (Liu et al., 2022) introduced elastic binarization function with scale factor and threshold 3:
aL Sw
ÖL dp.
ÖL 0q: 0q. Sw
aL dr
aL dr! Dr Dw
Σ
Üp: Dw
Or dw
= Σ (
log
Pi
+
0p:
Pi 0g:
Ow
qi Ow
X - X/ = = ClCipp(
0,1))
(14)
òr:
dr!
a
Σ 2(r - r1)
2(r
r
Ow
Bw
However, dividing the input by a is impractical for SNN in- ference. Hence, following the same rationale as PTsoftmax, we approximate a with the nearest power of two, = 2ha Accordingly, the elastic binarization function becomes:
(17)
Then we integrate BSPN and PTssottmx step by step into the model. For each step, we perform model distillation. Finally, the quantized model with these mergy-efffcieett components is transformed into the Sorbet model by passing it through spiking neurons. This entire procedure is summarized in Algorithm 3.
X = CClip ((Xr= B) >> ka. 0.1 < ka
(15)
Inspired by (Liu et al., 2022), we adopt hybrid training strategy that combines standard knowledge distillation with the distillation of intermediate activations. The overall loss function is L = Lugits + Treps, where Ligits employs the Kullback-Leibler (KL) divergence to facilitate learning from the teacher model to the student model while Lyeps is used to accelerate convergence and improve transfer and general- ization capabilities (Aguilar al., 2020). Concretely,
Algorithm 3 Multi-step distillation
Input: Full-precision model Mo, dataset D
Output: Sorbet S
3: Mj + Quantize(Mo) {Quantize My to 1-bit weight
4-bit activation)
4: M2 +- M1 with PTsoftmax replacing softmax
5: M3 + M2 with BSPN replacing LN
6: for i = 1 do
Taggis KL(P,q), q), Laeps
Σ ||r 1212 (16)
Maacher +- M,-1, Mstudent <- M, ModelDistill Masudent, Miestvery D) end for
where denotes the output distribution of the teacher model, and represents the output of the student model. rē and rf are the corresponding transformer block output activations
10: Convert M3 to SNN and obtain Sorbet S
11: return S
6

---

Sorbet: A Neuromorphic Horpaaie-Compattible Transformer-Ban Based Spiking Language Model
The distillation process yields a quantized model with higher accuracy. Furthermore, as shown in Table 3, our proposed PTsoftmax and BSPN can also be applied and trained di- rectly.
Figure 2. Energy of different operations. Each value repre- sents single execution with an input dimension of 128. Based on 45nm technology. FIX8 division requires 0.59 pl, whereas bit-shift operation requires only 0.024 pJ.
Softmxx
5. Result
PTsofmma
10:75
In this section, we present the performance of Sorbet on seven datasets from the GLUE benchmark, standard eval- uation suite used by plenty of language models. Because SNNs are relatively limited when applied to NLP tasks, there are few existing SNNs evaluated on GLUE, so we compare Sorbet against both SNN and quantized ANN baselines. Our experiments use BERT-base as the initial teacher model for distillation. We also provide comprehensive analysis of the model's energy and power efficiency. All experiments were conducted on three Nvidia RTX A100 GPUs, each with 80GB of memory. The number of timesteps used for all results in this section is 16.
SHIFT
LN-
171.52
DIV
MLL1
RSPN
EXP
50
100
150
200
250
300
Energy Consumption (pJ)
Figure 3. Spike firing rate the output of each block on SST-2
and STS B datascts.
SST-2
0.4
SIS-8
Moun of SST-2:0.13
Spike Rate
0.3
Mean of STS-B 0.15
5.1. Comparing with the Baseline
0.2
Table 2 shows Sorbet's performance on the GLUE bench- mark, indicating that it maintains competitive results overall. achieves state-of-the-art performance on four datasets and attains comparable results on the rest. Compared to binary neural networks such as BiT, Sorbet has the same model size and comparable performance but offers better energy efficiency and compatibility with neuromorphic hardware.
0.1
9
11
Block Index
Nsoorer highly related the spike rate r. For example, the SST-2 and STS-B datasets, the observed average spike rates are 0.13 and 0.15, respectively, as illustrated in Figure 3. Notably, spike rates can be higher when using symmetric quantization. With these observed spike rates, we calculated Sorbet's energy consumption and compared it with other baselines Table 4. Sorbet reduces energy consumption by 27.16 compared to BERT and by 3.16x compared SpikeLM.
Additionally, we also evaluated two existing SNNs, namely, SpikeLM and SpiKingBERT, on GLUE with 1-bit weight quantization. Although they claim to be SNN architectures with spike-generation methods, both depend heavily on LN and softmax operations, which are incompatible with SNN hardware. By contrast, Sorbet avoids these constraints and consequently more suitable for neuromorphic devices.
The simplification operations we propose also substantially reduce the computational load of these functions. Using the reported energy costs of DIV, EXP, and SHIFT opera- tions (You et al., 2020; Nadh et al., 2023; Wu et al., 2019), we calculated the energy consumption of our PTsoftmax and BSPN compared to the standard softmax and LN in Figure 2. The results show that PTsoftmax achieves approximately 27. 27.62x better energy efficiency than conventional softmax, while BSPN achieves 12.4x better energy efficiency than LN. Detailed calculations are provided in Appendix E.
5.2. Energy Saving Analysis
Sorbet achieves substantial energy efficiency in three ways. Firstly, SNN reduces overall energy consumption due to its event-driven nature, activating neurons only when necessary. Secondly, PTsofttax and BSPN replace traditional func- tions with lower-cost operations, further reducing energy use. Finally, the use model quantization further reduces computational load and power requirements.
Compared ANNs, the key energy-saving advantage of SNNs lies in replacing multiplications with additions. The numbers of addition needed in Sorbet( (aabeet((Sotee)) to substitute matrix multiplication in BERT(NBERR) can be expressed as:
5.3. Ablation Study
To evaluate the contribution of our proposed components, we conducted a series of ablation experiments. Specifically, we focused on the effectiveness of the PTsoftmax and BSPN modules. We conducted two ablation studies to evaluate the impact of our proposed modifications. First, we replaced
N Sorbet = NBBET
(18)
where 7 is the timestep and r the spike rate. From Eq. 18,
7

---

Sorbet: A Neuromorphic Hardware-Compattibe Wyansormer-Basse Spiking Language Model
Model
Size
QQP
MNLI-m
SST-2
QNLI
RTE
MRPC
STS-B
BERT base (Devlin al., 2019)
418M
91.3
84.7
93.3
91.7
72.6
88.2
89.4
DistilBERT (Sanh, 2019)
207M
88.5
82.2
91.3
89.2
59.9
87.5
86.9
TinyBERT, (Jiao et al., 2020)
207M
84.6
93.1
90.4
70.0
87.3
83.7
Q2BERT (Zhang et al., 2020)
43.0M
67.0
47.2
80.6
61.3
52.7
68.4
4.4
BiT (Liu et al., 2022)
13.4M
82.9
77.1
87.7
85.7
58.8
79.7
71.1
SpikingFormer (Zhou et al., 2023)
83.8
67.8
82.7
74.6
58.8
74.0
72.3
Spi (Bal & Sengupta, 2024)
50M
86.8
78.1
88.2
85.2
66.1
79.2
82.2
SpikeLM (Xing et al.)
87.9
76.0
86.5
84.9
65.3
78.7
84.3
1-bit SpikingBRRT (Bal & Sengupta, 2024)
83.8
75.4
86.7
80.5
75.8
1-bit SpikeLM (Xing et al.)
87.2
74.9
86.6
84.5
65.7
78.9
83.9
Sorbet $
13.4M
83.4
75.8
89.6
84.6
59.2
78.4
73.6
Sorbet
13.4M
86.5
77.3
90.4
86.1
60.3
79.9
78.1
Table Comparison with the baseline the GLUE benchmark. denotes that the size not reported in the original work. We report Spearman correlations for the STS-B datasct and accuracy for all other datasets. denotes further quantizing BSPN's ' scaling factor to power of two.
Table 3. Ablation study on PTsoftmax and BSPN in full precision ANNs
Model
QQP
MNLI-m
SST-2
QNLI
RTE
MRPC
STS-B
Avg.
BERT-softmax-LN
91.3
84.7
93.3
91.7
72.6
88.2
89.4
87.3
BERT- BERT-PTssftmax-LN
90.8
83.9
91.4
90.8
71.5
85.3
87.6
85.9
BERT-PTsoftmax-BSP
89.7
80.9
91.7
87.4
69.0
81.9
84.4
83.6
Table Energy cost comparison with various baselines. Sorbet is first evaluated on the STS-B dataset, while denotes usage the SST-2 datasct.
Table 5. Ablation study on the impact PTsoftmax and BSPN. is the accuracy drop compared to model using Softmax and LayerNorm the same precision. All models use 1-bit weights; 'Bits' in this table denotes the activation quantization bit-width.
Model
BERT
LIF-BERT
SpikeLM
PTsoftmax
BSPN
Accuracy (%)
FP32
51.41
7.98
3.98
Bits 4
FP16
15.21
3.55
1.77
91.5
Ours
90.8
0.7
1-Bit
0.65 / 0.56
X
91.2
0.3
90.9
0.6
the softmax and LN components in the full-precision BERT model with our PTsoftmax and BSPN, respectively. The per- formance results of this replacement detailed in Table 3. The impact caused by the components is equivalent to the model performance. Compared to our main result Sorbet in Table 2, the accuracy drop from full precision BERT to Sorbet is mainly caused by the quantization of weight and spike generation process, not by the replacement of softmax and normalization. Exploring more accurate ways to perform model quantization and spike generation can be potential future work.
Bit
X
X
81.2
V
x
80.0
1.2
X
79.9
1.3
79.8
ablation study on quantization level and timestep with conversion loss on SST-2 dataset in Appendix D.2.
6. Discussion
Second, we tested the effectiveness of our components in highly quantized BERT models on the SST-2 datasets. The results are presented Table 5. Our proposed PTsoftmax and BSPN can maintain a good performance on full pre- cision and highly quantized models. We also performed
Although we do not have access to physical neuromorphic chips, to demonstrate the neuromorphic hardware compat- ibility of our proposed model. we have implemented and validated the PTsoffmax and BSPN layers using the Lava
8

---

Sorbet: A Neuromorphic Horppaie-CCompatible Transformer-Bant Spiking Language Model
framework, targeting Intel's Loihi architecture. We further synthesized PT'softmax and BSPN layers in Verilog for commercial 22 nm process to estimate power consumption as in Appendix E. While these methods demonstrate sub- stantial energy-efficiency gains, they cannot fully capture real hardware effects. Future work will prioritize deploy- ment on actual neuromorphic platforms to obtain empirical power and latency measurements, validate our simulation fidelity, and guide hardware-aware refinements.
References
Aguilar, G., Ling, Y., Zhang, Y., Yao, B., Fan, X., and Guo, C. Knowledge distillation from internal representations. In Proceedings of the AAAI conference on artificial intel- ligence, volume 34, pp. 7350-7357, 2020.
Akopyan, F., Sawada, J., Cassidy, A., Alvarez-lcaza, R., Arthur, J., Merolla, P., Imam, N., Nakamura, Y., Datta, P., Nam, G.-J., et al. Truenorth: Design and tool flow of 65 mw 1 million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 34(10):137-11557 2015.
7. Conclusion
In this paper, we presented Sorbet, the first fully neuro- morphic hardware-compatible, transformer-based spiking language model. Sorbet addresses the challenge of adapting transformer- models for energy-efficient computation by replacing traditional energy-intee -inerggint operations like soft- max and layer normalization with our novel P'ssfftmax and BSPN. This challenge has been largely overlooked in previ- ous works in this field. Furthermore, by using techniques such as knowledge distillation and model quantization, we achieved a highly compressed binary weight model, fur- ther optimizing the model for real-world deployment on neuromorphic hardware. When evaluated the GLUE benchmark, Sorbet achieved performance comparable to state-of-the art models with substantial energy savings.
Bai, H., Zhang, W., Hou, L., Shang. L., Jin, J., Jiang, X., Liu, Q., Lyu, M., and King. I. Binarybert: Pushing the limit of bert quantization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pp. 4334- 4348, 2021.
Bal, M. and Sengupta, A. Spikingbert: Distilling bert to train spiking language models using implicit differentia- tion. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 10998 1006. 2024.
Benjamin, B. V., Gao, P., McQuinn, E., Choudhary, S., Chandrasekaran, A. R., Bussat, J.-M., Alvarez-Icaza, R., Arthur, J. V., Merolla, P. A., and Boahen, K. Neurogrid: A mixed-analog aaad--annlo--gitttt multichip system for large-scale neural simulations. Proceedings of the IEEE, 102(5): 699-716, 2014.
At the time of writing this paper, DeepSeek has become a global phenomenon. DeepSeek-V3 (De, 2024) uses quantization to FP8 on Nvidia GPUs and standard floating point softmax and normalization operations, while DeepSeek-R1 (DeepSeek-AI, 2025) innovates in the train- ing and distillation processes to achieve energy and resource- efficient models. We believe that the techniques of Sorbet can be adapted to make them even more so, especially for inference on customized embedded architectures. This shall be the focus of our future work.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Acknowledgment
Bu. T., Fang, W., Ding, J.. DAI, P.. Yu, Z., and Huang, T. Optimal ann-snn conversion for high-accuracy and ultra- low-latency spiking neural networks. In International Conference on Learning Representations.
This research is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2024).
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficiiet exact attention with io-awareness. Advances Neural Information Process- ing Systems, 35:16344 16359, 2022.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Davies, M., Srinivasa, N., Lin, T.-H.. Chinya, G., Cao, Y., Choday, S. H., Dimou, G., Joshi, P., Imam, N., Jain, S., et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38( 38(1):22-9, 2018
DeepSeck-Ae e. a. DeepSeck V3 technical report. arXiv preprint arXiv:2412. 2024.

---

Sorbet: A Neuromorphic Horpaaie-Comppatible Transformer-Ban Baaeed Spiking Language Model
DeepSeek-Al, e. a. DeepSeek-RI: Incentivizing reason- ing capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., and Keutzer, K. I-bert: Integer-only bert quantization. In International conference on machine learning, pp. 5506- 5518. PMLR, 2021.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- Proceedings of the 2019 confer- ence of the North American chapter of the association for computational linguisties: human language technologies, volume 1 (long and short papers), pp. 4171-4186, 2019.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. teaeeneee---eeeencee pre-training for natural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pp. 7871-7880, 2020.
guage understanding.
Bart: Denoising
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x 16 words: Transformers for image recognition scale. In International Conference Learning Representations, 2020.
Li, Y., Lei, Y., and Yang, X. Spikeformer: Training high- performance spiking neural network with transformer. Neurocomputiing 574:127279, 2024.
Li, Z. and Gu, Q. I-vit: Integer-only quantization for effi- cient vision transformer inference. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 17065-17075, 2023.
Eldan, R. and Li, Y. Tinystories: How small can lan- guage models be and still speak coherent english? arXiv preprint arXiv:2305.07754 2023.
Liu, Z., Oguz, B., Pappu, A., Xiao, L.,, Yih, S., Li, M., Krishnamoorthi, R., and Mehdad, Y. Bit: Robustly bi- narized
Guo, Y., Huang, X., and Ma, Z. Direct learning-based deep spiking neural networks: review. Frontiers in Neuroscience, 17:1209799 2023.
transformer. Advances in neural information processing systems, 15:144333-43166 2022.
Han, D., Pan, X., Han, Y., Song, S., and Huang, G. Flatten transformer: Vision transformer using focused linear at- tention. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5961-5971, 2023.
Lobov, S. A., Mikhaylov, A. N., Shamshin, M., Makarov, V. A., and Kazantsev, V. B. Spatial properties of stdp in a self-learning spiking neural network enable controlling a mobile robot. Frontiers in neuroscience, 14:88, 2020.
Horowitz, M. 1.1 computing's s energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC), pp. 10-14. IEEE, 2014.
Lu, J., Yao, J., Zhang. J., Zhu, X., Xu, H., Gao, W., Xu, C., Xiang, T., and Zhang, L. Soft: Softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34:21297-21309 2021.
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163- 4174, 2020.
Lv, C., Li, T., Xu, J., Gu, C., Ling. Z., Zhang, C., Zheng, X., and Huang, X. Spikebert: A language spikformer trained with two-stage knowledge distillation from bert. arXiv preprint arXiv:2308. 15/22, 2023.
Nadh, J. N., Sk, A., Veeramachaneni, S., and Sk. N. M. Energy efficient approximate floating point divider. In 2023 International Conference Computer, Electronics & Electrical Engineering & their Applications (IC2E3), pp. 1-5, IEEE, 2023.
Kasabov, N.. Dhoble, K., Nuntalid, N., and Indiveri, G. Dynamic evolving spiking neural networks for on-line spatio-and
pattern recognition. Neural Networks, 41:188-200 2013.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165. PMLR, 2020.
Raffel, C., Shazeer, N. Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67,29) 2020.
Kim, H., Hwang, S., Park, J., Yun, S., Lee, J.-H., and Park, B.-G. Spiking neural network using synaptic transistors and neuron circuits for pattern recognition with noisy images. IEEE Electron Device Letters, 39(4):630-6) 633, 2018.
Sanh, V. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of Thirty-third Conference on Neural Information Processing Systems (NIPS2019), 2019.
10

---

Sorbet: A Neuromorphic Horptabe-Compattibl Transformer-Ban Based Spiking Language Model
Shen, S., Yao, Z.,, Gholami, A., Mahoney, M., and Keutzer, K. Powernorm: Rethinking batch normalization in trans- formers. In International conference machine learning, pp. 9741-8751. PMLR, 2020.
Yao, M., Hu, J., Zhou, Z., Yuan, L., Tian, Y., Xu, B., and
Li, G. Spike-driven transformer. Advances in Neural Information Processing Systems, 36, 2024.
You, H., Chen, X., Zhang, Y., Li, C., Li, S., Liu, Z., Wang. Z., and Lin, Y. Shiftaddnet: A hardware-inspired deep network. Advances in Neural Information Processing
Shi, X., Hao, Z., and Yu, Z. Spikingresformer: Bridging resnet and vision transformer in spiking neural networks. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 5610-5619, 2024.
Systems, 13:2771-2788, 2020.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama:
An open-source small language model. arXiv preprint arXiv:2401.02383 2024.
Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., and Zhou, D. Mobilebert: compact aggosticc bert for resource- limited devices. In Proceedings of the 58th Annual Meet- ing of the Association Computational Linguistics, pp. 2158 2170, 2020.
Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., and Liu, Q. Ternarybert: Distillation-wwar ultra-low bit bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2020.
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: large language model for science. arXiv preprint Xiv::2211.09085 2022.
Zhang. Y., Zhang, Y., Peng. L., Quan, L., Zheng, S., Lu, Z., and Chen, H. Base-2 softmax function: Suitability for training and efficient hardware implementation. IEEE Transactions on Circuits and Systems I: Regular Papers, (909))3005-361, 2022.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint (.2.::23....3777// 2023.
Zhou, C., Yu, L., Zhou, Z., Ma, Z., Zhang, H., Zhou, H., and Tian, Y. Spikingformer: Spike-driven residual Iearn- ing for transformer- based spiking neural network. arXiv preprint arXiv:2304. /1954, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Zhou, Z., Zhu, Y., He, C., Wang, Y., Shuicheng, Y., Tian, Y., and Yuan, L. Spikformer: When spiking neural net- work meets transformer. In The Eleventh International Conference on Learning Representations.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and anal- ysis platform for natural language understanding. In International Conference Learning Representations.
Zhou, Z.., Che, K., Fang, W., Tian, K., Zhu, Y., Yan, S., Tian, Y., and Yuan, L. Spikformer v2: Join the high accuracy club imagenet with an snn ticket. arXiv 2024.
Wu, D., Chen, T., Chen, C., Ahia, O., San Miguel, J., Lipasti, M., and Kim, Y. Seco: A scalable accuracy approximate exponential function via cross-layer optimization. In 2019 IEEF/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1-6. IEEE, 2019.
preprint
Zhu, R.-J,, Zhao, Q., Li, G., and Eshraghian, J. Spikegpt: Generative pre-trained language model with spiking neu- ral networks. Transactions on Machine Learning Re- search.
WU, X., Song, Y., Zhou, Y., Bai, Y., Li, X., and Yang. X. Stca-snn: self-attention-bassee temporal-channel joint attention for spiking neural networks. Frontiers in Neuro- science, 17: 17:1261543, 2023.
Xing, X., Zhang. Z., Ni, Z., Xiao, S., Ju, Y., Fan, S., Wang, Y., Zhang, J., and Li, G. Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms. In Forty-first International Conference on Machine Learning.
Yan, Z., Tang, K., Zhou, J., and Wong, W.FF Low latency conversion of artificial neural network models to rate- encoded spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2025.

---

Sorbet: A Neuromorphic Horpaale-Compattibl Transformer-Bant Based Spiking Language Model
A. Spike Generation
provide detailed spike generation method we adopted in algorithm 4, where why is the weight of layer from neuron to neuron j. bt is the bias of the neuron in layer 1, st is the input spike train of the neuron i, and 7 is the time window size.
Algorithm 4 Average IF model
1: Input: Weight w'y, bias bý, input spike train s/ threshold 6, membrane potential (t( at timestep t, time window size
T:
2: Output: Spike train s;i
(1()) - E,(wl st b))
4: A ME U///U( (
5: (0) + 0, 0,( < 0;
6: fort 1
7:
Vi(t) + (1 - + A
8: (1)+ + 1, V1(()= 20
(0, otherwise
9: VI( + Vi(()) 8, (t( 2 8 (t((),
otherwise
end for 11: return )
1+1
(Output the ASG spike train of neuron i}
B. Theoretical Results
B.1. Proof BSPN
Assumption B.1 (Power-of-Two Condition on Normalization Input). For a Transformer-based SNN, the input to the normalization layer X satisfies
[log, (S(X)) 2 0,
where
1 n Σ X,/- S(X)
This condition is mild and consistent with empirical observations in Transformer-basee SNNs. The empirical results shown in Appendix D.1.
Based on these results, we have the following proof of Theorem 4.2 and Lemma 4.3.
Proof of Theorem 4.2. Since LBSPN = LPN (LNN(((X)) (X)), based on chain rule, we have:
ÖLBSPN (X)
ÖLPN (x())
($(X)
(19)
0X
(4())
0X
(
8X ØX
2 [[(x)))
8 2 foo,, (S(X)) TX (2 ((())))
X
(20)
0X
Since
0X
1,
and
2
log, (og,(((())
0
(almost everywhere),
(21)
0X
0x
we obtain
(Φ(
In
(22)
0X
[S(X)
(S(X))
12

---

Sorbet: A Neuromorphic Horptabe-Compatiibe Transformer-Ban Based Spiking Language Model
Thus, the gradient of LBSPN X)
can be expressed
ÖLBSPN (X)
@Lpx(9(X))
(23)
OX
2 logg (S(X))
(x()
As the gradient of any X is bounded for (sxesee section 4.2 in (Shen et al., 2020)):
[ (X)
S
2o,g (S(X)) C
(24)
0x
2
Now, under the mild Assumption B.I, we have:
(log(((())) 2 0.
(25)
Thus,
2 (8(x)]
2o, (S(X)) S 1.
(26)
Therefore, conclude:
SC
(27)
OX
This establishes the gradient bound.
Proof of Lemma 4.3. Let kx [log2 (S(x)) and ky = [log2 (S(Y)) under Assumption B.I, we have kx, ky 2 0, hence
1
S 1.
(28)
2
2ky
Therefore, we have:
X
Y
2kv X 2xx
xxx) - - =
2
2ky
=
2bx 24x
S 12 X -
(29)
< max(2kr 2/xx )||x - Y|l s ||x Yll.
(30)
B.2. Proof of P'Tsoftmax
In this section, we provide detailed proof the approximation error of PTsoftmax and base-2 softmax.
We claim the Proof of Lemma 4.5. Let a; F2(2)), b, Fa([xa]), a 2[=1]- where & [1o83 (E}=1?%) ). following inequalities:
1. bb S a. < 2h:-
2. J < by S /20,.
< at + we consider the worst-case scenarios. For the right side, the worst case occurs when Ij = [x,] + e. where e is an arbitrarily small value, no greater than 1, and [x,] 2z for all j. Denote [ææ] as E E.
Proof of 1: Since vi < [xt]
Then [x,] = n; + 1, so we have:
21
2.
6
(31)
5= 20,
2nite + 2
E/=1 2
- +
2 n
2
2a
(32)
2==1 2n
N
2j=1 2n
N
13

---

Sorbet: A Neuromorphic Honptaie-Compattible Wransormer-Basce Spiking Language Model
For the left side, the worst case occurs when and we have:
[ang] 3i and 2y ny + for some j. where ny Lary] Then [2y]]
ny
2
2
bi =
(33)
ΣN j==
2ns+1
20:+1 + 2z.
2. Σ j==1
2
2
2
2
2 di
(34)
2 Σ 2nd
2.
Given that l = [0gg E== 2*, ]}. we have the following bounds:
Proof of 2:
k - 0.5 < logz
Σ 2|25]
< le + 0.5.
(35)
Thus, the ratio
bj
2*
(36)
C
Σ 2/=1
2|0²]
f=1
satisfies
2
bi
2
<
22
(37)
V2
2.+0
2.5
C. Evaluation Benchmark
We evaluate our Sorbet on 7 distinct datasets in the GLUE benchmark as follows:
MNLI: The MNLI (Multi-Genre Natural Language Inference Corpus) is involved in natural language inference tasks.
It consists of a collection of sentence pairs annotated for textual entailment through crowdsourcing.
QQP: The QQP (Quora Question Pairs) pertains to tasks involving similarity and paraphrase identification, focusing pairs of questions from the community Q&A website, Quora. The primary objective of this task is to ascertain whether pair of questions are semantically equivalent.
The QNLI (Question-answering Natural Language Inference) is task in natural language inference. QNLI is derived from another dataset, the Stanford Question Answering Dataset (SQuAD 1.0), which is a preestion-paaggrapp pair question-answerte dataset where the paragraphs are sourced from Wikipedia.
QNLI:
SST-2: The SST-2 (Stanford Sentiment Treebank) is single-sentence classification task that involves sentences from movie reviews and their sentiment annotations by humans. This task requires classifying the sentiment of given sentence into positive and negative sentiment.
STS-B: The STS-B (Semantic Textual Similarity Benchmark) comprises collection of sentence pairs extracted from sources such as news headlines, video titles, image captions, and natural language inference data. is a regression task.
The RTE (Recognizing Textual Entailment datasets) are from natural language inference tasks. It consolidates datasets from a series of annual textual entailment challenges, with data samples constructed from news sources and Wikipedia.
RTE:
MRPC:
The MRPC (Microsoft Research Paraphrase Corpus) is involved in similarity and paraphrase tasks. It consists of sentence pairs automatically extracted from online news sources, with human annotations to determine the sentences are semantically equivalent. The categories are not balanced, with 68% of the samples being positive instances.
14

---

Sorbet: A Neuromorphic Hardware-Compatibbe Transformer-Ban Based Spiking Language Model
D. Extra Results
D.1. Empirical Results
We present the empirical results for [log, [S collected from different layers of Sorbet in Figure 4, These results confirm that Assumption B.1 holds, all values are greater than zero.
Figure 4. Distribution of S(X) measured from various Sorbet layers. The strictly positive values support the assumption made in Assumption B.I.
4
3
Values
2-
0
0
2
3
4
5
Layer Index
8
6
Values
4
0
6
7
8
10
11
Layer Index
D.2. Ablation on Quantization Levels
We provide additional results for different quantization levels and timesteps as a supplement to our ablation study on the SST-2 dataset, shown in Table 6.
15

---

Sorbet: A Neuromorphic Hardaaiee-Compatible Transformer-Bant -Breeddee----dde------ Spiking Language Model
Table Accuracy comparison between quantized ANNs and SNNs under different activation bits timesteps. ó accuracy drop when converting quantized ANNs into SNNs.
Activation Bits Timestep
1/2
2/4 4
3/8
4
16
Quantized ANN
79.8
87.9
90.1
90.9
Accuracy(%)
SNN
78.7
87.4
89.3
90.4
1.1
0.5
0.8
0.5
E. Energy Analysis
E.1. Computational cost for whole model
For assessment of the energy consumption of the whole model, we calculate based on the energy usage of multiplications and additions. For sorbet, the weights are binary while the timestep 16, so the maximum accumulation can represented with 1-bit FIX and 4-bit FIX. We tested on 22nm technology 0.01215pl. As all the baselines are under 45nm process technology, make it fair comparison, we approximate accumulation in Sorbet as ÉACC 0.0243pJ.
Note that for original ANN BERT, use EMAC 4.6pJ (Horowitz, 2014) for multiplication. Thus, with the spike firing rate and the total number of time steps T, the energy cost of Sorbet can be calculated as:
FACC
EsSurbet T r- EBERT
(38)
EMAC
E.2. Computational cost for operations
In Table 7, we list for input with dimension d, the needed operations for different softmax and normalization:
Table 7. Computational cost comparison of the PTsoftmax and BSPN with their equivalents.
X
2
v
LUT
softmax
PTsoftmax
n
1
LayerNorm
3n 2
2n
2n
11 + 2
1
BSPN
2n
2n 1
For our energy calculation in Figure 2, we count the major operations as 0.03pJ for addition, 0.2pJ for multiplication, 0.024pJ for shifting (You et al., 2020), 0.59pJ for division (Nadh et al., 2023) and 1.7pJ for exponential (Wu et al., 2019). To make it fair comparison, we use 8-bit integers for all these operations. For the results in Figure 2, we take 128. For example, the energy cost of applying softmax once can calculated as:
Exotmax = d. (EADD + EDIV + ÉEXP) 128 (0.03 + 0.59 + 1.7) 296.96p.J
(39)
16


Reviews:
[{"review_id": "9L5edK6kzS", "reviewer": "Reviewer_GqJG", "summary": "This paper proposed a Spiking Transformer language model, named Sorbet, designed for neuromorphic hardware. Sorbet introduces two approximations, PTsoftmax and BSPN, to replace traditional softmax and layer-wise normalisation.  The aim of them is to make the model neuromorphic compatible and energy-efficient. PTsoftmax replaces softmax's exponential and division operations with bit-shifting.  BSPN approximates the L1 norm to the nearest power of two, avoiding square and square root operations.  Sorbet also integrates binary weight quantization with knowledge distillation to further reduce the model's computational cost.  On the GLUE benchmark, Sorbet can achieve competitive performance, yet with much reduced energy consumption.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable suggestions. Below are our responses, which we hope will address your concerns.\n\n---\n> Q1: Hardware focus and evaluation on actual chips\n\nWe appreciate your suggestions. To demonstrate the neuromorphic hardware compatibility of our proposed model, we have implemented and validated the PTsoftmax and BSPN layers using the Lava framework, targeting Intel's Loihi architecture. We created an AbstractProcess along with its corresponding PyLoihiProcessModel, deployed within Lava's simulation environment Loihi1SimCfg. \n\nWe do not have access to physical neuromorphic chips. However, beyond the simulation, we implemented our design in Verilog and evaluated its power consumption using a commercial 22nm FD-SOI technology process. The results shows that PTsoftmax and BSPN achieve approximately $27.63\\times$ and $12.4\\times$ better energy efficiency than conventional operations. We will include the result into our final version.\n\n---\n\n> Q2: Compare with more baselines\n\nThank you for the suggestion, we have added the following comparison and will include them in our result section. In terms of FLOPs, TinyBERT\\_6 and DistilBERT\\_4 reduce energy by 2.0× and 3.0×, respectively, compared to BERT, while Sorbet achieves a remarkable 27.16× reduction:\n\n| Model      | Size(MB) | QQP  | MNLI-m | SST-2 | QNLI | RTE  | MRPC | STS-B |\n| ---------- | -------- | ---- | ------ | ----- | ---- | ---- | ---- | ----- |\n| BERT_base  | 418      | 91.3 | 84.7   | 93.3  | 91.7 | 72.6 | 88.2 | 89.4  |\n| DistilBERT | 207      | 88.5 | 82.2   | 91.3  | 89.2 | 59.9 | 87.5 | 86.9  |\n| TinyBERT_6 | 207      | -    | 84.6   | 93.1  | 90.4 | 70.0 | 87.3 | 83.7  |\n| Sorbet     | 13.4     | 83.4 | 75.8   | 89.6  | 84.6 | 59.2 | 78.4 | 73.6  |\n\nWe did not include SpikeBERT and SpikeGPT because they used different datasets except the SST-2. A comparison of these two models using SST-2 would be:\n\n| Model  | Size(MB) | Energy(mJ) | Acc  |\n| ---------- | -------- | ---------- | ---- |\n| SpikeGPT   | 216    | -   | 88.8 |\n| SpikeBERT  | -    | 28.54      | 85.4 |\n| **Sorbet** | 13.4     | 0.56       | 89.6 |\n\nRegarding ANN to SNN conversion methods, we performed the conversion after quantizing activations to 4 bits. This approach aligns with the mainstream ANN-to-SNN conversion techniques, such as QCFS [1] and [2]. The core idea behind [1] and [2] is to clip and quantize the activations to make the ANN model behave more like an SNN, thereby minimizing conversion loss. Our work follows this approach, incorporating advanced multi-step distillation inspired by [3] to obtain a quantized model with improved performance.\n\n[1] Bu T, Fang W, Ding J, et al. Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks[C]. ICLR, 2023.\n\n[2] Shen G, Zhao D, Li T, et al. Are conventional snns really efficient? a perspective from network quantization[C]. CVPR, 2024.\n\n[3] Liu Z, Oguz B, Pappu A, et al. Bit: Robustly binarized multi-distilled transformer[J]. Neurips, 2022.\n\n---\n\n> Q3: Results in Table 2\n\nIn Table 2, we reported 2 results for SpikingBERT and SpikeLM. The results of their original paper are in lines 336 and 337. We noticed they further quantized their models to 1-bit ( SpikeLM reported in their original papers, while SpikingBERT reported separately in [4]). To make a fair comparison, we included quantized results in lines 338 and 339, denoted as 1-bit models.\n\n[4] Bal M, Jiang Y, Sengupta A. Exploring Extreme Quantization in Spiking Language Models[C]// ICONS, 2024.\n\n---\n\n> Q4: Ablation for different timesteps\n\nThe timestep used for all results reported is 16. We will make this explicit in the paper. We performed an ablation study for timesteps on SST-2 dataset:\n\n| Timestep | 2  | 4 | 8  | 16 |\n| ------ | ---- | ---- | ---- | ---- |\n| Accuracy | 78.7 | 87.4 | 89.3 | 90.4 |\n\n---\n\n> Q5: Small typos and code accessibility\n\nThank you for pointing out. We fix them and take your suggestion to replace Figure 3 with a table. The repository had expired, but we have extended it. The code is available now in the same link provided in the manuscript.\n\n---\n\n> Q6: Use Sorbet as a multimodal LLM\n\nThe components we have designed can be used on any model that uses the transformer mechanism. Therefore, it should be easily extendable to multimodal LLMs with customized training processes.\n\n---\n\n> Q7: Calculation of energy saving\n\nWe calculate our energy savings as:\n$$\nN_{\\text{saving}} = \\frac{E_{\\text{BERT}}}{E_{\\text{Sorbet}}} = \\frac{15.21}{0.56} = 27.16\n$$\n\nWe use 15.21mJ for FP16 BERT_base from SpikeLM (Xing et. al, 2024). For $E_\\text{Sorbet}$, as in Appendix D.1:\n\n$$\nE_{\\text{Sorbet}} = T \\cdot r \\cdot E_{\\text{BERT}} \\cdot \\frac{E_{AC}}{E_{MAC}}.\n$$\nwhere $r$ is $0.13$ and $T$ is $16$. When using ${E_{MAC}} = 4.6pJ$, the $E_{\\text{BERT}}$ should be FP32 BERT as 51.41mJ. Our operations are essential for SNNs but contribute less to energy savings, so we excluded them from the full model evaluation.", "claims_and_evidence": "One of the key claims of this study is a neuromorphic hardware compatible model.  Hence evidence provided should have a hardware focus, which is not the case in the paper.  Can the model be evaluated on actual chips, such as Loihi, IBM TrueNorth, or NeuroGrid? It is more important than showing high efficiency.\n\nThe choice for baseline methods needs a bit update to substantiate the claim of high performance high efficiency of Sorbet. There are some efficient models like DistilBERT [1] or TinyBERT [2] which are designed for edge devices. How would Sorbet compare in terms of energy efficiency and performance trade-offs against these efficient models? In addition,  recent spiking transformers should be considered in the comparison, such as spikeBERT [3].  Also ANN to SNN conversion methods should be considered as well such as QCFS [4].  SpikeGPT is mentioned in the paper, but not included in the comparison.\n\n[1] Sanh et al., “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”, NeurIPS 2019. \n[2] Jiao et al., “TinyBERT: Distilling BERT for Natural Language Understanding”, EMNLP 2020.  \n[3] Lv et al., Spikebert: A language spikformer learned from bert with knowledge distillation, AAAI 2024\n[4] Bu, et al. Optimal ann-snn conversion for high-accuracy and ultra-low-latency spiking neural networks, ICLR 2022. \n\nIn Table 2, the performance of SpikingBERT is list as: 83.8 75.4 86.7 80.5 - 75.8 - on the seven GLUE datasets.  However the original paper reported better results: 86.82 78.10 88.19 85.20 66.06 79.17/85.15 82.20/81.90 respectively.  Why is there such discrepancy?  The results from SpikeLM are also different to that on Xing's paper.\n\nAnother point is the time step.  It is not mentioned in the main results on Table 2.  Does Sorbet fix it to 16?  If so, what would happen for a different timestep?", "methods_and_evaluation_criteria": "The proposed approximation methods, PTsoftmax and BSPN, are interesting and promising.  In this sense, this study may lead to a practical way to replace energy-intensive operations in Transformers, which could be a critical step for enabling language models on neuromorphic hardware.\n\nThe evaluation is performed on the GLUE benchmark, comparing Sorbet's performance with several baselines, including quantised models and other SNN-based language models.  This study is solid in this regard.", "theoretical_claims": "It is nice to see theoretical proofs in the paper.  The analysis confirms that BSPN maintains bounded gradients, making it a robust and efficient alternative to LN.  Also, although PTsoftmax does not strictly sum to 1, the analysis shows that this discrepancy has a minor impact on performance.\n\nIn addition, the ablation studies confirm that the use of PTsoftmax and BSPN introduces minimal performance degradation.", "experimental_designs_or_analyses": "The paper’s experimental design and analyses appear to be sound and comprehensive.  The comparison is performed using the well established GLUE benchmark. Several baselines, e.g., BERT, SpikeLM, and other quantized models, are involved.  The energy saving analysis and theoretical proof are included in the paper.  In addition, ablation studies are performed to show the contributions of PTsoftmax and BSPN.", "relation_to_broader_scientific_literature": "This work could contribute towards high-performance transformers designed for neuromorphic hardware, offering theory and practice of efficient neural network design.", "essential_references_not_discussed": "See the claims section.  More references should be added.", "other_strengths_and_weaknesses": "See above", "other_comments_or_suggestions": "SpikeLM by Xing et al., \"SpikeLM: Towards general spike-driven language modeling via elastic bi-spiking mechanisms\" is published in ICML 2024, not just an arXiv article.\n\nFigure 3. is not necessary, a simple table would be clear and more concise.\n\nTwo latex problems in Table 1 caption, '+'  --> `+'\n\nLine 370 Where -> where", "questions_for_authors": "How to use Sorbet as a multimodal LLM?  That is an important extension to be considered.\n\n\"27.16× energy savings compared to BERT\" was mentioned multiple times.  How was it calculated?\n\nAlso see questions in the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RgyVeev6aj", "reviewer": "Reviewer_ck8m", "summary": "The authors propose Sorbet: A transformer-based spiking language model optimized for neuromorphic hardware, enhancing energy efficiency while maintaining strong performance. It introduces BitShifting-based PowerNorm (BSPN) for normalization and Power-of-Two softmax (PTsoftmax) as a hardware-friendly alternative to softmax. Through binary weight quantization via knowledge distillation, Sorbet achieves 27.16x energy savings over BERT and 3.16x compared to SpikeLM while remaining competitive on the GLUE benchmark.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable suggestions. Below are our responses, which we hope will address your concerns.\n\n---\n\n> Q1: Deployment of SNNs on neuromorphic hardware\n\nWe have evaluated the hardware compatibility with the Lava framework to simulate Loihi chip. However, the platform does not provide the energy cost. So we independently implemented our designs of the two functions in Verilog and tested power consumption using a commercial 22nm FD-SOI technology process and reported in the manuscript. The results indicate that our proposed functions PTsoftmax and BSPN achieve approximately 27.63x and 12.4x better energy efficiency compared to conventional implementations.\n\n---\n\n> Q2: Latency and inference time\n\nA direct comparison between ANNs and SNNs on the issue of latency is not straightforward. It will depend on model and hardware parameters, as well as circuit implementation and optimization. Also, below certain thresholds required by the application, it becomes a non-issue.\n\nAll things being equal, the latency of SNNs increases with the timestep. We experimented with setting the Sorbet timestep to 2 and achieved an accuracy of 78.7 on the SST-2 dataset, demonstrating the robustness of Sorbet. \n\nCompared to an ANN, even using the same model size, the hardware circuits of an SNN would be simpler than an ANN. This would translate to a higher frequency. While it may take a few cycles for an SNN to produce what an ANN can do in a single cycle, because of the higher frequency, the SNN may still perform better on the end-to-end latency, or at least be able to satisfy the application's requirements.\n\n---\n\n> Q3: Hardware Dependency\n\nWhile SNNs are still not widely used, many companies are actively exploring neuromorphic hardware such as Intel, IBM, BrainChip, Qualcomm, and so on, reflecting the strong interest and potential in this emerging field. We believe that once SNN models can achieve comparable performance of advanced ANNs and be efficiently deployed on neuromorphic hardware using techniques such as what is proposed here, these chips will be more widely adopted as viable alternatives to overcome the power and latency limitations of traditional digital computing.\n\n---\n\n> Q4: Limitation\n\nWe will add a discussion section on the limitations of Sorbet. Sorbet is designed for SNN deployment specifically on edge devices. Sorbet optimizes the model with constrained computational resources. Hence, it probably will not outperform larger, more complex models in environments without resource constraints.\n\n---\n\n> Q5: Figure Caption\n\nThanks for pointing this out, we will fix it in our paper. We have extended the repository, and the code is available at the same link provided in the manuscript.", "claims_and_evidence": "Yes, the claims appear well-supported by the experimental results and visualizations.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, like theorems have proof in the appendix section. I read them and seemed justified enough.", "experimental_designs_or_analyses": "Yes, the experiments look sound. Like, the paper evaluates Sorbet against multiple SOTA methods and includes ablation studies to highlight the effectiveness of the PTsoftmax and BSPN modules.", "relation_to_broader_scientific_literature": "The paper contributes within the domains of small language models for edge devices, SNNs for energy efficiency, and transformer-based SNNs for NLP tasks. It also builds on research in quantized BERT models and simplified transformer architectures. The authors acknowledge prior work in these areas and present Sorbet as a solution to key limitations in existing approaches, particularly for NLP tasks on neuromorphic hardware.", "essential_references_not_discussed": "I believe the authors have covered a range of relevant references related to SNNs, transformer-based models, and quantization techniques.", "other_strengths_and_weaknesses": "Strengths:-\n\n1.) The paper is well-written and articulated.\n\n2.) The authors propose the first transformer-based spiking language model that removes softmax and Layer Normalization.\n\n3.) PTsoftmax and BSPN are proposed to replace softmax and layer normalization, using bit-shifting instead of costly operations, making Sorbet more efficient for neuromorphic hardware. \n\n4.) They propose Sorbet, a binary spiking language model derived from BERT, integrating full quantization and design refinements to enable low-power, high-efficiency inference comparable to ANN models.\n\n5.) A broad range of experiments (multiple datasets, models, baselines, and ablations) are conducted, and their experimental results demonstrate the effectiveness of Sorbet over existing methods. It achieves 27.16x energy savings over BERT and 3.16x over SpikeLM while maintaining stable performance on the GLUE benchmark.\n \n\nWeaknesses:-\n\n1.) The authors did the theoretical energy estimation of SNN architectures rather than empirical validation. The efficacy of their approach (PTsoftmax and BSPN) would have been more realized if the authors had deployed the converted SNN on neuromorphic hardware —such as Intel Loihi, TrueNorth, or BrainChip Akida —and provided measured power consumption data.  \n\n2.) While SNNs offer low power consumption, they inherently introduce additional latency due to spike processing over T timesteps. Although the proposed Sobret is made quantized and more efficient for neuromorphic accelerator, it still results in higher latency compared to ANNs. For real-world applications where inference time is critical, it remains unclear how SNNs can effectively address this challenge. A quantitative latency analysis, particularly in time-sensitive scenarios, would have been valuable. Additionally, evaluating the inference time could have provided deeper insights into the Sobret’s practical impact.\n\n3.)  Hardware Dependency: The practical deployment of SNN models rely on neuromorphic hardware, which seems limited in accessibility and widespread adoption. The authors should have addressed this limitation in their study.\n\n4.)  A discussion of the potential limitations of the Sorbet approach would enhance the paper's credibility.", "other_comments_or_suggestions": "Correct the orientation of captions of Figs. 2 and 3.", "questions_for_authors": "I would like all points under weaknesses to be addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JQwTU6IanP", "reviewer": "Reviewer_MRPY", "summary": "This paper argues that current Transformer-based SNN language models are difficult to deploy on neuromorphic chips due to the presence of softmax and layer normalization operations. To address this challenge, the authors propose Sorbet, a model that is more compatible with neuromorphic hardware. Sorbet is based on the concept of shifting and integrates novel PTsoftmax and BitShifting-based PowerNorm algorithms, which avoid complex operations like division and exponentiation. Additionally, the paper introduces techniques such as knowledge distillation and model quantization to enhance model performance and energy efficiency. Sorbet achieves comparable performance to other SNN language models on the GLUE benchmark.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable suggestions. We hope our response below can address your concerns.\n\n---\n> Q1 & 2: Pretraining, fine-tuning, and distillation\n\nWe use a pre-trained and fine-tuned BERT from HuggingFace as a starting point. We then applied distillation and quantization techniques, as in Algorithm 4, without further fine-tuning. After these, the weights are directly converted into a corresponding SNN. This part is not the main contribution of Sorbet. **In our Sorbet experiments, the teacher model refers to BERT_base**. We will clarify this in the updated version.\n\n---\n> Q3: Multi-step distillation(MSD)\n\nThe MSD method we use is based on BiT[1]. Section 5.5 of that paper showed that MSD yields better performance. MSD is also adopted in other recent works[2]. We performed distillation in three steps because we applied three major modifications to the model before SNN conversion.\n\n[1] Liu et al. Bit: Robustly binarized multi-distilled transformer. Neurips, 2022.\n\n[2] Han et al. Amd: Automatic multi-step distillation of large-scale vision models.ECCV, 2024.\n\nWe will add the hyperparameters in appendix. We included them in the same link of our code.\n\n---\n> Q4: Skipping activation quantization\n\nActivation quantization is essential for SNNs from ANN-to-SNN conversion to reduce timestep and maintain accuracies. A quantized ANN can resemble an SNN, allowing the model to learn how to represent quantized values in the equivalent SNN. Recent studies have also proposed lossless conversion methods through activation quantization[3].\n\n[3] Shen et al. Are conventional snns really efficient? A perspective from network quantization. CVPR 2024.\n\nAs an ablation study, we convert a full-precision BERT into an SNN with T=16. The model achieved only 50.92 accuracy on the SST-2 dataset, indicating a failure to represent the data.\n\n---\n> Q5: Two versions of Sorbet in Table 2\n\nThe two versions of Sorbet only differ in their **quantization strategies for BSPN**. For the Sorbet$\\ddagger$(line 340), we quantize BSPN’s weight to powers of two. These weights refer to the scaling factor $\\frac{\\gamma}{\\psi}$ in line 15 of Algorithm 1. We will clarify this in our revised manuscript.\n\n---\n> Q6: 'Bits' in Table 4\n\nIn Table 4, 'Bits' refers to the quantization bit-width of activations. For 'Bits=4', weights are 1-bit and activations are 4-bit. For 'Bits=1', both weights and activations are 1-bit.\n\nAs quantizing the activations is necessary to realize the spiking neurons, the energy-saving impact of quantization and spiking cannot be evaluated separately. However, the loss due to spiking neurons after quantization can be measured as follows:\n\n| Activation Bits/Timestep | 1/2  | 2/4  | 3/8  | 4/16 |\n| --- | ---- | ---- | ---- | ---- |\n| Acc (Quantized ANN)  | 79.8 | 87.9 | 90.1 | 90.9 |\n| Acc (SNN)  | 78.7 | 87.4 | 89.3 | 90.4 |\n| Loss  | 1.1  | 0.5  | 0.8  | 0.5  |\n\n---\n> Q7: Why ASG & ablation for spike neuron\n\nThe traditional IF model loads weights, computes membrane potentials, and generates spikes at every timestep, thus requiring multiple reads of the weights. Algorithmically, the IF model can be optimized by first summing inputs across all $T$ timesteps to reduce repeated weight accesses. However, this optimization would require extra storage for membrane potentials at each timestep. ASG avoids storing intermediate data. Furthermore, we conducted an ablation study on the SST-2 dataset. For T=2, ASG achieves 78.7% accuracy, while IF achieves 57.1%. For T=4, ASG reaches 87.4%, while IF is at 79.7%. These results demonstrate that ASG outperforms the IF model, further justifying our choice of ASG.\n\n---\n> Q8: Other suggestions\n\nWe will add the explanation in preliminary: Surrogate gradients approximate the gradients during backpropagation, enabling SNN training despite their discrete nature. These gradients smooth out non-differentiable spike events. ANN-to-SNN conversion involves transforming a trained ANN model into an SNN by mimicking ANN neuron behavior with the spike. **In Sorbet, we use ANN-to-SNN conversion.**\n\nWe will fix the missing reference in the appendix. The extended repository and code are now available at the same link as in the manuscript.\n\n---\n> Q9: k and \\psi_B in Algorithm 1\n\nThank you for pointing out. There is a mistake in line 10, Algorithm 1:\n\n$$\\sigma_B^2 \\gets \\frac{1}{B}\\sum_{i=1}^{B}\\mathbf{X}_i^2$$\n\nshould be \n\n$$\\psi_B^2 \\gets \\frac{1}{B}\\sum_{i=1}^{B}\\mathbf{X}_i^2$$\nWe hope that clarifies.\n\n---\n> Q10: Energy saving proportion\n\nSNNs are energy efficient because of their sparsity and low-bitwidth operations. Quantization and spiking neurons contribute to more than 99% of the energy-saving. Even though BSPN and PTSoftmax by themselves do not consume much energy, without them, it is not possible to realize true SNNs and the benefits that SNNs bring.\n\n---\n> Q11: Explanation of Assumption B.1\n\nThe plot for the distribution of the input of normalization can be found in the code repository.", "claims_and_evidence": "Yes. The claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes. The proposed Sorbet and evaluation criteria make sense for the problem.", "theoretical_claims": "Yes. The theoretical claims include 1. bit-shifting-based step maintains PowerNorm's gradient boundness 2. the approximation error of PTsoftmax remains within a constant factor of the traditional softmax. In my opinion, both of them are clear and there are no obvious errors. \nMoreover, I am concerned about the credibility of Assumption B.1. Line 651 states, \"In practical scenarios, the activations in Transformer-based SNNs are typically non-trivial, ensuring that their L1 norm remains above 1.\" However, in Figure 1 (c) and (d), the input X of  of the BSPN module is not binary (0-1). I hope the authors can provide a more detailed explanation, whether through theoretical analysis or experimental evidence.", "experimental_designs_or_analyses": "I checked the soundness/validty of the experimantal designs and analyses. The main results, energy saving analysis and ablation study are necessary due to the claim of energy efficiency and competitive performance.\nHowever, I believe that the experiments and explanations should be further supplemented. Overall, the proposed method integrates module design, quantization, distillation, and spiking neurons, making the training process relatively complex. Therefore, the authors should clearly describe the experimental setup of these components and provide sufficient ablation studies; otherwise, readers may find it confusing. I would like to raise the following questions regarding the experiments and hope the authors can address them and revise the paper accordingly. This is important for me to reconsider the rating.\n1. The experiments are based on the BERT model, which consists of two stages: pre-training and fine-tuning. However, the paper does not explicitly mention these two critical stages, even in Supplementary A's Algorithm 4. This makes it unclear whether Algorithm 4 entirely pertains to the pre-training phase. For example, when is the spiking neuron introduced—right after pre-training, or only after fine-tuning? Additionally, does the fine-tuning phase still involve knowledge distillation?\n2. In Line 306(left column), who is the teacher model? Is it BERT_base?\n3. In Algorithm 4, why is knowledge distillation conducted in three steps? Is there any literature supporting this approach? What are the hyperparameters (e.g., learning rate, epochs) for each step of distillation? Unfortunately, I could not find any details on experimental hyperparameters in the paper.\n4. Since Sorbet is ultimately an SNN model with activations taking only 0 or 1, why is 4-bit activation quantization performed first? Would it be possible to skip activation quantization and directly convert the model to an SNN? If activation quantization is necessary, please provide ablation studies to support its significance.\n5. In Table 2, two versions of Sorbet are listed. What are the differences between them in terms of weight and activation quantization? In Line 344, does \"a power of two\" refer to the 1-bit weight quantization mentioned in Line 260 (right column)? If so, what is the relationship between the Sorbet models in Lines 340 and 341?\n6. In Table 4, does \"Bits\" refer to weight quantization or activation quantization? In Line 423 (left column), the authors state that \"the accuracy drop from full-precision BERT to Sorbet is mainly caused by the quantization of weight and spike generation process, not by the replacement of softmax and normalization.\" This may suggest that Table 4 refers to weight quantization. However, given that the model also uses 4-bit activation quantization, what is its impact on performance? Additionally, how does the spike generation process affect the model’s accuracy? I could not find any ablation study on introducing spiking neurons.\n7. In Line 134 (left column), the authors adopt a novel ASG model instead of the traditional IF model, arguing that \"ASG accumulates input spikes and calculates membrane potential in a single pass, requiring only one access to the weights.\" However, based on Algorithm 3, it seems that the IF model could also load weights in a single pass by computing the summed inputs for t from 1 to T, similar to Step 3, but without averaging as done in Step 4. The authors' reasoning here might be misleading. Furthermore, I would like to see ablation studies on how different spiking neuron models affect performance.", "relation_to_broader_scientific_literature": "A key computational characteristic of SNNs is sparse operations, primarily based on addition, which has been demonstrated in models like SpikFormer and Spike-driven Transformer. Recently, there has been a surge of work on Transformer-based SNN language models, focusing on the critical challenge of handling the Attention mechanism. Some approaches, like SpikeBERT, attempt to discard the Softmax operation similarly to SpikFormer; however, doing so in language tasks often leads to poor performance unless distillation techniques are applied. On the other hand, SpikeLM retains the Softmax operation. This work explores an alternative approach—constructing an Attention mechanism using a shift-based method while preserving a Softmax-like structure, making it a valuable contribution to the field.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "1. In the section of Spiking Neural Networks within the \"Preliminary\" chapter, the author mentions terms such as \"surrogate gradients\" and \"ANN-to-SNN\" without providing explanations or references, which may lead to confusion. Furthermore, which specific method does the SNN in this paper employ?\n2. In Line 810, The energy cost are from (). Maybe the content in () is overlooked.\n3. When I checked the code in Suplementary material E, I found the repository is expired.", "questions_for_authors": "1. What is the relationship between k and \\psi_B in Algorithm 1?\n2. What is the proportion of energy consumption reduction contributed by efficient methods including BSPN, PTSoftmax, quantization, and spiking neurons? Which one plays the primary role?\n3. The seven questions of experiment mentioned in Experimental Designs Or Analyses, which are important.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Sorbet, a spiking language transformer optimized for neuromorphic hardware. It incorporates a shifting-based softmax (PTsoftmax) and a BitShifting-based PowerNorm (BSPN) to replace the key operations softmax and layer normalization (LN) are difficult to implement on neuromorphic hardware. The proposed Sorbet achieves competitive performance with 27.16 times theoretical energy  savings compared to BERT.

After the rebuttal, three reviewers rate 3, 3, 4, respectively. All of the reviewers vote to accept. Reviewer GqJG thinks the proposed PTsoftmax and BSPN are interesting and promising. Reviewer MRPY thinks this work constructs an attention mechanism using a shift-based method while preserving a Softmax-like structure, making it a valuable contribution to the field. There are also some concerns remain. Reviewer GqJG and ck8m believe that it would be better if the authors deploy Sorbet on neuromorphic hardware. I think these reviews are justified. Although there are still some problems, I believe that this paper is worth accepting. Therefore, my recommendation is to accept this paper.
Decision: Accept (poster)