SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation
Authors: Sangwoo Hwang
Seunghyun Lee
Dahoon Park
Donghun Lee
Jaeha Kung
Venue: NeurIPS 2024 poster (2024)
Keywords: Spiking Neural Network
ANN-to-SNN conversion
Transformer
Neuromorphic
Primary area: neuroscience_and_cognitive_science
TL;DR: We provide a direct transformer-to-SNN conversion that requires no training and no architectural change, achieving SOTA SNN accuracy of 80.0% by converting Swin-T (42% less energy) and only 0.3% accuracy loss by converting BERT (58% less energy).

Abstract: Event-driven spiking neural networks(SNNs) are promising neural networks that reduce the energy consumption of continuously growing AI models. Recently, keeping pace with the development of transformers, transformer-based SNNs were presented. Due to the incompatibility of self-attention with spikes, however, existing transformer-based SNNs limit themselves by either restructuring self-attention architecture or conforming to non-spike computations. In this work, we propose a novel transformer-to-SNN conversion method that outputs an end-to-end spike-based transformer, named SpikedAttention. Our method directly converts the well-trained transformer without modifying its attention architecture. For the vision task, the proposed method converts Swin Transformer into an SNN without post-training or conversion-aware training, achieving state-of-the-art SNN accuracy on ImageNet dataset, i.e., 80.0\% with 28.7M parameters. Considering weight accumulation, neuron potential update, and on-chip data movement, SpikedAttention reduces energy consumption by 42\% compared to the baseline ANN, i.e., Swin-T. Furthermore, for the first time, we demonstrate that SpikedAttention successfully converts a BERT model to an SNN with only 0.3\% accuracy loss on average consuming 58\% less energy on GLUE benchmark. Our code is available at Github ( https://github.com/sangwoohwang/SpikedAttention ).

SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation
Sangwoo Hwang
Electrical Engineering and Computer Science
Daegu Gyeongbuk Institute of Science and Technology (DGIST)
nemesis0523edgist.co
Seunghyun Lee
Dahoon Park
School of Electrical Engincering
School of Electrical Engineering
Korea University cooer kr
Korea University
manyteacherr98korea... ac kr
Donghun Lee
Jaeha Kung"
School of Electrical Engineering
School of Electrical Engineering
Korea University dhleeids@korea. ac.kr kr
Korea University
j jhkungkkorea.co ac.kr
Abstract
Event-driven spiking neural networks (SNNs) are promising neural networks that reduce the energy consumption of continuously growing AI models. Recently, keeping pace with the development of transformers, transformer-based SNNs were presented. Due the incompatibility of self-attention with spikes, however, exist- ing transformer- based SNNs limit themselves by either restructuring self-attention architecture conforming to non-spike computations. In this work, we propose a novel transformer-to-S9 -SNN conversion method that outputs an end-to-end spike- based transformer, named SpikedAttention.. Our method directly converts the well-trained transformer without modifying its attention architecture. For the vi- sion task, the proposed method converts Swin Transformer into an SNN without post-training conversion-awee training, achieving state-of-the-art SNN accuracy on ImageNet dataset, i.e., 80.0% with 28.7M parameters. Considering weight ac- cumulation, neuron potential update, and on-chip data movement. SpikedAttention reduces energy consumption by 42% compared to the baseline ANN, i.c.. Swin-T. T. Furthermore, for the first time, demonstrate that SpikedAttention converts a BERT model to an SNN with only 0.3% accuracy loss on average con- suming 58% less energy GLUE benchmark. Our code is available at Github
successfully
Introduction
Recently, many practical AI applications such conversational question answering ( intelligent code completion (2), weather forecasting (3), and text-to-image generation (4) have rapidly evolved with the advances in artificial neural networks (ANNs) based transformer architecture. Transform-
J. Kung is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

20M:
50M:
SNN-1o-SNN, Conversion
Direct Training
85
W
W
W
Swin-T (ANN)
Neuron
Neuron
Neuron
Sg
S
Accuracy (%)
80
This Work
Neuron
Meta-Spine Meta-SpikeFormer
S
MST
75
SDSA
WOSS Softs-- Neuron
SpikFormer
Trace-driven
Accumulation
Pazzhald
Dot Product
70
Threshold
Neuron
60
160
Energy (mJ) E mddi 2 m
(v. SynOP!)
MARTOR
S
Ie 19yy V)
(00)
(a)
(b)
Figure 1:
(a) Accuracy, energy consumption (refer to Appendix A.1). and parameter size of the propsoed SpikedAttentin and other -pike-based SNNs on ImageNet classification (15). (b) The structure of the proposed fully spike-based attention module (more details are in Section
ers are powered by the self-attention mechanism, which extracts long-range dependencies between input tokens, e.g., words (5), image patches (6), both (7). However, deploying state-of-the-are transformers requires a large amount of computations and a huge memory footprint. For instance, generating single token on GPT3-175B model requires least 350 GFLOPs (8).
In order to reduce the ever-growing computational overhead of transformers, there were some efforts to improve their energy efficiency by utilizing spiking neural networks (SNNs). The major strength of an SNN, which mimics biological neuron, is that only weight accumulations (ACs) required when input spikes encountered (event-driven), instead of power-hungry multiply- accumulate (MAC) operations in ANNs. The energy consumption of 32-bit MAC is 5x higher than that of 32-bit accumulation (9. To exploit the high energy efficiency of SNNs, many CNN models based on SNNs (IO) TT) have been proposed, and recently, SNN-based transformers have emerged. Implementing SNN-based transformer is done either by direct training via surrogate gradient 112 13) or converting well-trained transformer to an SNN (14).
When implementing the attention module in spike- based transformer, there are two types of matrix multiplication. One is associated with generating Q, K, and V where static weight matrices (WQ+ WK. and Wv) used. In this case, when input neuron fires, its associated weight value gets accumulated the output ncuron's 's potential. Another type is the multiplication between dynamically generated matrices, c.g., Q K , In previous works 12 [4), logical AND operations performed between two "rate-coded" spike trains, e.g., SQ(i.)) and S &)( for each dot product. However, due to the probabilistic nature of the rate coding, requires long timestep (T 128) maintain high ANN-to-SNN conversion accuracy (14). Unfortunately, this translates to high energy consumption in running SNNs (refer to MST in Fig.
Another challenge with realizing spike-based transformers is the softmax operation. In the field of ANN quantization, there have been many efforts to approximate softmax operations (16) (1).. I-BERT (16) approximates the exponential function with a second-order polynomial, successfully converting it to integer operation (i-exp), and FQ-VIT (17) combines i-exp with logarithmic quantization. In baN-based transformers, however, softmax has not been converted to a spike-based operation due to the presence of exponential functions. Since SNNs have binary inputs and outputs, previous approaches in ANN quantization cannot be directly applied.
In this work, we present an end-to-end spike-basse transformer, named SpikedAr- tention, solving the abovementioned challenges to achieve state-of-the-ar accuracy (0.6% accuracy loss compared to ANN) among spike-based transformers (Fig. a In addition, it consumes the minimum energy, which is 42% lower than the original Swin Transformer (6), for image classification task. Furthermore, for the first time, we demonstrate that BERT model can be successfully converted
Our Contribution:

---

Non-Spike Comp
Non-Spika Comp
S
OK?
Attn
S,
K'V
4
4
2
>
2
X
X
X
AND Operation
Softmax
AND Operation
(a)
(b)
Figure 2:
The computation of attention in previous SNN-based transformers (a) with the softmax (1).
and (b) without the softmax operation (12). Both involve non-spike computations. Total timestep T of each spike tensor is set to for simplicity.
to an SNN, i.c., Attettein,nn with negligible accuracy loss (0.3% on average) while consuming 58% less energy on GLUE benchmark. Note that the SpikedAttention does not require any additional training since it is created by the direct vansformer-to-SNN conversion method thanks to the proposed fully spike-based attention module (Fig. (b)). The main contributions are summarized below:
SpikedAttention encodes external inputs and all interme- diate features in spikes. Especially, each element in intermediate features is encoded as single spike for given timestep I' to minimize the energy consumption.
Fully spike-based transformer:
Trace-driven matrix multiplication: To reduce the required timestep ' in performing multiplication between two dynamically generated matrices, we propose a trace-driven matrix multiplication. Owing to the reduced timestep, we can reduce energy consumption.
Exponent-free spike-based softmax: We present a winner-oriented spike shift that controls the output spike timing realize the spike-based softmax. This allows keep the vanilla self-attention architecture of the original transformer to maintain high accuracy.
2
Related Work
Most SNN studies have focused replacing CNNs to improve energy efficiency in vision tasks (10) 18]. With the recent advent of transformers, there have been increasing efforts to convert transformers into SNNs (14) directly train spike-based transformers (12) 13 19 20). When realizing spike- based transformer, self-attention is the major hurdle in converting the transformer to a fully spike- driven network. Here, "fully spike-driven" means that no real-valued multiplications nonlinear functions are involved any operations. Fig. 21a) illustrates the required operations in vanilla self-attention (VSA) (5) but with Q, K, and V in the form of spikes (14).
In the original VSA module, real-valued multiplications (Q KT and Alln V) and a softmax operation (Attm so poofmmr(() Q - KT) /d) are required. MST (14, the authors encode Q, K, and V as spike trains (denoted by Sq. SK. and Sy in Fig. 2 a)). By doing the matrix multiplication (Q KT) simply becomes AND operations followed by floating-pon accumulations. To convey meaningful information (in form spikes) to the next layer, however, long timestep 7 128 required in (14). This is due to the fact that the simultaneous firing of spikes at both SQ and Sxr rare. This problem gets worse for temporal coding or phase coding, which produces much fewer spikes than the rate coding. Moreover, in MST, the complex softmax operations done to compute attention maps, which requires non-spike computations (i.c., expensive exponential functions).
To remove the burden of computing the softmax, some previous studies on direct training of SNNs for transformers have restructured the attention module [3). As shown in Fig. (bb). Spikformer (12) removes the softmax operation since 'SQ SxT or 'S Sv" only produces non neg-nvee values. There was another attempt to modify the attention module, such as replacing 'Sq SKT* with Hadamard product followed by the column summation to obtain mask vector that masks out less important channels in Sy (3). Since these prior works restructure the network topology, they need to train the model from scratch with surrogate gradients to improve the performance.
3

---

3 Preliminaries
3.1
Neuron Model
There has been a long-standing effort in neuroscience to mathematically model biological neurons 21 Neuron models with high biological plausibility are computationally expensive, while models with low biological similarity are energy efficient. Due to a high volume of parameters and significant computations involved in ANNs, energy efficiency has become more of priority than biological plausibility when replacing them with SNN counterparts. Therefore, the most energy efficient neuron model, i.e., the leaky integrated-ani and fire (LIF) model (23) is widely used. The LIF neuron model is defined as
(22).
Norc
1y(1) = Au (1 1)+ Σ
wiyS
Sy(1) 1, 1y(t) = (t()) - Vo when uy(t)= Vo,
where or j is the index of a pre- or post-synaptic neuron, v is the potential of the neuron j. and A is the leak factor of potential that causes the v, gradually decay the time proceeds. The way is the synaptic weight between the neuron i and j converted from a pre-trained ANN or directly trained. The (C) represents the binary spike (0 1) from the neuron at time 1. The pre is the number of pre-synaptic neurons. The pre-synaptic neurons that generate spikes activate the weight accumulation on (t).. When the uy (t) exceeds the threshold Vp. the neuron j fires a spike (S₃ (t) 1) and its potential resets to (t) Vg. Note that decreasing the potential by threshold instead of resetting it to zero is a common technique in ANN-10-SNN conversion methods (I0 24).
3.2 Spike Coding Schemes
To achieve higher SNN performance, it is essential to determine how real-valued inputs are encoded into sequence of spikes, known as 'spike coding'. The spike coding is a scheme that determines whether to generate a spike at time f when converting continuous values into a set of spikes. The most widely used coding scheme is rate coding, which generates multiple spikes at each neuron via Poisson sampling over pre-defined timestep T [10, 255). The rate coding generates more spikes as it converts larger values into spikes. However, increasing the number of spikes and the timestep for higher resolution leads to higher energy consumption due to more weight accumulations in Eq. (
Unlike the rate coding, which generates multiple spikes at each neuron, temporal coding generates a single spike per neuron. In the temporal coding, each spike represents a different value at a different spike time for a given total timestep T. The spike at earlier time represents larger value. For instance, an earlier spike can represent a linearly larger value as T-t
T 26) or an exponentially larger value as " (7).. The temporal coding energy efficient but requires a long timestep T to increase the precision/resolution.
The alternative coding scheme, known as phase coding 28 29), combines the characteristics of the temporal coding and the rate coding. Like the temporal coding, the phase coding assigns each timestep t a different value, i.e., B-t where B is the base of the phase. As proven in Appendix B the phase coding can be realized by replacing the leak factor (A) of the widely used LIF neuron model in Eq. ( with B, making the spike at one timestep earlier has the Bx larger impact on the potential increase. When B = 2, we call it a binary coding. Unlike the temporal coding, the phase coding allows multiple spikes when encoding a value. For fixed (total) timestep T, the phase coding provides a higher resolution than other coding schemes. Recently proposed one-spike phase coding maximizes the energy efficiency, where the sequence of spikes is approximated by the first spike (30). Note that the neuron ceases to update its potential after the first spike being generated because it is unnecessary one-spike phase coding, Vo is tuned to the midpoint between two consecutive phase-bases based weights, ie.. (B- B-t t- 1/2, to reduce the approximation error. The authors in (30) have proposed a technique to reduce the approximation error by decreasing the base B of phase coding. However, decreasing the base only guaranteed to reduce the error when is large enough. In SpikedAAteention, therefore, we use the one-spike phase coding by carefully selecting the proper base B and total timestep T to balance well between the accuracy and the energy efficiency. The details on selecting B and T are presented in Appendix E
fire the following spikes at the cost of approximation error. In the

---

(t)).(())
of
Stored: B
*xxoo())
B
S(t) =1
S(0) 1 x(O
Sit) =
M(t) 0
Load
x(t) >
Phase-based
(1)
(z)
(0)
+ (3
BB
x²(()
S (t (t)
x*(t)
(
Ph Weight
Load
xx)
x (4)
(OK)),
(t)
(QK)
F92 Thoostop
Threstop
B(t) = xx() >
S(t) 1
S(U
0
B
x(t) >
(t)
(0 (t)
Accumulated Traces
Accumulated Traces
S Aq (1)
( (1)
Stored:
B
Timasdep
Timsatop
5(0= 1 X(I)
S(t) 1
S( = 1
t = 2
t 3
t= 4
Figure 3: The proposed trace-driven dot product in SpikedAttention, A global trace decays by its base (B) at each timestep and the trace is transferred to each neuron's 's local memory when its first spike, from either Sq or Sk, is observed. The (111))(111) is the potential of the neuron (QK?"), ),1-
4 Fully Spike-based Attention: SpikedAttention
The main goal of designing the SpikedAttention was to encode external inputs and all intermediate features by binary spikes without altering the VSA structure at all. Prior work on spike- based vision transformers (12 13 14) feed in floating-poins inputs, i.e., 2D images, to the patch partition module instead of binary spikes. Also, the nonlinear nature of softmax and the high sparsity in Sq. Sk, and Sy make things more challenging in making fully saske-based transformers. In the proposed SpikedAttention. all intermediate neurons fire a single spike using the one-spike phase coding (B < 2). Only the external input allows multiple spikes with the binary coding, i.e., phase coding with B = 2. Until now, previous transformer-to-SNT conversion methods (14) could not achieve fully spike-based computations due to softmax operations, and they require long timestep due to extremely high sparsity after AND operations between two spike-basee vectors. The direct SNN training methods (3) simplified the VSA structure by removing softmax, which degrades the performance. Therefore, we present two novel techniques to (i) minimize the spike timestep by performing trace-driven matrix multiplication and (ii) exponent-fre free spike-based softmax to realize the taanformer-to-SNN conversion method without any training.
4.1 Trace-driven Matrix Multiplication
We focus on developing an efficient spike-based computing scheme for the multiplication between two dynamically generated spike-based matrices. We utilize global trace that tracks the phase at time to consider the values associated with every spike. With the rate coding, probabilistic multiplications are performed with logical AND between two spike trains. However, since we use one-spike phase coding, each spike must carry its information to the next layer. Consider an example of performing Sq SXT. The output value at (i-j), denoted as (QK*),, ).. is computed by the dot product which is Σx SQ(.,)) Each spike train in Sq (or Sx) consists of only one spike which represents the value B B- where (Q(i,)) is the spike time of SQ(.A) and B is the base.
Thus, the dot product output should become Σx B B-tqu. B (...).)
As mentioned earlier, our work exploits the 'spike trace model' record the value associated with the spike that fires first among the neuron pair, e.g., SQ((,) and SKG.)). In neuroscience research, spike trace is a popular method to determine the correlation between connected neurons by relative spike timing (31). Typically, the trace of a neuron 'n', i.e.. 2n, (t), reflects the history of spikes during timestep T. We simplify the trace model to globally track the value B- spike time which is
x (t) I 1)/B,
(2)
5

---

Threshold
Potential
Base
Timestep T
5/(=3)
B
B
Winner Spike
soasy (t)
Blocked
B
mosag (0:
Blocked
8
8,(1-1)
Activate (+1)
Potental
(Nocel 7
Inhibittinn
Thrahold Shif
Patential
IV (ooo
Inhibition
Thrashoid Shift
Vanse
Stope: - log,
Decay By B
Inhibit (-1)
3 Te0 T+1 T+2 Te5
T+0 T+1 T+2
T+3
Activate (+1)
#1 Potential Update #2 Spike Generation Timestep
#1 Potential Updale #2 Spike Goneration
Timestep
(a)
(b)
(c)
Figure 4: The proposed winner-oriented spike shift (WOSS) for approximating softmax. (a) Each WOSS neuron increases its potential (Vwoss,) by the incoming input spike (z() The first (winner) spike among (t) activates (global) inhibitory neuron which depresses all neurons. (b) Potential at neuron (winner) and its corresponding output spike at t = T O. We are shifting the first spike time from
O. The following spikes are time-shifted by the same amount. (c) Potential at neuron j (non-winee -winner) and its corresponding output spike at t = T + 2 (effectively, t t= 2) due to the global inhibition and the threshold shift.
where (t) is the global trace which is initialized to 1 at = O. With the one-spike neuron model, when neuron encounters the first spike at I, we store B-t in its local memory as its own trace x,, (l), which will be used later to compute the dot product. We intentionally leveraged the trace-driven method since neuromorphic chips such as Loihi (32) already have the hardware module, e.g., trace memory, for updating neuron traces for widely used STDP learning.
Fig. 3 illustrates the process of a dot product for computing (QK"),,1 with spike traces. If SQ(.1) 3 is stored in the local memory of
generates spike earlier (t = 3) than SK(1,I), the trace value B-3 the neuron SQ(11)- When the neuron SK(1,1) exceeds the threshold and generates the spike at 4, the trace of SQ(1.1) (= 3-3 propagates to update the potential of (QKT)), Since the current phase is B-1 at / = 4, propagation of B-3 at = 4 (colored in red) is equivalent to 'B-4 x B-3 There might be situation in which multiple neuron pairs observe the second spike simultaneously. For instance, SQQ1.2)-Sk11.
and SQ(1.3)-Sx(C
pairs have the second spike at / = 3 (it may coincide both pairs already have their stored
with the first spike like the Squ1,3)-kk (1(33))) pair). Thus, at trace for one of two neurons (Sq or Sk) ) that fired the carlier spike in the pair. Then, pre-stored traces from those pairs that share the second spike time will be accumulated together, i.e., (B-2 + B 3) in this case (colored in blue). The accumulated traces
weighted by the phase of the current timestep, i.e. B-3 By accumulating traces first, we can bound the number of multiplications to the timestep T of the neuron model irrespective of the number of input neurons. Usually, input spike tensor SQ or SK lies in R. xxT where N is the number of image patches (or token length), D is the embedding dimension, and T is the total timestep. The accumulations happen across the dimension D, making the proposed approach energy efficient when T < D, which is the general case for transformers.
4.2 Winner-Oriented Spike Shift (WOSS) for Softmax
The main issue with the typical softmax operation is the presence of exponential functions. Our work uses the normalized softmax similar to (33) which is expressed as follows:
exp(e:)
exp(z:) EnD exp(z;))
(2x))
exp(max(e )
Σ == (Ma())
exp(2;)
where RN and (24) is the softmax output with respect to the th neuron. Our SNN approximates exp(z:)
[ by using the proposed WOSS neuron model and scales the threshold (V₂) of the following
layer by their sum (y exp(max(z)) ) leading to the final softmax result. Note that multiplying the threshold by the sum term has the effect of dividing inputs by the same amount.
exp($s)

---

match the scale of the potential after the "potential update" stage. The log, B term is to consider Since SpikedAttention generates a single spike per neuron, the objective is to approximate exp[max(2]] with B E [0. 1]. Before converting it to spike-based representation, we extract the maximum value of z, i.e., M2. by running proxy training dataset. Then, by setting = Ma2 and by applying logg on the normalized exponential term, it can be expressed as
exp(=1)
exp (M224) logg exp (rsp(max((M)) (xp(max((2z')
(z) () -tis
log, B
where ts is the required spike time neuron i approximate the left-hand side of Eq. 44 The z' arrives at WOSS neurons by spikes S₂ from the previous layer, i.c., each row of S(OKT), with the form of the one-spike phase coding.
First, we need to compute '2. max(z')' from the input spikes, i.e., S². To do so, we first find max(z') by detecting the first incoming spike, called winner spike. In Fig. aa), the winner spike is observed at neuron i. For the neuron i, its potential increases by at ( = (Fig. Ab). Inspired by inhibitory neurons in (25). the winner spike actívates the global inhibition so that potentials of all neurons (including i) within the same softmax group decrease by The global inhibition performs the subtraction by max(z') in Eq. (4). After the global inhibition at t 1, non-winner neurons receive an input spike at > 1. Fig. (c) shows how the potential of non-winner neuron j changes. At t = 3, the neuron j receives the imput spike, increasing the potential by 1. At each timestep t, the potential decays by the base B (at = 2 and f = 3 in Fig. 4c). which is identical to the leak factor À in Eq. I. This decay factor allows to assign a dedicated phase B to each timestep. Thus, after the first "potential update" stage (t 2 7), each neuron i has the fixed potential equal to
(woss() (t 2 T) = ((T-1) max(z')).
(5)
Now, need to generate single output spike at each neuron by comparing its potential to the threshold (i.e., "spike generation" " stage in Fig. (b)).. The threshold is initially set to so that the winner-spike is time-shifted and fires at 0 for more precise approximation of softmax, i.e., utilizing full timestep T (Fig. 4b). The desired time 4 that each neuron should fire approximately equals to (max(z²) M2 log,
To find the 4, we shift the threshold by following
dV = -B -1 1og.-1log... B
(6)
dt
M2
and generate output spike comparing the threshold with the neuron's potential. Note that the right-
hand side of Eq. (6) is a constant and thus can be pre-computed. The appearance of BT-1 is to
M,
the M2/ log, B in Eq. ( Then, by comparing the Eq. (5) with the shifted threshold (ta), we can generate a spike if the threshold becomes more negative (Fig. (()) that satisfies
Mr
1 > (max(z')= =x() log, B
The output spike from the WOSS neuron, i.c., Swoss. approximates exp(x.)
exp(max(z)) Thus, it needs to exp[max(a)) to complete the softmax calculation. This is done by amplifying the
be divided by Σ exp(s.)
threshold at the following trace-driven dot products (i.e., SAn SSy by the sum term, which is the summation of spikes from the WOSS neurons in the same group, i.e., each row of Swoss. Similar to Eq. ( the threshold is incremented by Swoss(t) and multiplied by B at each timestep until it
reaches Σ (x)())
exp(x;)
which is expressed follows:
Ve(i.) (t +1)= = B x (o(ra)tt + Σ (woss(.s))tt,
(8)
JEZ:
where Vo(t = 0) is initially set to 1. Note that all elements at each row i share the same threshold
Vola.)- Finally, the output neuron at the final trace-driven dot product in Fig. Tb) generates spikes that
approximate 'softmax(Sg SKT / Vd) Sv'. Note that implementing WOSS neurons for softmax in the neuromorphic hardware incurs 9.88% area and 12.35% power overheads compared to the
hardware only with LIF neurons with no support of softmax (details are discussed in Appendix C).
7

---

Decoded Spike
0.5
0.8
0.0
-3.5
-0.5
1.0000
-05
0.0
0.
10-10
-05
0.0
0.5
1.0
0.0
02 04
06
08
1.0 ao
0.2
044
0.8
0.8
1.0
-1.
-1.0
-0.5
00
0.5
1.0
Trace-driven QK
Trace-driven QK
WOSS-Softmax
WOSS-Softmax
Last layer
(First Block)
(Last Block)
(First Block)
(Last Block)
(Classiffer)
Figure 5: Scatter plots showing the correlation between the actual activation values in Swin Trans- former (X-axis) and the decoded spike values in converted SpikedAttention (y-axis). The parameters are set to T 40, 1.15.
Experimental Results
5.1 Conversion Swin Transformer to Attentin
The proposed SpikedAttention is implemented by SpikingJelly (34) and PyTorch computing with four NVIDIA GeForce RTX 4090 GPUs. For evaluation, Swin Transformer (6) is selected as a baseline and converted transformers require GeLU and layer normalization, which are non-spike computations. Thus, as a pre-trained model, we replaced GeLU with ReLU and layer normalization with batch normalization according to (35). Since some activations are not followed by the ReLU function in Swin-T, those cannot be converted unsigned/positive spikes. To cover both positive and negative values, utilized a signed neuron (36) that generates positive spike (+1) if the potential is greater than Va and negative spike (-1) if the potential less than Vg. Therefore, we trained two baselines, i.e., one with ReLUs at every layer and another with no change. to convert them to SNNs. Fig. illlustrats the correlation between the decoded spike values and the actual activations at trace-driven matrix multiplication and softmax layers of SpikedAAtention. The carlier layers (c.g., First Block) have little conversion error, but the deeper layers (e.g.. Last Block) have higher conversion error. Nevertheless, the classifier exhibits a high correlation between the actual activations and the decoded spikes.
SNN, i.e., SpikedAttentionn,
perform image classification. Conventional
Table 1:
Comparison between Spiked Spiked/ttention Attention and the prior work in terms of the parameter size, the energy consumption, the required timestep, and the accuracy on ImageNet classification task
Model
Method
Param (M)
Energy (mJ)²
Timestep
Ace (%)
Spikformer 112
Direct Training
66.3
21.5
74.8
SDSA (3))
Direct Training
66.3
6.1
77.1
Formeer
37
Direct Training
55.4
52.4
80.0
Meta-Spikeeorrmer 37
Direct Training
55.4
13.0
79.1
MST (4)
ANN-to-SNN
28.5
158.6
128
77.9
Ours (w/o ReLU)
ANN-to-SNN
28.7
3.0
40
80.0
Ours (w/ ReLUU)
ANN-to-SNN
28.7
1.8
77.4
fFor the fair comparison with the prior work, only the energy consumption for weight accumulations is included.
Table D compares performance between the prior work and the proposed SpikedAttention (denoted as "Ours" in the table). For the fair comparison, we estimate the energy consumption of only weight accumulations (discussed in Appendix A.T) for the entire timestep similar the prior work [22 (3). Previous SNNs based on vision transformers demonstrated better performance than CNN-based SNNs (IO TI  24). However, existing SOTA transformer-baat based SNNs consume 2 53x higher energy than SpikedAttention (w/o ReLU) while presenting lower accuracy (< 80%). Other SNNs feed external inputs as floating point numbers, resulting higher energy consumption. It is noteworthy that SpikedAttentionn minimizes the energy consumption even with T = 40 by allowing only one spike per neuron. To perform hardware-realistic energy comparison between ANNs and SNNs. we estimated the total energy consumption by using the energy metric presented in (38) which considers the data movement and membrane potential update as well (details can be found in Appendix A.2). As result, SpikedAttenntion reduces the energy consumption by 42% with only 0.6% accuracy loss when converting an ANN (w/o ReLUU which consumes 45.4mJ. Similarly, the energy consumption of SpikedAAtentiin is 47% lower than the ANN (w/ ReLU) which consumes

---

Table 2: Conversion results of pre-trained models from MST (4) to SpikedAttention
Dataset
Model
Param (M)
Energy (mJ)
Timestep
Accuracy (%)
CIFAR-10
ANN (w/ ReLU)
27.5
42.1
1
97.5
MST (Unsigned)
27.5
1089.9
64
96.3
Ours (Unsigned)
27.5
21.3
24
97.3
CIFAR-100
ANN (w/ ReLU)
27.6
47.1
87.7
MST (Unsigned)
27.6
1157.1
64
85.4
Ours (Unsigned)
27.6
23.7
24
86.3
ImageNet
ANN (w/ Rel.U)
28.3
55.5
79.3
MST (Unsigned)
28.3
1836.7
128
77.9
Ours (Unsigned)
28.3
31.7
48
77.2
27.5 mJ. To directly compare with the previous ANN-to-SNN conversion method, i.e., MST (14), the pre-trained ANNs from MST were converted to SpikedAttention as well (Table In Table we computed the energy consumption of ANNs and SNNS using the hardware- realistic energy metric (Appendix A..2) Since the pre-trained models of MST embed ReLU in each layer. they were converted SpikedAttention using unsigned neurons. The MST consumes significantly higher energy (2 48x) than SpikedAttention because multiple spikes are generated for a longer timestep 7 64 or 128. In addition, the first convolution layer based floating point-based MACs instead of spike-based accumulations. Compared to the MST. SpikedAttention achieves higher accuracy with shorter T thanks to the trace-driven matrix multiplication. As a result, we achieve SOTA SNN accuracy on ImageNet without any training or modifying the architecture while minimizing energy consumption.
5.2 Conversion of BERT to SpikedAttention
Table 3: Comparison between Spiked/tteention and other BERT models on GLUE Benchmark
Dataset
CoLa
MNLI
MRPC
QNLI
QQP
RTE
SST-2
WNLI
STS-B
MA-BERT (39
(ANN)
Accuracy (%)
59.8
84.7
84.3
91.4
91.2
64.6
92.6
56.3
Energy (mJ)
189.7
189.7
189.7
189.7
189.7
189.7
189.7
189.7
189.7
SpikingBERT (40)
(SNN)
Accuracy (%)
78.1
79.2
85.2
86.8
66.1
88.2
82.2
Ours (SNN)
Timestep
24
24
16
24
24
16
16
16
24
Accuracy (%)
59.3
84.4
84.1
91.0
90.8
65.0
92.1
36.3
83.9
Energy (mJ)
81.5
82.1
77.5
81.6
82.1
79.1
79.9
77.7
79.9
Our proposed SpikedAttentiing is also applicable to NLP since it converts attention modules into spike-based computations. However, there are some functions, such as GeLU and LayerNorm, that are difficult compute with SNN for now. Since MA-BERT (39) replaces GELU with ReLU, converts LayerNorm to BatchNorm, and fuses normalization layers into adjacent linear layers, MA-BERT can be easily converted SpikedAttention without any modification. Therefore, we converted MA-BERT (base-uncased) with 11MM parameters to SpikedAttentionn
Note that accuracy loss of MA-BERT compared to traditional BERT only 0.1%. Even though MA-BERT approximates the sofumax with a two-layer neural network, we converted MA-BERY with the original softmax to StteeoAttentoon (thanks WOSS). The output of the token embedding is binary coded and fed our SNN model. By converting the pre-trained MA-BERT for text classification on GLUE benchmark (41). we observed only 0.3% accuracy loss on average without any additional training (Table (3). Note that Matthews correlation coefficient is reported for CoLA, while accuracy is reported for the other tasks. By estimating the energy consumption as presented Appendix A.2, SpikedAttention reduces the energy by 58% on average compared to MA-BERT. The SpikedAttention is the first work that demonstrates the transformer-to-SNe conversion for NLP tasks. SpikingBERT 40) directly trains SNN as a student using knowledge distillation from pre-trained BERT as a teacher. Compared to

---

SpikingBERT (T' = 125). SpikedAttention was able to achieve 3.6% higher accuracy average as it directly converts well-traine BERT into an SNN without training.
Conclusion
This paper presented trace-driven matrix multiplication and winner-oriented spike shift to convert the attention module with spike-basse based computations. Our proposed methods accurately approximate all activations in the self-attention module spikes without performing expensive exponent compu- tations. Thus, our work outperforms previous transformer- much less energy (25.6mJ for vision tasks and 80.2mJ for language tasks average) without struc- tural modifications or direct training. By converting ANNs with a high number of multiplications into addition-only SNNs without any training iterations, our work makes Al more accessible on energy-constrained devices. Since the presented conversion method does not require any additional training, we can obtain SNNs without increasing the amount of carbon emissions. However, there are some limitations: first, the timestep required for Spiked longer than directly trained SNNs. Thus, our next goal would be reducing the timestep by learning the per-layer base for better one-spike phase coding. The second limitation is that SpikedA do not support GeLUU and LayerNorm making it difficult to be generalized to any language models. Nevertheless, our work is an essential step towards converting large language models to SNNs, and converting LayerNorm and GeLU to spike-based computations remains as our future work.
SNNS in accuracy and consumes
to maintain high accuracy is
Acknowledgments and Disclosure of Funding
This work was partially supported by the Institute of Information & Communications Technology Planning & Evaluation (ITPP) funded by the Ministry of Science and ICT under Grant 2022-0-01 170 and Grant RS-2023-002298499 in part by the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT under Grant RS-2023-00258277.
References
[1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding." in Proceedings of the Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (NAACL-HLT), pp. 4171-4186, 2019.
[2] S. Gunasekar, Y. Zhang, Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi, et al., "Textbooks are all you need," orXiv:2306.1164 2023.
[3] Wu, H. Zhou, M. Long, and Wang, "Interpretable weather forecasting for worldwide stations with a unified deep model," Nature Machine Intelligence, pp. 1-10, 2023.
[4] W. Peebles and S. Xie, "Scalable diffusion models with transformers," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195-4205, 2023.
[5] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 2017.
[6] 7. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, 7. Zhang. S. Lin, and B. Guo, "Swin Transformer: Hierarchical vision transformer using shifted windows." in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2021.
[7] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, et al., "Learning transferable visual models from natural language supervision." Proceedings of International Conference on Machine Learning (ICML), pp. 8748-8763, PMLR, 2021.
[8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neclakantan, P. Shyam, G. Sastry, A. Askell, et al., "Language models are few-shot learners," in Proceedings of the International Conference on Neural Information P Processing Systems (NeurIPS), pp. 1877-1901, 2020.
[9] M. Horowitz, "1.1 Computing's 's energy problem (and what we can do about it)," in Proceedings of IEEE International Solid State Circuits Conference (ISSCC), 2014.
10

---

[10] B. Han, G. Srinivasan, and K. Roy, "RMP-SNN: Residual membrane potential neuron for enabling deeper high-aceuracy and low-latency spiking neural network." in Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 13555-13564, 2020.
[11] B. Han and K. Roy. "Deep spiking neural network: Energy efficiency through time based coding," in Proceedings of European Conference on Computer Vision (ECCV), pp. 388-404, Springer, 2020.
[12] Z. Zhou, Y. Zhu, C. He, Y. Wang, S. YAN, Y. Tian, and L. Yuan, "Spikformer: When spiking neural network meets transformer," Proceedings of International Conference on Learning Representations (ICLR), 2023.
[13] M. Yao, Hu, Zhou, L. Yuan. Y. Tian, X. Bo, and G. Li, "Spike-driven transformer." Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 2023.
[14] Z. Wang. Y. Fang, J. Cao, Q. Zhang, Z. Wang, and R. Xu, "Masked spiking transformer," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[15] Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, leee, 2009.
[16] S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer. "I-BERT: Integer-only BERT quantization." " in Proceedings of International Conference on Machine Learning (ICML), pp. 5506-5518, PMLR, 2021.
[17] Y. Lin, T. Zhang, P. Sun, Z. Li, and S. Zhou, "FQ-ViT: Post-training quantization for fully quantized vision transformer." in Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), pp. 1173 1179, 2022.
[18] H. Zheng, Y. Wu, L. Deng, Y. Hu, and G. Li, "Going deeper with directly-trained larger spiking neural networks," in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 11062-11070. 2021.
[19] C. Lv, T. Li. J. Xu, C. Gu, Z. Ling, C. Zhang, X. trained with two-stage knowledge distillation from bert," arXiv preprint arXiv:2308.1572 2023.
Zheng, and X. Huang. "Spikebert: A language spikformer
[20] R.-J. Zhu, Q. Zhao, and J. K. Eshraghian, "Spikegpt: Generative pre-trained language model with spiking neural networks," arXiv preprint arXiv:2302.1 13939, 2023.
[21] A. L. Hodgkin and A. F. Huxley, "Currents carried by sodium and potassium ions through the membranc of the giant axon loligo." The Journal of Physiology, vol. 116, no. 4, pp. 449-472, 1952.
[22] E. M. Izhikevich, "Which model use for cortical spiking neurons?," IEEE Transactions on Neural Networks (TNN), vol. 15, pp. 1063-1070, Sept. 2004.
[23] N. Fourcaud-Troente D. Hansel, van Vreeswijk. and Brunel, "How spike generation mechanisms determine the neuronal response to fluctuating inputs." " The Journal of Neuroscience, vol. 23, no. 37, pp. I 11628-11642 16400, 2003.
[24] T. Bu, W. Fang, J. Ding, P. Dai, Z. Yu, and T. Huang, "Optimal ANN-SNN conversion for high-accuracy and ultra-low- spiking neural networks," in Proceedings of International Conference on Learning Representations (ICLR), 2021.
[25] P. U. Diehl and M. Cook, "Unsupervised learning of digit recognition using spike-timing-dependent plasticity," Frontiers in Computational Neurssience,, vol. 9, p. 99, Aug. 2015.
[26] B. Rueckauer and S.-C. Liu, "Conversion of analog to spiking neural networks using sparse temporal coding," in Proceedings of IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-5, 2018.
[27] S. Park, S. Kim, B. Na, and S. Yoon, "T2FSNN: deep spiking neural networks with time-to-fist--piike coding," in Proceedings of ACM/IEEE Design Automation Conference (DAC), pp- 1-6, 2020.
[28] J. Kim, H. Kim, S, Huh. J. Lee, and K. Choi, "Deep neural networks with weighted spikes," Neurocomput- ing, vol. 311, pp. 373-386, 2018.
[29] M. Zhang, Z. Gu, N. Zheng, D. Ma, and G. Pan, "Efficient spiking neural networks with logarithmic temporal coding," IEEE Access, vol. 8, pp. 98156-98167 2020.
[30] S. Hwang and J. Kung, spik- snn: Single-spike phase coding with base manipulation for ann-to-sm conversion loss minimization." " IEEE Transactions Emerging Topics in Computing, 2024.
11

---

[31] J.-P. Pfister and W. Gerstner, "Triplets of spikes in model of spike timing-dependent plasticity," Journal of Neuroscience, vol. 26, Sept. 2006.
[32] Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Cao, H. Choday, Dimou, P. Joshi, N. Imam, Jain, et al., "Loihi: A neuromorphic manycore processor with on-chip learning." " IEEE Micro, vol. 38, no. 1, pp. 82-99, 2018.
[33] R. Hu, B. Tian, S. Yin. and S. Wei, "Efficient hardware architecture of softmax layer in deep neural network," in Proceedings of International Conference Digital Signal Processing (DSP), pp. 1-5, IEEE, 2018.
[34] Fang. Y. Chen, Ding, Z. Yu, T. Masquelier, D. Chen, Huang, H. Zhou, G. Li, and Tian, "Spikingjelly: An open-source machine learning infrastructure platform for spike- based intelligence," Science Advances, vol. 9, no. 40, eadi 1480, 2023.
[35] Z. Yao, Y. Cao, Y. Lin, Z. Liu, 7. Zhang, and H. Hu, "Leveraging batch normalization for vision transform- ers," " in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 413 422, 2021.
[36] B. Rueckauer, I.A. Lungu, Y. Hu, M. Pfeiffer, and S.-C. Liu, "Conversion of continuous-valuc deep networks to efficient weent-driven networks for image classification," Frontiers in Neuroscience, vol. 11, p. 682, 2017.
[37] M. Yao, J. Hu, T. Hu, Y. Xu. Z. Zhou, Y. Tian, B. XU, and G. Li, "Spike-driven Transformer V2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips." in Proceedings of International Conference on Learning Representations (ICLR), 2024.
[38] F. Ouati, C. Gao, Q. Chen, G. Brignone, M. R. Casu, J. K. Eshraghian, and L. Lavagno, "To spike or not to spike: A digital hardware perspective on deep learning acceleration," IEEE Journal on Emerging and Selected Topics Circuits and Systems (JETCAS), 2023.
[39] N. W. Ming. 7. Wang, C. Liu, R. S. M. Goh, and Luo, "Ma-bert: Towards matrix arithmetic-only bert inference by eliminating complex non-linear functions," " in Proceedings of International Conference on Learning Representations (ICLR), 2022.
[40] M. Bal and A. Sengupta, "Spikingbett Distilling bert to train spiking language models using implicit differentiation," in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 38, pp. 10998- 11006, 2024.
[41] A. Wang. A. Singh, J. Michael, F. Hill, Levy, and Bowman, "Glue: multi-task benchmark and analysis platform for natural language understanding." in Proceedings of International Conference on Learning Reprexentations (ICLR), 2018.
[42] R. Yin, A. Moitra, A. Bhattacharjee, Y. Kim, and P. Panda, "Sata: Sparsity- aware training accelerator for spiking neural networks," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), 2022.
[43] T. Ajayi and D. Blaauw, "Openroad: Toward self-driving, open-source digital layout implementation tool chain," in Proceedings of Government Microcircuit Applications and Critical Technology Conference (GOMACTech), 2019.
[44] H. Wang. Z. Wang, M. Du, F. Yang. Z. Zhang, S. Ding, Mardziel, and X. Hu, "Score-CAM: Score- weighted visual explanations for convolutional neural networks." Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 119, 2020.
[45] P. Dichl, D. Neil, J. Binas, M. Cook, S. Liu, and M. Pfeiffer, "Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing," in Proceedings of International Joint Conference on Neural Networks (ICCNN, 2015.
[46] S. Kim, S. Park, B. Na, and S. Yoon, "Spiking-yolo: spiking neural network for energy-efficient object detection," in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
[47] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000+ questions for machine compre- hension of text," in Proceedings of the the Conference on Empirical Methods in Natural Language Processing (EMNLP) (J. Su, K. Duh and X. Carreras, eds.), (Austin, Texas), pp. 2383 2392, Association Computational Linguistics, Nov. 2016.
12

---

A Energy Metrics
A.1 Simple Energy Metric Considering Only Synaptic Operations
For the fair comparison with the prior work 12 13 energy cost of synaptic operations is estimated based on the power consumption of 32-bit floating-point arithmetic units synthesized in 45mm CMOS technology (2). The synaptic operations of the ANN architecture are MACs, i.e., input activations multiplied by the corresponding weights and partial sums are accumulated. We estimate the energy consumption of an ANN (denoted as EANN) by using the number of synaptic operations associated with the layer of the ANN (S) which is defined as:
NL Σ
EANN (Eadd + Emait)
SynOP!
where Fodd + Emoutt 3. for single 32-bit floating point MAC operation.
Unlike ANNs that require both multipliers and adders, SNNs only require additions. Only weights that are connected to pre-synaptic neurons that fire spikes are accumulated together. Therefore, the energy consumption of an SNN can be estimated by:
T Nu ΣΣ
ESNN
EEad''
Gr SynOPP)
(10)
where of is the ratio of spikes at timestep i and Fradd is 0.9p.J for a single 32-bit floating point addition. Note that the energy numbers reported in the Table I for the entire timesteps in SNN models.
A.2 More Realistic Energy Metric Considering Hardware
The energy metric described in Appendix A.I only considers the computing energy. However, large portion of energy is being consumed by repeatedly reading/writing activations and weights, as well as updating the neuron potential over multiple timesteps. The data movement highly depends on the underlying hardware architecture, e.g., on-chip memory capacity and interconnect topology. For instance, membrane potentials can be fetched from DRAM, whereas the neuron model is computed on the core (42). In that case, the energy cost of fetching membrane potentials becomes significant. On the contrary, Intel's s Loihi (32) places a large on-chip memory to store membrane potentials or traces inside the chip. Therefore, perform and SNNs, we estimated the total energy consumption by using the energy metric presented (38). which considers the energy of reading weights and spikes, the energy of computing the neuron model, and the energy writing spikes back to the memory.
hardware-agnostee energy comparison between ANNs
First, the energy consumption of ANN is computed by:
EANN Erdra
EMMA + Eof fmap:
N₂
Erdeat
2EEd
SSnOOp
Nr. 7' SynOP P!
(11)
EMAC = (Fadd + Emmust)
Eof fmap
Eur
Σ Act'
where Frd... is the energy for reading weights and activations, EMAC is the energy for arithmetic operations, and Eof fmap is the energy for writing MAC results to the scratchpad memory. The Act' is number of output activations at layer I and Era/E,rr is set to 5pJ for the 32-bit read/write operation (9). Also, the activation sparsity considered by the term by for the fair comparison with the SNN.
13

---

Next, the energy consumption of an SNN is computed as follows:
ESNN Erdest + EAC + ENeuron + Bof frop:
Nu
Erdiot
(1+ Brd
: SynOPt
2 Nv.
EMAC = Eaadd
ΣΣ
t SynOP!
(12)
ENeuron (Erd + Emout + Eudd + Ecomp + Eaub + Eur)
Σ
Act',
1=1
1=1
N₁
Bur
Eoffmap
Act'
32
where Erdeg is the energy for reading weights and input spikes, EAC is the energy for synaptic operations (i.e., additions), ENeuron the energy for computing neuron models and foffmap is the energy for writing output spikes to the scratchpad memory. In Erd,.. and Eof fmop. the energy of reading/writing 1-bit spike is E3a/32 (or /3, since E is the read/write energy for 32-bit data. The Eveeron consists of retrieving the neuron's potential (Erz), multiplying leak factor (Fmutti), adding input to potential (Fadd), comparing with the threshold (Fcomp). subtracting the potential by the threshold (F.uh), and finally writing the potential back to the memory (Fwr) As in (38), we defined Esub and Ecomp to have the same energy value as Badd = 0.9pJ. Note that SpikedAttention stops generating spikes neurons that fired already implying that there is no energy consumed by those neurons afterwards. The energy overhead of computing the neuron model was 30~40% of the total energy,
though the total timestep of SpikedAttention converted from Swin-T was T 40. This is because the number of synaptic operations (M SymOP!) is larger than the number of output activations (E Act'). Therefore, even with the overhead of computing the neuron model, SpikedAttention,, which generates a single spike for the entire timestep (i.e., DL oi I 1), is energy efficient since the sparsity of synaptic operations the most critical factor in improving energy efficiency.
B LIF Model with Phase Coding
In this work, our phase coding with the base 'B' follows the phase coding scheme presented in (2)) which fixes the base to 2. The dynamics of neuron in the phase coding can be computed by Eq. by changing the leak factor of the LIF the selected base B.
To express the equivalence clearly, we formulate as the following theorem:
Theorem B.1. Given the following neuron dynamics of the phase coding with the base B
Nave
(1(() = uz - 1)+ Σ
Wiz WiyB (.)().
(A1)
Sy(t) = 1, uf Vo B
0, otherwise
can be formulated to the following LIF-based neural dynamics:
(, Vo 00 otherwise
01(1) B u,(t +)+} Σ w.y (.),,((),S(())
(A2)
Proof. The equivalence of the two dynamics lies in the equivalence of the conditions, (t))) 2 B B- and (t()) Vo. i.e.. (t(t) uy(t) B ft I....,T. We prove this using mathematical induction. For t = 1, we have (1) (1)) B-1 = 0. Assume that u, (t) vy(t) B-t holds for t I,..,
14

---

next we prove that uy(++1) = + 1) B ((((1)
prc
- uy(() + Σ
Wij B (++1) (e(1) Si(t+1) +1)
Npre
-,,(1BB
Wiih
(-++) +1)
(13)
Npra
B B.us(l) B-+1+
wij wi)S((++1)
B (++1)
(1+
C Hardware Overhead of Implementing WOSS Neurons
To implement WOSS in the neuromorphic hardware, we need some modifications to the hardware structure of the existing LIF neuron. The recent Loihi2 (32) allows users to select a neuron model among many neuron models implemented in hardware. Thus, we designed WOSS neuron in Verilog and synthesized the hardware module in 45nm CMOS technology using OpenROAD (45) to estimate the area/power overhead when realizing it top of the commercial neuromorphic hardware. Please note that WOSS neurons are only required for the softmax layer accounting for 19% of all neurons at each attention block of Swin-T. The remaining 81%
implemented typical LIF neurons. As summarized in Table. compared to a general LIF model, the proposed WOSS neuron increases the area by 52% and the power consumption by 65%. This translates to 9.88% larger and 12.35% higher power consumption to support the proposed WOSS method for softmax operations.
Table Area and power overheads of a hardware module for supporting the WOSS neuron
Tech Node: 45nm
Area (Jm=F)
Power (μWW
General LIF
807
542
WOSS LIF
1227
894
Overhead (%)
52
65
D Visualization of Attention Map
Fig. 6 presents the attention maps, highlighting the of activation's interest in the image the outputs of several self-attention layers. The attention maps from the pre-trained Swin Transformer and the SpikedAttenttin were extracted using Score-CAM (44). The attention maps of SpikedAttention from the 9th to 12th attention layer are similar to those of Swin-T, even though the conversion error increases deeper layers. This visualization of attention maps implies that our ANN-to-SNN conversion method successfully transforms attention modules into spike-based operations.
E Selecting the Optimal Base B for a Given Timestep T
To achieve the optimal performance, SpikedAttention needs to set an optimal base. For selecting the optimal base, we define an error function to find the optimal base B, which presents the smallest difference between approximated spike values and true activations. The most common error evaluation is an absolute or relative error. However, we need something more suitable for our work. It is not effective use absolute error like MSE i.c., Σ (x 2 where z is the actual value and å is the approximated value, since larger error gets accumulated for large æ as our spike approximation is log-based. The relative error like ARE, i.e., Σ ش- )] is also not a good candidate, since the same relative difference may imply different timestep scale which depends on the base B. For instance, for a given x, 1.1 x æ 1.95 timesteps away from when the base is 1. 1.05. When the base increases to 1.07, the timestep difference between the two values decreases to 1.41.
15

---

Attn #9
Attn #10
Attn #11
Attn #12
Swin-T (ANN)
Input
m
Ours (SNN)
Figure 6: Comparison of attention maps based on Score-CAMe 44) between Swin-T (baseline ANN) and SpikedAttention (Ours; B 1.16 and T 40) ImageNet- IK at the output of various attention blocks.
We can accurately estimate the approximation error between the real-valued activation and the decoded spike value using the logarithmic error E, (x) as follows:
B 1+
if if
(x)
B L - [1 if -12L2-7
(14)
B 0.5,
otherwise.
where the sampled activation from X of ANN and LL logg x log (max(X)) is the logarithm of normalized activation. Note that as the weights are normalized 45 46) by max(X), i.e., the maximum value of X in subset of the training dataset, each activation approximated by spike as "max(X) x B grr In other words, spike time t represents - L].. Error with values less than "max(X) B is fixed 0.5, since log, is not defined.
Timestep
E (x)
Accuracy
1.00
0.60
Timestep
0.75
0.55
0.80
Accuracy (%)
Ex
Fixed Error
0.50
0.78
-E 1
+
0.50
0.45
0.76
0.40
0.74
0.25
0.35
0.72
0.00
0.30
0.70
B-
B-
B-2
8
1.00
1.05
1.10
1.15
1.20
1.25
1.30
1.35
1.40
1.45
1.50
Normalized Activation x
Base (b)
(a)
Figure 7: (a) Normalized logarithmic error Eg(ar) with respect to the normalized activation
(X)
given timestep T 4. (b) Logarithmic error (Fey(zr) and the accuracy the ImageNet classification task at various base values at a given timestep T 40.
4. The Eq. first computes the -based absolute error, i.c., - [,]], where L is 'log x log (xax By doing this, can accurately measure the error when approximating an activation with a spike having B value. Two exception ranges (i) L -1 and (ii) L < -T and they correctly handled to estimate the enror conservatively. Then, this log based error term is multiplied by B to set the timestep scale properly.
Fig. ([a) shows the error function for a given timestep
To find the optimal base, our method sets the initial base Bg in the midpoint of the search space (1.0, 1.5). Then, we compute the Σ Ex() for both B Bo and B Bo + and
16

---

update the base to one of them that provides a smaller enror. The optimal base with the smallest error can be found within a given search space by iterating the process several times. Fig. 7 presents the logarithmic error (3)) and the accuracy according to the base B. The accuracy loss follows the same trend to our logarithmic error function, i.e., as gets smaller, the accuracy increases while our enror term decreases. If B gets too small, however, the accuracy decreases (equivalently, our error term increases) due to the increased number of underflows. As a result, we utilized the log -based absolute enror multiplied by the base when determining the optimal base for given benchmark.
F Performance on Question Answering
Table 5: Performance of SpikedAttennion on Question Answering
Question Answering
Dataset
Model
Param (M)
Energy (mJ)
Timestep
FI Score (%)
SQuAD
MA-BERT (w/o ReLUU
110
833.8
1
88.3
Ours (Signed)
110
337.9
24
87.2
To demonstrate that SpikedAttention can reduce energy consumption for more complex tasks, converted MA-BERT for question answering to SpikedAttention. We trained the existing MA-BERT on the SQuAD dataset (47) for question answering and converted to spike-based computations. As result, Spiked SpikedAttention achieves an energy reduction of 59.5% with only 1. % accuracy loss, as presented on Table 5
G Impact of Total Timestep on Accuracy and Energy Consumption
(a) w/o ReLU on Swin-T (ImageNet)
(b) w/ ReLU on Swin-T (ImageNet)
Accuracy Energy
Accuracy
Energy
30
Accuracy (%)
0.5
25
Accuracy Loss (%)
0.5
25
(rw) Energy
20
0.3
15
0.3
0.2
0.2
10
(nu)
0.1
0.1
0.0
0.0
56
48
40
24
56
40
32
24
Timestep
Timestep
Figure 8: Accuracy loss and the energy consumption at various timesteps when converting Swin-T to an SNN for ImageNet classification. (a) SpikedAttention from Swin-T without any modification, and (b) SpikedAttention from Swin-T with inserted ReLUs.
Regarding the accuracy, as discussed in Appendix the larger the timestep T, the smaller the base value of one-spike SNN can be. A smaller base reduces the conversion loss, leading to higher accuracy. Regarding the energy consumption, incurs higher energy consumption due more neuron model computations and data movements. To summarize, a longer timestep increases the energy consumption while reducing the accuracy loss. Fig. shows the accuracy loss and the energy consumption at various timesteps when converting Swin-T to SNN for ImageNet classification.. It clearly shows the trade-off between the total timestep and the accuracy/energy consumption. Note that even T 24 the accuracy loss less than 0.5%.
discussed in Appendix A.2 a longer timestep
17

---

Energy consumption of ANN/SNN varies with the input token length for NLP tasks. To evalute this, we varied the input token length MA-BERT (i.e., target ANN) from 64 to 256 on SST-2 dataset. Also, the same lengths of input tokens are fed into the converted SpikedAttention model for the Accuracy Loss
Energy Reduction Ratio
Energy
SpikedAttention Energy
0.6
59.0%
480
Accuracy Loss (%)
0.5
400
58.4%
0.4
320
Energy (mJ)
0.3
240
57.8%
0.2
160
57.3%
0.1
80
0.0
64
128
192
256
Maximum Input Length
Figure 9: Energy consumption of MA-BERT and SpikedAttenniion, and accuracy loss of BERT-to- SNN conversion on the GLUE SST2 dataset according to the maximum input length (i.e., 64, 128, 192 and 256).
H Energy Consumption at Various Input Token Lengths
energy evaluation. Fig. shows energy consumptions at various input lengths for both ANN and SpikedAttention.. In addition, the accuracy losses due to the ANN-10-SNN conversion presented which are negligible (< 1)).
For both ANN and SpikedAttention, the energy consumption increases the maximum input length increases from 64 to 256. This because the number of computations increases in the attention module with the increase in the input length. For instance, MA-BERT with the input length 128 consumes 189.7mJ energy, while MA-BERT with the input length of 256 consumes 458.9m.J (2.4x). Since Spiked/tteention benefits from fully spiked-based based computations, the energy consumption for input lengths 128 and 256 79.9m.J and 188.0m.J, respectively. These energy numbers imply that SpikedAttention is 2.4x more energy-efficient compared to the ANN, regardless of the input token length. As we optimized the spiked-based computation in the attention module with WOSS and trace-based matrix multiplication, the energy reduction ratio of SpikedAttention compared to ANN slightly increases as the input length increases.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See line 51-68 in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See line 307-313 in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
See Appnedix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 5 and Appendix
E
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
See supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / ssiniioiicyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
See Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification: We do not observe large enought variance in our inference of conversion from pre-trained model.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See line 241 Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See line 304-307 in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: We properly cited the pre-trrined models (Swin- Transformer.Maskked Spiking Transformer and MA-BERT) on which our work is based.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: Our work does not release assets
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: Our work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
Our work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "3gvnnIU8Od", "reviewer": "Reviewer_XrJp", "summary": "This paper proposes a transformer-to-SNN conversion method without modifying its attention architecture. To minimize the energy consumption, the authors apply one-spike phase coding, Trace-driven matrix multiplication, and winner-oriented spike shift for softmax. They evaluate their conversion method on vision and NLP tasks including ImageNet classification and GLUE Benchmark.", "questions": "- In WOSS, the winner-spike is time-shifted and fires at t=0. It seems that the output spike always fires at t=0 regardless of when the winner-spike fires. I wonder if there is any information loss from shifting\n- It would be better if the authors could provide the firing rate for the ImageNet dataset. This will be a good comparison with direct coding which is generally used in previous SNN transformers [1,2].\n- In Appendix C, the authors compare the area and power overhead of the general and WOSS LIF in the softmax layer. How about the performance of general and WOSS LIF? It would be better to connect the overheads and performance.\n\n[1] Zhou, Zhaokun, et al. \"Spikformer: When spiking neural network meets transformer.\" arXiv preprint arXiv:2209.15425 (2022).\n\n[2] Yao, Man, et al. \"Spike-driven transformer.\" Advances in neural information processing systems 36 (2024).", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Solid paper structure and intuitive figures. Easy to follow the paper.\n- The proposed methods are well-structured for energy efficiency.\n- Comparable performance with low energy consumption\n- As well as theoretical energy consumption, the authors provide hardware-realistic energy including data movement and the update of membrane potential.\n.", "weaknesses": "- In the vision task, the authors evaluate their method on only the ImageNet dataset. It would be helpful if there are other type of datasets such as neuromorphic datasets.\n- In Table 1, the meaning of without ReLU is ambiguous. Please elaborate the details.\n- In figure 2(b), weights should be $W_Q$, $W_K$, $W_V$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Neuromorphic Datasets)** Thanks for the comment on expanding experiments to the event-based dataset. In this work, we proposed an ANN-to-SNN conversion method for tasks that provide high accuracy using ANNs. Therefore, when submitting the paper, we did not train ANNs on event-based data to convert them to SNNs. Probably, directly training SNNs might be more suitable for event-driven tasks. However, we agree with the reviewer’s suggestion that demonstrating performance on event-based datasets would make our method more comprehensive. In **Table R1** below, we compare the performance of the Spiked-Attention (Ours) with the Masked Spiking Transformer (MST) on CIFAR10-DVS [1] and CALTECH 101 [2]. Input spikes from each event-based dataset are fed into the model over ten timesteps. The results demonstrate that the proposed Spiked-Attention successfully converts ANNs trained on event-based datasets as well. Note that we were unable to report the number of parameters and the energy consumption of MST since the pre-trained MST model on event-based data is not publicly available. Analyzing the energy consumption by considering the neuron model and the data movement of membrane potentials as provided in [3], we can achieve 2.4~2.7x higher energy efficiency than ANNs with less than 0.8% accuracy loss in event-driven datasets. \n* We will add **Table R1** to Appendix in the paper.\n\n| Dataset | Model | Param (M) | Energy (mJ) | Timestep | Accuracy (%) |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| CIFAR10-DVS | ANN (w/ ReLU) | 27.6 | 20.6 | 1 | 88.6 |\n|  | MST (Unsigned) | - | - | 128 | 86.6 |\n|  | Ours (Unsigned) | 27.6 | 8.4 | 48 | 88.3 |\n| N-CALTECH 101 | ANN (w/ ReLU) | 27.6 | 23.5 | 1 | 91.6 |\n|  | MST (Unsigned) | - | - | 64 | 84.7 |\n|  | Ours (Unsigned) | 27.6 | 8.6 | 48 | 90.8 |\n**Table R1: Experiment results on event-based dataset**\n\n[1] Li, H et al., CIFAR10-DVS: An event-stream dataset for object classification. Frontiers in Neuroscience, 2017.\n\n[2] Orchard, G et al., Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience, 2015.\n\n\n\n**Notation in Figure 2)** In Figure 2, $S_Q$, $S_K$, and $S_V$ are query ($Q$), key ($K$), and value ($V$) within an attention layer in spike forms, not the weights required to generate $Q$, $K$, and $V$. Thus, the notation ‘$S$ ’ represents a spike train. In other words, these are activations generated by the “weights” you mentioned. \n\nFigure 2(a) is the attention operation used in [3], which performs the AND operation between spike-encoded $Q$ and spike-encoded $K^T$, followed by the floating-point softmax operation to obtain an attention map. Figure 2(b) is the attention operation presented in [4], which uses the AND operation between $Q$, $K$, and $V$ in spike forms. However, it omits the softmax operation which cannot be generalized to any transformer models.\n\n[3] Z. Wang, et al., “Masked spiking transformer,” ICCV’23.\n\n[4] M. Yao, et al., “Spike-driven transformer,” NeurIPS’23. \n\n\n**Information Loss in WOSS)** We appreciate the comment about information loss due to WOSS-based softmax. As mentioned in the paper, the output spikes are shifted to always have a winner spike at t=0. Since the winner spike generated at t=0 is the first output spike, no spike occurs before the winner spike. Therefore, shifting the spikes based on the winner spike does not result in any information loss. Rather, a winner-spike-based shift increases the range of time steps over which spikes can be generated for a given total timestep T. In short, the proposed WOSS method improves the precision of the softmax operation by extending useful time steps rather than losing information.\n\n**Spike Rate on ImageNet)** Thanks for the comment on the spike rate for the ImageNet dataset. When converting Swin-T (‘w/o ReLU’) to an SNN using SpikedAttention on ImageNet, we observed an average spike rate of 1.8%. For the comparison, we re-implemented MST [3] and obtained the average spike rate of 30%. It means that our conversion method provides the SNN model which fires much less resulting in higher energy efficiency. \n* We will add this comparison on firing rate between SpikedAttention and MST [3] in the paper.\n\n**Performance of WOSS Hardware)** Thanks for the comment on the performance of the general LIF and the WOSS LIF hardware modules. The general and WOSS LIF hardware modules presented in Appendix C have the same performance, in terms of latency, because they are synthesized and operate at the same clock frequency, i.e., 333MHz. It means that the membrane potential of each neuron, either general or WOSS, gets updated in 3ns.\n* We will add this information in Appendix C.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4p1oN2dHCr", "reviewer": "Reviewer_xJgY", "summary": "The paper presents SpikedAttention, a novel method for converting pre-trained transformers into spiking neural networks (SNNs). The method introduces two key techniques: trace-driven matrix multiplication and winner-oriented spike shift (WOSS) for softmax. These techniques enable the conversion of attention modules into spike-based computations without altering the original transformer architecture or requiring any additional training. The authors demonstrate the effectiveness of SpikedAttention by converting Swin Transformer for image classification and BERT for natural language processing tasks, achieving state-of-the-art accuracy and significant energy reduction compared to previous SNN-based transformers and the original ANN models.", "questions": "1. Impact of timestep on performance: How does the choice of timestep affect the accuracy and energy efficiency of SpikedAttention, and is there a trade-off between these factors?\n\n2. Performance on more complex tasks: How does SpikedAttention perform on more challenging vision and language tasks, such as object detection, machine translation, or question answering?\n\n3. Hardware implementation and optimization: What are the specific hardware considerations and potential optimizations for deploying SpikedAttention on neuromorphic chips or other specialized hardware?", "limitations": "The paper discusses limitations in the final section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel techniques. The paper introduces innovative methods for implementing attention mechanisms in SNNs, addressing the challenges of softmax computation and matrix multiplication between spike-based matrices.\n\n2. State-of-the-art accuracy and energy efficiency. The proposed SpikedAttention achieves impressive results on both vision and language tasks, demonstrating superior accuracy and energy savings compared to existing SNN-based transformers.\n\n3. Direct conversion without training. The method enables the direct conversion of pre-trained transformers into SNNs without requiring any additional training or architectural modifications, making it a practical and efficient approach.\n\n4. Applicability to both vision and language tasks. The authors showcase the versatility of SpikedAttention by successfully converting both Swin Transformer and BERT models, highlighting its potential for broader applications in various domains.", "weaknesses": "1. Longer timestep compared to directly trained SNNs. The paper acknowledges that SpikedAttention requires a longer timestep than directly trained SNNs to maintain high accuracy, which could impact its latency and efficiency in certain scenarios. Could the authors draw a figure to show the accuracy and energy with different timesteps?\n\n2. Limited support for certain functions. The current implementation of SpikedAttention does not support GeLU and LayerNorm, limiting its applicability to most models.\n\n3. There is no ablation study in this work. What if trace-driven matrix multiplication is absent? What if WOSS softmax is missing?\n\n4. Limited discussion on hardware implementation: While the paper mentions the potential for hardware implementation, a more in-depth analysis of the hardware implications and optimizations would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Impact of Timestep on Accuracy/Energy)** Thanks for the comment about the trade-off between the total timestep and the accuracy/energy efficiency. Regarding the accuracy, as discussed in Appendix E, the larger the timestep T, the smaller the base value of an one-spike SNN can be. A smaller base reduces the conversion loss, leading to higher accuracy. Regarding the energy consumption, as discussed in Appendix A.2, a longer timestep incurs higher energy consumption due to more neuron model computations and data movements. To summarize, a longer timestep increases the energy consumption while reducing the accuracy loss.\n\nAs requested, we attached **Figure R1** (on PDF of global rebuttal) showing the accuracy loss and the energy consumption at various timesteps when converting Swin-T to an SNN for ImageNet classification. It clearly shows the trade-off between timestep and accuracy/energy consumption.\n* We will add this figure to Appendix in the paper. \n\n**Ablation Study)** Thanks for the comment on the ablation study. We also considered adding experiment results without either trace-driven matrix multiplication or WOSS. However, those two methods are indispensable to realize the end-to-end spike-based computation for transformers (i.e., without any one of them, it is impossible to run the model without structural modification [1,2] or involving floating-point computations [3]). First, when we use AND operation instead of the trace-driven multiplication in SpikedAttention, the activation values floor to zero as the layers get deeper because the one-spike SNN is highly sparse, and the accuracy collapses to zero. Second, without WOSS, the softmax operation cannot be implemented by spike-based computations, and the attention module of the ANN needs to be restructured (e.g., removing softmax) and trained from scratch. Since our most important goal was achieving “training-free ANN-to-SNN conversion”, converting the softmax to spike-based computations without WOSS is impossible.\n\n[1] Z. Zhou, et al., “Spikformer: When spiking neural network meets transformer,” ICLR’23.\n\n[2] M. Yao, et al., “Spike-driven transformer,” NeurIPS’23.\n\n[3] Z. Wang, et al., “Masked spiking transformer,” ICCV’23.\n\n**Hardware Deployment)** SpikedAttention is not a model targeted for a specific type of hardware but designed to support a diverse set of hardware architectures. For example, Loihi [4], i.e., Intel’s neuromorphic chip, implements a trace model in hardware that decays and tracks the per-neuron trace over time. The hardware module for updating neuron traces is implemented on a neuromorphic chip to support spike-timing-dependent plasticity (STDP), i.e., a well-known unsupervised learning rule. We intentionally designed the trace-driven matrix multiplication to utilize the already available hardware resources for neuron traces. Therefore, trace-based matrix multiplication is fully supported on neuromorphic chips. In addition, recent Loihi2 allows users to select a neuron model among various neuron models implemented in hardware. To support WOSS neurons, we can modify the existing LIF neuron hardware and hardware overhead details are provided in Appendix C of the submitted paper.\n\nSpikedAttention has a longer timestep compared to directly trained SNN models, but only generates one spike within the timestep (highly sparse). SpikedAttention has an average spike rate under 2%, while the previous works typically have a spike rate of 30%. Since it has significantly higher sparsity than other SNNs, it can be computed with low latency and high energy efficiency if sparse GEMM accelerators [5,6,7,8] are used. The throughput of spGEMM accelerators is determined by the total number of non-zeros (spikes) amortizing the impact of a longer timestep. For instance, if the spike rate is 10x lower, we can keep the performance at the same level even with a 10x longer timestep. Recently, accelerators targeted for sparse SNNs have been proposed to maximize the throughput and efficiency when running SNNs [9,10]. \n\n[4] Davies, M., et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro’18.\n\n[5] Z. Zhang, et al., “Sparch: Efficient architecture for sparse matrix multiplication,” HPCA’20.\n\n[6] Jinkwon Kim, et al., “Harp: Hardware-based pseudo-tiling for sparse matrix multiplication accelerator”. MICRO’23.\n\n[7] N. Srivastava, et al., “Matraptor: A sparse-sparse matrix multiplication accelerator based on row-wise product,” MICRO’2020.\n\n[8] G. Zhang, et al., “Gamma: Leveraging gustavson’s algorithm to accelerate sparse matrix multiplication,” ASPLOS’21.\n\n[9] S. Narayanan, et al., “Spinalflow: An architecture and dataflow tailored for spiking neural networks,” ISCA’20\n\n[10] R. Yin, et al., “Loas: Fully temporal-parallel datatflow for dual-sparse spiking neural networks,” 2024.\n\n\n**Performance on Complex Task)** Thanks for the comment on performance on more complex tasks. To demonstrate that SpikedAttention can reduce energy consumption for more complex tasks, we converted MA-BERT for question answering to SpikedAttention. We trained the existing MA-BERT on the SQuAD dataset [11] for question answering and converted it to spike-based computations. As a result as presented on **Table R1** (on PDF of global rebuttal), SpikedAttention achieves an energy reduction of 59.5% with only 1.1% accuracy loss. \n\n* We will add **Table R1** to Section 5.2 (Conversion of BERT to SpikedAttention) in the paper.\n\n[11] Pranav Pajpurkar, et al., “SQuAD: 100,000+ questions for machine comprehension of text.” ACL’16.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g4v1skFWK4", "reviewer": "Reviewer_uL88", "summary": "The paper introduces a novel method for converting Swin Transformer and BERT into SNN without additional training, achieving high accuracy and energy reduction. Key innovations include fully spike-based encoding, trace-driven matrix multiplication, and an exponent-free spike-based softmax using winner-oriented spike shift. The method maintains the original attention architecture, achieving state-of-the-art accuracy on ImageNet with a 42% energy reduction and only 0.3% accuracy loss on GLUE benchmarks while reducing energy consumption by 58%.", "questions": "1. In Table 1, why are there two Meta-SpikeFormer?\n2. In Table 1, why SpikedAttention w/o ReLU overperforms SpikedAttention w/ ReLU in terms of Acc?\n3. How will the energy consumption change when the input length increases in NLP tasks?", "limitations": "1. SpikedAttention requires a longer timestep to maintain high accuracy compared to directly trained SNNs, which could limit its efficiency and responsiveness in real-time applications.\n\n2. The method does not currently support functions like GeLU and LayerNorm, making it difficult to generalize to all types of language models and potentially limiting its applicability across diverse neural network architectures.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. SpikedAttention allows for direct transformer-to-SNN conversion without requiring additional training, preserving the original attention architecture and simplifying the deployment process of energy-efficient neural networks.\n\n2. SpikedAttention significantly reduces energy consumption by 42% compared to baseline models like Swin Transformer for image classification and by 58% for NLP tasks on the GLUE benchmark.\n\n3. The method achieves state-of-the-art accuracy for SNNs on the ImageNet dataset (80.0%) and maintains high performance with minimal accuracy loss (0.3% on average) when converting BERT models.", "weaknesses": "1. SpikedAttention requires a longer timestep to maintain high accuracy compared to directly trained SNNs, which could limit its efficiency and responsiveness in real-time applications.\n\n2. The method does not currently support functions like GeLU and LayerNorm, making it difficult to generalize to all types of language models and potentially limiting its applicability across diverse neural network architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Long Timestep of SpikedAttention)** Thanks for the comment about the total timestep. As you mentioned, SpikedAttention requires a longer timestep than a directly trained SNN. However, directly trained prior works [1,2] have 2x more weight parameters to achieve the accuracy of 80%. More parameters mean that a higher number of off-chip memory accesses is required which is a significantly more energy consuming operation. Also, fetching data from off-chip DRAM takes longer which increases the latency. Note that a model in [1] with a similar number of parameters as SpikedAttention is >3% less accurate than SpikedAttention.\n\nIn addition, SpikedAttention has a longer timestep but only generates one spike within the timestep (highly sparse). SpikedAttention has an average spike rate under 2%, while the previous works typically have a spike rate of 30%. Since it has significantly higher sparsity than other SNNs, it can be computed with low latency and high energy efficiency if sparse GEMM accelerators [3,4,5,6] are used. The throughput of spGEMM accelerators is determined by the total number of non-zeros (spikes) amortizing the impact of a longer timestep. For instance, if the spike rate is 10x lower, we can keep the performance at the same level even with a 10x longer timestep. Recently, accelerators targeted for sparse SNNs have been proposed to maximize the throughput and efficiency when running SNNs [7,8]. \n\n[1] M. Yao, et al., “Spike-driven Transformer V2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips,” in Proc. of ICLR, 2024. \n\n[2] M. Yao, et al., “Spike-driven transformer,” NeurIPS’23.\n\n[3] Z. Zhang, et al., “Sparch: Efficient architecture for sparse matrix multiplication,” HPCA’20.\n\n[4] Jinkwon Kim, et al., “Harp: Hardware-based pseudo-tiling for sparse matrix multiplication accelerator”. MICRO’23.\n\n[5] N. Srivastava, et al., “Matraptor: A sparse-sparse matrix multiplication accelerator based on row-wise product,” MICRO’20.\n\n[6] G. Zhang, et al., “Gamma: Leveraging gustavson’s algorithm to accelerate sparse matrix multiplication,” ASPLOS’21.\n\n[7] S. Narayanan, et al., “Spinalflow: An architecture and dataflow tailored for spiking neural networks,” ISCA’20.\n\n[8] R. Yin, Y. Kim, D. Wu, and P. Panda, “Loas: Fully temporal-parallel datatflow for dual-sparse spiking neural networks,” 2024. \n\n**Meta-SpikeFormer Results in Table 1)** Thanks for the comment on experimental results of the prior work, i.e., Meta-SpikeFormer. We trained two different versions of Meta-SpikeFormers, i.e., one with high accuracy and one with low energy consumption, for the comparison with the proposed SpikedAttention. For the higher accuracy, we trained Meta-SpikeFormer which has total timestep of 4. For the higher energy efficiency, we trained Meta-SpikeFormer with the total timestep of 1.\n\n**Energy Consumption w.r.t. Input Token Length)** We agree with the reviewer that energy consumption of ANN/SNN varies with the input token length for NLP tasks. Therefore, we varied the input token length of MA-BERT (i.e., target ANN) from 64 to 256 on SST-2 dataset. Also, the same lengths of input tokens are fed into the converted SpikedAttention model for the energy evaluation. The attached **Figure R2** (on PDF of global rebuttal) shows energy consumptions at various input lengths for both ANN and SpikedAttention. In addition, the accuracy losses due to the ANN-to-SNN conversion are presented which are negligible (< 1%). \n\nFor both ANN and SpikedAttention, the energy consumption increases as the maximum input length increases. This is because the number of computations increases in the attention module with the increase in the input length. For instance, MA-BERT with the input length of 128 consumes 189.7mJ of energy, while MA-BERT with the input length of 256 consumes 458.9mJ (2.4x). Since SpikedAttention benefits from fully spiked-based computations, the energy consumption for input lengths of 128 and 256 is 79.9mJ and 188.0mJ, respectively. These energy numbers imply that SpikedAttention is ~2.4x more energy-efficient compared to the ANN, regardless of the input token length. As we optimized the spiked-based computation in the attention module with WOSS and trace-based matrix multiplication, the energy reduction ratio of SpikedAttention compared to ANN slightly increases as the input length increases.\n\n* We will add this relationship between input token length and the energy consumption in Appendix of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and insightful comments which have helped improve our paper. We address a few common points in this response. All other questions are addressed in reviewer specific responses.

**Meaning of ‘w/o ReLU’)** *( for Reviewer XrJp and Reviewer uL88 )* Since a spike in SNNs can only represent {0,1}, the binary spikes as an activation map in SNNs cannot represent negative values, thus having limited information capacity compared to the activation map of ANNs. Since CNNs normally have positive activations with ReLU as an activation function at all conv layers, it is easier to convert them using binary spikes. However, simply converting transformers to SNNs may cause accuracy loss because many layers are not followed by ReLU, e.g., normalization layers and linear layers. Therefore, in the experiments of SpikedAttention, we tested two transformer versions for the ANN-to-SNN conversion, i.e., one with ReLU inserted at all layers to always generate positive-only activations (denoted as ‘w/ ReLU’) and another without such modification (denoted as ‘w/o ReLU’). Thus, transformers without inserting additional ReLUs but keeping the originally placed ReLUs are called ‘w/o ReLU’ in the paper. Since most layers are keeping both positive and negative activations in the original model (i.e., ‘w/o inserted ReLUs’), it shows higher accuracy than the model ‘w/ inserted ReLUs’. 

In our Swin-T experiments, for instance, Swin-T without inserted ReLUs is converted to an SNN with positive and negative spikes making the SNN ternary similar to [1]. However, this increases the number of spikes in the converted SNN model increasing the energy consumption (refer to Table 1; 3mJ vs. 1.8mJ). To avoid this, we may add ReLUs at all intermediate layers, i.e., Swin-T w/ ReLU, and convert the model to an SNN (like MST [2]). Then, since all activations are positive, we can easily represent them using binary spikes. 

Since our final goal is to convert transformers without any modification or additional training, converting ANN w/o ReLU is desirable. Therefore, we converted MA-BERT w/o (inserted) ReLUs and reported the results in Table 3.
* We will edit the paper accordingly in order not to confuse the readers regarding the term ‘w/o ReLU’.

[1] Y. Guo, “Ternary Spike: Learning Ternary Spikes for Spiking Neural Networks”, AAAI’ 24.

[2] Z. Wang, et al., “Masked spiking transformer,” ICCV’23.

**Support of GeLU and LayerNorm)**  *( for Reviewer xJgY and Reviewer uL88 )* We appreciate your comment about supporting fully spike-based GeLU and LayerNorm. As pointed out by reviewers, the conversion method proposed in SpikedAttention does not support GeLU and LayerNorm yet. As mentioned in “Conclusion”, SpikedAttention has focused on converting the attention module and the softmax operation to fully spike-based operations, which are fundamental building blocks of all transformer models. Please consider this work as the first yet big step towards supporting all transformers by fully spike-based computations including yet to be conquered GeLU and LayerNorm layers. This remains as our future work.

Meta-review: The paper has received positive reviews. It has a good energy efficiency perspective and exposes SNNs to attention mechanism for vision and language tasks.
Decision: Accept (poster)