SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks
Authors: Rui-Jie Zhu
Qihang Zhao
Guoqi Li
Jason Eshraghian
Venue: Accepted by TMLR ()

Abstract: As the size of large language models continue to scale, so does the computational resources required to run them. Spiking Neural Networks (SNNs) have emerged as an energy-efficient approach to deep learning that leverage sparse and event-driven activations to reduce the computational overhead associated with model inference. While they have become competitive with non-spiking models on many computer vision tasks, SNNs have proven to be more challenging to train. As a result, their performance lags behind modern deep learning, and until now, SNNs have yet to succeed at language generation on large-scale datasets. In this paper, inspired by the Receptance Weighted Key Value (RWKV) language model, we successfully implement `SpikeGPT', a generative language model with binary, event-driven spiking activation units. We train the proposed model on two model variants: 46M and 216M parameters. To the best of our knowledge, SpikeGPT is the largest backpropagation-trained SNN model when released, rendering it suitable for both the generation and comprehension of natural language. We achieve this by modifying the transformer block to replace multi-head self-attention to reduce quadratic computational complexity $\mathcal{O}(T^2)$ to linear complexity $\mathcal{O}(T)$ with increasing sequence length. Input tokens are instead streamed in sequentially to our attention mechanism (as with typical SNNs). Our experiments show that SpikeGPT remains competitive with non-spiking models on tested benchmarks, while maintaining 32.2$\times$ fewer operations when processed on neuromorphic hardware that can leverage sparse, event-driven activations. Our code implementation is available at https://github.com/ridgerchu/SpikeGPT.

Published in Transactions on Machine Learning Research (06/2024)
SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks
Rui-Jie Zhu
reba48
Deportment of Electrical and Computer Engineering University of California, Santa Cruz
Qihang Zhao
shaoghanng@kzaissoo..co com
Kuaishou
Guoqi Li
guoqi.li@ia.oco
Institute of Automation Chinese Academy of Sciences
Jason K. Eshraghian
jan edu
Deportment of Electricol and Computer Engineering University of California, Santa Cruz
Reviewed on OpenReview: https: // openreview. net/ form?? id= gcf Dcf1ee
Abstract
As the size of large language models continue to scale, so does the computational resources required run them. Spiking Neural Networks (SNNs) have emerged as an energy-efficient approach deep learning that leverage sparse and event-driven activations to reduce the computational overhead associated with model inference. While they have become competitive with non-spiking models on many computer vision tasks, SNNs have proven to be more challenging to train. As a result, their performance lags behind modern deep lcarning, and until now, SNNs have yet to succeed at language generation on large-scale datasets. In this paper, inspired by the Receptance Weighted Key Value (RWKV) language model, we successfully implement 'SpikeGPT', generative language model with binary, event-driven spiking activation units. We train the proposed model on two model variants: 46M and 216M parameters. To the best of our knowledge, SpikeGPT is the largest backpropagattion-trainee SNN model when released, rendering it suitable for both the generation and comprehension of natural language. We achieve this by modifying the transformer block to replace multi-head self-attention to reduce quadratic computation atipuaational complexity O(T2) to linear complexity (TT) with increasing sequence length. Input tokens are instead streamed in sequentially to our attention mechanism (as with typical SNNs). Our experiments show that SpikeGPT remains competitive with non-spiking models on tested benchmarks, while maintaining 32.2x fewer operations when processed neuromorphic hardware that leverage sparse, event-driven activations. Our code implementation is available at https://gitaie com/ ridgerchu//pikkoP"
1 Introduction
Artificial Neural Networks (ANNs) have recently achieved widespread, public-facing impact in Natural Language Processing (NLP), but with a significant computational and energy consumption burden across training and deployment. As an example, training GPT-3 was projected to use 190,000 kWh of energy (Brown) Anthony al. 2020 Spiking neural networks (SNNs), inspired by neuroscientific
et al.
2020
Dhar 2020

---

Published in Transactions on Machine Learning Research (06/2024)
models of neuronal firing, offer more energy-efficient alternative by using discrete spikes to compute and transmit information (Maass 1997). Spike-based computing combined with neuromorphic hardware holds great potential for low-energy Al (Davies et al.
2018 Merolla et al. 2014 Sun et al. 2022 Nath et al. 2024).
and its offectiveness in integration with deep learning has been demonstrated through numcrous studies (Roy
et al. 2019: Pleifler & Pleil 2018 Fang et al. 2021 Eshraghian et al.) 2021 Wu et al. 2018 Zhang et al. 2020
While SNNs have shown competitive performance in computer vision tasks such as classification and object detection (Barchid et al. 2023 Kim et al. 2020 Cordone al. 2022), they have yet to attain similar success in generative models. With respect to language generation, this can be attributed to several reasons: i) the absence of an effective language encoding technique for SNNs, ii) the difficulty of training large-scale SNNs due to the extreme constraint on layer-to-layer bandwidth (i.e., binarized spike activations) (Eshraghian et al. 2022), and the lack of informative gradient signals in excessively sparsified models (Eshraghian Lu 2022], and iii) the inherent recurrence of SNNs incompatible with self-attention, where input token are passed to the model in parallel (Vaswani et. al. 2017). These issues mean that training large-scale SNNs via error backpropagation is extremely challenging, leading to a lack of performant SNNs in language generation.
Despite the difficulties faced by recurrent networks in NLP, the sequential structure of linguistic data presents a unique advantage for SNNs. To address these problems, we propose three techniques. First, rather than using an encoder to project information into an additional temporal dimension, SpikeGPT aligns the sequence dimension of language with the temporal dimension of SNNs. This eliminates the need for an encoder as with most prior attention- based SNNs (Zhou et al., 2023 Li et al. 2022 Deng et al., 2024 QQu et al. 20244. Second, apply autoregressive training rather than accumulating spikes the output to caleulate the loss, as is common in modern SNNs. Third, although binary activations constrain layer-to-layer bandwidth, we show that using stateful neurons can greatly reduce the adverse impact of binarization. By leveraging the capabilities of the RWKV model and integrating these three techniques, we have developed the SpikeGPT language model. This was the first SNN to achieve language gencration at the time of writing and code release of SPikeGPT. It also the largest SNN trained to date in terms of parameter count when it released; the largest version has 216M parameters, which 3x more than the previous largest SNN (Zhou et al., 2023).
The implementation of SpikeGPT is based on integrating recurrence into the Transformer block such that it is compatible with SNNs and eliminates quadratic compulational complexity, allowing for the representation of words as event-driven spikes. Combining recurrent dynamics with linear attention enables our network to stream incoming data word-by-word, and commence computation before a sentence has been completed, while still retaining long-range dependencies present in complex syntactic structures. Our experiments show that SpikeGPT achieves competitive performance on all tested datasets while consuming significantly less energy compared to traditional ANN models. Our contributions in the field of NLP and language generation can be succinetly described as follows:
1. we provide the first demonstration of langunge-generation using direct-SNN training;
2. we achieve performance comparable to that of ANNs, while preserving the energy efficiency of spike- based computations and reducing quadratic computational complexity O(T²) to linear complexity O(T);
3. our results demonstrate that small-scale variant of the SpikeGPT model with 46 million parameters performs competitively against similar transformer models. Furthermore, it achieves this performance with an estimated 3 less energy consumption on asynchronous hardware.
2 Related Works
Although language generation has not previously been achieved with SNNs, this section provides an overview of how SNNs have been used in basic NLP tasks, and the ways in which transformers have been adopted for SNNs.
Spiking Neural Networks for Natural Language Processing. Xiao al. (022)) proposes a bi- directional SNN for sentiment classification and machine translation tasks. Their approach uses spiking
2

---

Published in Transactions on Machine Learning Research (06/2024)
encoders, which replace costly multiplication operations with much cheaper additive operations to significantly reduce computational energy consumption. Similarly, Lv et al. (203bb) presents a two-step method to train SNNs for text classification with a simple way to encode pre-trained word embeddings as spike trains. Their results indicate that converted SNNs achiove comparable results to their ANN counterparts and are more robust against, adversarial altacks. Furthermore, Diehl et al. (2016) demonstrate the train-and-consttrain methodology that enables the mapping of machine-learned recurrent neural networks (RNNs) on substrate of spiking neurons. The authors achieve 74% accuracy on question classification task using less than 0.025% of the cores one TrueNorth chip (Merolla et al., 2014), showcasing the potential for SNNs in classification tasks in NLP. For the pre-train language model side, two works [Bal & Sengupta (0023 and Lv et al. 2023a
offer a direct-trained spiking Bidirectional Encoder Representations from Transformers (BERT) model via
knowledge distillation.
Transformer in Spiking Neural Networks. The Transformer model, first introduced in Vaswani et al, (017), has shown significant success in various NLP tasks. However, the application of the Transformer model to SNNs has been relatively limited. The first Spiking Transformer model was proposed in [Zhou el al. (2023). which proposes spiking self-attention to model visual features using sparse Query, Key and Value matrices. Li et al. (2022) proposes another variant on Transformer- based SNNs, adopting spatial-temporal attention instead of spatial or temporal wise attention to better incorporate the attention mechanism within the Transformer.
While Transformers were initially proposed to solve NLP lasks, SNN-based Transformers have only succeeded with vision tasks. An additional temporal dimension must be added which increases the computational complexity from quadratic to the cubic order (O(T3)), which makes training more expensive. The additional challenges of extreme sparsity, non-differential operators, approximate gradients, and single-bit activations that are characteristic of SNNs make training convergence more challenging. The demonstrated image classification tasks have a far smaller number of output classes, which shrinks the scale of demonstrated networks. Image classification also does not exploit the inherent long-range learning capacity self-attention. Therefore, there is under-explored potential in the application of Translormer models in other SNN-based applications beyond vision tasks. the following sections, we demonstrate how this computational complexity is reduced to enable scaled-up models that are capable of language generation.
3 Methods
3.1 Leaky Integrate-and-Fire Neuron
We employ the Leaky Integrate-and-Fire (LIF) neuron as the default spiking neuron of our model, widely used model for SNNs often trained via error backpropagation [Maass 1997). The LIF dynamics are represented as follows:
Uit] H []] 8 (H[t 1] Ureset)) Sii (U Uehreshold)
(1)
H = U -())(tt)
where 3 a decay factor, is the membrane potential (or hidden state) of the neuron, is the spiking tensor with binarized elements, Y denotes the output of the previous series RWKV block (see Eq. 7 O(-) denotes the Heaviside function, and represents the resct process after spike emission. We set Unreeshold 1,
Uresat and 3 = 0.5 done in Zhu et al. (2022) and Yao al. 22021 2023).
To overcome the non-differentiable problem during back-propagation caused by the Heaviside step function (()), we employ the arctangent surrogate function during the backward pass. The arctangent function of (z) = a=+( (a Sigmoid-like shape) is applied as a 'surrogate gradient" for backward propagation provide a biased gradient estimator (Fang al., 2021 Neftci al. 2019). Additional details on the derivation and training of LIF neurons using backpropagation provided in Appendix
3

---

Published in Transactions on Machine Learning Research (06/2024)
Spikdngg RWVV
X(0(5plk))
Spiking (B33)p
Linear
AHK
MI Mi
KIGKNG
R
XL
VIA
PTA Mu (E 28) MM, x
Add & Norm
My. Max My
Tiasspoc
Spiking RFFN
RO
A[d
VIAI
Add & Norm
Nx
Spiking RWKV
My CAE
Ste]
[c]
A19
(Sparse)
Binary Embedding
Hodamand Product
Ste]
Spkking
S Salking
Input
Neuron
Sigmoid
Y(+)
r(e]
Figure 1: Model Architecture. The left portion displays the block-level structure. The middle and right illustrations demonstrate the Spiking RWKV and Spiking RFFN architectures, respectively. Spiking RWKV serves as a token mixer and Spiking RFFN functions as a channel mixer. These components are arranged in a loop with residual connections a manner akin to a Transformer architecture.
3.2 Model Architecture
The high-level architecture of SpikeGPT is shown Fig. Given an embedded input E Rxxd we first use Binary Embedding (BE) layer embed the input to binarized representation, Xo (Eq. 2). Using Spiking
RWKV, X, is passed to the L-th SpikeGPT layer. Similar to the standard Transformer block, SpikeGPT block consists of Spiking RWKV (SRWKV) unit and Spiking Receptance Feed-Forward Networks (SRFFN) unit. Residual connections are used in both the SRWKY and SRFFN blocks using the SEW-RResNet (Fang et al. 2021 configuration, which is well-established standard form within the Spiking ResNet framework with sparse integer spikes. Once the data has traversed through all layers, the model is directed towards the generation head (GH) for the purpose of generating the next token. For natural language understanding (NLU) tasks, the model uses classification head instead. Sec. 3.6 provides further information.
X, (E))),
I RTxd Xo Txd X/ E Rxd 1...L X1 E Txd .1 1= 1...I
X/ = SX X1-1
(3)
X, SRFFN(X)) + +X,,
(4)
Y GH(X)).
5
In Section 3.3 detailed information about Binary Embedding (BE) will be provided. Sec. 3.4 will primarily focus on the SRWKV block, while Sec. 3.5 will delve into the Spiking Receptance Feed-Forward Network (SRFFN) block. Finally, in See 3.6 we will describe the training and inference details employed for natural language generation (NLG) and natural language understanding (NLU), At the commencement of each block, token-shift is applied which is detailed in Appendix C.1
4

---

Published in Transactions on Machine Learning Research (06/2024)
3.3 Binary Embedding
To maintain consistency with the binary activations of SNNs, propose binary embedding step to convert the continuous outputs of the embedding layer into binary spikes. The conversion is performed using a Hcaviside function for feed-forward propagation which maps the continuous values to binary spikes. This allows us to convert continuous embedding values into spikes using non-differentiable functions, while still being able to perform backpropagation and update the weights of the embedding layer (Neftci et al. (2019). During backpropagation, the arctangent surrogate function is applied as is the case with LIF neurons.
3.4 Efficient Processing of Variable-Length Sequences Using Spiking RWKV
In this section, we revisit the self-attention mechanism, which endows Transformers with the ability to process variable-length sequences, enabling advanced language modeling capabilities. Subsequently, we highlight fundamental challenge: the incompatibilityy of sell-attention with SNN language modeling. Self-attention relies on access to the entire sequence for effective language modeling and is not inherently recurrent, making it incompatible with SNNs. Furthermore, it involves dynamic matrix-matrix multiplication operations. The associated computational overhead compromises the objective of reducing computational costs via SNNs. To address these limitations, we introduce the Spiking RWKV (SRWKV) module as replacement for the self-attention module. This module draws inspiration from the RWKV model (Peng et al., 2023), an RNN model renowned for achieving Transformer-level proficiency across spectrum of model sizes. This module serves the same essential function of facilitating information exchange across token dimensions but achieves this through clement-wist products rather than matrix-matrix multiplication, while also accommodating the recurrent nature of SNNs.
Recall Aelf-Attention. In the context of Transformers, the self-attention mechanism operates on an input sequence denoted as X R and employs a scaled dot product attention tochnique. Mathematically, self-attention is formally expressed as follows:
f(x) softmax QQ()* V.Q XMo, K XMR, V X dd
Here, Me Rdxds, Mx Rdxds and My Bdxd, represent linear transformations, while d& and de signify the dimensions of the key and value vectors, respectively. This mechanism enables dynamic information mixing across token dimensions and accommodates sequences of variable lengths by leveraging dynamic matrix-matrix multiplication (Q(K)?) and the softmax function (which necessilates access to the entire sequence).
Nevertheloss, in the case of a typical SNN characterized by its event-driven nature, the recurrent structure of the network poses a challenge. SNNs can only generate spikes based on information from previous time-sleps, making it impossible access the entire sequence at once, as required by self-attention. One potential solution involves introducing
additional temporal dimension, denoted as Tadditional, to allow spiking neurons to feed forward. However, this approach would substantially increase the model's size and necessitate additional computations, scaling proportionally with Tadditional- This is not practical, especially for large-scale language models reaching the scale of hundreds of billions of parameters. Therefore, the optimal approach is not to introduce a new dimension, but to leverage the existing sequence dimension for forward propagation. This is why a spiking transformer would benefit from a recurrent alternative self-attention.
To address the challenges posed by self-attention in the context of language modeling, introduce the Spiking RWKV, novel approach that incorporates both recurrent and block-level spiking features. The foundation of the vanilla RWKV is inspired by the Attention Free Transformer (Zhai et al. 2021), serving as a replacement for the traditional self-attention mechanism. For comprchensive information on vanilla RWKV, including its overall structure and parallelization techniques, we refer the reader to Appendix C.I. In contrast to self-attention, which operates on the entire sequence dimension, the spiking input of Spiking RWKV is unrolled as X(t) E Bixd where represents the time step index, rather than the entire sequence X E Txx, Similar self-attention, Spiking RWKV initiates by applying linear transformations: R XiI]Mn, K = MK, and V =X[] where Mn, Mr. My RdxH with H
Spiking RWKV.
5

---

Published in Transactions on Machine Learning Research (06/2024)
indicating the hidden size. Notably, all imputs to the three MLP layers are in the form of spiking activations. The subsequent step involves generating the output using the following equation:
Y 1 SN (R( c
exp(W,) (xp( @ (V[4) + A|D]
(7)
exp(W)) cxp B(t)
The hidden states A and B are defined as:
A[d] exp( (V()) + exp(Wa) o Alt
(8)
and
B|d] exp(K - 1]) exp(Wa) @ B[t -
99)
Here, SN refers to spiking neuron layers, the @ symbol represents element wise multiplication, Wa E Rixd and Wy E Rixd are deray vectors, where W responsible for weighting the current time-step's information, and Wa plays a role in decaying the influence of information from previous time-steps. Details of Wa and Wy can be found in Appendix C.4 Notably, Spiking RWKV diverges from self-attention by not employing matrix-matrix multiplication to dynamically adjust the attention map according to the input. Instead, it utilizes the learnable vectors Wa and W to recurrently blend the token dimensions of the input. While self-attention dynamically reweights token dimensions based on the input, Spiking RWKV adopts continuous decay strategy rather than a dynamic weighted attention strategy. Because the hidden states A[t] and B(t] encapsulate information from the previous time-step, this module can effectively blend information across token dimensions, much like self-attention. Additionally, this approach is particularly well-suited for language modeling. When using tention in casual language models, it's necessary to mask out half of the attention map to prevent information leakage. In contrast, Spiking RWKV combines the inherent recurrent properties of SNN with RWKV, and both naturally operate in unidirectional manner.
3.5 Spiking Receptance Feed-Forwer Networks (SRFFN)
Each block in our model contains fully con- nected feed-forward network with gating mech- anism (SRFFN), which is applied to normalized and token-shifted outputs of each Balking-RWWK mod- ule. This SRFEN module consists of three linear transformations follows:
Trport:
31
X
Impa:
Kg X
Label:
X5
SpikeCPT Model
Label proochhnn
Ingut
Xs
Y'M MM)
((1)))
Output
(10)
SpilecPPT Modil
where SW represents spiking neuron layers, [' de-
Poding
notes the output of SRFEN at time-step L. which
Output:
y5
y4.
is then passed to the spiking neuron (Eq. D
$ Drae Freray C
[Mp, Ma, Ms} E RdxH are learnable parameters of the linear transformations, and Activation represents the activation function. Depending on the specífic SpikeGPT variant being used, we employ either the ReLU SN activation function. SRFFN is vari- ant of the Gated Linear Unit (GLU) Dauphin et al.) 2017), which can control the degree of information flowing into the model by a (Mp.((t) In order to maintain consistency between SRFFN and GEGLU parameters (Shazeer) 2020), we set the size of H from the SRFFN to 4d. After the feedforward process in the SRFEN layer, the resulting output Y'[t] will be subsequently fed into LIF neuron, as depicted Eq. [ In this context, Y[t] replaced with the updated value Y'[ preserve the bloek-level spiking and binary characteristics, thereby ensuring the maintenance of the desired sparsity.
Tabat:
SyikeGaT genertion lesks.
SPI dirdeeiiitioon
Figure 2: Training SpikeGPT for NLG and NLU tasks.

---

Published in Transactions on Machine Learning Research (06/2024)
3.6 Training& & Inference
Our Training procedure consists of two stages. The first slage is pre-training on large-scale corpus to build high-capacity language model, and the next stage is specific fine-tuning to perform downstream tasks, such as natural language generation (NLG) and natural language undorstanding (NLU).
NLG Tasks. We adopt a decoder-only pre-training paradigm similar to GPT to train the model. Specifically, our model utilizes SRWKV and SRFFN modules to process the imput token sequence and generate an output distribution for each target token. Formally, given a token sequence C = {ci, 02, ,r,,, we use the standard language modeling objective to maximize the following likelihood:
P(a) = softmax(X"((WW))
(11)
Cp Σ logP(c/l,,..."' ;G-1)))))))))))))
(12)
where W is the token embedding matrix, and θ is the set of all model parameters.
After pre-training the model using the loss in Eq. 12 model parameters fine-tuned to adapt to different downstream tasks in NLG and NLU. For natural language generation tasks, we define new dataset Da. where each sample of data consists of a sequence of input tokens. The consistency between the pre-training process and the NLG task allows for the fine-tuning procedure to reuse the method and objectives adopted in the pre-training pipeline (Eq. 11 and Eq. 12 which maximizes the likelihood of the target token based on the previous information of the target position.
NLU Tasks. As for NLU tasks, such sentiment classification, the fine-tuning process requires several modifications to the top-level of the pre-trained SpikeGPT model to adapt to NLU tasks, as shown in Fig, 2 Specifically, given new dataset for the downstrcam task Du, where each instance consists of a token sequence C, and label 4, the following objective is maximized:
LNLU Σ 4 logP(C)
(13)
(c.d)
where P(C:) is defined as:
P(C.) = softmax(Y
(14)
WT. includes the learnable parameters of the MLP module in the top layer, and Y is generated by passing the input token sequence C, through the model and then average- soooggg-pooingggggo the embedding of each target, which is formalized as:
Y = AvgPooling [Xz[1], Xz Xx(TT)
(15)
In the inference phase, we directly give a prompt for the NLG lask, and let the model continue and calculate chaaaaer (BPC) and perplexity (PPL) as an evaluation metric. For the NLU task, we pass the inputs through our model to obtain the embedding over target positions, and then apply mean-pooling operation to all embedded tokens to predict the label of each instance.
3.7 Theoretical Energy Consumption Analysis
The key altribute of SNNs here their remarkable energy elficiency due dynamical sparsity. We provide theoretical energy consumption analysis of SpikeGPT's constituent components, while evaluating its feasibility for deployment on asynchronous hardware. The block-level theoretical energy consumption analysis is shown in Tab. M
Spiking RWKV. The SRWKV block comprises two integral components: the Spiking MLP layer, which receives sparse spike inputs, and the full-precision clement-wise product. Within the Spiking RWKV, akin self-attention mechanisms, the spike input denoted as X undergoes projection through MLP layers, resulting in three distinct variables, namely, R, K, and V. Notably, all these MT.P layers operate spike inputs,
7

---

Published in Transactions on Machine Learning Research (06/2024)
facilitating the transformation of matrix multiplication into sparse addition. It is worth highlighting that the non-binary element-wise product, as depicted in Tab. D constitutes relatively minor proportion compared to the MLP layers. Importantly, it is essential to note that all these operations can be efficiently supported by neuromorphic chip technology.
SRFFN. The SRFFN block primarily consists of ML.P layers and element-wise product operations. However, an alternative operation is introduced when employing the ReLU activation function instead of spiking neurons, While ReLU activations are not binary, they induce dynamical sparsity. We can further employ quantization techniques to convert them into integers, common practice in LLMs (Liu et al. 2023 Dettmers ct al. 2022b a) and SNNs (Venkatesh et al., 20244 Shen et al. 2023). where 8-bit activations and weights can be used with no noticeable loss in performance (Xiao et al. 2023). Moreover, we investigated the possibility of using LIF to replace Re in Sec. [.44 which demonstrates that the impact of this substitution on performance is negligible. With regard to integer sparse spikes, modern neuromorphic chips (Orchard et al. 2021 Parpart et al.] 2023) accommodate graded spikes that support up to 32-bit integer, rendering the entire SRFEN block compatible with neuromorphic hardware, requiring solely addition operations for its execution.
Table I: Energy Evaluation: FLMEP represents the Floating-Pori Operations (FLOPs) of the MLP layers in the ANNs, while R denotes the spike firing rates (indicating the proportion of non-zero elements in the spike matrix) within the spike matrices. In computing operation counts, models are parameterized with T = 3072, d 512, ft = 0.15, all derived from real test data of SpikeGPT-46M. Additionally, energy consumption is characterized by assuming EMAC 4.5pJ and EAC 0.9pJ based on Horowitz (2014) and Rathi & Roy 22021). Other than element-wise operation involving dot products in SpikeGPT, all other module inputs are in spike form, Icading to a significant reduction in operations by a factor of up to 32.2x overall. This is reflected in the Vanilla GPT-Lo-SpikeGPT (V/S) ratio.
Vanilla GPT (Radford et al.
2018
SpikeGPT
Energy Consumption (pJ) Vanilla GPT
Ratio V/S
(with GLU
Dauphin et al.
(This work)
SpikeGPT
Q/R, K.V /QQ/R,K,V K. Scale
EMAc-33 3
Eac Rr
1.09 x 1010
3.25 x 10
33.3x
EMAC- 2 d
EMAC MT4 7Td
8,50 x 10"
4.95 x 10?
1.71x
Attention
EMAC T
4.25 107
Softmax
EMAC- 21
8.50 x 107
Layer Layer 2
EMAC FLMIPI
EAC R. FLMMPI
3.62 10"
1.09 x 10
33.3x
FFN
EMAC. FLULPE
Esc R. FLMIPP
1.45 x 10
4.35 x 108
33.3x
Layer
EMAC FLMIPS
EAC A. FLMLPS
3.62 x 10
1.09 x 10a
33.3x
Overall
3.29 10
1.02 x 10
32.2x
4 Experiments
4.1
Experiment Settings
We test two variants of the 46 million parameter model; one where T = 1, 024 and another where T = 3, 072. We used the Enwik8 dataset to conduct both training and testing in 46M scale, and our most extensive model with 216 million parameters was trained using the OpenWeb'lex22 (Gao et al.) 2020) corpus for pre-training. In our 46M model, we employ a character-level tokenizer, as has been done in previous works Zhai et al. (202T). J. To evaluate the performance of our model, we calculate its BPC metrics. To mitigate the issue of overfitting, we incorporate dropout after the output of each SRFFN block and set the dropout ratio to 0.03. The 216M model with pre-training consists of 18 layers and feature dimension d = 768. We employ the Byte Pair Encoding (BPE) tokenizer and share the same prper-parameters GPT-NeoX (Black et al. 2022 Due to the availability of sufficient data for pre-training, we do not incorporate dropout as we did in our 46M model and remove the binary embedding but use the first layer neurons for encoding. To facilitate better convergence, we utilize a warmup technique during the first 500 training steps. For both the 46M and 216M models, we use the Adam optimizer, and set the learning rate to 6 10-4 and 4 x 10-4, respectively.
All experiments were conducted on four NVIDIA V100 graphic cards. For the models of 46M and 216M, we trained them for 12 and 48 hours respectively.
8

---

Published in Transactions on Machine Learning Research (06/2024)
We evaluated SpikeGPT on two major language-related tasks: Natural Language Generation (NLG) and Natural Language Understanding (NLU).. For NT.G, we evaluated the text generation performance of SpikeGPT on three classic text generation datasets: Enwik8 Mahonnyy] 2011), WikiTexx-2 (Merity et al. 2017), and WikiText-103 Meerity ct al.) 2017). These datasets are widely used to mcasure a model's ability to perform compression and for benchmarking. For NLU, we evaluated the performance of SpikeGPT on four classic text classification datasets: MR (Pang & Lee 2005), SST-5 (Socher et al. 2013). SST-2 (Socher et al. 2013) and Subj (Pang& & Lee, 2004). These datasets cover sentiment analysis and subjective/objeettie classification. Our implementation is based PyTorch (Paszke al.) 2019) and SpikingJelly (Fang et al., 2020). For detailed information regarding the datasets and baseline methods, please refer to Appendix
B Our zero-shot
text samples of this experiment can be found in Appendix
4.2 Results on Natural Language Generating Tasks
A summary of results are provided in Tabs. and This includes the BPC and PPL achieved on NLG tasks using SpikeGPT trained on Enwik8, WikiText-103, and WikiText-2 compared to several baselines, including 46M and 216M parameter variants.
Table 2: Enwik8 results. Measured in Bits Per Character (BPC): the lower the better. Baseline comparisons are made with Reformer (Kitaev et al.) 2020), Synthesizer (Tay 2020), Linear Transformer (Katharopoulos et al. 2020a Performer (Choromanski et al. 2020), Stacked LSTM (Graves] 2013) and SHA-LSTM (Merity 2019 L, d and T denote the number of blocks (network depth), dimension of features, and sequence length, respectively. Both Linear Transformer and Performer are implemented with customized CUDA kernels
(github. cot.iii//cmmdiia/faatttttt htddapp/Eaatttrranfffrressss transformers),
and all other models are implemented in native Pytorch.
Method
Spiking
L
T
Train BPC
Test BPC
Complexity
Params.
Transformer
12
512
1024
0.977
1.137
O(T² .d)
43.0M
Reformer
X
12
512
1024
1.040
1.195
O(TlogT d)
40.1M
Synthesizer
12
512
1024
0.994
1.298
OTT dd)
42.8M
Linear Transformer
12
512
1024
0.981
1.207
O(--d))
43.0M
Performer
12
512
1024
1.002
1.199
O(T dloodd
43.0M
Stacked LSTM
7
1.420
1.670
-
SHA-LSTM (no attention)
4
1024
1024
1.330
O(T-d)
SpikeGPT 46M
12
1024
1.113
1.283
O(T
46.1M
SpikeGPT 46M
12
512
3072
0.903
1.262
O(T
46.1M
As shown in Tab. the generative performance of SpikeGPT has far surpassed that of models with LSTM backbone, and can approach even surpass some simplified variants of the Transformer, such as Linear Transformer and Synthesizer. However, it should be pointed out that there is still a certain gap between SpikeGPT and the vanilla Transformer. While increasing the size of T significantly reduces the training BPC of SpikeGPT, its test BPC has not changed significantly, indicating that SpikeGPT potentially suffering from over-fitting
We also compared the Perplexity of SpikeGPT and GPT-2 based WikiText-2 and WikiText- 103 datasets text generation tasks. The results shown in Tab. 3 In the interest of fairly comparing models of similar scales, we selected GPT-2 small and GPT-2 medium RRafford et al. 2019 with parameter sizes similar to those of the fine-tuned 216M SpikeGPT. We found that after fine-tuning, the performance of SpikeGPT on WikiText-2 2 has surpassed that of GPT-2 series. Unfortunately, the performance of SpikeGPT on the larger WikiText-100 dataset has fallen behind the GPT-2 series models, which suggests a potential need for refined, and perhaps more sophisticated, training methodologies for SpikeGPT when dealing with larger scale corpora, e.g. knowledge distillation (Bal & Sengupta 2023

---

Published in Transactions on Machine Learning Research (06/2024)
Table 3: Results on WikiText-2 and WikiText-1033 measured in token-level perplexity. Lower values indicate better performance. We report the perplexity when the lowest value achieved the validation datasets. Note that for WikiText-2, use both the 1ikiTexx-1033 and WikiText-2 training sets to extend the training corpus.
WikiText-103
WikiTexd-2
Method
Parameters
Val.
Test.
Val.
Test.
GPT-2 Small
Radford et al. 2019
124M
29.41
37.50
GPT-2 Mediuum Radlord cl al. 2019
346M
26.37
22.76
SpikeGPT With Pre-training
216M
39.92
39.75
19.17
18.01
4.3 Results on Natural Language Understanding Tasks
For NL.U tasks, we utilize SpikeGPT as a dynamic embedding generator that constructs embeddings based on context. We compare SpikeGPT with text classification algorithms of similar scales, including LSTM (Hochreiter & Schmidhuber 1997), TexICNN (Kiml 2014), BERT (Kenton & Toutanova 2019), and the latest SNN-based text classification algorithm TextSCNN (Lv al. 2023b). The accuracy four datasets is shown in Tab. 4 The fine-tuned 216M SpikeGPT achieves the second-highest performance among the models, only surpassed by BERT. BERT is a bidirectional Transformer encoder that uses masked training to obtain a high-quality text embedding. However, unlike SpikeGPT, BERT does not have the capability to generate text directly. Our 46M model without any tne-tuning also achieves competitive results compared to the baseline models, indicating the potential of SpikeGPT NLU tasks. We also analyze the complexity of each method and show that SpikeGPT can achieve linear complexity by using spiking neurons and recurrent structure. Unlike TextSCNN, our model does not require additional temporal dimension for processing, it uses the sequence dimension iteratively during the forward-pass pass through the spiking neurons.
Table 4: Results of NLU tasks on four text classification datasets using both SNN and ANN methods. Measured classification accuracy: the higher the better. the 'complexity per layer' column, we compute the complexity of each method using the following notation: T is sequence length, d is the embedding dimension, K the convolution kernel size, and Taddational the additional temporal dimension for feed- forward processing. Our model combines recurrent and spiking features and does not need extra temporal dimension for feed-forward processing, as it exploits the inherent temporal dimension in the language model.
Method
Spiking
Recurrent
Complexity per layer
SST-2
SST-5
MR
Subj.
TexCCNN [ii
2014 BAINA
O(K- -d)
81.93
44.29
75.02
92.20
TextSONN Threet Trmining [Lx et
2023h 1CL8-2059
OlTatiiimal K.T.a
75.73
23.08
51.55
53.30
TexSSONN-ANN CNNNONNNNNNNNNNNNN Fine-tune LV
2023b
ICLR-2005
O(Tuttitonal K.T.
80.91
41.63
75.45
90.60
LSTM Tai al. 2015
x
O(T O(T?. d)
84.92
46.43
$1.60
BERT Kenton & Iontanmva 2019
x
91.73
53.21
86.72
SpikeGPT 46M
O(T d))
80.39
37.69
69.23
88.45
SpikeGPT 216M
O(T -4)
$2.45
38.91
65.1
89.10
SpikeGPT 216M Pre-training
O dd)
88.76
51.27
85.63
95.30
4.4 A Study of SpikeGPT and RWKV Variants
We conducted a study of variants of SpikeGPT and RWKV variants to evaluate how the binarization of activations impacts performance. The following models are tested:
1. Vanilla RWKV: The original RWKV model as reported in Peng et al. 2023
2. Heaviside RWKV: The above RWKV model where neuron activations are binarized.
3. SpikeGPT-B: The model above, where the second-MLP activation is swapped from ReLU2 to spiking neuron LIF,
4. SpikeGPT: The model described and proposed in this paper.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Results for those experiments are shown in Tab. 5 For the Heaviside RWKV variant, we applied the Hcaviside function directly to the RWKV layer for binarization, in lieu of the LIF neuron. Specifically, we employed the Heaviside function for the feed-forward process, and use the arctangent surrogate function to address the non-differentiable nature of the Hcaviside function during the backward pass, similar to the original SpikeGPT. For SpikeGPT-B, we substituted the middle activation function the SRFEN layer (as illustrated in Eq. 10 with the spiking neuron LIF introduce layer-lovel binary spiking.
The test BPC of SpikeGGT-B is marginally lower (and thus, better) than that of SpikeGPT. This shows that REFN layers tolerant to binarized activations, provided that recurrent dynamics included. In absence of recurrent dynamics, the Teaviside-RWKV experiment shows the worst performance, thus indicating that single-order recurrence is necessary to compensate for the performance degradation of binarized activations.
Table 5: An ablation study showcases the impact of different, architectural modifications the performance of SpikeGPT models, with all experiments conducted under the consistent settings of L 12, T 1024, and d = 512. The evaluation includes Vanilla RWKV as the baseline model without any modifications, Hcaviside RWKV, which replaces all spiking neurons with Heaviside functions, SpikeGPT-B featuring a variation where the middle activation in the SRFFN block is also a spiking neuron rather than ReLU2, and SpikeGPT, representing the default configuration of SpikeGPT.
Model
Vanilla RWKV Heaviside RWKV SpikeGPT-B
SpikeGPT
Train BPC
1.014
1.318
1.109
1.113
Test BPC
1.201
1.403
1.306
1.283
4.5 Scaling SpikeGPT
216M
430M
10
IB2
Loss
0.0
0.2
0.4
0.6
0.8
Number Tokens (Btokens)
Figure 3: Training Loss for Different Model Sizes on 0.93 Tokens.
Neural scaling laws posit that model error decreases power function of training set size and model size, and have given confidence in performance. Such projections become important as training becomes increasingly expensive with larger models. A widely adopted best practice in LLM training is to first test. scalability with smaller models, where scaling laws begin to take effect (Kaplan et al.) 2020 Hoffmann et al, 2022: Yang al.] 2023 Zhu al. 2024). The GPT-4 technical report revcaled that prediction model just. 1/10, 000 the size of the final model can still accurately forecast the full-sized model performance (Achiam) et al. 2023
11

---

Published in Transactions on Machine Learning Research (06/2024)
We evaluate the scaling capability of SpikeGPT by training models of three different sizes: 216M, 430M and 1B2, using the MiniPile (Kaddour 2023) dataset with 0.9B tokens. Due to the constrain on the computational resources, we cannot fully train the 430M and 1B2 model with OpenWeblexx22. As shown in Fig. ] the training loss decreases as the model size increases, demonstrating SpikeGPT's ability to scale effectively. The 216M model achieves a final training loss of 3.20, while the 430M model reaches loss of 3.00, and the 1B2 model attains the lowest loss of 2.88. These results suggest that SpikeGPT follows the neural scaling laws, with larger models exhibiting better performance when trained on the same dataset.
Furthermore, we observe that the rate of improvement in training loss slows down the model size increases, which is consistent with the diminishing returns predicted by scaling laws. The performance gain from 216M to 430M is more significant than the improvement from 430M to 1B2, indicating that the benefits of increasing model size gradually taper off. This insight can help guide decisions on model size selection, balancing performance gains with computational costs.
5 Conclusion
Our results remonstrate that event-driven spiking activations are not only capable of language generation, but they can do so with fewer high-cost operations. We develop techniques that promote lightweight models for the NLP community, and make large-scale models for the neuromorphic and SNN community more effective. We demonstrate how large SNNs can be trained a way that harnesses advances in transformers and our own serialized version of the attention mechanisms. We expect this research can open new directions for large-scale SNNs.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar preprinl arXiv:2303.0877 087774 2023.
Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. Carbontrackor: Tracking and predicting the carbon footprint of training deep learning models. orXiv preprint arXiv:2007.03051, 2020.
Malyaban Bal and Abhronil Sengupta. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. arXiv preprint arXiv:2308.1087. 2023.
Sami Barchid, José Mennesson, Jason Eshraghian, Chaabane Djéraba, and Mohammed Bennamoun. Spiking neural networks for frame-based and event-based single object localization. Neurocomputing, pp. 126805, 2023.
Sid Black, Stella Biderman, Fric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leaby, Kyle McDonell, Jason Phang, et al. Gpt-neox-22 20b: An open-source autoregressive language model. arXiv preprint Xrx::22004.00 06745, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877 1901, 2020.
Nicolas Bruncl and Mark CW Van Rossum. Lapicque's 1907 paper: From frogs to integratc-and-fire Biol. Cybern. 97(5))3337-33) 2007.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint (xXi:22000.47000 2020.
Loic Cordone, Benoît Miramond, and Philippe Thierion. Object detection with spiking neural networks on automotive event data. In International Joint. Conference Neural Networks (IJCNN), pp. 1-8, 2022.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference. on Machine Learning (ICML.) volume 70 of Proceedings of Machine Learning Research, pp. 933 941, 2017. URL http://proceedinns... .ir ppepss//70/dauphinl/la http:/ppoeaaagg...ppeee/////aaaa html
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82 99, 2018.
Haoyu Deng, Ruijie Zhu, Xuerui Qiu, Yule Duan, Malu Zhang, and Liang-Jian Deng. Tensor decomposition based attention module for spiking neural networks. Knowledge- Basee Systems, 295:111780, 2024.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems (NeurIPS), 35:30318-30333
2022a.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zeltlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208. 07933,,08.07399, 2022b.
Payal Dhar. The carbon impact of artificial intelligence. Nature Machine Intelligence, 2:423 5, 2020.
Peter U Diehl, Guido Zarrella, Andrew Cassidy, Bruno U Pedroni, and Emre Neftei. Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. In IEEE International Conference on Rebooting Computing (TCRC), pp. 1-8, 2016.
Jason K Eshraghian and Wei D Lu. The fine line between dead neurons and sparsity in binarized spiking neural networks. ar preprint orXiv:2201.11911 2022.
Jason K Eshraghian, Max Ward, Emre Neftei, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei D) Lu. Training spiking neural networks using lessons from deep 2021.
learning. Xiv preprint arXiv:2109.1188
Jason K Eshraghian, Xinxin Wang, and Wei D Tu. Memristor-based binarized spiking neural networks: Challenges and applications. IEEE Nanotechnology Magazine, 16(2):14-23, 2022.
Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Timothée Masquelier, Yonghong Tian, and other contributors. Spikingjelly. httggs//ggtao.cccom////eee1112// Accessed: 2022-05-21.
2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems (NeurIPS), 34: 21056-21066 2021.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.000 000000 2020.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5484-5495, 2021.
Alex Graves. Generating sequences with recurrent neural networks. arXie preprint orXiv:1308.00 0850, 2013.
Sepp Hochreiter and Jürgen Schmidluber. Long short-term memory. Neural Computation, 9(8):1735 1780, 1997.
Alan Hodgkin and Andrew F Huxlcy. A quantitative description of membrane current and its application conduction and excitation in nerve. The J. of Physiol., 117(4):500 544, 1952.
Jordan Hoffmann, Scbastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. ar Xie preprint orXim:2203.15556 2022.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Mark Horowitz. 1.1 computing's energy problem (and what do about it). In 2014 IEEE international solid-state circuils conference digest of technicol papers (ISSCC), pp. 10-14. TEEE, 2014.
Jean Kaddour. The minipile challenge for data-efficient language models. arXiv preprint arXim:2804.08444. 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888 2020.
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rmns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning (ICML), 2020a. URL https: al-iett-l-1imm20000---1 pdf
A Angelos Katharopoulos, A poorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Ma- chine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1156-5165, 2020b. URL http: http:://proceedings. mlr. ppeess//w1//kktaaroooolllol.3000 html
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HIT (NAACL), pp. 4171-4186, 2019.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: spiking neural network for energy-efficient object detection. In Proceedings of the AAAI conference on artificial intelligence (AAAT), volume 34, pp. 11270-11277 2020.
Yoon Kim. Convolutional neural networks for sentence classification. In Alessandro Moschitti, Bo Pang, and Walter Daclemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), pp. 1751, 2014. URL https://doi...... 1111..11111111111111111111
Nikita Kilaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th Interna-
tional Conference on Learning Representations (ICLR), 2020. URL attps::/oppenrevies..eee/
TkgNKKHtvB
Louis Lapique. Recherches quantitatives sur l'excitation electrique des nerfs traitec comme une polarization.
J. of Physiol. and Pathology, 9:620 635, 1907.
Yudong Li, Yunlin Lei, and Xu Yang. Spikeformer: A novel architecture for training high-performance low-latency spiking neural network. arXiv preprint orXiw:2211.10686 2022.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. ar Xie preprint arXin:2305.17888 2023.
Changze Lv, Tianlong Li, Jianhan Xu, Chenxi Gu, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Spikebert: A language spikformer learned from bert with knowledge distillation. 2023a.
Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. In The Eleventh International Conference on Learning Representations (ICLR), 2023b.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Newral Networks, 1059::16991161,, 1997.
Matt Mahoney. Large text compression benchmark, 2011.
Stephen Merity, Single headed attention RNN; stop thinking with your head. CoRR, abs/ Abs/1911.11122, 2019.
URL http://arxiv. org//bs......... abs//911 1911. 11423
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference Learning Representations (TCLR), 2017.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filippp Akopyan, Bryan I. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345(6197):668 673, 2014.
Shimul Kanti Nath, Sujan Kumar Das, Sanjoy Kumar Nandi, Chen Xi, Camilo Verbel Marquez, Armando Rúa, Mutsunori Uenuma, Zhongrui Wang, Songqing Zhang, Rui-Jie Zhu, et al. Optically tunable electrical oscillations in oxide-based memristors for neuromorphic computing. Advanced Materials, pp. 2400904, 2024.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- basedd optimization to spiking noural networks. IEEE Signal Processing Magazine, 36(6):51-63, 2019.
Garrick Orchard, E Paxon Frady, Daniel Ben Dayan Rubin, Sophia Sanborn, Sumit Bam Shrestha, Friedrich T Sommer, and Mike Davies. Efficient neuromorphic signal processing with loihi 2. In 2021 IEEE Workshop on Signal Processing Systems (SiPS), pp. 254-259. IEEE, 2021.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based minimum cuts. arXiv preprint cs cs/0409058, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075, 2005.
Gavin Parpart, Sumedh Risbud, Garrett Kenyon, and Yijing Watkins. Implementing and benchmarking the locally competitive algorithm the loihi 2 neuromorphic processor. In Proceedings of the 2023 International Conference on Neuromorphic Systems, pp. 1-6, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnss for the transformer era. arXiv preprint arXiv:2305.1000 19048, 2023.
Michael Pfciffer and Thomas Pfeil. Deep learning with spiking neurons: Opportunitics and challenges. Frontiers in Neuroscience, 12:774, 2018.
Xuerui Qiu, Rui-Jie Zhu, Yuhong Chou, Zhaorui Wang. Liang-jian Deng, and Guoqi Li. Gated attention coding for training high perffrrrfoee and efficient spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 601-610, 2024.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language understanding by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Newral Networks and Learning Systems, 2021.
Frank Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Rev., (6):3888,, 1958.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607 617, 2019.
Noam Shazeer. GLU variants improve transformer. CoRR, abs/ 22020.0002, 2020. URL https //arxivv
15

---

Published in Transactions on Machine Learning Research (06/2024)
Guobin Shen, Dongcheng Zhao, Tenglong Li, Jindong Li, and Yi Zeng. Is conventional smn really efficient? perspective from network quantization. arXiv preprint arXin:231 1. 101080, 2023.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1631-1642, 2013.
Pao- Pao-Sheng Vincent Sun, Alexander Titterton, Anjlee Gopiani, Tim Santos, Arindam Basu, Wei D Lu, and Jason K Eshraghian. Intelligence processing units accelcrate neuromorphic learning. arXie preprint arXiv:2211.1072) 2022.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint WrXiv:1503.000% 2015.
Y Tay, D Bahri, D Metzler, D Juan, Z Zhao, and C' Zheng. Synthesizer: rethinking self-attention in transformer models. Xivv preprint #rXi:22055000453 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (NeurIPS), pp. 5998-6008, 2017.
Sreyes Venkatesh, Razvan Marinescu, and Jason K Eshraghian. Squat: Stateful quantization-aware training in recurrent spiking neural networks. Newro-Inspire Computational Elements (NICE), 2024.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temppral backpropagation for training high-ppeffrmance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning (ICML), pp. 38087-38099. PMLR, 2023.
Rong Xiao, Yu Wan, Baosong Yang, Haibo Zhang, Huajin Tang, Derek F Wong, and Boxing Chen. Towards energy-preserving natural language understanding with spiking neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:439 447, 2022.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiw:2300.11 10305, 2023.
Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li. Temporal-wise attention spiking neural networks for event streams classification. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10221 10230, 2021.
Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE Transactions Pattern Analysis and Machine Intelligence, PP: 1-18, 01 2023.
Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An attention free transformer. arXio preprint arXiv:2105.14 2021.
Youhui Zhang, Peng Qu, Yu Ji, Weihao Zhang, Guangrong Gao, Guanrui Wang, Sen Song, Guoqi Li, Wenguang Chen, Weimin Zheng, et al. A system hicrarchy for brain-inspired computing. Nature 586 (7829):378-) 384, 378-384,, 2020.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng YAN, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In The Fleventh International Conference on Learning Representations (ICLR), 2023.
Rui-Jie Zhu, Qihang Zhao, Tianjing Zhang, Haoyu Deng, Yule Duan, Malu Zhang, and Liang-Jian Deng. Teja-snn: Temporal-channel joint attention for spiking neural networks. arXiv preprint arXiw:2206.1017 10177, 2022.
Rui-Jic Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, and Jason K Eshraghian. Scalable matmul-free language modeling. arXiv preprinl arXiv:2406.0255 02528, 2024.
17

---

Published in Transactions on Machine Learning Research (06/2024)
APPENDIX
A Details about LIF Neuron
A.1 Derivation and Analysis of the LIF Neuron
ANNs and SNNs can model similar network topologies, but SNNs use spiking neurons instead of artificial ones (Rosenblatt 1958). Spiking neurons, like artificial ones, work on a weighted sum of imputs. Instead of using a sigmoid or ReLU nonlinearity, this sum affects the neuron's membrane potential U().. When the potential exceeds threshold 0, the neuron spikes, sending signal to connected neurons. Most inputs are brief electrical spikes, unlikely to arrive simultancously, indicating temporal dynamics that sustain the membrane potential.
Louis Lapicque, in 1907, demonstrated that spiking neurons resemble low-pass filter circuit with a resistor I and capacitor C, known as the leaky integrate-and-free (LIF) neuron (Lapique 1907 Brunel & Van Rossum 2007p. Physiologically, the capacitance comes from the lipid bilayer membrane, and the resistance from ion channels (Hodgkin & Huxley 1952). This can be modeled as:
dU -U(( 4m))))))))))
(16)
di
where RC' is the time constant. For constant input current, the solution is:
U(t) = Iin R+ UUo Rle
(17)
This shows exponential relaxation of (t) to steady state. For a sequence-based neural network, the forward Euler method approximates the solution:
UI ( - (1 mm
(18)
Here, B = e-/r is the decay rate. In deep learning, the input weight is usually a learnable parameter. Simplifying, the input current coefficient becomes learnable weight W, leading to [n][(] W W[t.. Including spiking and reset, get:
Uld = - W Sout 1/0
(19)
Soullt E (0,1) 1} is the neuron's spike output. If Sout = 1, the reset term subtracts the threshold 6; otherwise, it has no effect. A spike is generated if:
Uf 0, otherwise
1,
Sont
(20)
A.2 Backpropagation through LIF Neuron
Here we describe how backpropagation can be applied to LIF neurons. At time-step f, H' and U represent the membrane potential after neuronal dynamics and after reset, respectively. The vectors U and U Useset indicate the threshold and reset. potential, respectively. The weighted inpuls from the preceding layer are the given by x w P. The output spike at time-step is represented by S = [st where st 81g We j-th neuron fires a spike, otherwise si = 0. The gradients propagating back from the next layer are as:
18

---

Published in Transactions on Machine Learning Research (06/2024)
recursively compute the gradients follows:
0L OL 0H
0L.
(21)
HH, HH OH,
OH;
0H 0H au;
(22)
aH; au; HH:
aL, ar, as;
(23)
HH; 05; OH;
Using Eq. 1
we obtain:
OH 1+1 = 1 B
(24)
0U;
OU;
Hi) Us,
S + (U
(25)
HH
reset
OH;
00;
O'H
-Ui))
(26)
0H
0H 3
(27)
0x;
Note that d= O when t 2 T, and U²_, = U reset We use the derivative of the surrogate function oxx) to as; define the derivative of the Heaviside function ((z). For more details, please refer to Interactive tutorial notebooks on s
B Datasets and Baselines
B.1 Datasets
We conducted experiments on two major types of tasks, Natural Language Gencration (NIG) and Natural Language Understanding (NLU).
For NLG tasks, chose the following 3 classic text classification datasets to evaluate the text generation performance of SpikeGPT: Enwik8 (Mahoney 2011), WikiText-2 (Merity et al. 2017) and WikiText- 103 Merity et al. 2017).
Enwik8. The Enwik8 dataset is a subset of the English Wikipedia XML dump from March 2006. It contains the first 100 million bytes of the dump and typically used measure a model's ability compress data. The dataset is based on the Hutter Prize, competition for lossless compression of human knowledge. We split the tokens into three subsets: 90% for training, 5% for validation, and 5% for testing.
Wik WikiText-2 is natural language dataset comprising collection of 2 million tokens derived from Wikipedia articles. This dataset is commonly utilized for benchmarking various natural language processing models.
WikiText-1003 The Wikitext-103 103 dataset is large collection of text extracted from Wikipedia articles that are verified as Good or Featured. It contains over 100 million tokens and covers wide range of topics and domains. The dataset is suitable for language modeling tasks that require long-term dependencies and rich vocabulary. The Wikitext-103 dataset is a larger and more diverse version of the Wikitext-2 dataset.
19

---

Published in Transactions on Machine Learning Research (06/2024)
For NLU tasks, chose the following 4 classic text classification datasets to evaluate the performance of our proposed SpikeGPT: MR (Pang & Lee 2005), SST-5 (Socher et. al. 2013), SST-2 (Socher et al. 2013). Subj. Pang & Lee 2004
MR (Pag & Lee 2005p. It consists of movie review files, labeled based on their overall sentiment polarity (positive or negative) or subjective rating.
SST-5. The Stanford Sentiment Tree Library 5 includes 11855 sentences extracted from movic reviows for sentiment classification (Socher et al.) 2013). There are different categories (very negative, negative, neutral, positive, and very positive)
SST-2 (Socher et al. 2013 Tt. is binary version of SST-5, with only two classes (positive and negative).
Subj (Pang Lee 2004 Classify sentences in the datasct as subjoctive or objoctive.
The sample sizes and text lengths of these datasets vary. If there is no standard training test segmentation, we will follow Lv et al. (003bb and randomly select 10% of the samples from the entire dataset as the test set.
B.2 Baselines
To verify the effectiveness NLG and NLU lasks of our proposed SpikeGPT, compare it with the following representative baselines:
For NLG, we list the baselines that we have selected as follows:
Stacked LSTM. A model architecture that stacks multiple LSTM modules together.
SHA-LSTM MMeriyy 2019). An LSTM model that follows by a single attention head layer.
Transformer Vaswani et al. 2017). Transformer is statc-of-the-art neural network architecture,
leveraging self-allention mechanisms to capture global dependencies of sequential data.
Reformer (Kitaev et al,, 2020). Reformer is an extensible variant of the Transformer model. By introducing the invertible sheaf and using the local sensitive hash mechanism, it solves the problem of low memory and computing efficiency of the traditional Transformer, and realizes efficient processing of long sequences.
Synthesizer TTay et al., 2020). Synthesizer is also variant of Transformer, which is a model that learns to synthesize attention weights without token-token interaction.
Linear Transformer (Katharopoulos et al., 2020b). Linear Transformer is lightweight variant. of Transformer that uses linear transformation layers to construct self attention mechanism.
Performer (Choromanski et al., 2020 variant of Transformer that does not depend on sparsity or low-rankness assumptions and could use linear complexity accurately estimate attention weights.
GPT-2 (Radford et al., 2019). GPT-2 is a transformer- based language model that specifically functions as a decoder. is an extensively trained, large-scale generative model using the autoregres- sive paradigm. To ensure compatibility with the parameter sizes SpikeGPT, we selected GPT-2 medium and GPT-2 small as suitable alternatives.
For NLU, the basclines we have selected are as follows:
LSTM (Hochreiter & Schmidhuber 1997). LSTM model is a type recurrent neural network with the ability to capture and utilize long-term dependencies in input sequences.
20

---

Published in Transactions on Machine Learning Research (06/2024)
TextCNN (Kim, 2014). TextCNN is convolutional neural network architecture specifically
designed for text classification tasks, leveraging convolutional layers to capture local patterns and features in textual data.
TextSCNN (Lv et al., 2023b) A variant of Tex model that combines spiking neural networks.
BERT (Kenton Toutanova 2019). BERT is a bidirectional language model based on the Transformer Encoder-only architecture and an auto-encoding training paradigm.
C Details of RWKV
C.1 Token Shift
Given imput. X, we perform loken shift operation as follows:
X, (eroPPadp.o....111
W shift
(
n//N i 1, E
(28)
x= W
X + Wahatt X,
where ZeroPad denotes the zero padding operation, Wahift represents learnable shift mask, E is the embedding size of each token, is the current block, and T is the total number of blocks.
C.2 General RWKV
Inspired by the Attention Free Transformer (Zhai et al. 2021). RWKV a replacement for self-attention. It reduces computational complexity by swapping matrix-matrix multiplication with convolution that sweeps along the time dimension. We subsequently modify this step to instead operate recurrently input data. This modification enables compatibility with recurrent SNNs, thus making it more manageable to run on limited resources.
Given an input shieneshifted embedding vector X, similar to self-attention, RWKV first applies a linear transform R = X'Mn, K = XMRx V M X' time-varying embedding (varying over the sequence), 3
and RG K, are also time-varying. Fig. 1 depicts the sequence unrolled into set of 2-D matrices.
MR. Mx and My consist of learnable parameters, where K and V can be likened to the key and value matrices of self-attention. R is referred to as the receptance matrix, where each element indicates the acceptance of past information.
Next, the following operation is applied:
E exp(W(rti-g) O exp(Ki) o V, Ei exp(e O exp(K,)
Y, = o(,) O
(29)
where @ is the element-wise product, T is the sequence length, is the non-linearity applied to R with the default being Sigmoid: W E RXXE is the positional weight decay matrix (represented vector unrolled over time in Fig. D. W encodes the sequential importance of given word subsequent words. It is not directly learnable, but it varies over time with learnable dynamics. Long-range dependence can be captured when the dynamics are learnt to decay slowly. The parallel version of RWKV is also shown in Fig.
Intuitively, as time increases, the vector Y, dependent on a longer history, represented by the summation of an increasing number of terms.
clips the top row and 1 zero-pads the bottom 3 3MM, MR, Mv} EXXH where H denotes hidden RWKV, H.
"The subscript [0,0, 0, 1,1] written with PyTohh syntax in mind, where
21

---

Published in Transactions on Machine Learning Research (06/2024)
For the target position t, RWKV performs weighted summation in the positional interval of [1, and takes the Hadamard product of the weighted result with the receptance a (Re). By taking the Sigmoid of Rs, the receptance acts as a 'frgget gate" by eliminating unnecessary historical information.
C.3 Self-Attention and RWKV
Distinct from the method of calculating the matching degrod between tokens by the self-attention mechanism, RWKV decomposes the calculation of matching degree into: 0g (R,) exp(Wr-++1) exp(Ky); where Oi E & is vector. Each element in aıj, that is Oijk, represents the matching degree the k-th position of the embedding of the i-th and j-th tokens. In other words, it can be seen as multi-headed RWKV with E' heads, each of which has a hidden size=1, which is similar to the multi-headed self-attention (MHA) mechanism.
To gain more comprehensive understanding of SpikeGPT, we conducted visualizations of the spike and membrane potential patterns in the Spiking RWKV layer and Spiking Receptance Feed-Forward Networks (SRFFN) layers (Fig. 1. These visualizations clearly reveal distinct differences between the Spiking RWKV and SRFFN layers, indicating diverse information representation patterns. Notably, the SRFEN layer exhibits higher firing rate, suggesting that it may retain more information similar to Transformer FFN layers (Geva eet al. 2021). It is worth noting that in various studies, outliers in ceresforme-bbsse language models have been shown to significantly impact performance, making them a crucial concorn in the quantification of large language models (Wei et al. 2022 ) Therefore, understanding and addressing the implications of outliers in SpikeGPT are of utmost importance, particularly in the context of optimizing its overall performance and reliability. However, due to the binary nature of SNNs, these outliers cannot be expressed in activation values in ANNs. Nevertheless, notable observation is the presence of prominent outliers in the membrane potential of individual neurons, many of which are negative. This finding suggests that SpikeGPT employs different approach to accommodate and preserve outliers.
C.4 Positional Weight Decay
The positional weight bias matrix W is determined by three matrices, Wa, Wa and Wy, parameterized as follows:
Wa ln(W,), W. Fx1
(30)
W. ((T++2) (-T+3) (-T+4) -1 EI Wy [In(ps) In(ps) In(px)] gExt
(31)
(32)
where W, a pre-calculated matrix dependent on the layer and size of E, the vector Wa contains the decay factors for each time-step, We is an indicator of time-step, and W is the initial decay factor to avoid the constant decay phenomenon of RNNs. Wa and W, are both learnable, and W. is a static, pre-calculated matrix based on training time-step. Ps is hyperparameter, which is set to 0.3 in this paper.
Visualization of Spike and Membrane Potential
To gain a more comprehensive understanding of SpikeGPT, we conducted visualizations of the spike and membrane potential patterns in the Spiking RWKV layer and Spiking Receptance Feed-Forward Networks (SRFFN) layers (Fig- 4 These visualizations clearly reveal distinct differences between the Spiking RWKV and SRFFN layers, indicating diverse information representation patterns. Notably, the SRFFN layer exhibits higher firing rate, suggesting that it may retain more information similar to Transformer FFN layers (Geva that in various studies, outliers in Transformer-basee language models have been shown to significantly impact performance, making them a crucial concern in the quantification of large
let al.
2021 It is worth noting
4 scalar self-attention, ag === Q.KT
22

---

Published in Transactions on Machine Learning Research (06/2024)
Sa dnng
Shiingg
Membern porerial of Spiing IWVKV
(h) Manbhrane potential SRIVN
3
Figure 5: demonstration of the Parallelized RWKV model adeptly handling computations in- volving e and c V through the application of large-kernel convolution operation. Notably, during the convolution's sliding window process, the model implements decay mechanism to ef- fectively manage temporal dependencies. For this demonstration, the sequence length set to = 4, and the embedding size is configured E 3.
(cl Spike outpuz of Spricing RWLVV layer
(6) Membrnnc peternzal of SRPPN laye:
Figure Visualization of spike and membrane potential of neurons. Figure (a) and (b) depict the membrane potential of the Spiking RWKV layer, while figure (c) and (d) display the spike patterns observed in the SRFFN layer, where each dot represents spike event.
language models (Wei et al. 2022 Therefore, understanding and addressing the implications of outliers in SpikeGPT are of utmost importance, particularly in the context of optimizing its overall performance and reliability. However, due to the binary nature SNNs, these outliers cannot be expressed in activation values as in ANNs. Nevertheless, a notable observation is the presence of prominent outliers in the membrane potential of individual neurons, many of which are negative. This finding suggests that SpikeGPT employs a different approach to accommodate and preserve outliers.
E Generation Examples
you? I an the AT. Who are you? I Jack.
Who are you?
Mho are you?
Generated Ansver
Generated Ansver 2
Any.
the in book.
The amazing thing is,
you can tell stories about what happened to ne.
Figure 6: Example of text generated by SpikeGPT. The model is trained on Ope2
23

---

Published in Transactions on Machine Learning Research (06/2024)
Context -
<start>
Generated Ansver "so what da you think ne? asked.
standing behind ne,
looking posters,
and suddenly knew what he was asking.
what you think of me?
looked up hin the corners of his lips pressed together. "I don't know."
looked at and all could do was watch him, him, listen
staring hin,
realizing he doing.
voice,
and then sould fall back to the stage and look at hin.
Figure Example of text generated by SpikeGPT, The model is trained on BookCorpus.
Context
In shocking rinding, acientiat diacovered a herd of dragona Living in a remote, previoualy unexplored valley, the that dragens spoke perfect Chineee.
Tibet. Even wore surprising to
Generated Ansver
extinct. They did condition of the tree. The discovery of a Chinese zoo was the latest study by archaeologiet Peter Herovitz, the discovery great example. researchere
separate case, they
earliest
pbysical
sho led researchers also exanined the significance of the artifacts of the bonas, the humancid skeleton, the skull, the call parts and the outer jaza. The apeciea is believed be located in the nountains and provide information about the the aurrounding
University Texas,
artifacta the island. The discovary shows that this large shich has a huge influx of the populations.
local island,
Figure Example of text generated by SpikeGPT. The model trained on OpenWebText2.
Pinky
The pink ghost's AI designed to "feel" opposite the red ghoet's behavior. Pinky actually atteupte to get front of Pac-Van. This accomplished by setting the target tiles ahoad of Pac-Man's current location in the diraction that Pac-Man in travelling. One exception to this ia whan Pac-Man is travaling up. Due to an equal the expected up offeet.
bug
code, calculation includes left
Gemerated Ansver
This vhere the AI is able to see how much the AI is still in notion. This is because the player can control player by noving
location of player in screen forvard towards the player. This is why the gane is so poverful when it ceses to setting the target up. means that player
predict
happening in the world, This allovs the player to see shere they are at and hau far the player is heading. The gane can also be used player.
exactly whore the player
track the location of the player, also track the location of enemies in the and detect targets. It also also monitor the location of onenies in the area, such as the locations of buildings, schoole, schools, or even buildings where player is player Dae of the nost interesting festures of the gane is the ability to control the direction of the player. This fly, or distance location. on. This allous players to see the diraction of the player, as vell as where they vant to go. The also supporta navigation, and can alao track the location of vehicles in the aky.
even track
location
addition to this, the gans
allovs players choose betveen
player they
eneny
Figure 9: Example of text generated by SpikeGPT. The model is trained on OpenWebText2..
24

---

tasted Published in Transactions on Machine Learning Research (06/2024)
Contort
Aaron loves nint chocolata cake, but be requires that it be paired with mini chocolata chipa, so I thras some had few Peppermint Jos hand
those betveen the layers. sone of thoge in along with some crushed neringue cookies because, shy not? It's a total enorgasbord of miaty how
then up
chocolate chippy cookie crunchy goodness. didn't the finished product, wieh had added bark, etc. And don't be afreid to use a heavy hand. Texture good.
topping used, You add anything you want- crushed candy canes, peppermint
Pour out of the cupcako, cut out some vanilla butter, then add sugar and vanilla. Reneve from heat until thick and anooth, then place into the pan ninutea until golden brown. Turn on and off, stirring until the cake's browned, if it were black chocolate and orange eyrup.
Figure 10: Example of text generated by SpikeGPT. The model trained on OpenWebText2.
Context
Corporal Michael F. Goeldin vas an unskilled laborer fron Ireland when enlisted in Company in Novenber 1860. Goldein survived the Corporal Patrick D'Nlell, also from Ireland, first enlisted in 1854 and served with Coapany L, 3d U.S. Artillery, in Oregon. He returned to the Bast Coast and enlisted in the company in 1860. o'Neal served until 1874, shen Corporal Benjanin Bromne was a shoerazer from Orange County, Nle York. In Auguat 1862, he enliated in the kas
named superintendant of National Conatery Willets Point, New York.
oix nen traneferred Broune reenlisted in the company in February 1864 while it was canped at Brandy Station. Be returned to civilian life after completing his ealistment in 1867. Dn June, Artificer Willian Collina was promoted to corporal, probably fill conbat lesderchip void for the crossing of the Janes River. Collins's service record does not reflect the quilities he denonstrated to carn this promotion, but had abvicusly overcone some serious problens. age teanty-two, and
nexly forned 124th New
Volunteers,
Company
Born Sacketta Harber, Nau Collins enlisted in the company Dacenber 1853 reenlisted in December 1858. Just a nonth before the war began in April 1861, Collins went "over the hill" and company 22 1864,
was caught
years
Returned
Generated Ansver
tried possession of marijuana atteapted firet degres nurder. For nore information on Collins on Tuitter visit wwv. faceboo.......o cawcok..cc//ooeo/o con/IoeCollin: April 1879, Collins vas sentenced to prison for tuo counts degres first degres marder, second degree second degree second dagres murder,
degree murder: first degrea
Figure 11: Example of text generated by SpikeGPT. The model trained OpenWebTex12.
Context -+
incradible array prahistoric heasts on the rough nodern France 36, 36,000 years ago. Now, with the help cutting-edge technology, those works art in the Chauvet-Pont- Ponte---Pot---eeee Are Cave have been reproduced create the biggest replica cave in the world. The nannade naned the Caverne Pent-d'Arc has been built few niles from the original site in Vallon-Pont-D'se in Southern France and contains 1,000 patnstakingly-reprrott features... Cavenen a cave 36, 16,000 years ago and now replica has been created (pictured) that shou bow beautiful the cave vas in the 70a and 8Da
Prahistoric
valls
dravings as well as around 450 bonas and other rough linestone walls
aketched incredible array prebistoric beasts
Scroll
Grouth of a rare cave on a cliff near the river level near the Great Wall of Barcelona (pictured right): The sorld's stona cave Italy has discovered part hacause ancient cave arosion. has bacoze In an opan latter dated 19 Juna, archaeclogista fron the Univernity of Barcelena say they found the excavation and excavation the aite 'significant'.
part nodern archasology to deternine how the pyranids lived before the Stone Age.
Figure 12: Example of text generated by SpikeGPT. The model trained on OpenWebText2.
25


Reviews:
[{"review_id": "NL8s3UMnTL", "reviewer": "Reviewer_KfdC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n- enabling efficient training and scaling of SNN is a topic of clear interest to the audience of this journal\n- first demonstration of adapting the RWKV architecture for SNN training\n- comparable results (typically moderately behind) in terms of Bits Per Characters, perplexity, and accuracy against non-spiking NN of similar size on a selection of downstream tasks\n- presented training and inference workflows for both NLG and NLU tasks, adapted for SNN\n- theoretical analysis of energy consumption complements well the aforementioned results, and, as expected, is highly favorable towards SNN (estimate of 32.2x vs. comparable non-spiking NN)\n\nWEAKNESSES:\n- what drove the architectural choice of settling on a 46M and a 216M variants? Also, I may have missed it but I couldn't find indicated the number of layers and hidden size dimension for the 216M model\n- Sec 3.7 mentions the option to \"further employ quantization techniques to convert them [the activations] to integer\" but it is not clear whether this is actually done in this study, and what implementation is considered by the energy consumption analysis. It should be noted that the references cited here all refer to non-spiking NN works\n- Sec 4.4 \"Ablation studies\" can be confusing. It compares two SNN variants to a non-spiking RWKV and an unmodified spiking RWKV. The latter is called SpikeGPT-R in this section only, so it wasn't clear this was the configuration used in the rest of the paper\n- Sec 4.4 also states \"Notably, SpikeGPT-S exhibits superior training loss compared to SpikeGPT-R, indicating that\nspiking neurons can enhance model performance in certain cases.\" when comparing network with LIF in SRFFN (SpikeGPT-S) instead of ReLU2 (SpikeGPT-R) but training performance are at best comparable (1.109 vs. 1.113), especially as no dispersion over multiple seeds is reported, and test performance of SpikeGPT-S are inferior. So this statement appears overblown. Also calling this an \"extensive ablation\" study is not appropriate.\n- The whole Sec 5 \"Discussion\" reads as an appendix and the authors may want to consider moving it there:\n    - Sec 5.1 is superficial, just noting that the membrane potential appears different in SRWKV and SRFNN layers, and that it results in SRFNN having higher firing rate. It's an interesting observation but it isn't really discussed.\n    - Sec 5.1 is also repeated verbatim in its entirety with appendix B.3 \n    - Sec 5.2 consists of one paragraph discussing parallelization efforts of RWKV. Again, interesting (although very limited in scope) but not a discussion.\n\nOTHER MINOR OBSERVATIONS:\n- Fig 3 presents SRFNN to the left, followed by SRWKV on the right. It's slightly misleading as the SRFNN blocks follows SRWKV in the SpikeGPT architecture\n- Sec 5.2 refers to equation 18 which is not part of the main text but is in an appendix\n- typo in Sec 3.3: \"binary spikes, This allows\" -> \"binary spikes. This allows\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents efficient training of Spike Neural Network (SNN)-based transformers on natural language tasks, obtained by adapting the RWKV architecture to SNN. This is achieved primarily by replacing the multi-head self-attention module with a Spiking RWKV module, which achieves linear complexity with sequence length. They perform autoregressive training on sizeable SSNs (up to 216M parameters) demonstrating reasonable performance against non-spiking networks of similar size on some Natural Language Generation and Natural Language Understanding tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested clarifications and polishing as per \"Weaknesses\". Consider expanding Sec 5 or moving it to the appendix.", "broader_impact_concerns": "Not addressed in the manuscript. No concerns on my end.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xGw5EEi2xg", "reviewer": "Reviewer_odWR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The paper successfully integrates spiking neural networks with the transformer model architecture, a novel approach in the field of natural language processing. This integration allows the model to leverage the energy efficiency of SNNs while handling complex language generation tasks.\n\n- By modifying the transformer architecture to replace multi-head self-attention with a mechanism that supports streaming inputs, the authors reduce the computational complexity from quadratic to linear, which is important for scaling up models both in terms of size and efficiency.\n\n- The paper demonstrates that SpikeGPT can achieve substantial reductions in energy consumption on neuromorphic hardware. In the meanwhile,  SpikeGPT remains competitive with conventional non-spiking models on tested benchmarks compared with transformer models.\n\n**Weaknesses**:\n- Even though the paper presents a successful implementation, training SNNs is inherently complex due to their non-differentiable nature and the use of surrogate gradients. This complexity might limit wider adoption or replicability by other researchers without substantial expertise in neuromorphic computing.\n\n- While the model scales up to 216 million parameters, the challenges associated with scaling SNNs even further (to the size of the largest ANNs in use today, e.g., billions of parameters with transformer models) are not thoroughly discussed. Potential bottlenecks and computational constraints could be significant as the model size increases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduced SpikeGPT, an energy-efficient generative language model employing Spiking Neural Networks (SNNs). This model integrates modifications to the traditional transformer architecture, reducing computational complexity from quadratic to linear by adapting a streaming input mechanism suitable for SNNs. The authors applied innovative techniques tailored for SNN training, including aligning the language sequence with SNN temporal dynamics, autoregressive training that accumulates spikes, and utilizing stateful neurons to overcome binary activation constraints. Experimental results show that SpikeGPT can achieve competitive performance against similar-size transformer models with less energy consumption.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- To mitigate the steep learning curve associated with training SNNs, the authors could provide detailed documentation, tutorials, and open-source code. This would include clear explanations of the surrogate gradient approach and its implementation. Such resources can help demystify the process, making it more accessible to researchers without a deep background in neuromorphic computing.\n\n- The paper could include a more thorough analysis of the scalability challenges encountered when increasing the size of SNNs. This should cover both theoretical and practical constraints, such as memory usage, processing speed, and the efficiency of surrogate gradients at scale.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1nXVGbTDPC", "reviewer": "Reviewer_zsxD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\nThis paper is overall well written, well motivated, and technically sound. Being the first paper to achieve GPT-level model on SNN and achieving competitive benchmark results, the paper would have significant impact to the field of SNN and general efficienct deep learning algorithms and systems. Overall a very solid work.\n\n## Weakness\n\nThe paper can be further improved by discussing the following topic:\n1. As the first paper to present a SNN implementation of GPT, this paper would have a broad impact in the TMLR audiance. However, this paper lacks discussion on fundamental related work of SNN and the discussion of basic SNN operation. Adding a subsection in related work and add a few diagrams/equations to explain SNN operation will make the paper more accessible to broader audiences.\n2. It would be interesting to see if the proposed SpikeGPT can benefit from migrating the weights of pretrained non-spiking LLM or distill from a pretrained model, rather than doing the full pretraining from scratch", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes SpikeGPT, the first GPT implementation with the Spiking Neural Network architecture. To achieve its goal, the paper resolves the issue of language encoding for SNN, training large-scale SNN, and the incompatible of SNN with self-attention. SpikeGPT leverage the architecture of the RWKV language model, with novel techniques resolving the encoding and training issues. The final outcome leads to an implementation of a 46M and a 216M model, achieving competitive benchmark results with non-spiking models while maintaining 32.2x fewer operations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness part.", "broader_impact_concerns": "No concern on broader impact", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers unanimously recommend the acceptance of the manuscript in its present form, as (i) all claims are convincingly supported by appropriate experiments, and (ii) the overall contribution (the “first demonstration of spiking neural networks” applied to language modeling) is significant and of interest to the TMLR audience.
Decision: Accept (poster)