Title: SpikeBERT: A Language Spikformer Trained with Two-Stage Knowledge Distillation from BERT
Venue: NeurIPS 2023
Accepted: no
Keywords: Spiking neural networks

Text classification

Knowledge distillation
Abstract: Spiking neural networks (SNNs) offer a promising avenue to implement deep neural networks in a more energy-efficient way.
However, the network architectures of existing SNNs for language tasks are too simplistic, and deep architectures have not been fully explored, resulting in a significant performance gap compared to mainstream transformer-based networks such as BERT.
To this end, we improve a recently-proposed spiking transformer (i.e., Spikformer) to make it possible to process language tasks and propose a two-stage knowledge distillation method for training it, which combines pre-training by distilling knowledge from BERT with a large collection of unlabelled texts and fine-tuning with task-specific instances via knowledge distillation again from the BERT fine-tuned on the same training examples.
Through extensive experimentation, we show that the models trained with our method, named SpikeBERT, outperform state-of-the-art SNNs and even achieve comparable results to BERTs on text classification tasks for both English and Chinese with much less energy consumption.

1

Abstract

Spiking neural networks (SNNs) offer a promising avenue to implement deep neural networks in a more energy-efficient way. However, the network architectures of existing SNNs for language tasks are too simplistic, and deep architectures have not been fully explored, resulting in a significant performance gap compared to mainstream transformer-based networks such as BERT. To this end, we improve a recently-proposed spiking transformer (i.e., Spikformer) to make it possible to process language tasks and propose a two-stage knowledge distillation method for training it, which combines pre-training by distilling knowledge from BERT with a large collection of unlabelled texts and fine-tuning with task-specific instances via knowledge distillation again from the BERT fine-tuned on the same training examples. Through extensive experimentation, we show that the models trained with our method, named SpikeBERT, outperform state-of-the-art SNNs and even achieve comparable results to BERTs on text classification tasks for both English and Chinese with much less energy consumption.

1 Introduction

Modern artificial neural networks (ANNs) have been highly successful for a wide range of natural language processing (NLP) and computer vision (CV) tasks. However, it requires too much computational power and energy to train and deploy state-of-the-art ANN models, leading to a consistent increase of energy consumption per model over the past decade. The energy consumption of large language models, such as ChatGPT[OpenAI, 2022] and GPT-4[OpenAI, 2023], is unfathomable even during inference. In recent years, spiking neural networks (SNNs), arguably known as the third generation of neural network [Maas, 1997], have attracted a lot of attention due to their high biological plausibility, event-driven property and low energy consumption [Roy et al., 2019]. Like biological neurons, SNNs use discrete spikes to process and transmit information. Nowadays, neuromorphic hardware can be used to fulfill spike-based computing, which provides a promising way to implement artificial intelligence with much lower energy consumption.

Spiking neural networks have achieved great success in image classification task [Hu et al., 2018, Yin 27

et al., 2020, Fang et al., 2021, Ding et al., 2021, Kim et al., 2022a, Zhou et al., 2022] and there have 28

been some works [Plank et al., 2021, Lv et al., 2023, Zhu et al., 2023] that have demonstrated the 29

efficacy of SNNs in language tasks partially. However, the backbone networks employed in SNNs for 30

language tasks are overly simplistic, which significantly lowers the upper bound on the performance 31

of their SNN models. For instance, the SNN used by Lv et al. [2023], which is built upon TextCNN 32

[Kim, 2014], demonstrates a notable performance gap compared to those built on Transfomer-based 33

(a)

Spikformer

SPS

Conv 2D

BN

MP

Images

D-SSA

BN

Conv 2D

Encoder Block

×𝐿

×2

MLP

Words

Word

Embedding

(b)

SpikeBERT

N-SSA

LN

Linear

Encoder Block

×𝐿 ′

×2

MLP

Spiking Neuron Layer

Element-wise Add

BN

LN

Layer Normalization

SSA

SPS

Spiking Patch Splitting

Spiking Self Attention

MP

[Vaswani et al., 2017] large language models like BERT [Devlin et al., 2019] and RoBERTa [Liu 34 et al., 2019] on multiple classification benchmarks. 35

Recently, Spikformer was proposed by Zhou et al. [2022], which first introduced Transformer 36

architecture to SNNs and significantly narrowed the gap between SNNs and ViT [Dosovitskiy et al., 37

2020] on ImageNet [Deng et al., 2009] and CIFAR-10. We think that Spikformer provides the 38

possibility to construct complex language representation models. As shown in Figure 1, considering 39

the discrete nature of textual data, we improve the architecture of Spikformer to make it suitable 40

for language tasks. we replace certain modules that were originally designed for image processing 41

with language-friendly modules. Please see Section 3.2 for details on the improvement in network 42

architecture. In general, a deeper ANN model often implies better performance. Increasing the 43

depth of a ANN allows for the extraction of more complex and abstract features from the input data. 44

However, Fang et al. [2020a] have shown that deep SNNs directly trained with backpropagation 45

through time (BPTT) [Werbos, 1990] using surrogate gradients (See Section2.1) could suffer from 46

the problem of gradient vanishing or exploding due to 'self-accumulating dynamics'. Therefore, we 47

proposed to use knowledge distillation [Hinton et al., 2015] to train language Spikformers so that the 48

deviation of surrogate gradients in Spikformer would not be rapidly accumulated[Qiu et al., 2023]. 49

Inspired by the widely-used 'pre-training + fine-tuning' recipe [Sun et al., 2019, Liu, 2019, Gururan50

gan et al., 2020], we present a two-stage knowledge distillation strategy. In stage 1 , we choose BERT 51

as teacher model and the improved Spikformer as student model. We utilize a large collection of 52

unlabelled texts to align features produced by two models in the embedding layer and multiple hidden 53

layers. In stage 2 , we use a BERT fine-tuned on a task-specific dataset as teacher and the model 54

that completes stage 1 as student. At this stage, we first do the data augmentation for task-specific 55

dataset and then employ the logits predicted by the teacher model to further guide the student model. 56

After two-stage knowledge distillation, a spiking language model, named SpikeBERT, can be built 57

by distilling knowledge from BERT. The experiment results show that SpikeBERT not only can 58

outperform the state-of-the-art SNNs-like frameworks in text classification task but also achieve 59

competitive performance to BERTs. The experiments of the ablation study (Section 4.5) also show 60

that 'pre-training distillation' plays an important role in training SpikeBERT. 61

The major contribution of this study can be summarized as follows: 62

63

64

65

66

67

We improve the architecture of Spikformer for language processing and propose a two-stage, 'pre-training + task-specific' knowledge distillation training method, in which the improved Spikformers are pre-trained on a huge collection of unlabelled texts before they are further fine-tuned on task-specific datasets by distilling the knowledge of feature extractions and predictive powers from BERTs.

68

69

We empirically show that SpikeBERT achieved significantly higher performance than existing SNNs on 6 different language benchmark datasets for both English and Chinese.

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

This study is among the first to show the feasibility of transferring the knowledge of BERTlike large language models to spiking-based architectures that can achieve comparable results but with much less energy consumption.

2 Related Work

2.1 Spiking Neural Networks

SNNs use discrete spike trains instead of continuous decimal values to compute and transmit information. Spiking neurons, such as Izhikevich neuron [Izhikevich, 2003] and Leaky Integrate-and-Fire (LIF) neuron [Wu et al., 2017], are usually applied to generate spike trains. However, due to the non-differentiability of spikes, training SNNs has been a great challenge for the past two decades. Currently, there are two mainstream approaches to address this problem.

ANN-to-SNN Conversion ANN-to-SNN conversion method [Diehl et al., 2015, Cao et al., 2015, Rueckauer et al., 2017, Hu et al., 2018] aims to convert weights of a well-trained ANN to its SNN counterpart by replacing the activation function with spiking neuron layers and adding scaling rules such as weight normalization [Diehl et al., 2016] and threshold constraints [Hu et al., 2018]. This approach suffers from a large number of time steps during the conversion.

Backpropagation with Surrogate Gradients Another popular approach is to utilize surrogate gradients [Neftci et al., 2019] during error backpropagation, enabling the entire procedure to be differentiable. Multiple surrogate gradients functions have been proposed, including the Sigmoid surrogate function [Zenke and Ganguli, 2017], Fast-Sigmoid [Zheng and Mazumder, 2018], and ATan [Fang et al., 2020a]. Backpropagation through time (BPTT) [Werbos, 1990] is one of the most popular methods for directly training SNNs[Shrestha and Orchard, 2018, Kang et al., 2022], which applies the traditional backpropagation algorithm [LeCun et al., 1989] to the unrolled computational graph. In recent years, several BPTT-like training strategies have been proposed, including SpatioTemporal Backpropagation (STBP) [Wu et al., 2017], STBP with Temporal Dependent Batch Normalization (STBP-tdBN) [Zheng et al., 2020], and Spatio-Temporal Dropout Backpropagation (STDB) [Rathi et al., 2020]. These strategies have demonstrated high performance under specific settings. For more detailed information about Backpropagation Through Time (BPTT), please refer to Appendix A.

2.2 Knowledge Distillation

Hinton et al. [2015] proposed the concept of knowledge distillation by utilizing the 'responsebased' knowledge (i.e., soft labels) of the teacher model to transfer knowledge. However, when this concept was first proposed, the features captured in the hidden layers were neglected, as they only focused on the final probability distribution at that time. To better learn from teacher models, some works [Zagoruyko and Komodakis, 2016, Heo et al., 2019, Chen et al., 2021] have advocated for incorporating hidden feature alignment during the distillation process. In addition, relation-based knowledge distillation has been introduced by Park et al. [2019], demonstrating that the interrelations between training data examples were also essential.

Recently, there have been a few studies [Kushawaha et al., 2020, Takuya et al., 2021, Qiu et al., 2023] in which knowledge distillation approaches were introduced to train SNNs. However, most of them focused on image classification task only, which cannot be trivially applied to language tasks. In this study, we propose a two-stage knowledge distillation approach to train the proposed SpikeBERT for text classification tasks, which is among the first ones to show the feasibility of transferring the knowledge to SNNs from large language models.

3 Method

In this section, we describe how we improve the architecture of Spikformer and introduce our twostage distillation approach for training SpikeBERT. Firstly, we will depict how spiking neurons and surrogate gradients work in spiking neural networks. Then we will show the simple but effective

modification of Spikformer to enable it to represent text information. Lastly, we will illustrate 116 'pre-training + task-specific' distillation in detail. 117

118

119

120

121

122

3.1 Spiking Neurons and Surrogate Gradients

Leaky integrate-and-fire (LIF) neuron [Wu et al., 2017] is one of the most widely used spiking neurons. Similar to the traditional activation function such as ReLU, LIF neurons operate on a weighted sum of inputs, which contributes to the membrane potential U t of the neuron at time step t . If membrane potential of the neuron reaches a threshold U thr , a spike S t will be generated:

We can regard the dynamics of the neuron's membrane potential as a resistor-capacitor circuit [Maas, 123 1997]. The approximate solution to the differential equation of this circuit can be represented as 124 follows: 125

where X t are inputs to the LIF neuron at time step t , W is a set of learnable weights used to integrate 126 different inputs, I t is the weighted sum of inputs, β is the decay rate of membrane potential, and 127 U t -1 is the membrane potential at time t -1 . The last term of S t -1 U thr is introduced to model the 128 spiking and membrane potential reset mechanism. 129

130

131

132

133

134

135

136

137

138

139

140

141

where α defaults to 2 .

3.2 SpikeBERT Architecture

Spikformer [Zhou et al., 2022] is the first hardware-friendly Transformer-based spiking neural network, whose architecture is shown in Figure 1 (a). The most crucial module is the Spiking Self Attention (SSA), which utilizes discrete spikes to implement the self-attention mechanism without employing a softmax function:

where S is Heaviside step function like Equation 1, X s ∈ R T × L × D is the input of SSA, T is number of time steps, BN is batch normalization, τ is a scaling factor. Outputs of SSA and Q s , K s , V s are all matrix containing 0 and 1 . W Q s , W K s , W V s and MLP are all learnable decimal parameters.

We modify Spikformer so that it can effectively process textual data. Firstly, we replace Spiking 142 Patch Splitting (SPS) module with a word embedding layer and a spiking neuron layer so that it can 143 process sentences. Meanwhile, we find that the shape of the attention map in vanilla Spikformer is 144 D × D where D is the dimensionality of the hidden layers, which is unreasonable in language tasks. 145 For language tasks, the features shared with words in different positions by attention mechanism 146 are more important than those in different dimensions. Therefore, we reshape the attention map in 147 Spiking Self Attention (SSA) module to N ∗ N where N is the length of inputs. Lastly, we use linear 148 layers and layer normalization (LN) instead of convolution layers and batch normalization(BN). We 149 show the architecture of SpikeBERT in Figure 1 (b). 150

In addition, we follow Fang et al. [2020b] and use Arctangent-like surrogate gradients function, which regards the Heaviside step function (Equation 1) as:

Therefore, the gradients of S in Equation 3 are:

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

pos:87% neg:13%

pos:100% neg: 0%

Stage 2

Logits Loss:

𝐿𝑙𝑜𝑔𝑖𝑡𝑠

Cross-entropy Loss:

𝐿𝑐𝑒

We like this movie!

…

T

Feature Alignment Loss:

𝐿 𝑓𝑒𝑎

𝑖

Embedding Alignment Loss:

𝐿𝑒𝑚𝑏

…

T

Feature Alignment Loss:

𝐿 𝑓𝑒𝑎

1

We like this movie!

…

…

Teacher: BERT

Student: SpikeBERT

Backpropagation

Feature Transformation

Loss

Optional Loss

BERT Feedforward

Transformed Features of Student

Embeddings of Student

Features of Teacher

Embeddings of Teacher

Spiking Neuron Layer

Spiking Features of Student

pos:64% neg:36%

Stage 1

Two-stage Knowledge Distillation

For SpikeBERT

3.3 Two-stage Distillation

Two-stage distillation is the key to enabling the student model with language processing ability. The first stage is to align the embeddings and hidden features between BERT and the improved Spikformer using a large-scale corpus. The second stage is to distill logits and cross-entropy information on a task-specific dataset from a fine-tuned BERT to the model finishing stage 1. We show the overview of our method in Figure 2.

3.3.1 Stage 1. Pre-training Distillation

Given a pre-trained BERT [Devlin et al., 2019] irrelevant to downstream tasks as teacher TM and an improved Spikformer as student SM , our goal in this stage is to align the embeddings and hidden features of TM and SM with a collection of unlabelled texts. We will introduce embedding alignment loss and feature alignment loss in the following.

Feature Alignment Loss This loss L fea is to measure the similarity of features between TM and SM at every hidden layer. However, the shape of the student model's feature F sm at every layer is T × N × D but that of BERT's feature F tm is N × D , where T is the number of time steps, D is the dimensionality of hidden layers and L is sample length. What's more, F sm is a matrix only containing 0 and 1 but F tm is a decimal matrix. To address the issue of different dimensions between F tm and F sm , as well as the disparity between continuous features of TM and discrete features of SM , a transformation strategy is necessary. We follow the feature transformation approaches of Heo et al. [2019], Chen et al. [2021], Qiu et al. [2023] to map the features of TM and SM to the same content space:

However, we find it hard to align the features generated by the student model with those generated by 171 BERT for the first few layers in this stage. We think that's because the student model might require 172 more network layers to capture the essential features via the interaction among the inputs. As shown 173 in Figure 2, we choose to ignore some front layers when calculating feature alignment loss. Assume 174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

BERT contains B Transformer blocks (i.e., B layers) and assume the student model contains M Spike Transformer Block. Therefore, we will align features every ⌈ B M ⌉ layers if B > M . For layer i in student model, its feature alignment loss is L i fea = || F ′ tm -F ′ sm || 2 .

Embedding Alignment Loss As discussed in Section 3.2, the embeddings of the input sentences are not in the form of spikes until they are fed forward into the Heaviside step function. Define E tm and E sm as the embeddings of teacher and student, respectively so the feature alignment loss is L i fea = || E tm -MLP( E sm ) || 2 . The MLP layer is a transformation playing a similar role as that in Equation 6.

To sum up, in stage 1 , the total loss L 1 is the sum of chosen layer's feature alignment loss:

where the hyperparameters σ 1 and σ 2 are used to balance the learning of embeddings and features.

3.3.2 Stage 2. Task-specific Distillation

In stage 2 , we take a BERT fine-tuned on a task-specific dataset as the teacher model, and the model completed stage 1 as the student. To accomplish a certain language task, there should be a task-specific head over the basic language model as shown in Figure 2. For example, it is necessary to add an MLP layer over BERT for text classification. Besides, data augmentation is a commonly used and highly effective technique in knowledge distillation[Jiao et al., 2019, Tang et al., 2019, Liu et al., 2022]. In the following, we will discuss our approach to data augmentation, as well as the logits loss and cross-entropy loss.

Data Augmentation In the distillation approach, a small dataset may be insufficient for the teacher model to fully express its knowledge[Ba and Caruana, 2013]. To tackle this issue, we augment the training set in order to facilitate effective knowledge distillation. We follow Tang et al. [2019] to augment the training set:

Firstly, we randomly replace a word with [MASK] token with probability p mask .

Secondly, we replace a word with another of the same POS tag with probability p pos .

Thirdly, we randomly sample an n -gram from a training example with probability p ng , where n is randomly selected from { 1 , 2 , ..., 5 } .

Logits Loss Following Hinton et al. [2015], we take logits, also known as soft labels, into consideration, which lets the student learn the prediction distribution of the teacher. To measure the distance between two distributions, we choose KL-divergence: L logits = ∑ c i p i log ( p i q i ) , where c is the number of categories, p i and q i denote the prediction distribution of the teacher model and student model.

Cross-entropy Loss Cross-entropy loss can help the student model learn from the samples in task-specific datasets: L ce = -∑ c i ˆ q i log ( q i ) , where ˆ q i represents the one-hot label vector.

Therefore, the total loss L 2 of stage 2 contains three terms:

where λ 1 , λ 2 , λ 3 , and λ 4 are the hype-parameters that control the weight of these loss.

For both stages, we adopt backpropagation through time (BPTT), which is suitable for training spiking neural networks. You can see the detailed derivation in Appendix A if interested.

4 Experiments

We conduct four sets of experiments. The first is to evaluate the accuracy of SpikeBERT trained with the proposed method on 6 datasets of text classification datasets. The second experiment is to compare the theoretical energy consumption of BERT and that of SpikeBERT. The third experiment is an ablation study about the training process. The last experiment is to figure out how the performance of SpikeBERT is impacted by the number of time steps and model depth.

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

4.1 Datasets

As mentioned in Section 3.3.1, a large-scale parallel corpus will be used to train student models in Stage 1. For the English corpus, we choose the '20220301.en' subset of Wikipedia 1 and the whole Bookcorpus[Zhu et al., 2015], which are both utilized to pre-train a BERT [Devlin et al., 2019]. For the Chinese corpus, we choose Chinese-Wikipedia dump 2 (as of Jan. 4, 2023). Additionally, we follow Lv et al. [2023] to evaluate the SpikeBERT trained with the proposed distillation method on six text classification datasets: MR[Pang and Lee, 2005], SST2 [Socher et al., 2013], SST5 , Subj, ChnSenti, and Waimai. The dataset details are provided in Appendix B.

4.2 Implementation Details

Firstly, we set the number of encoder blocks in SpikeBERT to 12 . Additionally, we set the threshold of common spiking neurons U thr as 1 . 0 but set the threshold of neurons in the spiking self-attention block as 0 . 25 in SpikeBERT. In addition, we set decay rate β = 0 . 9 and scaling factor τ as 0 . 125 . We also set the time step T of spiking inputs as 4 and sentence length to 256 for all datasets.

To construct SpikeBERT, we use two Pytorch-based frameworks: SnnTorch [Eshraghian et al., 2021] and SpikingJelly [Fang et al., 2020b]. Besides, we utilize bert-base-cased 3 from Huggingface as teacher model for English datasets and Chinese-bert-wwm-base 4 [Cui et al., 2019] for Chinese datasets.

In addition, we conduct pre-training distillation on 4 NVIDIA A100-PCIE GPUs and task-specific distillation on 4 NVIDIA GeForce RTX 3090 GPUs. Since surrogate gradients are required during backpropagation, we set α in Equation 3 as 2 . In stage 1 , we set the batch size as 128 and adopt AdamW [Loshchilov and Hutter, 2017] optimizer with a learning rate of 5 e -4 and a weight decay rate of 5 e -3 . The hyperparameters σ 1 and σ 2 in Equation 7 are both set to 1.0. In stage 2 , we set the batch size as 32 and the learning rate to 5 e -5 . For data augmentation, we set p mask = p pos = 0 . 1 , p ng = 0 . 25 . To balance the weights of the four types of loss in Equation 8, we set λ 1 = 0 . 1 , λ 2 = 0 . 1 , λ 3 = 1 . 0 , and λ 4 = 0 . 1 .

4.3 Results

We report in Table 1 the accuracy achieved by SpikeBERT trained with 'pre-training + task-specific' distillation on 6 datasets, compared to 2 baselines: 1) SNN-TextCNN proposed by Lv et al. [2023]; 2) improved Spikformer directly trained with gradient descent algorithm using surrogate gradients.

Table 1 demonstrates that the SpikeBERT trained with two-stage distillation achieves state-out-ofart performance across 6 text classification datasets. Compared to SNN-TextCNN, SpikeBERT achieved up to 5 . 42% improvement in accuracy ( 3 . 49% increase on average) for all text classification benchmarks. Furthermore, SpikeBERT outperforms TextCNN, which is considered a representative

251

252

253

254

255

256

257

258

259

260

261

262

263

264

artificial neural network, and even achieves comparable results to the fine-tuned BERT by a small drop of 4 . 13% on average in accuracy for text classification task. What's more, Table 1 demonstrates that SpikeBERT can also be applied well in Chinese datasets (ChnSenti and Waimai).

Fang et al. [2020a] propose that, in image classification task, surrogate gradients of SNNs may lead to gradient vanishing or exploding and it is even getting worse with the increase of model depth. We found this phenomenon in language tasks as well. Table 1 reveals that the accuracy of directly-trained Spikformer is noticeably lower than SpikeBERT on some benchmarks, such as MR, SST5 , and ChnSenti. This is likely because the directly-trained Spikformer models have not yet fully converged due to gradient vanishing or exploding.

4.4 Energy Consumption

An essential advantage of SNNs is the low consumption of energy during inference. We compare the theoretical energy consumption per sample of fine-tuned BERT and SpikeBERT on 6 test datasets and report the results in Table 2. The way to calculate floating point operations (FLOPs), synaptic operations (SOPs), and the theoretical energy consumption (Power) is shown in Appendix C.

It is worth noting that the energy consumption of SpikeBERT is significantly lower than that of 265 fine-tuned BERT, which is an important advantage of SNNs over ANNs in terms of energy efficiency. 266 As shown in Table 2, SpikeBERT demands only 25 . 00% of the energy that fine-tuned BERT needs to 267 achieve comparable performance on average. Moreover, on the Subj dataset, SpikeBERT can reduce 268 energy consumption by up to 77 . 17% compared to fine-tuned BERT for predicting each text example. 269 This indicates that SpikeBERT is a promising candidate for energy-efficient text classification in 270 resource-constrained scenarios. 271

272

273

274

275

276

277

278

279

280

281

282

4.5 Ablation Study and Impact of Hyper-parameters

In this section, we conduct ablation studies to investigate the contributions of: a) different stages of the proposed knowledge distillation method, and b) different types of loss in Equation 8.

As we can see in Table 4.5, SpikeBERTs without either stage 1 or stage 2 experience about 3 . 20% performance drop on average. Therefore, we conclude that the two distillation stages are both essential for training SpikeBERT. Furthermore, we observed that the average performance dropped from 76 . 30 to 73 . 27 when excluding the logits loss, demonstrating that the logits loss L logits has the greatest impact on task-specific distillation. Meanwhile, data augmentation (DA) plays an important role in Stage 2 , contributing to an increase in average performance from 75 . 54 to 76 . 30 .

We investigate how the performance of SpikeBERT is affected by the two important hyperparameters: time steps T and model depth. To this end, we conduct two experiments: (a) varying the number of

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

the time steps of spike inputs when training SpikeBERT; and (b) training a variant of SpikeBERT with different encoder block depths, specifically 6 , 12 , 18 , using our proposed two-stage method.

Figure 3 (a) shows how the accuracy of SpikeBERT varies with the increase of time steps. We find that, with the increase of time steps, the accuracy increases first, then remains unchanged, and reaches its maximum roughly at T = 4 . Theoretically, the performance of SpikeBERT should be higher with bigger time steps. However, the performance of models with 8 and 12 time steps is even worse than that with 4 time steps on ChnSenti and Waimai datasets. A plausible explanation is that using excessively large time steps may introduce too much noise in the spike trains.

1

2

4

8

12

Time Steps

80

85

90

95

sst2

ChnSenti

Waimai

(a)

MR

SST2

SST5

Dataset Name

40

60

80

Depth 6

Depth 12

Depth 18

(b)

In addition, as we can see from Figure 3 (b), the accuracy of SpikeBERT is generally insensitive to the model depths and even gets lower in some datasets. We think that's because more spike Transformer blocks mean more spiking neurons (See Section 2.1), introducing more surrogate gradients when error backpropagation through time. Higher model depth often brings better model performance for traditional deep neural networks. However, it seems that deeper spiking neural networks cannot make further progress in performance. Many previous SNNs works [Zheng et al., 2020, Fang et al., 2020a, Kim et al., 2022b] have proved this deduction.

5 Conclusion

In this study, we extended and improved Spikformer to process language tasks and proposed a new 299 promising training paradigm for training SpikeBERT inspired by the notion of knowledge distillation. 300 We presented a two-stage, 'pre-training + task-specific' knowledge distillation method by transferring 301 the knowledge from BERTs to SpikeBERT for text classification tasks. We empirically show that 302 our SpikeBERT outperforms the state-of-the-art SNNs and can even achieve comparable results to 303 BERTs with much less energy consumption across multiple datasets for both English and Chinese, 304 leading to future energy-efficient implementations of BERTs or large language models. 305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

References

OpenAI. Introducing chatgpt. 2022. URL https://openai.com/blog/chatgpt .

OpenAI. GPT-4 technical report. 2023. URL https://arxiv.org/abs/2303.08774 .

Wofgang Maas. Networks of spiking neurons: the third generation of neural network models. Neural Networks , 14:1659-1671, 1997.

Kaushik Roy, Akhilesh R. Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature , 575:607-617, 2019.

Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems , 2018.

Bojian Yin, Federico Corradi, and Sander M. Boht'e. Effective and efficient computation with multiple-timescale spiking recurrent neural networks. International Conference on Neuromorphic Systems 2020 , 2020.

Wei Fang, Zhaofei Yu, Yanqing Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Neural Information Processing Systems , 2021.

Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. ArXiv , abs/2105.11654, 2021.

Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks. ArXiv , abs/2201.10355, 2022a.

Zhaokun Zhou, Yue sheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Liuliang Yuan. Spikformer: When spiking neural network meets transformer. ArXiv , abs/2209.15425, 2022.

Philipp Plank, A. Rao, Andreas Wild, and Wolfgang Maass. A long short-term memory for ai applications in spike-based neuromorphic hardware. Nat. Mach. Intell. , 4:467-479, 2021.

Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. In The Eleventh International Conference on Learning Representations , 2023.

Rui-Jie Zhu, Qihang Zhao, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2302.13939 , 2023.

Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP , 2014.

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv , abs/1706.03762, 2017.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv , abs/1810.04805, 2019.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv , abs/1907.11692, 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv , abs/2010.11929, 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

Wei Fang, Zhaofei Yu, Yanqing Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2641-2651, 2020a.

Paul J. Werbos. Backpropagation through time: What it does and how to do it. Proc. IEEE , 78: 1550-1560, 1990.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv , abs/1503.02531, 2015.

Haonan Qiu, Munan Ning, Li Yuan, Wei Fang, Yanqi Chen, Changlin Li, Tao Sun, Zhengyu Ma, and Yonghong Tian. Self-architectural knowledge distillation for spiking neural networks, 2023.

Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In China National Conference on Chinese Computational Linguistics , 2019.

Yang Liu. Fine-tune bert for extractive summarization. ArXiv , abs/1903.10318, 2019.

Suchin Gururangan, Ana Marasovi´ c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. ArXiv , abs/2004.10964, 2020.

Eugene M. Izhikevich. Simple model of spiking neurons. IEEE transactions on neural networks , 14 6:1569-72, 2003.

Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience , 12, 2017.

Peter Udo Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. 2015 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2015.

Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision , 113:54-66, 2015.

Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in Neuroscience , 11, 2017.

Peter Udo Diehl, Guido Zarrella, Andrew S. Cassidy, Bruno U. Pedroni, and Emre O. Neftci. Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. 2016 IEEE International Conference on Rebooting Computing (ICRC) , pages 1-8, 2016.

Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine , 36:51-63, 2019.

Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural Computation , 30:1514 - 1541, 2017.

Nan Zheng and Pinaki Mazumder. A low-power hardware architecture for on-line supervised learning in multi-layer spiking neural networks. 2018 IEEE International Symposium on Circuits and Systems (ISCAS) , pages 1-5, 2018.

Shrestha and G. Orchard. Slayer: Spike layer error reassignment in time. ArXiv , abs/1810.08646, 2018.

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

Taewook Kang, Kwang-Il Oh, Jaejin Lee, and Wangrok Oh. Comparison between stdp and gradientdescent training processes for spiking neural networks using mnist digits. 2022 13th International Conference on Information and Communication Technology Convergence (ICTC) , pages 17321734, 2022.

Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation , 1:541-551, 1989.

Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In AAAI Conference on Artificial Intelligence , 2020.

Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. ArXiv , abs/2005.01807, 2020.

Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ArXiv , abs/1612.03928, 2016.

Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1921-1930, 2019.

Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5006-5015, 2021.

Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3962-3971, 2019.

K. Kushawaha, S. Kumar, Biplab Banerjee, and Rajbabu Velmurugan. Distilling spikes: Knowledge distillation in spiking neural networks. 2020 25th International Conference on Pattern Recognition (ICPR) , pages 4536-4543, 2020.

Sugahara Takuya, Renyuan Zhang, and Yasuhiko Nakashima. Training low-latency spiking neural network through knowledge distillation. 2021 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS) , pages 1-3, 2021.

Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors. Spikingjelly, 2020b. Accessed: YYYY-MM-DD.

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings , 2019.

Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy J. Lin. Distilling task-specific knowledge from bert into simple neural networks. ArXiv , abs/1903.12136, 2019.

Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao. Multi-granularity structural knowledge distillation for language model compression. In Annual Meeting of the Association for Computational Linguistics , 2022.

Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS , 2013.

Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV) , pages 19-27, 2015.

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL , 2005.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP , 2013.

Jason Kamran Eshraghian, Max Ward, Emre O. Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Bennamoun, Doo Seok Jeong, and Wei D. Lu. Training spiking neural networks using lessons from deep learning. ArXiv , abs/2109.12894, 2021.

Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. Pre-training with whole word masking for chinese bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29:3504-3514, 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2017.

Youngeun Kim, Joshua Chough, and Priyadarshini Panda. Beyond classification: Directly training spiking neural networks for semantic segmentation. Neuromorphic Computing and Engineering , 2 (4):044015, 2022b.

Indiveri, Federico Corradi, and Ning Qiao. Neuromorphic architectures for spiking deep neural networks. 2015 IEEE International Electron Devices Meeting (IEDM) , pages 4.2.1-4.2.4, 2015.

Appendix 450

A Backpropagation Through Time in Spiking Neural Networks 451

The content of this section is mostly referred to Lv et al. [2023]. 452

Given a loss function L like Eqution 7 and 8, the losses at every time step can be summed together to 453 give the following global gradient: 454

where i and j denote different time steps, and L t is the loss calculated at time step t . No matter which 455 time step is, the weights of an SNN are shared across all steps. Therefore, we have W 0 = W 1 = 456 · · · = W , which also indicates that ∂W j ∂W = 1 . Thus, Equation (9) can be written as follows: 457

Based on the chain rule of derivatives, we obtain: 458

where ∂L i ∂S i is the derivative of the cross-entropy loss at the time step i with respect to S i , and ∂S i ∂U i can 459 be easily derived using surrogate gradients like Equation 3. As to the last term of ∑ j ≤ i ∂U i ∂W j , we can 460 split it into two parts: 461

From Equation (2), we know that ∂U i ∂W i = X i . Therefore, Equation (9) can be simplified as follows: 462

By the chain rule of derivatives over time, ∂U i ∂W j can be factorized into two parts: 463

It is easy to see that ∂U i ∂U i -1 is equal to β from Equation (2), and Equation (9) can be written as: 464

We can treat ∂U i -1 ∂W j recurrently as Equation (12). Finally, we can update the weights W by the rule of 465 W = W -η ∂L ∂W , where η is a learning rate. 466

B Datasets 467

The benchmark we used in Table 1 includes the following datasets: 468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

MR : MR stands for Movie Review and it consists of movie-review documents labeled with respect to their overall sentiment polarity (positive or negative) or subjective rating [Pang and Lee, 2005].

SST5 : SST5 contains 11 , 855 sentences extracted from movie reviews for sentiment classification [Socher et al., 2013]. There are 5 categories (very negative, negative, neutral, positive, and very positive).

SST2 : The binary version of SST5 . There are just 2 classes (positive and negative).

Subj : The task of this dataset is to classify a sentence as being subjective or objective 5 .

ChnSenti : ChnSenti comprises about 7 , 000 Chinese hotel reviews annotated with positive or negative labels 6 .

Waimai : There are about 12 , 000 Chinese user reviews collected by a food delivery platform for binary sentiment classification (positive and negative) 7 in this dataset.

C Theoretical Energy Consumption Calculation

For spiking neural networks (SNNs), the theoretical energy consumption of layer ξ can be calculated as

where 0 . 9 pJ is the energy consumption per synaptic operation (SOP) [Indiveri et al., 2015, Hu et al., 2018, Zhou et al., 2022]. The number of synaptic operations at the layer ξ of an SNN is estimated as

where T is the number of times step required in the simulation, γ is the firing rate of input spike train of the layer ξ , and FLOPs ( ξ ) is the estimated floating point operations at the layer ξ .

For classical artificial neural networks, the theoretical energy consumption required by the layer ξ can be estimated by

Note that 1 J = 10 3 mJ = 10 12 pJ.

D Discussion of Limitations

In the image classification task, spiking neural networks have demonstrated comparable performance to ViT on CIFAR-10-DVS and DVS-128-Gesture datasets, which are neuromorphic event-based image datasets created using dynamic vision sensors. We think that the performance gap bewteen SNNs and ANNs in language tasks is mainly due to the lack of neuromorphic language datasets. It is unfair to evaluate SNNs on the datasets that were created to train and evaluate ANNs because these datasets are mostly processed by continuous values. However, it is quite hard to convert language to neuromorphic information without information loss. We hope there will be a new technology to transfer senteces to neuromorphic spikes.

In addition, GPU memory poses a limitation in our experiments. Spiking neural networks have an additional dimension, denoted as T (time step), compared to artificial neural networks. Increasing the number of time steps allows for capturing more information but results in an increased demand for GPU memory by a factor of T . During our experiments, we observe that maintaining the same number of time steps during training requires reducing the sentence length of input sentences, which significantly constrains the performance of our models. We remain optimistic that future advancements will provide GPUs with sufficient memory to support the functionality of SNNs.

Peer reviews:
comments: SUMMARY:
This paper proposes SpikeBERT, a spiking-based BERT model for text classification. It employs LIF spiking neurons an surrogate gradients for backpropagation. The training method consists of a two-stage distillation process (pre-training + task-specific). The experiments conducted on different benchmarks for English and Chinese languages show that the proposed SpikeBERT achieves higher accuracy than prior spike-based language models and lower energy consumption than the (non-spiking) BERT.

STRENGTHS:
1.	The proposed method is novel and relevant to the community.
2.	The technical sections are described clearly.
3.	The experiments provide good-quality results.

WEAKNESSES:
1.	The design decisions are not discussed in detail.
2.	Unlike non-spiking BERT, there are scalability issues when increasing the depth.

QUESTIONS:
1.	In Section 3, please discuss the design decisions made to devise the proposed method. Why choosing LIF neurons instead of other neuron models? Why these choices for the architectural parameters?
2.	Since the SpikeBERT does not achieve a significant accuracy increase when increasing the depth, can the proposed method scale up and acquire similar properties like the emergent abilities in (non-spiking) large language models?

LIMITATIONS:
The limitations have not been discussed by the authors, but there are no major limitations in this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

comments: SUMMARY:
This paper presents SpikeBERT, an improved version of the Spikformer spiking transformer model for language tasks. SpikeBERT utilizes a two-stage knowledge distillation method that combines pre-training with BERT and fine-tuning with task-specific data. Experimental results show that SpikeBERT outperforms state-of-the-art spiking neural networks and achieves comparable results to BERT on text classification tasks for English and Chinese, while consuming significantly less energy.

STRENGTHS:
1 The writing is commendable.
2 The proposed approach is captivating and innovative, and I believe it will generate significant interest within the machine learning community.
3 The results are promising, particularly in achieving comparable performance to BERT in text classification.

WEAKNESSES:
1 Although SpikeBERT significantly reduces energy consumption during inference, the two-stage knowledge distillation process may introduce additional costs. It would be helpful if the authors could provide insights on this matter.

2 Considering other model architectures such as the OPT model families, does SpikeBERT possess the potential to be extended to these models? It would be interesting to explore the applicability of SpikeBERT beyond the BERT architecture.

3 I am curious about the evaluations of SpikeBERT on additional tasks such as GLUE, RACE, and SQuAD. It would provide valuable insights into the model's performance across a broader range of language processing tasks.

QUESTIONS:
See Weaknesses

LIMITATIONS:
No limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents SpikeBERT, an implementation of BERT-based models on a Spiking Neural Network (SNN) architecture, motivated by theoretical energy efficiency benefits.\n\nThe paper presents the transformer architecture and a two-stage distillation approach which first distills a general purpose BERT model into a general purpose SpikeBERT, then finetunes the SpikeBERT by distilling a finetuned BERT model.\n\nThe approach is evaluated on several text classification tasks on English and Chinese datasets, resulting in accuracies lower but comparable than a standard finetuned BERT classifier.\n\nA theoretical energy efficiency improvement is calculated, reporting improvements, which however I find dubious (see weaknesses)\n\nSTRENGTHS:\n- Interesting work on an unusual architecture\n\nWEAKNESSES:\nI am especially concerned about the claims of improved energy efficiency, which serve as the main motivation of the paper.\nStarting from the introduction, where the author claim: \"However, it requires too much computational power and energy to train and deploy state-of-the-art ANN models, leading to a consistent increase of energy consumption per model over the past decade. The energy consumption of large language models, such as ChatGPT[OpenAI, 2022] and GPT-4[OpenAI, 2023], is unfathomable even during inference.\"\nIt is clearly not true that \"it requires too much computational power and energy to train and deploy state-of-the-art ANN models\" since these models are in fact trained and deployed.\n\nMore concerning is the theoretical energy comparison of SpikeBERT and BERT (Section 4.4 and Appendix C), where the authors compare FLOPs for BERT and SOPs (spiking operations) for SpikeBERT, multiply by theoretical energy costs and declare SpikeBERT the winner. The theoretical energy costs seem to be copied from other papers, and following the citation chain they seem to come from Yao et al. 2022 \"Attention Spiking Neural Networks\" where they are computed using data from Horowitz 2014 \"1.1 computing’s energy problem (and what we\ncan do about it)\", with the assumption of 32-bit floating point operations on 45nm hardware. Modern GPUs use 7nm hardware, and inference is often done with 8-bit floating point operation or less, therefore I wonder whether these number are obsolete.\n\nQUESTIONS:\nTable 2: mJ is a measure of energy, not power\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This work develops SpikeBERT, which extends Spikformer to perform language processing tasks, and proposes a two-stage knowledge distillation method for better training it. Experiments validate the improved accuracy of SpikeBERT over previous SNNs and the improved efficiency over vanilla BERT.

STRENGTHS:
1. This work is the first transformer-based SNNs for language processing tasks.

2. Experiments validate the achieved efficiency improvement over vanilla BERT.

WEAKNESSES:
I have the following concerns about this work:

1. The novelty and technical contributions of this work are limited: the modifications from Spikformer to SpikeBERT are minor, and similar distillation schemes have been widely studied and adopted in efficient BERT works, e.g., TinyBERT, TernaryBERT, FastBERT, DistilBERT, etc. It is hard to tell the key technical contribution of this work.

2. The experimental validation is insufficient: It only validates the improved efficiency over vanilla BERT while the aforementioned efficient BERT variants are not benchmarked, making it not clear whether SpikeBERT is a practical efficient BERT option. In addition, the theoretical power is not enough for indicating the real-device efficiency and on-device measurement is highly desirable for benchmarking the aforementioned efficient BERT variants.

3. This work may violate the formatting regulations, i.e., it adds an extra appendix in the main manuscript. In addition, the citation format seems to not follow the official template.

QUESTIONS:
I have listed my questions in the weakness section.

LIMITATIONS:
This work may violate the formatting regulations, i.e., it adds an extra appendix in the main manuscript.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors have proposed SpikeBERT which is an energy efficient Spiking Neural Networks(SNN) for natural language representation. The architectural design for SpikeBERT is inspired from Spikformer which is an SNN for computer vision, with the following major changes:\n\t- Spiking Patch Splitting for images is replaced by embedding layer to encode words/tokens into vectors.\n\t- BatchNorm replaced by LayerNorm\n\t- Shape of Spiking Self Attention is changed to adapt to language tasks by changing its size based on length on input instead of dimensionality of hidden layers.\n\nThe model is trained using a two-step knowledge distillation approach:\n\t- General purpose: The model is trained using embedding and intermediate hidden layer representations of BERT on unlabeled natural language texts.\nTask specific: The model is tuned using task specific logits from a fine-tuned BERT model.\n\nSTRENGTHS:\n- The paper is well written and easy to follow. The authors provide the necessary background on SNNs including the advantages and challenges in training them.\n\t- The motivation is clear and the energy consumption presented in the results section helps convey the same.\n\t- The metrics indicate that the proposed approach outperforms the previous spiking network based baseline developed for natural language understanding and standard SNN training mechanism using surrogate gradients.\n\t- The authors present a thorough ablation analysis for all the contributions presented in the paper.\n\nWEAKNESSES:\n- The novelty of the paper is limited:\n\t\t○ The architecture is mostly derived from Spikformer: the usage of word embeddings instead of image patches, and using layer normalization in transformer  is a standard approach.\n\t\t○ The two-step knowledge distillation approach has been used widely in the past for distilling BERT and GPT style transformer models to smaller/specific architectures.\n\t\t○ The usage of hidden layer representations in the distillation process is also a standard practice for BERT style models.\n\t\t○ Most of the formulations and approaches related to spiking neurons, its derivatives and feature transformations are adapted from previous work.\n\nQUESTIONS:\n- BERT was introduced several years ago, and since then, multiple adaptions of its architecture and training tasks have been proposed that outperform it such as RoBERTa. Why have the authors chosen BERT as a teacher model for SpikeBERT?\n\nLIMITATIONS:\n- The datasets used for evaluation seem limited. With models like BERT, it's a standard approach to present results on all GLUE tasks or at least the ones such as MNLI as they are considered to be good & reliable indicators of model quality.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}