Rethinking Deep Spiking Neural Networks: A Multi-Layer Perceptron Approach
Authors: Luziwei Leng
Boyan Li
Ran Cheng
Shuaijie Shen
Kaixuan Zhang
Jianguo Zhang
Jianxing Liao
Venue: Submitted to ICLR 2023 (2023)
Keywords: spiking neural network
multi-layer perceptron
image classification

Abstract: By adopting deep convolution architectures, spiking neural networks (SNNs) have recently achieved competitive performances with their artificial counterparts in image classification, meanwhile with much lower computation cost due to event-driven and sparse activation. However, the multiplication-free inference (MFI) principle makes SNNs incompatible with attention or transformer mechanisms which have shown significant performance gains on high resolution vision tasks. Inspired from recent works on multi-layer perceptrons (MLPs), we explore an efficient spiking MLP design using batch normalization instead of layer normalization in both the token and the channel block to be compatible with MFI. We further strengthen the network’s local feature learning ability with a spiking patch encoding layer, which significantly improves the network performance. Based on these building blocks, we explore an optimal skip connection configuration and develop an efficient multi-stage spiking MLP network combining global receptive field and local feature extraction,  achieving full spike-based computation. Without pre-training or other advanced SNN training techniques, the spiking MLP network achieves 66.39% top-1 accuracy on the ImageNet-1K dataset, surpassing the state-of-the-art directly trained spiking ResNet-34 by 2.67% under similar model capacity meanwhile with shorter simulation steps and much less computation cost. Another larger variant of the network achieves 68.84% top-1 accuracy, rivaling the spiking VGG-16 network with 4 times smaller model capacity. Our work demonstrates the effectiveness of an alternative deep SNN architecture combining both global and local learning abilities. More interestingly, finally we show a close resemblance of the trained receptive field of our network to cells in the cortex. Code will be publicly available.

Under review as a conference paper at ICLR 2023
RETHINKING DEEP SPIKING NEURAL NETWORKS: A MULTI-LAYER PERCEPTRON APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
By adopting deep convolution architectures, spiking neural networks (SNNs) have recently achieved competitive performances with their artificial counterparts image classification, meanwhile with much lower computation cost due to event- driven and sparse activation. However, the multiplication-Fin free inference (MFT) principle makes SNNs incompatible with attention or transformer mechanisms which have shown significant performance gains on high resolution vision tasks. Inspired from recent works on multi-layer perceptrons (MLPs), we explore efficient spiking MLP design using batch normalization instead of layer normal- ization in both the token and the channel block be compatible with MFI. We further strengthen the network's 's local feature learning ability with a spiking patch encoding layer. which significantly improves the network performance. Based on these building blocks, we explore an optimal skip connection configuration and develop an efficient multi-stage spiking MI.P network combining global receptive field and local feature extraction, achieving full spike- -ased computation. With- out pre-training or other advanced SNN training techniques, the spiking MLP net- work achieves 66.39% top-1 accuracy the ImageNet-IK dataset, surpassing the state-of-ther the-hhhtttt art directly trained spiking ResNet-34 34 by 2.67% under similar model capacity meanwhile with shorter simulation steps and much less computation cost. Another larger variant of the network achieves 68. 88.84% top-1 accuracy, rivaling the spiking VGG-16 network with 4 times smaller model capacity. Our work demon- strates the effectiveness of an alternative deep SNN architecture combining both global and local learning abilities. More interestingly, finally we show close resemblance of the trained receptive field of our network to cells in the cortex. Code _cod_wwlll be wtwpeeeppplllaeeeeeeeeeeeeeellle aaaliicllyavallale....
INTRODUCTION
Spiking neural networks (SNNs) Maass 1997) have been proposed as models for cortical simula-
tion (Izhikevich] 2004 Brette & Gerstner 2005 Deco et al.] [008 Gerstner et al. [004]] Korcsak-
Gorzo et al. 22222 and candidates for solving problems in machine learning (Tavanaci et al.) 2019
Roy et al.) 2019). Nevertheless, SNNs following exact biological topology and constraints such as Dale's law have not been demonstrated to be equally effective as artificial neural networks (ANNs)
in practice, especially when scaling up. By adopting structures and adapting learning algorithms from their artificial counterparts, SNNs have improved their performances and recently achieved
higher accuracy in benchmark image classification problems (Shrestha & Orchard 2018 Wu al.,
2019 Sengupta et al.) 2019 Li et al. 2021 Fang et al.) 2202 Deng et al.] 2022). Deep convolutional neural networks (CNNs) are the current de-facto architectures adopted by SNNs in various vision tasks (Kim et al. 2020; 2021 Rançon et al. 2021 (0222221 Zhu et al.) (2022). Recently, ANNs with visual at- tention and transformer mechanisms (Dosovitskiy al. 2020
10200Liu Liu et al. (02T) have surpassed pure CNNs by learning global dependency of the image. However, these mechanisms usually involve matrix multiplication and softmax function which make them contradict to the multiplication-Fi free inference (MFI) principle of SNNs (Roy al. 2019 Rathi & Roy 2021).
In this work, we explore a spike-based implementation of an alternative structure more compatible with this principle, i. multi-layer perceptrons (MLPs), which have recently been demonstrated to be equally efficient as transformers (Tolstikhin et al.] 202T). The original MLP-Mixer architecture for ANNs still involves real-valued matrix multiplication which violates MFI. To this end, we design

---

Under review as a conference paper at ICLR 2023
spiking MLP-Mixer architecture using the MFI-friendly batch normalization (BN) with lightweight axial sampling in the token block. With the spiking MLP-Mixer as a basic building block, we pro- pose multi-stage spiking-MLI network achieving full spike- based computation. To enhance the local feature extraction of the MLP network, we propose a spiking patch encoding module based on directed acyelic graph structure to replace the original patch partition for downsampling. In addi- tion, we identify the crucial role of skip connection configuration for an optimal spiking MLP-Mixer design. To our best knowledge, this is the first work to explore full spike-based ooken-sampling MLP architectures in the field of SNNs. To be specific, our contributions can be summarized as follows:
We develop an efficient spiking MLP-Mixe with MFI-friendly BN and lightweight axial sampling in the token block. In addition, identify the crucial role of skip connection configuration for an optimal spiking MLP-Mixer design.
We propose a spiking patch encoding module to enhance local feature extraction and for downsampling, based on which we construct multi-stage spiking- -MLP network achieving full spike-based computation.
Our network achieves 66.39% top-1 accuracy on the classification of ImageNet-1K with 2.67% improvement compared to the current state-of-the-an deep spiking ResNet-34 net- work, meanwhile with similar model capacity, 2/3 of its simulation steps and much lower computation cost. With an equal simulation steps the same network improves to 69.09% accuracy achieving 5.37% improvement, slightly surpassing spiking VGG-16 network with 5.5 times smaller model capacity, demonstrating the effectiveness of an alternative archi- tecture design for deep SNNs.
Finally, our networks pre-trained on ImageNet create new SNN records when fine-tune on CIFARIO and CIFAR 1100 datasets, achieving 96.08% and 80.57% accuracy, demonstrating the general usage of our architecture as pre-trained models.
2 RELATED WORK
2.1 SPIKING NEURAL NETWORKS IN DEEP LEARNING
Originated from computational neuroscience, SNNs have been widely used for modeling brain func- tion and dynamics (Izhikevich 2004 Brette & Gerstner 2005) Deco et al. 2008 Gerstner et al., 2014 Koresak- Gorzo et al., 2022). The success achieved and challenges faced by deep ANNs in solving machine learning problems have led trend to use SNNs as an alternative and explore func-
tional benefits of their bio-inspired properties in solving similar problems. has been demonstrated that by adapting learning algorithms (Bohte et al. 2000; Wu et al., 2018 Nefici et al., 2019 Bellec itt al. (020)) and adopting efficient architectures from ANNs, such as Boltzmann machines (Ack- ley et al. 1985) in the early phase of deep learning and the current dominant CNNs (LeCun et al.
1989).
SNNs can achieve competitive performances rivaling their artificial counterparts (Petrovici
al.
2016 Nettci et al.) 2014 Leng al. 2018)) Shrestha & Orchard 2018) Wu et al., 2019 ZZang 2020
& Li
Li et al.] 2021 Fang et al. [202]] Deng et al. 2022). Several recent works have applied neural architecture search (NAS) to obtain task-specific cells network structures for SNNs (Na)
et al.
2022), Kim et al.) 2022). Despite that these networks have achieved further spike reduction
and new sfate-of-the-art results in image classification, their architectures are still deep CNNs.
2.2 BIOLOGICALLY PLAUSIBLE ATTENTION
The recent success of attention and transformer mechanisms in speech (Vaswani et al. 2017) and vision tasks (Dosovitskiy et al.) (020)) has motivated explorations of similar mechanisms in more biologically plausible forms. Works in Widrich et al. (2020); |amssuer al. (2020) showed that the attention mechanism of transformer is equal to the update rule of modern Hopfield networks with continuous states and demonstrated its high storage capacity in large scale multiple instance learn- ing. However, same the original transformer, it involves matrix multiplication and softmax func- tion which are not compatible with spike- computation. The pioneering work of MLP-Mixer Tolstikhin et al.] 2021 proposed alternative architecture based exclusively on MLPs without convolution or self-attention layers. The approach contains two MLP blocks with a token mixing block applied on sliced image patches similar to vision transformer (VIT) (Dosovitskiy et al. 2020
2

---

Under review as a conference paper at ICLR 2023
and channel mixing block applied across patches. Without the inductive biases of the local con- nectivity and the self-attention, the token- coken-mixing MLP is more flexible and has a stronger fitting capability (Zhao et al. 202T Liu et al. 2022). However, it is also more prone to over-fitting and relies on pre-trained models on large-scale datasets to achieve competitive performances on image classification benchmarks comparable ViT and CNNs. [Tang al.) (2022) proposed sparseMI.P to solve the over parametric and over-fitting problem of MLP-Mixer by adopting a multi-stage pyramid network structure and applying axial sampling instead of full sampling in the token mixing MLP. Similar concepts were also shared in other MLP variants (Hou et al. 2022 Tatsunami & Taki 2021 Wang et al.) p.22). The structural simplicity and effectiveness of MLPs suggest a promising network paradigm. Li et al. (2022) implemented MLPs with non-spiking full-precision neurons. Neverthe- less, the network is not typical SNN since avoids spike-based communication and fails to fulfill
the MFI principle.
3 METHOD
3.1 TRAINING OF SNN
We adopt the leaky integrate-and-fire (LIF) neuron model with hard threshold and decay input de- scribed by
u pre = u²=1 + z' - u "'1
(1)
y' g(u"-pro) u' = (1-y')u"ar
(2)
(3)
where denotes the time step, T is the membrane time constant, u the membrane potential denoted by bold italic letter representing vector, y denotes the spike output, is threshold function, z' Wy'epre is the synaptic input with W denoting the weight matrix, and y'rre denotes the afferent spikes from pre-synaptic neurons. A neuron will fire spike and transmit it the post- synaptic neuron when its membrane potential exceeds threshold Vaa, followed by hard reset of the membrane potential; otherwise the neuron transmits signals, given by
v =
{
u otherwise
if p.re 2 Vn
In this work, set and Va. 1. Given loss L and using chain rule, the weight update of an SNN can be expressed as:
aL aW Σ dL Oy' pue Oz
(5)
Oy -ure Oz Ow
where ay'
au Spre is the gradient of the firing function, which is zero everywhere except at the threshold. The surrogate gradient method uses continuous functions to approximate the gradients, such as rectangular (Zheng al., (021), triangular (Bellec et al. 2018), exponential curves (Shrestha & Orchard Zenke & Ganguli 2018), etc. We adopt sigmoid function our experiments.
3.2 NETWORK ARCHITECTURE
The MLP-Mixer (Tolstikhin al. 202T) adopted single-stage or "isotropic" network design where input and output resolutions were kept the same across different layers. This structure leads to excessive number of parameters which can cause an over-fitting of the network when training on medium-scale dataset such as ImageNet-IK. K. To alleviate this issuc, adopt multi-stage pyramid network architecture as Liu et al. (2021); Chu et al. (2021): Tang et al. (2022). Figure Iillustrates the architecture of our network.
Spiking Patch Encoding Given an input RGB image with size 3xHx W, the original MLP-Mixer divides it into overllalgg patches of size p and linear projects the new channel dimension of 3p² to a hidden dimension of C, This process is equal to a 2D convolution operation with kernel and stride size of p and output dimension of Ct. Compared to convolution operation, the MLP has
3

---

Under review as a conference paper at ICLR 2023
Output
Rearrange
Rea Rearrange
LIF
FC
Skip Add
Skip Add
LIF
BatchNorm
BatchNorm
Cony 1x1
FC
NorchNorm
Concat
Spiking MLP
FC
Rearrange
Rearrange
(UWW) x
(CW) x
(CN) x w
M x
Ng x
Channel Block
(c) Soiking Channel Block
Spiking MLP
Spiking MLP
Taken Block
(CW) x
(CN) W
LI
SPC
Rearrange
Rearrange
BatchNorm
C x H x W
FC
Inout
(a) Architecture
(b) Spiking Taken Block
(d) Spiking MLP
Figure 1: Overall network architecture (a) and the spiking MLP-Mixer (b-d). The multi-stage net- work is downsampled with spiking patch encoding (SPE) module at each stage. Within each stage, the SPE is followed by a sequential of spiking MI.P-Mixers with identical architecture each con- taining spiking token block with axial sampling (b) and spiking channel block with full sampling (c). For simplicity, we denote the repeated FC-BatchNorm-LIIF block as spiking MLP layer (d).
2C H W J''
(C, W)
H W 3 3..'
(2C, 7'
2C # W
3 2 2
Conv2d BatchNorm
Identity
UF
Figure 2:
Spiking patch encoding with a directed acyclic graph structure. The structure follows MFI principle with additions performed on BN states and multiplications performed between convolution weights and binary spikes.
global receptive field but in turn lacks the inductive bias to learn from local features. To address this issue, we construct a spiking patch encoding (SPE) module to replace the original patch partition approach. The SPE is implemented with a spike-based directed acyelic graph inspired from a recent work (Anonymous), as shown in Figure 2 We call this structure a spiking cell since it is originated from DARTS (Liu et al. 2018). A cell consists of nodes, where each node receives the same input from the previous stage with convolution operation followed by BN. In addition, node pairs (1,2) and (2.3) are interconnected with convolution followed by BN, and node pair (1,3) is connected with identity connection. Within each node, multiple operations are added after BN, and the following spiking activation is used as output of the node. The output the cell is the concatenated spiking states of all nodes.
The output of the spiking cell is then fed into a sequential of spiking MixeMixers which finally output spiking feature map of the same shape. In the following stages, this process is repeated
4

---

Under review as a conference paper at ICLR 2023
with the input feature consecutively downsampled in spatial dimension and expanded in channel dimension, both with a ratio of 2. The output of the final stage is fed into a spiking classification head with spiking MLP block consisting of fully connected (FC) layer followed by BN and spiking activation, with the FC layer performing full sampling on the flattened spatial dimension of the feature map. Finally, linear classifier projects the output of spiking MIP block to label layer. The loss function is a cross entropy fuuction with the network output averaged over simulation time steps.
3.3 SPIKING MLP-MIXER
direct adoption of existing design of MLP blocks in ANNs by replacing the real-valued activation function with spiking activation function (SAF) leads to a contradiction to MFI principle of spike- based computation. In both of the token and channel block of the original MLP-Mixer, the FC operation is performed on the layer normalization (LN) state of the feature map, leading to real- valued matrix multiplication. To address this issue, we relocate the normalization after the FC operation and replace LN with BN, such that the parameters of the latter can be merged into linear projection weights during inference to be compatible with the MFI principle (loffe & Szegedy 2015 Rueckauer et al.) 2017).
A spiking MLP-Mixer consists of a token block and a channel block, each includes multiple FC layers with spiking activation, as illustrated in Figure Tp.c.
Spiking Token Block Given input image patches I of shape C x H x W. the token block of the p original MLP-Mixer fattens its spatial dimension to form a 2D tensor and performs full sampling on the token dimension with weights shared across the channel dimension, thus leading to a global receptive field which can be heavily parameterized given small patch sizes. Inspired from previous works TTtsunammi & Taki 2021 Hou et al., 2022 Tang et al., 2022 Wang et al. 2022). we design two branch structure for the spiking token block to separately encode the feature representation along the horizontal and vertical spatial dimension which learns long-range dependencies along one direction meanwhile preserving positional information along the other direction. As shown in Figure Te. each branch consists of an FC layer followed by a BN and an SAF, Afterwards, the two binary feature maps are concatenated along the channel dimension together with identity connection from the input feature map, and projected back to C through an FC layer with BN and an SAF. Additionally, here we add skip connection from the BN states of the SPE.
Spiking Channel Block The channel block consists of two spiking FC layers and receives the transposed output from the token block. The first layer expands the channel dimension by a certain ratio a and the second layer recovers to the original dimension. A skip connection is added between the BN state of the second second layer and the summed BN state of the token block.
To alleviate the gradient vanish problem of deep SNN, we add skip connections between the last BN states of the token and the channel block, together with the BN states of the SPE. We show in ablation studies that this design is crucial for an efficient network. Within a stage, denoting matrix multiplications by two bold capital letters, the first spiking MLP-Mixer can be formulated as:
X Concatt (X11, Xn2: Xna)
(6)
I S(X)
(7)
Un BN(W,,
(8)
U = BN(W,,, ... IT")
(9)
Y BN ENN (onn +X
(10)
1 = S(Y)
(11)
V Y BBN(
(12)
I3 = f(V)
(13)
where BN denotes batch normalization, XSs the state hefore the spiking activation of the ith node, ie summed up values of multiple combined operations, denotes an SAF, W,, and W... are the token FC weights operating on the height and width dimension with shape H x H and W x W respectively, W, is the branch fusion weights with shape C' x 3C, superscripts h. w and denote the matrix reshape operation preserving the height, width and channel dimensions with I.. I
5

---

Under review as a conference paper at ICLR 2023
and I2 of shape H CW W x CH and C' x HW respectively, "denotes the reshape operation F
3e flattening the spatial dimension with U, and UT of shape C' x HW W and Wez are the channel p² FC weights with shape aC' C' and C' x oC respectively, where a is the channel expansion ratio. Note that all matrix multiplications are between a realevalued matrix and a binary matrix, such that the spiking MLP-Mixer is essentially multiplication-free For the following spiking MLP-Mixers, the input tensor becomes the output from the channel block of the previous Mixer.
n²
4 EXPERIMENTS
We evaluate model benchmark image classification datasets of ImageNet-1K IK (Krizhevsky et al. (017) and CIFARIO/100 (Krizhevsky et al.). We use SpikingJelly (Fang et al.) 2020) with its cupy backend to speed up the simulation of LIF neuron. In the SPE module, for convolution operations we use a kernel size of 3 x 3 with stride and I for input to node and node to node connections, respectively.
4.1 IMAGENET-IK
The ImageNet- IK dataset consists of a training set, validation set and test set with 1.28M, 50K and 100K 224 images respectively. Most of our training settings are inherited from Tang et al. (022) including data augmentation. We use cosine decay learning rate strategy. The initial learning rate is 1 and gradually drops to in 100 epochs. The optimizer is SGD with a momentum of 0.9. For the first stage of the spiking MLP network patch size 4 and the rest stages with SPE module for downsampling. We build three variants of spiking MLP network, namely spiking MLP-SPE-T/S/B, The architecture of these two models are: spiking MLP--PPEEE
the original patch partition approach with
C1 78, number of layers in each stage {2: 8: 14: 2}. spiking MLP-SPE- S: C1 96, number of layers in each stage = (2: 8; 14: 2}, spiking MLP-SPE-B. C₁ = 108, number of layers in each stage = (2: 10; 24; 2}. The expansion ratio of the channel FC layer is set a = 3. To study the influence of the SPE module, we also train an MIP-S model with the original patch partition approach while other parts of the network remaining the same. The simulation step of SNN is set to T steps, we use time inheritance training (TIT) (Deng et al. 2022) and obtain the T = 6 result for MLP-SPE-T, T, where we initialize the network with the trained MLP-SPE-T (T = 4) model and fine-tune for 50 epochs with T= 6 using cosine learning rate decaying from 0.1 to O. We compare our model with other state-of-the-ar SNNs including directly training and ANN-to-SNN conversion methods. The result is in table | where we report the top-1 accuracy along with the number of model parameters and simulation steps. We can see that ANN-SNN conversion methods can achieve
To better compare with other SNNs with larger time
Table 1: Comparison on ImageNet-IK. T denotes simulation length.
Method
Architecture
Model Size
T
Accuracy [%]
ANN-SNN (Hu al. 2018)
ResNet-34
22M
768
71.6
ANN-SNN (Songupta et al. 2019
VGG-16
138M
2500
69.96
Hybrid training (Rath el al. 2020
ResNet-34
22M
250
61.48
Hybrid training LLee et al.
2020
VGG-16
138M
250
65.19
STBP-dBN (Zheng et al. 2021
ResNet-34
22M
6
63.72
TET (Deng et al. 2022
ResNet-34
22M
6
64.79
STBP-dBN (Zheng et al.
2021
ResNet-34-large
86M
67.05
Diet-SNN (Rathi & Koy 2021
VGG-16
138M
5
69.00
Spiking (
(our model)
MLP-SPE-T
25M
4
66.39
Spiking MIP (our model)
MLP-SPE-T
25M
6
69.09
Spiking MI.P (our model)
MLPPS
34M
4
63.25
Spiking MLP (our model)
MLP-SPE-S
38M
4
68.84
Spiking MLP (our model)
MLP-SPE-B
66M
6
71.64
high accuracy however with extremely long simulation steps. Hybrid training methods reduce the simulation steps but are still significantly longer than directly training methods. Among directly trained methods, under similar network capacity, MIPP-PP-T significantly surpasses STBP-tdBN

---

Under review as a conference paper at ICLR 2023
34 by 2.67% with shorter simulation time steps. After TIT the model achieves accuracy of 69.09% with = 6, surpassing BesNee-33--argge and VGG-16 16 model by 2.04% and 0.09% with more than 3 and 5.5 times smaller model capacity, demonstrating the superiority of our architecture. The improvement of MIPP-SPES from MLP-S shows that the SPE module can significantly improve network performance, demonstrating the effectiveness of combining global receptive field and local feature extraction. Note that it is more fair to compare our methods with the STBP-tdBN method since we use the same conventional temporal averaged loss function. The TET method used temporal mosent-wise loss function that can outperform temporal averaged one as demonstrated in Deng et al. (2022).
4.2 CIFAR
Both the CIFARI0/100 datasets contain 50K training images and 10K testing images with size of 32 32 pixels. We use random resized crop and random horizontal flip for data augmentation as other works. We adopt the pre-training and fine-tuning paradigm other transformer and MLP works (Dosovitskiy et al. 20200 Tolstikhin et al. 202I). We use the pre- pre-trained spiking MLP-SPE- T network the ImageNet- IK dataset the pre-trained model. We reset the final classification output layer of the network and fine-tune it on the resized 224 x 224 CIFAR10/106 dataset. For fine-tuning, use cosine decay learning rate with initial value of 0.1 and gradually drops to 0 in 100 epochs. The optimizer is SGD with a momentum of 0.9. We compare the result with other state-of-the-ard SNNs shown in table 2 With larger network size, our model sets new records on both datasets, significantly surpassing existing SNNs. It might be a bit unfair to compare with directly trained SNNs, nevertheless the results demonstrate the usefulness of the spiking MLP network as a pre-trained model for other datasets.
Table 2: Comparison on CIFAR. denotes simulation length.
Dataset
Method
Architecture
T
Accuracy[%]
Diet-SNN Rathi & Roy
2021
ResNet-20
10
92.54
STBP-tdBN Zheng ct al.
2021
ResNet-19
6
93.16
STBP-tdBN Zheng al.
2021
ResNet-19
4
92.92
CIFARIO
TET (Deng
ResNet-19
94.44 + 0.08
Spiking MLP (our model)
MLP-SPE-T
96.08
Dict-SNN (Rathi & Roy
202
ResNet-20
64.07
STBP-tdBN ZZeng ct al.
2021
ResNet-19
71.12 + 0.57
STBP-1dBN (Zheng el al. 2021
ResNet-19
4
70.86 + 0.22
CIFARI00
TET (Deng et al.
2022
ResNet-19
6
74.72 + 0.28
TET (Deng 2022
ResNet-19
4
74.17
+ 0.15
Spiking MLP (our model)
MLP-SPE-T
80.57
4.3 ABLATION STUDY
In this section we further study the influence of different configurations of skip connections on the performance of the network. As shown in Figure 3 we distinguish different skip connections, ie patch block token block (PT), patch block to channel block (PC), token block to channel block (TC) and channel block to the next token block (CT), with corresponding indices and colors. The skip connection from the patch block the first token block is set as default. To shorten the simulation time, we set the initial channel number to 60 and directly train networks of different skip connection configurations from scratch on the CIFARIO 10 dataset for 100 epochs. The results are collected in Table 3 Networks without skip connections from the initial patch block to sequential token and channel blocks perform much worse than the others. This could be due to long range skip connections alleviate the gradient vanish problem of deep SNNs during training. Our spiking MLP-mixe P-xieer adopts the optimal skip connection configuration (PT, PC, TC).
7

---

Under review as a conference paper at ICLR 2023
P
Figure 3: Potential skip connections for the spiking MLP-Mixer.
Table Network performance under different skip connection configurations on CIFAR1O.
PT(I) PC(2) TC(3) CT(4)
Accuracy[%]]
80.53
81.35
f
79.46
V
10.53
V
10.00
stage
2nd stage
stage
4th stage
Sparsity
0.10
0.05
0.00
Layer Index
Figure 4: Mean network sparsity of spiking MLP-SPE-T on the ImagNet- 1K test set. We distinguish each stage with different colors.
4.4 NETWORK SPARSITY AND COMPUTATION COST
SNNs are considered be much more energy efficient than dense-computing ANNs, due their inherent event-driven and sparse computing property. In addition to that, the MFI principle enables SNNs be fully based on additions, which further decreases their potential energy consumption comparing with ANNs relying on real-valued matrix multiplications. In this section, we measure the sparsity as well as the computation cost the spiking MPP-SPE-T model and compare with spiking ResNet-34. We plot the mean sparsity of each spiking MLP-Mixer on the ImageNet- 1 test set in Figure 4 Our model shows varying sparse activity across different layers with a mean network sparsity of 0.14. Following (Li et al. 2021 Rathi & Roy 2021), we count the number of addition operations of the SNN by A where is the mean sparsity (total number of spikes divides total feature map size), T is the simulation time step and A is the addition number. The result is Table 4 The small amount multiplication from the input layer which receives floating-vau -vauue images. We estimate the energy consumption of the network following the study of Horowitz (2014)
8

---

Under review as a conference paper at ICLR 2023
THumo
24
11
0.5
43
0.3
68
0.2
-13
0.0
-20
20
26
[a] Stage
(b) Stage
[c] Stage
(d) Weights selected from stage
Figure 5: Axial sampling weights from token blocks across different stages after training on ImageNet-1K 1K. (a-c) Randomly selected token weights ploted
x H or W x W 2D matrix. 1, 2 and 4 weight matrices from stage 1, 2 and are displayed. The diagonal distribution of values early stages indicates local sampling. (d) Weights of several neurons from the same token block in stage 2. For simplicity only plot several neurons here, this 'M' shaped weight kernel is found universal in early stages of the network.
45mm CMOS technology adopted by previous works (Li et al. 2021; Rathi & Roy 202T), with one addition operation in SNN costing 0.9pJ and consuming 4.6pJ. With similar model capacity, our network has a computation cost almost twice smaller than spiking ResNet-34.
accumulate (MAC) operation in ANNs
Table 4: The computation and estimated energy cost.
Model
#Param.
Accuracy[%]
#Add.
#Mult.
Energy
Spiking ResNet-34
22M
63.72
1.85G
118M
2.21 mJJ
Spiking MLP-SPE-T
25M
66.39
1.18G
12M
1.12 mJ
4.5 VISUALIZATION
Finally, we visualize weights from token blocks across different stages of spiking MLP-SPE-T after training on ImageNet-Iee IK and plot them in Figure Interestingly. can be observed that weights in the early stage concentrate on local areas while those of latter stages gradually sample more global areas. Without inductive bias, the SNN naturally learns hierarchically arranged receptive fields. In addition, receptive fields of spiking neurons in early stages resemble those 'off-center on-surround' receptive fields of cortex cells (Hubel Wiesel 1962). with central inhibitory area surrounded by an excitatory area. This 'M' shaped weight kernel is found universal in early stages of the network. would interesting to see similar phenomena can observed ANNs.
CONCLUSION
In this work, we constructed MLP architectures with spiking neurons following the MFI princi- ple. The spiking MLP-Mixer adopted batch normalization instead of layer normalization in both the token and the channel block to be compatible with MFI. We further strengthened the network's s local feature learning ability with spiking patch encoding layer, which significantly improved the network performance. Based on these building blocks, we explored an optimal skip connection con- figuration and developed an efficient multi-stage spiking MLP network combining global receptive field and local feature extraction. Under similar model capacity, our networks significantly outper- form state-of-the-art mainstream deep spiking convolutional networks on ImageNet-IK dataset in terms of a balance between accuracy, model capacity and computation cost, demonstrating the ef- fectiveness of such an alternative architecture for deep SNNs. Our work suggests the importance of integrating global and local learning for optimal architecture design of SNN. Finally, the trained receptive fields our network closely resemble those cells in the cortex. future comparison with ANNs would be interesting.

---

Under review as a conference paper at ICLR 2023
REFERENCES
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 911):147-165 1985.
Anonymous. Differentiable hierarchical and surrogate gradient search for spiking neural networks, in submission. Accepted by NeuralPS 2022.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-tor -e-ein in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neu- rons. Nature communications, 11(1):1-15, 2020.
Sander Bohte, Joost N Kok, and Johannes A Poutré. Spikeprop: backpropagation for networks of spiking neurons. In ESANN, volume 48, pp. 419-424. Bruges, 2000.
Romain Brette and Wulfram Gerstner. Adaptive exponential integrate-and-fire model an effective description of neuronal activity. Journal of neurophysiology. 94(5):3637-3642, 2005.
Xiangxiang Chu, Zhi Tian, Yuqing Wang. Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34:9355-9366 2021.
Gustavo Deco, Viktor K Jirsa, Peter A Robinson, Michael Breakspear, and Karl Friston. The dy- namic brain: from spiking neurons to neural masses and cortical fields. PLoS computational biology, 100002,,, 2008.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202. 1946, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors. Spikingjelly. https:: /gittb com/ fangwei123456/ spiking jelyy, 2020. Accessed: YYYY-MM-DD.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons networks and models of cognition. Cambridge University Press, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14. IEEE, 2014.
Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: A permutable mip-like architecture for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 2018.
10

---

Under review as a conference paper at ICLR 2023
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional archi- tecture in the cat's visual cortex. The Journal of physiology. 160(1):106, 1962.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Eugene M Izhikevich. Which model use for cortical spiking neurons? IEEE transactions on neural networks, 15()):1033-1070, 2004.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11270-11277, 2020.
Youngeun Kim, Joshua Chough, and Priyadarshini Panda. Beyond classification: Directly training spiking neural networks for semantic segmentation. arXiv preprint arXiv:2 2021.
Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks. arXiv preprint arXiv:2201. 10355, 2022.
Agnes Korcsak- Michael G Müller, Andreas Baumbach, Luziwei Leng, Oliver Breitwieser, Sacha J van Albada, Walter Senn, Karlheinz Meier, Robert Legenstein, and Mihai A Petrovici. Cortical oscillations support sampling-based computations spiking neural networks. PLoS computational biology, 18(3)=e1009753 2022.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). URL http:: httt::://ww.c.... toronto edu--kriz/ci html
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-9) 90, 2017.
Yann LeCun, Bemhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike- backpropagation for training deep neural network architectures. Fron- fiers in neuroscience, pp. 119, 2020.
Luziwei Leng, Roman Martel, Oliver Breitwieser, Ilja Bytschok, Walter Senn, Johannes Schemmel, Karlheinz Meier, and Mihai A Petrovici. Spiking neurons with short-term synaptic plasticity form superior generative networks. Scientific reports, 8(1):1-11, 2018.
Wenshuo Li, Hanting Chen, Jianyuan Guo, Ziyang Zhang, and Yunhe Wang. Brain-inspired multi- layer perceptron with spiking neurons. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 783-793, 2022.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng. Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv: 11880600055. 2018.
Ruiyang Liu, Yinghui Li, Linmi Tao, Dun Liang, and Hai-Tao Zheng. Are we ready for a new paradigm shift? survey on visual deep mlp. Patterns, 307):000520, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models.
Neural networks, 10(9):1659-1671, 1997.
11

---

Under review as a conference paper at ICLR 2023
Byunggook Na, Jisoo Mok, Seongsik Park, Dongjin Lee, Hyeokjun Choe, and Sungroh Yoon. Au- tosnn: Towards energy -fffcieeff spiking neural networks. arXiv preprint arXiv: 2022.
Emre Neftci, Srinjoy Das, Bruno Pedroni, Kenneth Kreutz-Delgado, and Gert Cauwenberghs. Event-driven contrastive divergence for spiking neuromorphic systems. Frontiers in neuroscience, 7:272, 2014.
Emre Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning spiking neural networks: Bringing the power of gradient-baset optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):5 1-63,,1-6366600031 2019.
Mihai A Petrovici, Johannes Bill, Iaja Bytschok, Johannes Schemmel, and Karlheinz Meier. Stochas- tic inference with spiking neurons in the high-conductance state. Physical Review E, 94(4):
042312, 2016.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks all you need. arXiv preprint (rXXx:2000.022177. 2020.
Ulysse Rançon, Javier Cuadrado-Anibarre Benoit Cottereau, and Timothée Masquelier. Stere- ospike: Depth learning with spiking neural network. arXiv preprint arXiv:2109, 109.:2100.,,51,, 2021.
Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.01800 2020.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- spike-based machine intelligence with neuromorphic computing. Nature, 575(7784)6077-617 2019.
Bodo Rueckauer, Iulia- Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chin Liu. Con- version of continuous- deep networks to efficient event-driven networks for image classifi- cation. Frontiers in neuroscience, 1 11:682, 2017.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time, arXiv preprint arXiv: 18Xivv18100.. 1et0.08810006 2018
Chuanxin Tang. Yucheng Zhao, Guangting Wang. Chong Luo, Wenxuan Xie, and Wenjun Zeng. Sparse mip for image recognition: is self-attention really necessary? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2344 2351, 2022.
Yuki Tatsunami and Masato Taki. Raftmlp: Do mlp- based models dream of winning over computer vision? arXiv e-prints, pp. arXiv-2108, 2021.
Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural networks, 1 111:47-63, 2019,
Iyya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34:24261- 24272, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Ziyu Wang, Wenhao Jiang. Yiming M Zhu. Li Yuan, Yibing Song, and Wei Liu. Dynamixer: vision mlp architecture with dynamic mixing. In International Conference on Machine Learning, pp. 22691-22701. PMLR, 2022.
12

---

Under review as a conference paper at ICLR 2023
Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Hol- zleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al. Modern hopfield networks and attention for immune repertoire classification. Advances in Neural Infor- mation Processing Systems, 33: 18832-18884, 2020.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training seghpperformance spiking neural networks. Frontiers in neuroscience. 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 33, pp. 1311-1318 2019.
Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 3066):1514-1541 2018.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems, 33: 33:12022-12033 2020.
Yucheng Zhao, Guangting Wang, Chuanxin Tang, Chong Luo, Wenjun Zeng. and Zheng-Jun Zha. battle network structures: empirical study of enn, transformer, and mlp. arXiv preprint arXiv:2108. 13002, 2021.
Hanle Zheng. Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11062-11070, 2021.
Lin Zhu, Xiao Wang. Chang, Jianing Li, Tiejun Huang, and Yonghong Tian. Event-based video reconstruction via potential-assistee spiking neural network. arXiv preprint arXiv:2201.1099 2022.
A APPENDIX
A.I DATA AUGMENTATION
For experiments on the ImageNet dataset, we inherited the default data augmentation from Tang al. (2022), which is different from the one He al. (2016) used in SNNs have compared with Zheng et al. 2021 Li et al. 2021 Deng et al. (022).. We denote the former as strong data augmentation while the latter simple data augmentation and perform the ablation study. The results are summarized in Table [ and the training curves of top-1 accuracy are plotted in Figure Our model performs slightly better with simple data augmentation, probably due suboptimal learning rate when we use strong data augmentation.
Table 5: Results comparison on ImageNet- 1K using simple and strong data augmentation.
Architecture
Augmentation
Accuracy[%]
MIP-SPP-T MLP-SPE-T
Strong
66.39
Simple
66.84
13

---

Under review as a conference paper at ICLR 2023
60
Test Top@1 Accurccy
50
40
30
Augmentation Method
20
Simple
Strong
10
40
Epochs
Figure 6: Training curves of top-1 accuracy of spiking MLP-SPE-T using simple and strong data augmentation.
14


Reviews:
[{"review_id": "pD4laKvZ1n", "reviewer": "Reviewer_2Yab", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a MLP based spiking neural network, termed spiking MLP-Mixer, which contains the Spiking Token Block, Spiking Channel Block, and Speaking MLP. The authors state that they achieve good performance on the Image-1k dataset, Cifar10, Cifar100. This work suggests the importance of integrating global and local learning for optimal architecture design of SNN. \n\n\n", "summary_of_the_review": "This paper proposes a MLP based spiking neural network, termed spiking MLP-Mixer, which contains the Spiking Token Block, Spiking Channel Block, and Speaking MLP. The authors state that they achieve good performance on the Image-1k dataset, Cifar10, Cifar100. This work suggests the importance of integrating global and local learning for optimal architecture design of SNN. But I think the proposed framework is not new, similar ideas can be found in previous works. \n", "strength_and_weaknesses": "Strength \n1. the direct training of deep SNN for energy-efficient classification is an interesting direction. \n\n\n\nWeaknesses \n1. the integration of SNN and MLP is not new in year 2022. [a]. Li, Wenshuo, et al. \"Brain-inspired multilayer perceptron with spiking neurons.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. \n2. the key problems this paper solved for deep SNN are not clear. In another word, the novelty of this paper is limited. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nC5K4ES2guS", "reviewer": "Reviewer_86ui", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces recent MLP-based artificial neural network architecture design to spiking neural networks. By replacing the commonly used layer normalization in MLP-Mixer with batch normalization, this paper proposes spiking MLP-Mixer with multiplication-free inference. A spiking patch encoding module is also proposed to enhance local feature extraction in the model. Experiments on static image classification tasks demonstrate competitive performance and energy efficiency.", "summary_of_the_review": "In summary, the experiment results are competitive, but the novelty and contributions are limited and there lack some important comparisons and experiments.", "strength_and_weaknesses": "Strengths:\n\n1. Experiment results demonstrate competitive performance and energy efficiency compared with convolution-based SNNs.\n\n2. Ablation study on skip connections and visualization of weights are studied to better understand the model.\n\nWeaknesses:\n\n1. The novelty and contributions of this paper are limited or hard to judge. There are two main designs in this paper: MLP-based design and spiking patch encoding module. The first one is directly adopted from recent ANN works, i.e. the recent popular patch-based methods such as transformers and MLP-Mixers. Replacing LN with BN is nothing new and has been studied in transformer architectures [1]. The design in this paper simply applies these techniques in ANNs to SNNs, without much specificity of SNNs. It is not clear what additional contributions this paper can bring. The second one, as said in the paper, is inspired from a recent work which is anonymous in the reference. However, this paper does not provide a copy manuscript in the supplementary material or discuss the difference in the main text, so it is hard to judge whether there is any novelty compared to that work.\n\n2. Experiments do not compare the spiking MLP-Mixer with the ANN counterpart, regarding performance and energy efficiency.\n\n3. There lacks comparison with some better results of previous SNN works, such as results with SEW-ResNet-34 architecture in [2,3] (67.04% and 68% on ImageNet), the state-of-the-art results on ImageNet with ResNet-34 architecture (68.19%) and VGG-16 architecture (71.24%) in [4], or the state-of-the-art results on CIFAR datasets in [5]. Since this paper focuses on network architecture design, the comparison with previous better architectures is required. The descriptions about the state-of-the-art results are not update-to-date.\n\nBesides, since transformer-like architectures usually requires more training techniques such as strong data augmentations, this should also be discussed if this paper uses a stronger training setting than previous works.\n\n4. There lacks ablation study on the proposed spiking patch encoding module. And also, since this module is inspired by the anonymous work, there should be comparison results to that work.\n\n5. There is no experiment on dynamic datasets commonly used for SNNs such as DVS-Gesture or DVS-CIFAR10.\n\n6. The authors should distinguish parenthetical and narrative types for references in the main text.\n\n[1] Yao, Z., Cao, Y., Lin, Y., Liu, Z., Zhang, Z., & Hu, H. (2021). Leveraging batch normalization for vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 413-422).\n\n[2] Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., & Tian, Y. (2021). Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34, 21056-21069.\n\n[3] Deng, S., Li, Y., Zhang, S., & Gu, S. (2021, September). Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In International Conference on Learning Representations.\n\n[4] Li, Y., Guo, Y., Zhang, S., Deng, S., Hai, Y., & Gu, S. (2021). Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 23426-23439.\n\n[5] Meng, Q., Xiao, M., Yan, S., Wang, Y., Lin, Z., & Luo, Z. Q. (2022). Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12444-12453).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aqqLzspzoRU", "reviewer": "Reviewer_saxA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors transplant the idea of spiking neurons to MLP. ", "summary_of_the_review": "In summary, this paper's attempt is worthy to be credited but it really lacks good motivation. ", "strength_and_weaknesses": "Strength: \n\n+ The authors ensure multiplication-free inference.\n\n+ This work may be the first attempt for combining LIF and MLP.\n\nWeakness: \n\n- The motivation for this work is not convincing for me. In the abstract, the authors claimed that the current MLP does not support MFI and that is why they want to improve spikingMLP. For me, I think the real advantage of applying SpikingMLP is whether it is extremely highly efficient compared to SpikingCNN in training or inference; whether SpikingMLP has higher transferability than SpikingCNN; or whether SpikingMLP has less gap between ANN and SNN accuracy than SpikingCNN. The current work looks like a simple combination. \n\n- Section 3.2 said  *This structure leads to excessive number of parameters which can cause an over-fitting of the network when training on medium-scale dataset such as ImageNet-1K. To alleviate this issue, we adopt a multi-stage pyramid network architecture as Liu et al. (2021); Chu et al. (2021); Tang et al. (2022).* This questions me again for the motivation of this paper. If simply applying the global architecture like MLP-Mixer the SNN will become much worse, does that mean spiking neurons are not suitable for MLP architecture (no inductive bias one) at all? More concretely, I am curious what is the compatibility of LIF neurons for MLP structure, is it strongly relied on inductive bias or it can learn inductive bias by itself?\n\n- I think the accuracy cannot be interpreted very well by readers. For example, in the original MLP Mixer results table, there are inference throughput and training days compared to CNN and transformer. If possible, can authors also provide these two hardware performances since they are completely different architectures? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PdzCpd2QNiP", "reviewer": "Reviewer_8C6v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors train spiking neural networks (SNNs) for image classification using the surrogate gradient method. They propose a MLP architecture, where (strided) convolutions are only found in the patch encoding stages. The rest of the network uses linear layers only (no attention either). They report results on CIFAR10, CIFAR100 and ImageNet.", "summary_of_the_review": "The method is competitive on CIFAR but not on ImageNet.", "strength_and_weaknesses": "STRENGTHS:\n\nAccuracy on CIFAR10 and 100 is excellent.\n\nWEAKNESSES:\n\nThe method is not competitive on ImageNet, suggesting that it does not scale well. Fang et al is cited, but does not appear in Table 1! Their SEW ResNet50 reaches 67.8% with 4 timesteps, vs 66.4% here.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CA9y6TTZGq", "reviewer": "Reviewer_BQAF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an interesting training algorithm for training SNNs from scratch. It uses a combination of skip connections and BN tools to get better accuracy on image recognition tasks.", "summary_of_the_review": "Interesting paper, but as expressed in my concerns- i am not very convinced about the novelty and especially the energy efficiency advantages of the work. ", "strength_and_weaknesses": "+ Very comprehensive results \n\n+ Simple yet effective idea\n\n- Since the authors use a BN technique, I am wondering if the authors can shed light on how their method differs from previous temporal BN methods proposed by prior works that have shown accuracy improvement while decreasing the total timesteps [1, 2].\n[1]Kim, Y., & Panda, P. (2020). Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, 1638.\n[2] Zheng, Hanle, et al. \"Going deeper with directly-trained larger spiking neural networks.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n\n- There is a plethora of works today on SNN algorithmic training-  precisely talking about how we can get improved accuracy with less timesteps. But, I am more concerned by the fact that in such large-scale settings, are SNNs going to be actually advantageous? The authors show some simplistic energy estimation results which is grossly approximate. For true energy estimation, they have to consider memory access and data access energy which turns out to expend a lot of computations in SNNs given their repeated time-wise computation. In a recent work [3], true energy estimation on a systolic accelerator precisely shows that SNNs are not very advantageous over ANNs because repeated timestep computations will lead to redundant access of weights and membrane potentials is going to further add to energy unless we really improve the sparsity rate. Can the authors kindly comment on this -  and it may be worthwhile for authors to include a discussion n the relevance of using more mainstream tools for energy estimation rather than just doing analytical modeling of FLOPS count?\n[3] Yin, Ruokai, et al. \"SATA: Sparsity-Aware Training Accelerator for Spiking Neural Networks.\" arXiv preprint arXiv:2204.05422 (2022).\n\n- Coming to my next point, there is a recent work [4] that explores sparse SNNs using lottery ticket hypothesis to truly take advantage of SNNs energy efficiency over ANNs. Can the authors comment on how their model in terms of parameter count compares to these sparse SNN models which in fact show SOTA accuracy on CIFAR10,100 with very low timestep count?\n[4] Kim, Youngeun, et al. \"Lottery Ticket Hypothesis for Spiking Neural Networks.\" arXiv preprint arXiv:2207.01382 (2022).\n\n- Finally, I think it is well known that SNNs will be more suited to DVS or event based tasks as compared to standard digital camera recognition models. Recent works have shown superiority of SNNs over ANNs on these neuromorphic datasets [5, 6, 7]. Can the authors run their model on one of these datasets and compare to [5,6,7]?\n[5] Li, Yuhang, et al. \"Neuromorphic Data Augmentation for Training Spiking Neural Networks.\" arXiv preprint arXiv:2203.06145 (2022).\n[6] Kim, Youngeun, and Priyadarshini Panda. \"Optimizing deeper spiking neural networks for dynamic vision sensing.\" Neural Networks 144 (2021): 686-698.\n[7] Kim, Y., Chough, J., & Panda, P. (2022). Beyond classification: directly training spiking neural networks for semantic segmentation. Neuromorphic Computing and Engineering.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject