Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient
Authors: Yuhang Li
Youngeun Kim
Hyoungseob Park
Priyadarshini Panda
Venue: Accepted by TMLR ()

Abstract: Spiking Neural Networks (SNNs) are recognized as the candidate for the next-generation neural networks due to their bio-plausibility and energy efficiency. Recently, researchers have demonstrated that SNNs are able to achieve nearly state-of-the-art performance in image recognition tasks using surrogate gradient training. However, some essential questions exist pertaining to SNNs that are little studied: Do SNNs trained with surrogate gradient learn different representations from traditional Artificial Neural Networks (ANNs)? Does the time
dimension in SNNs provide unique representation power? In this paper, we aim to answer these questions by conducting a representation similarity analysis between SNNs and ANNs using Centered Kernel Alignment (CKA). We start by analyzing the spatial dimension of
the networks, including both the width and the depth. Furthermore, our analysis of residual connections shows that SNNs learn a periodic pattern, which rectifies the representations in SNNs to be ANN-like. We additionally investigate the effect of the time dimension on SNN representation, finding that deeper layers encourage more dynamics along the time dimension. We also investigate the impact of input data such as event-stream data and adversarial attacks. Our work uncovers a host of new findings of representations in SNNs. We hope this work will inspire future research to fully comprehend the representation power of SNNs. Code is released at https://github.com/Intelligent-Computing-Lab-Yale/SNNCKA.

Published in Transactions on Machine Learning Research (04/2023)
Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient
Yuhang Li Yale University
yuhang.li@yole edu
Youngeun Kim Yale University
youngeun.kimm@@ykee.cmm edu
Hyoungseob Park Yale University
byyoungseob.prrk@yyll....
Priyadarshini Panda Yale University
prige.ponda@yole. edu
Reviewed on OpenReview: https: // openreview. net/ forum? id- s9ef0F9QW1
Abstract
Spiking Neural Networks (SNNs) recognized as the candidate for the next-generation neural networks due to their bio-plausibility and energy efficiency. Recently, researchers have demonstrated that SNNs are able to achieve nearly state-of-the-art performance in image recognition tasks using surrogate gradient training. However, some essential questions exist pertaining to SNNs that are little studied: Do SNNs trained with surrogate gradient learn different representations from traditional Artificial Neural Networks (ANNs)? Does the time dimension in SNNs provide unique representation power? In this paper, we aim answer these questions by conducting a representation similarity analysis between SNNs and ANNs using Centered Kernel Alignment (CKA). We start by analyzing the spatial dimension of the networks, including both the width and the depth. Furthermore, our analysis of residual connections shows that SNNs learn a periodic pattern, which rectifies the representation in SNNs to be ANN-like. We additionally investigate the effect of the time dimension on SNN representation, finding that deeper layers encourage more dynamics along the time dimension. We also investigate the impact of input data, such as event-stream data and adversarial attacks. Our work uncovers host of new findings of representation in SNNs. We hope this work will inspire future research to fully comprehend the representation power of SNNs. Code is released at https: wttpabbbcco/tteelleeeeteoooooote SNNCKA.
Introduction
Lately, Spiking Neural Networks (SNNs) (Tavanaei al., 2019; Roy et al., 2019; Deng al., 2020; Panda et al., 2020; Christensen et al., 2022) have reccived increasing attention thanks to their biology-inspired neuron activation and efficient neuromorphic computation. SNNs process information with binary spike representation and therefore avoid the need for multiplication operations during inference. Neuromorphic hardware such TrueNorth (Akopyan et al., 2015) and Loihi (Davies et al., 2018) demonstrate that SNNs can save energy by orders of magnitude compared to Artificial Neural Networks (ANNs).
Although SNNs can bring enormous energy efficiency in inference, training SNNs is notoriously hard because of their spiking activation function. This function returns zero-but-all gradient (i.e., Dirac delta function) and thus makes gradient-base optimization infeasible. To circumvent this problem, various training tech- niques have been proposed. For example, piike-tining-depeddee plasticity (STDP) (Rao & Sejnowski, 2001)

---

Published in Transactions on Machine Learning Research (04/2023)
Artificial Neural Network
Ciyy
Xu Xaj
RelMI
Refal
Preact Feature 1
Preact Feature 2
Presect Pecature 3
CKA Similarity
Correlation Gram matrix
Preact Feature 1
Preact Feature
Preant Feature
LIF
Test Images
ΣΣ
x
Spiking Neural Network
Figure 1: The representation similarity analysis workflow. The test images are fed into both ANN and SNN, then we record the intermediate feature for computing the correlation matrix, which is used for inferring the CKA similarity (Korublith et al., 2019).
either strengthens or weakens the synaplic weight based on the firing time; Line-to-first-spike (Mostafa, 2017) encodes the information into the time of spike arrival to get a closed-form solution of the gradients. However, these two methods are restricted et al., 2013; Bender et al., 2018; Wu et al., 2018; Bellec et al., 2018; Kim et al., 2022b; Li et al., 2022), on the other hand, can achieve the best task performance by applying agation. Combined with surrogate gradient, SNNs be optimized by Backpropagation Through Time (BPTT) (Neftei et al., 2019) algorithm, outperforming other learning rules in SNNs.
small-scale tasks and datasets. Surrogate gradient technique (Bengio
alternate function during backprop-
Despite increasing interest in pursuing high-performance SNNs with surrogate gradient training, there is limited understanding how surrogate gradient training affects the representation of SNNs. Investigating this fundamental question is critical since surrogate gradient- based BPTT algorithm mimies the way how compared other learning rules like STDP. Therefore, would
ANN learns and is less bbologicl-pplassin be intriguing to study whether surrogate gradient-based SNNs learn different representations than ANNs. Understanding the representation learned in SNN can also promote further research developments, e.g., designing spiking-frient friendly architectures (Kim et al, 2022a; Na et al., 2022) and exploring other ways to optimize SNNs (Bellec et al., 2020; Zhang & Li, 2020; Kim & Panda, 2021a).
More concretely, we ask, do SNNs optimized by surrogate gradient. BPTT learn distinct representations from ANNs? How do the width and depth of the neural network affect the representation learned in SNNs and ANNs? Does the extra temporal dimension in SNNs yield unique intermediate features? On neuromorphic datasets, how does the SNN process event-based data? In this paper, aim to answer these core questions through a detailed analysis of ResNets (He et. al., 2016a) and VGG-series (Simonyan & Zisserman, 2015) models using a representation similarity analysis tool. Specifically, we utilize the popular Centered Kernel Alignment (CKA) (Korublith et al., 2019) to measure the similarity between SNNs and ANNs. Fig. I demonstrates the overall workflow of our representation similarity analysis framework. Our analysis spans both the spatial and temporal dimensions of SNNs, as well as the impact of network architecture and input data.
Our contributions and findings include:
We analyze the representation similarity between SNNs and ANNs using the centered kernel alignment to determine whether SNNs produce diflerent feature representations from ANNs. We examine various aspects of representation similarity between SNNs and ANNs, including spatial and temporal dimensions, input data type, and network architecture.
Surprisingly, our findings show that SNNs trained with surrogate gradient have rather similar represen- tation to ANNs We also find that residual connections greatly affect the representations in SNNs.
2

---

Published in Transactions on Machine Learning Research (04/2023)
Meanwhile, we find that the time dimension in SNNs does not provide much unique representation. We also find that shallow layers are insensitive to the time dimension, where the representation in each time step converges together.
2 Related Work
Spiking Neural Networks (SNNs). SNNs have gained increasing attention for building low- low-power intel- be divided into two categories: (1) ANN-SNN conversion (Rueckauer et al., 2016; 2017; Han et al., 2020; Sengupta et al., 2019; Han & Roy, 2020) and (2) direct training SNN from scratch (Wu et al., 2018; 2019). Conversion- based methods utilize the knowledge from ANN and convert the ReLU activation to a spike activation mechanism. This type of method can produce an SNN in a short time. For example, in Rueckaner et al. (2017), one can find the percentile number and set it as the threshold for spiking neurons. Authors in Deng & Gu (2021) and Li et al. (2021a) decompose the conversion error to each layer and then propose reduce the error by calibrating the parameters. However, achieving near-lossless conversion requires a considerable amount of time steps to accumulate the spikes. Direct training from scratch allows SNNs to operate in extremely low time steps, even less than 5 (Zheng et al., 2020). To enable gradient-based learning, direct training leverages surrogate gradient to compute the derivative of the discrete spiking function. This also benefits the choice of hyper- parameters in spiking neurons. Recent works (Fang et al., 2021; Rathi & Roy, 2020; Kim & Panda, 2021b; Deng el al., 2022) co-optimize parameters, firing threshold, and leaky factor logether via gradient descent. Our analysis is mostly based on directly trained SNNs, as converted SNNs only contain ANN features and may be misleading for representation comparison.
ligence. Generally, the SNN algorithms to obtain high performance
Representation Similarity Analysis (RSA). RSA (Kriegeskorte al., 2008) was not originally de- signed for analyzing neural networks specifically. Rather, it is used for representation comparison between any two computational models. Prior works such as Khaligh-Raas
& Kriegeskorte (2014); Yamins et al. (2014) have used RSA to find the correlation between visual cortex features and convolutional neural network features. Authors of Seminar (2016); Raghu et al. (2017); Morcos et al. (2018); Wang et al. (2018) have studied RSA between different neural networks. However, recent work (Kornblith et al., 2019) argues that none of the above methods for studying RSA can yield high similarity even between two different initializa- tions of the same architecture. They further propose CKA, which has become powerful evaluation tool for RSA and has been successfully applied in several studies. For example, Nguyen et al. (2020) analyzes the reprosentation pattern in extremely deep and wide neural networks, and Raghu et al. (2021) studies the representation difference between convolutional neural networks and vision transformers with CKA. In this work, we leverage this tool to compare ANNs and SNNs.
3 Preliminary
3.1 ANN and SNN Neurons
In this paper, vectors/matrices are denoted with bold italic/capital letters (e.g. x and W denote the input vector and weight matrix, respectively). Constants are denoted by small upright letters. For non-linear activation function in artificial neurons, use the rectified linear unit (ReLU) (Krizhevsky et al., 2012), given by y max(0, Wæ). As for the non-linear activation function in spiking neurons, we adopt the well- known Leaky Integrate-and-Fing (LIF) model. Formally, given a membrane potential (!) at time step t and a pre-synaptic input j((+1) = Wx(++)), the LIF neuron will update as
u (++1),pr = ( + + g(+11) y (( = [1 -f1 ( otherwise
if (((++))pm > veh u -+1)= - (1 (1- (t+1))))).
(1)
Here, (t+)), the pre-synaptic membrane potential, the firing threshold, the LIF neuron will fire a spike (+(++) = 1) when the membrane potential exceeds the threshold; otherwise, it will stay inactive (y(t-1) 0). After firing, the spike output (1+1) will propagate to
constant leak factor within (0, 1). Let Via be
3

---

Published in Transactions on Machine Learning Research (04/2023)
the next layer and become the input x( of the next layer. Note that here the layer index is omitted for simplicity. The membrane potential will be reset to O if spike fires (refer to Eq. (1) the third sub-equation).
3.2 Optimize SNN with Surrogate Gradient
To enable gradient descent for SNN, we adopt the BPTT algorithm (Werbos, 1990). Formally, denote the loss function value as L, the gradient of the loss value with respect to weights can be formulated by
OL aW
aL 0yl K(O where K(O)_
((Potto (uu(0,p oi
Oul).e Sult-1) K(t k(-") (2)
oy(0) Ou(0)pre
0:00)
OW
dult-1) dult-1),pre
Here, the gradient is computed basod on the output spikes from all time steps. In each time step, we denote K as the gradient of pre-synaptic membrane potential with respect to weights aW which consists of the gradient of pre-synapiic input and the gradient of membrane potential in the last time steps.
Oyte
By
As a matter of fact, all terms in Eq. (2) can be easily differentiated except Fu which returns a zero- but-all gradient (Dirac delta function). Therefore, the gradient descent ends up either freezing the weights or updating weights infinity. To address this problem, the surrogate gradient proposed (Bender et al., 2018; Wu et al., 2018; Bellec el. al., 2018; Neftci el al., 2019; Li et al., 2021b) to replace the Dirac delta function with another function:
0y 1
1u pe mal<o
Or.p a where a is a ayper-parameter for controlling the sharpness and a = 1 the surrogate gradient becomes the Straight-Through Estimator (Bengio et al., 2013).
Compared to other methods such as ANN-SNN conversion (Deng & Gu, 2021) or spike--ining--eepeeeee plasticity (Caporale al., 2008), BPTT using surrogate gradient learning yields the best performance in image recognition tasks. However, from biological perspoctive, BPTT is implausible: for cach weight update, BPTT requires the use of the transpose of the weights to transmit errors backward in time and assign credit for how past activity affected present performance. Rumning the network with transposed weights requires the network to either have two-way synapses or use symmetric copy of the feedforward weights to backpropagate the error (Marschall et al., 2020). Therefore, the question remains whether the representation in SNNs learned with surrogate gradient- baade BPTT actually differs from the representation in ANNs.
3.3 Centered Kernel Alignment
Let X, E RRTTP and X, E mxpx contain the representation in an arbitrary layer of SNN with Pi hidden neurons across T time steps and the representation an arbitrary layer of ANN with Pa hidden neurons, respectively. Here m is the batch size and we concatenate features from all time steps in the SNN altogether. We intend to use similarity index s s(x,, X.) to describe how similar they are. We use the Centered Kernel Alignment (CKA) (Kornblith et al., 2019) to measure this:
HSIC(K, L)
1
CKA(K,L) HSICKK, K)HSIC(LLL)
HSIC(K, L)
tr (HLH))
(4)
(m 1)2
are the Gram matrices as shown in Fig. Each Gram matrix has the shape of m m, reflecting the similarities between pair of examples. For example, Kij indicates the similarity between the tt and jth example in the SNN feature X,, Further measuring the similarity between K and L, one can measure whether SNN has similar inter-example similarity matrix with ANN, Let H = I- _11 be the centering matrix, the Hilbert-Schmidt Independence Criterion (HSIC) proposed by Gretton et al. (2005) can conduct test statistic for determining whether two sets of variables are independent. HSIC = 0 implies independence. The CKA further normalizes HSIC to produce a similarity index between 0 and (the higher the CKA, the more similar the input pair) which is invariant to isotropic scaling. In our implementation, we use the unbiased estimator of HSIC (Song al., 2012; Nguyen al., 2020) to calculate it across mini-batches.
Here, K XXX,XX
4

---

Published in Transactions on Machine Learning Research (04/2023)
Artifi Rosket
60
Artificial ResNet-35
125
1.00
Artifcaal Resiver-5
150
Attifcal Reshet 110
300
Artifcal Nestet 114
-00
400
200
300
200
04
20
100
25
100
20
40
60
%
50
100
0
50
100
150
160
200
300
200
400
600
Spiking ResNet-20
Spiking ResNet-36
Spiking
ResNet-56
Spiking
ResNet-110
Spiking ResNet-164
Artitiiaal ResNeet220 x2
Artificial Rechet-220 -20 x4
Artificill ResNet-20 x8
Artificiall Rostet- 00000-20xx0 x16
Artificial Restet- 22
00
40
40
+0
04
20
20
2
70
20
40
40
60
60
Spiking
Spiking ResNet-22
Spiking ResNet-20 x4
Spiking ResNet-20 x8
Spiking ResNet-20 x16
0.5
0.9
1.0
0.0
100
120
0.7
Monebee /
0.6
0.0
0.5
0.5
QUVI Ho) (
1330
0.4
1.0
0.3
2e
ResNNet0
RosNe120
ResNet20
10
30
40
50
70
Layers
Layers
Figure CKA heatmap between SNNs and ANNs with different depth and width on the CIFAR-10 dataset. Top: the CKA cross-layer heatmap across different depths from 20 layers to 164 layers. Middle: the CKA caoss-layer heatmap across different widths from the original channel number 16 times. Bottom: visualizing only the corresponding layer, which is the diagonal of the CKA heatmap. We find generally SNNs and ANNs have relatively high similarity, and deeper/wided networks have susitive/negative effects on the representation similarity.
4 Do SNNs Learn Different Representation from ANNs?
In this section, we comprehensively compare the representation learned in SNNs and ANNs. Our primary study case is ResNet. with identity mapping block (He et al., 2016b) on the CIFARIO dataset, which the standard architecture and dataset in modern deep learning for image recognition. There are two differences between SNNs and ANNs. First, ANNs adopt the Batch Normalization layer (Ioffe & Szegedy, 2015), and SNNs use the lime-dependent Batch Normalization layer (Zheng el al., 2020), which normalizes the feature across all time steps (i.e. Xs). Second, the ANNs use ReLU activation, and SNNs leverage the LIF spiking neurons. For default SNN training, we use direct encoding, for the firing threshold, T = 4 for the number time steps, and a = 1.0 for surrogate gradient, which are tuned for the best training performance on SNNs. Detailed training setup and codes can be found in the supplementary material.
0.5 for the leaky factor, wth = 1.0
4.1 Scaling up Width or Depth
We begin our study by studying how the spatial dimension of model architecture affects internal represen- tation structure in ANNs and SNNs. We first investigate a simple model: ResNet-20, and then we either increase its number of layers or increase its channel number to observe the effect of depth and width, re- spectively. In the most extreme cases, we scale the depth to 164 and the width to 16x (see detailed network configuration in Table D.1). For each network, we compute CKA between all possible pairs of layers, includ-
We also provide RSA VGG-series networks in Sec. A.1 and RSA on CIFAR100 dataset in Sec. A.2.
5

---

Published in Transactions on Machine Learning Research (04/2023)
Table 1: The top-1 accuracy of SNNs and ANNs on CIFAR-10 dataset, as well as the gap A between ANN and SNN.
Network
Depth
Width
20
38
56
110
164
x1
x2
x4
x8
x16
ANN
91.06
92.34
92.98
92.37
93.00
91.06
93.36
94.52
94.78
94.78
SNN
89.67
91.14
91.94
91.83
92.05
89.67
92.00
93.96
94.48
94.73
A-S
1.39
1.20
1.04
0.53
0.95
1.39
1.36
0.56
0.30
0.05
ing convolutional layers, normalization layers, ReLU//IF layers, and residual block output layers. Therefore, the total number of layers is much greater than the stated depth of the ResNet, as the latter only accounts for the convolutional layers in the network. Then, we visualize the result as a heatmap, with the æ and y axes representing the layers of the network, going from the ipput layer to the output layer. Following Nguyen et al. (2020), our CKA heatmap is computed on 4096 images from the test dataset.
As shown in Fig. 2, the CKA heatmap emerges a checkerboard-like grid structure, especially for the deeper neural network. In ResNet-20, we observe a bright block in the middle and deep layers, indicating that ANNs and SNNs learn overlapped representation. As the network goes deep, we find the CKA heatmap becomes darker, meaning that representations in ANNs and those in SNNs are diverging. Notably, we find a large portion of layers artificial ResNet-164 exhibit significantly different representations than spiking ResNet-164 (<0.2 CKA value) which demonstrates that deeper layers tend to learn disparate features.
In Fig. 2 middle part, we progressively enlarge the channel number of ResNet-20. In contrast to depth, the heatmap of wider neural nelworks becomes brighter, which indicates the represenlations in SNNs and ANNs are converging. Interestingly, although the majority of layers are learning more similar representations between ANN and SNN in wide networks, the last several layers still learn different representations.
We further select only the diagonal elements in the heatmap and plot them Fig. bottom part. Because SNNs and ANNs have the same network topology, this visualization more specific and may accurately reveal the similarity between SNNs and ANNs at each corresponding layer. First, we can find that in Fig. 2 the CKA curve of ResNet-20 shows relatively high values. Most layers go above 0.5 and some of them can even reach nearly 1.0. Interestingly, we observe that deeper networks tend to derive a curve with a jagged shape. This means some layers in SNN indeed learn different representations when compared to ANN, however, the difference is intermillently miligaled. In later sections, we will show that the miligation of dissimilarity is performed by residual connection. As for width, we generally notice that CKA curves mostly become higher for wider networks, especially when comparing ResNet-20 and ResNet-20 x8, where most layers have above 0.8 CKA value.
Evaluating the Task Performance. We lay out the accuracy of ANNs and SNNs in Table 1. Additionally, we also calculate their accuracy difference across different width and depth configuratinns. We can find that the accuracy gap decreases if we scale up the width, ranging from 1.39% to 0.05%. However, as the depth increases, the accuracy gap does not consistently reduce as it does in wider networks. This explains that wider neural networks which bring more similar representations encourage less accuracy gap. However, SNNs do not achieve the same accuracy level as ANNs with different representations. This finding means that current, state-of-the-art SNNs relying on wider neural networks (e.g., ResNet-19 (Zheng et al., 2020) which is similar to our ResNet-20 8x) do not truly develop unique feature representations than ANNs.
4.2 The Effect of Residual Connections
In Fig. 2, the CKA curves appear with a periodic jagged shape. To investigate what causes this similarity oscillation, we igale each layer in a residual block. In Fig. 3 left, we plot the CKA curve of the ResNet- 110 and additionally sample two residual blocks, the 10-th and the 34-th block, whose architecture details are depicted in Fig. 3 right. Surprisingly, we find that every time when the residual conncction meets the main branch, the CKA similarity restores nearly to 1. Moreover, every time when the activation passes a convolutional layer or LIF layer, the similarity decreases. The BN layers, in contrast, do not affect the
6

---

Published in Transactions on Machine Learning Research (04/2023)
3.0
Averaged from blcoks
10-th Residua Block
34-th Residual Block
1.00
1.00
2.5
0.75
0.75
0.5
0,50
0.50
LIF/ReLU
2.0
Conv (td)8N
Similarity
0.25
1.5
0.00
70
244
246
1.0
0.5
LIF/ReLU
Conv
(td)BN
0.0
50
150
200
250
300
Layers
Figure 3: Emergence of periodic jagged CKA curve. Left: CKA curve of ResNet-110. We subplot the 10-th and the 34-th residual blocks ResNet-110, which forms a periodic jagged curve. Top right: The averaged CKA value in all blocks, Bottom right: The architecture specification of the residual block used.
Spiking
Spiking
Spiking Residual
Spiking Residual
150
150
150
150
Layers
100
100
50
50
50
0
100
150
0
100
0
100 Layers
0
0.0
Layers
Layers
Layers
Figure The effect of residual connections in SNNs. We remove residual connections in of three stages in the Re and show the CKA heatmaps. The non-residual stage is annotated with green square D.
similarity since it is linear transformation. These results substantiate that the convolutional layers and LIF layers in SNNs are able to learn different represenlations thon ANNs. However, the representution in the residual branch still dominates the representation in post-residuol layers and leads to the junction of ANN's and SNN's representation.
To further explore why residual connections can restore the representations in SNNs ANN-like, we conduct an ablation study, We select one of the three stages in the spiking ResNet-56 where the residual connections are disabled selectively. In Fig. y axes the same layers in SNN. The first heatmap demonstrates the full residual network, while the remaining three heatmaps show the partial residual networks, with the 1st, 2nd, and 3rd stages disabled, respectively. Our observations can be summarized as follows: (1) In terms of inter-stage similarity, residual connections can preserve the input information from the previous stage. In the 1st and 2nd heatmaps in Fig. 4, we find residual blocks can have high similarity with their former stage. The non-residual block, however, does not have this property. In the 3rd and 4th heatmaps, we can see that blocks without residual connections exhibit significantly different representation can find that residual connections preserve the representation in early layers. As such, if ANN and SNN learn similar representations in the first layer, the similarity can propagate to very deep layers due to residual connections. (2) In terms of intra-stage similarity, the non-residual stage's heatmap appears with a uniform representation across all layers, meaning that layers in this stage are similar. In contrast, residual stages share a grid structure.
visualize the CKA heatmaps of SNN itself, which means both and
when compared to their former stage. Therefore,
Next, we verify the accuracy of SNNs and ANNs when both are equipped with residual connections or not, under different network depths. As shown in Table 2, both the SNNs and ANNs successfully train very

---

Published in Transactions on Machine Learning Research (04/2023)
Table The impact of residual connections on accuracy.
Depth
Model
Type
20
38
56
74
ANN
W
residual connection
91.06
92.34
92.98
92.85
/o residual connection
91.32
91.17
89.62
21.07
residual connection
89.63
91.14
91.94
91.83
SNN
w /0 residual connection 86.50
82.64
33.61
10.00
deep networks if the residual connections enabled. In this case, though SNNs do not surpass the accuracy of ANNs, the gap is relatively small, with 1~2% accuracy degradation. However, if the residual connections are removed from SNNs, the gap between the accuracies of ANNs and SNNs significantly enlarges, ranging from 5-56%. Therefore, we can conclude that the residual connections help the gradient descent optimization SNNs and regularize the representations in SNNs to be similur to those in ANNs so that SNNs can have similor task performances with ANNs.
4.3 Scaling up Time Steps
The results the previous sections help characterize the effects of spatial structure on internal represen- tation differences between SNNs and A.Ns.. Next, we ask whether the time dimension helps SNN learn some unique information. To verify this, we train several spiking ResNet-20 with 118/16/32/64/11/ time steps and calculate the ANN-SNN CKA similarity. In Fig. 5, envision the CKA heatmaps and curves respectively between artificial ResNet-20 and spiking ResNet-20 with various time steps. Notably, cannot find significant differences among these heatmaps. Looking at the CKA curves, we also discover that many layers are overlapped, especially when we focus on the residual block oulput (the local maximumns.. We find similarities between different time steps reaching the same point, meaning that the time step variable does not provide much unique representation SNNs.
To further analyze the representation ion along the time dimension in SNNs, we compare the CKA among various time steps. Concretely, for any layer inside an SNN, we reshape the feature X, to [X(1) x ...., x()) where ( is the i-th time step's output. By computing the CKA similarity between arbitrary two time steps, i.e., CKA(X()) X()),, we are able to construct a CKA heatmap with x, y axes being the time dimension, which demonstrates whether the features are similar across different time steps. Fig. 6 illustrates such CKA heatmaps of outputs from all residual blocks in the spiking ResNet-20, with time steps varying from 4 to 32. In general, deeper residual block output exhibits darker CKA heatmaps and the shallower layers tend become yellowish. In particular, all the residual blocks from the first stage have an all-yellow CKA heatmap,
70
70
1.0
60
0
0
60
0.9
40
40
0.8
Attiicaa
Atticcal
20
20
10
Similarity
0.7
0.6
Spkking
Spiking Re:Net-20
70
70
Artiicaa Reziet 20
80
0.5
50
50
40
40
40
0.4
ResNet-20
ResNet-20
888
20
20
0.3
ResNet-22 T16
ResNet-20
ResNet-20 T128
ResNet:20 20 T64
15
0.2
60
20
30
40
ResNect 20
Spiking ResNet:20 T64
Spiking ResNet-20 T128
Layers
Figure 5: The effect of time steps in SNNs. Left: CKA heatmaps between ANNs and SNNs with the different number of time steps. Right: The CKA curve of corresponding layers (diagonal values as in left).
8

---

Published in Transactions on Machine Learning Research (04/2023)
2
10
06
2
00
20
20 20 20
20
20
20
Figure The similarity across times in SNN. Each heatmap shows the CKA among different time steps the output of the residual block. "s" means stage, and "b" means block. The top/ htp/middle/battonn rows are spiking ResNet-22 20 with 4/8/32 time steps, respectively.
Table 3: The sensitivity of time steps in SNNs.
# Time Steps
Model
Type
4
8
16
32
Full time steps
89.67
90.44
90.98
90.99
SNN
Reduce the time steps in the first stage
88.81
89.70
89.91
90.47
Reduce the time steps in the last stage
87.41
87.78
87.38
87.73
indicating extremely high similarity in these blocks. The second stage starts to produce differences across time steps, but they still share >0.8 similarities between any pair of time steps. The last stage, especially the last block, demonstrates around 0.5 similarities between different time steps. To summarize, the impact of time in SNN is gradually increased as the feature propagates through the network. In Appendix A.4, we provide the heatmap convolutional/LII layers and find a similar trend.
We further conduct an empirical verification verify the findings in Fig. More specifically, we define sensitivity metric and measure it by reducing the number of time steps to 1 in certain layers of SNN and recording the corresponding accuracy degradation. In Fig. 6 we find the first stage (sl) has the same representation in time dimension while the last stage (s3) exhibits a more diverse representation. Therefore, we choose to reduce the number of time steps to 1 either in s1 or in s3. To achieve this "mixed -mixed-time- step SNN', we repeat/averse /average the activation in time dimension after sl sl/bofore s3 to match the dimension. Table summarizes the sensitivity results. We observe that the first stage can have much lower accuracy degradation (<1%), while the last stage drop 2~4% accuracy. Moreover, if the last stage only uses 1 time step, then increasing the time steps for the other two stages does not benefit the accuracy at all. This indicates that LIF neurons more effective in deep layers than in shallow layers.
4.4 Representation under Event Data
In this section, evaluate the CKA similarity the event-based dataset. We choose CIFARIO-DVS (Li et al., 2017), N-Caltech 101 (Orchard et al., 2015) and train spiking/artifice ResNet-20. Since the ANN cannot process 4-dimensional spatial-temporal event data easily, we integrate all events into one frame for ANN training and 10 frames for SNN training. Fig. 7 provides the CKA heatmaps/ccuree on the event datasot, showing a different similarity distribution than the previous CIFAR-10 datasct. The heatmaps have a different pattern and the curves also do not appear with a periodic jagged shape. In addition, the similarity distribution differs at the dataset level, Lee, the CIFAR10-DVS and N-Caltech 101 share different

---

Published in Transactions on Machine Learning Research (04/2023)
CIFAR10-DVS
N-Caltech 101
Figure 7: CKA Similar- ity Event Dataset. We train spiking and artificial ResNet-20 on CIFARIO-DVS and N-Caltech 101, respec- tively. Left: the CKA heatmaps. Right:
13
Artffccaa ResNet-29
Artifici ResNet-20
0.6
-0
20
16
0.5
The CKA curves of corresponding lay- ers between ANN and SNN.
10
15
0.4
FIARO DVS tach lo1
60
0.0
20
26
30
Spiking ResNet-20 T10
Spiking ResNet-22 T10
Layers
Robustness of ANN and SNN
01.77
1.0
Similarity of ANN
1.0
Similarity of SNN
Figure 8 The robust- against adversarial attack. Left: The accu- racy of SNN and ANN af- ter atlack under different e. Right: The CKA curve be- tween clean images and ad- versarial images of ANN and SNN, respectively.
06.72
0.8
0.8
Accuracy
Similanty
Similantty Imilattty
0.6
0.6
40
0.4
0.4
20-
r-0,001
1-0.02
F-0.001
2-0.02
0.2
E=0.005
00005
0.2
E=00005
0.0.05
c-0.01
0.0
0.0
0.0
0.001
0.005
0.01
0.02
0.05
10
20
30
40 Layers
50
60
70
10
20
30
40
50
60
70
Layers
CKA curves and heatmaps. On N-Caltech 101, the SNN learns different feature representations compared with ANN in shallow and deep layers, but similar representation in intermediate layers. For CIFARI0-DVS, the similarity continues to decrease from 0.9 to 0.5 as the layers deepen. In summary, with the event-based dataset, SNNs and ANNs share different CKA pattern in comparison to the natural image dataset, which implies that SNNs may have further optimization space in this type of dataset. We put more CKA results on various models for the CIFARIO-DVS dataset in Appendix A.3.
4.5 Representation under Adversarial Attack Data
We next study the adversarial robustness of SNN and ANN using CKA. Inspired by quantized ANNs that are robust lo adversarial attack Lin el al. (2019), the SNNs could also inherit this property since their activations are also discrete. Previous works have explored understanding the inherent robustness of SNNs (Sharmin et al,, 2020; Kundu et al., 2021; Liang et al., 2021; ; Kim & Panda, 2021c). However, they either evalu- ate on converted SNN or using rate-encoded images. Here, we test Projected Gradient Descent (PGD) attack (Madry et al., 2017) the directly trained SNN and ANN using direct encoding. Formally, we generate the adversarial images by restricting the L-infinity norm of the perturbation, given by
k+1 a Pddv
= P[ (a + asign( (V a L( w. y)));
(5)
where zi is the generated adversarial sample at the k-th iteration. (m) projects the generated sample onto the projection space, the Tro neighborhood of the clean sample. is the attack optimization step size. With higher the adversarial image is allowed to be perturbed in a larger space, thus degrading task performance.
We evaluate the performance of spiking and artificial ResNet-20 on the CIFAR-10 dataset with values ranging from 0.001, 0.005, 0.01, 0.02, 0.05 generate adversarial images. We then compute the CKA value between the features of the clean images and the adversarially corrupted images. The results are summarized in Fig, (left). We find that although the clean accuracy of ANN is higher than that of SNN, SNN has higher robustness against adversarial attacks. For example, the PGD attack with 0.01 L-infinity norm perturbation reduces the accuracy of ANN by 43%, while only reducing the accuracy of SNN by 22%. We also invest igate the CKA similarity between clean and adversarial images, as shown in the second and third subplots of Fig. 8. We observe that the higher the robustncss against adversarial attacks, the higher the similarity
between clean and corrupted images. This intuition confirmed by the CKA curves, which show that SNN has higher similarity than ANN. We also observe several interesting phenomena. For example, the ANN
10

---

Published in Transactions on Machine Learning Research (04/2023)
suffers large decrease in similarity in the first block, even with small value of 0.001. Additionally, when we focus on the purple line (e = 0.02), we notice that ANN and SNN have similar perseverance in earlier layers, but ANN drops much more similarity than SNN in the last block. These results provide insight into model robustness and suggest that SNNs are more robust than ANN, espocially in their shallow and deep layers.
5 Discussion and Conclusion
Given that SNNs are drawing increasing research attention due to their bio-plausibility and recent progress in task performance, it is necessary to verify if SNNs, especially those trained with the surrogate gradient algorithms, can or have the potential to truly develop desired features different from ANNs. In this work, we conduct a pilot study to examine the internal representation of SNNs and compare it with ANNs using the popular CKA metric. This metric mcasures how two models respond to two different examples. Our findings can be briefly summarized as follows:
1. Generally, the layer-wise similarity between SNNs and ANNs is high, suggesting SNNs trained with surrogate gradient learn similar representation with ANNs. Moreover, wider networks like ResNet-20 8x even have > 0.8 similarities for almost all layers.
2. For extremely deep ANNs and SNNs, the CKA value would become lower, however, the residual con- nections play an important role in regularizing the representations. By conducting ablation studies, we demonstrate that the residual connections make SNNs learn similar representations with ANNs and help
SNNs achieve high accuracy.
3. The time dimension does not provide much additional representation power in SNNs. We also demonstrate that the shallow layers learn completely static representation along the time dimension. Even reducing the number of time steps to 1 in shallow layers does not significantly affect the performance of SNNs.
4. On other types of datasets, SNNs may develop less similar representations with ANNs, e.g., event-data.
Our results show that SNNs optimized by surrogate gradient algorithm do not learn distinct spatial-temporal representation compared to the spatial representation in ANNs. Current SNN learning relies on the residual connection and wider neural networks (for example, Zheng et al. (2020) use ResNeet-19 which is similar to our ResNet-20 8x) to obtain decent task performance. However, our study suggests that this task performance is highly credited to the similar representation with ANN. Furthermore, the time dimension brings limited static dalasets like CIFARIO and CIFAR100. In particular, the first
effect to the SNN representation stage of ResNets results in quite similar representation across time steps.
Nonethcless, our study is not a negation against SNNs. Our results are based on the surrogate-gradient BPTT optimization, which, as aforementioned, is inherently bio-implausible and resembles the optimization method for ANNss Therefore, it may not be surprising to see SNNs and ANNs have similar representations under a similar optimization regime. Additionally, we find that input data is also important in developing the representations. Indeed, the direct encoding used in SNNs inputs the same static images for each time step, again leading to less gap between the representations of ANNs and SNNs.
Here, we provide several directions worth studying in the future: a) Bio-plausible learning rule for SNNs: surrogate gradient training tends to learn an ANN-like representation in SNN, thus is necessary to develop an optimization method that suits SNN better. b) Spiking architecture design: a specialized SNN network architecture may avoid learning similar reprosentation, e.g., Kim et al. (2022a). c) Understanding the robustness of SNN: adversarial attack is inconsequential for human visual systems, which may be reflected in SNN as well. We believe the SNN robustness can be significantly improved. To conclude, work tries to understand the representation of SNNs trained with surrogate gradient and reveals some counter-intuitive observations. We hope our work can inspire more research in pushing the limit of SNNs.
Acknowledgements. This work was supported in part by CoCoSys, a JUMP2.0 center sponsored by DARPA and SRC, Google Rescarch Scholar Award, the National Science Foundation CAREER Award, THI (Abu Dhabi), the DARPA AI Exploration (AIE) program, and the DoE MMICC center SEA-CROGS (Award DFFSC0023199)
11

---

Published in Transactions on Machine Learning Research (04/2023)
References
Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-cczza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, et al. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 34(10):1537-1555 2015.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short- term memory and learning-to-leant in networks of spiking neurons. arXiv preprint #rXiv:180300977 2018.
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature commumicalions, 11(1):1-15 2020.
Gabriel Bender, Picter-Jan Kindormans, Barret Zoph, Vijay Vasudevan, and Quoe Lc. Undorstanding and simplifying one-shot architecture search. In ICML, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1008:3300.3442, 2013.
Natalia Caporale, Yang Dan, et al. Spike timing-dependent plasticity: a hebbian learning rule. Annual
review of neuroscience, (1(1):25-40 2008.
Dennis Valbjørn Christensen, Regina Dittmann, Bernabé Baaares-aarranoo, Abu Sebastian, Manuel Le Gallo, Andrea Redaelli, Stefan Slesazeck, Thomas Mikolajick, Sabina Spiga, Stephan Menzel, et al. 2022 roadmap on neuromorphic computing and engincering. Newromorphic Computing and Engineering, 2022.
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Iece Micro, 38(1):82-900 2018.
Lci Deng, Yujic Wu, Xing Hu, Ling Liang, Yufci Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xic. Rethinking the performance comparison between suns and anns. Newral Networks, 121:294 307, 2020.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. Xiviv preprint Stfi::20330047000 2021.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting, xriie preprinl or Xivv:2200. 11946, 2022.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEFE/CVE International Conference on Computer Vision, pp. 2661-2671, 2021.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical depen- dence with hilbert-schmidd norms. In International conference on algorithmic learning theory, pp. 63 -7.77. Springer, 2005.
Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In Proc. IEEE Eur. Conf. Comput. Vis. pp. 388 404, 2020.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognition, pp. 13558 13567, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on compuler vision, pp. 530-645, Springer, 2016b.
12

---

Published in Transactions on Machine Learning Research (04/2023)
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In TCML, 2015.
Seyed-Mahdi Khaligh-Raase Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS computational biology, 10(11):e1003915, 2014.
Youngeun Kim and Priyadarshini Panda. Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks, 144:686 698, 2021a.
Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, pp. 1638, 2021b.
Youngeun Kim and Priyadarshini Panda. Visual explanations from spiking neural networks using inter-spike intervals. Scientific reports, 11(1):19037, 2021c.
Youngeun Kim, Yuhang Li, Hyoungscob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural archilecture search for spiking neural networks. arXiv preprinl or (xX::2201.10355 2022a.
Youngeun Kim, Yuhang Li, Hyoungscob Park, Yeshwanth Venkatesha, Ruokai Yin, and Priyadarshini Panda. Exploring lottery ticket hypothesis in spiking neural networks. In Computer Vision -CCCV 2022: 17th European Conference, Tel Avio, Israel, October 23-27, 2022, Proceedings, Part XII, pp- 102-120. Springer, 2022b.
Simon Kornblith, Mohammad Norouzi, Honglak Lec, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519-3529. PMLR, 2019.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-conneeting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 25, 2012.
Souvik Kundu, Massoud Pedram, and Peter A Beerel. Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5209-5218, 2021.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifarl0-dvs: event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning, pp. 6316 6325. PMLR, 2021a.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable spike: Rethinking gradient-descent for training spiking ncural networks. Advances in Neural Information Pro- cessing Systems, 34:23426 23439, 2021b.
Yuhang Li, Youngeun Kim, Hyoungscob Park, Tamar Geller, and Priyadarshini Panda. Neuromorphic data augmentation for training spiking neural networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviu, Israel, October 23-27, 2022, Proceedings, Part VII, pp. 631-649. Springer, 2022.
Ling Liang, Xing Hu, Lei Deng, Yujie Wu, Guoqi Li, Yufei Ding, Peng Li, and Yuan Xie. Exploring adversarial attack in spiking noural networks with spiko-compatible gradient. IEEE transactions on neural networks and learning systems, 2021.
Ji Lin, Chuang Gan, and Song Han. Defensive quantization: When efficiency meets robustness. ar preprint arXio:1904.08444 2019.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXio preprint or 1 06003, 2017.
13

---

Published in Transactions on Machine Learning Research (04/2023)
Owen Marschall, Kyunghyun Cho, and Cristina Savin. A unified framework of online learning algorithms for training recurrent neural networks. Journal of machine learning research, 2020.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018.
Hesham Mostafa. Supervised learning based on temporal coding in spiking neural networks. IEEE transac- tions on neural networks and learning systems, 29(7):3227 3235, 2017.
Byunggook Na, Jisoo Mok, Seongsik Park, Dongjin Lee, Hyeokjun Choe, and Sungroh Yoon. Autosnn:
Towards energy-cffiient spiking ncural networks. arXiv preprint arXiv:2001.1279: 2022.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- optimization to spiking neural networks. IEEE Signal Processing Magazone, 36(6):51-66) 63, 2019.
Thao Nguyen, Maïthra Raghu, and Simon Korublith. Do wide and deep networks learn the same things? un- covering how neural network representations vary with width and depth. arXiv preprint orXiv:2010. 15327, 2020.
Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience, 9:437, 2015.
Priyadarshini Panda, Sai Aparna Aketi, and Kanshik Roy. Toward scalable, efficient, and accurate deep spik- ing neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in newral information processing systems, 30, 2017.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
Rajesh PN Rao and Terrence Sejnowski. Spiee-immiggdeppendent hebbian plasticity as temporal difference learning. Neural computation, 13(10):2221 2237, 2001.
Nitin Rathi and Kaushik Roy. Diet-smn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint 2Xiu:2008.06655 2020.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based based machine intelligence with neuromorphic computing. Nature, 575(7784):6077-61,, 2019.
Bodo Ruockauer, Inlia-Aexaandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv preprint arXiv: 04002, 2016.
Bodo Rucckaner, Alcxanna Lungu, Yuhuang Hu, Michael Pfciffer, and Shih-Chii Liu. Conversion of continnous-valuud deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience, 11:682, 2017.
Statistics Student Seminar. Convergent learning: Do different neural networks learn the same representa- tions? 2016.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in newroscience, 13:95, 2019.
Saima
Sharmin, Nitin Rathi, Priyadarshiní Panda, and Kanshik Roy. Inherent adversarial robustness of deep spiking neural networks: Effects of discrete input encoding and non-linear activations. In European Conference Computer Vision, pp. 399 414. Springer, 2020.
14

---

Published in Transactions on Machine Learning Research (04/2023)
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLRR 2015.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012.
Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Téeothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Newral networks, 111:47 63, 2019.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards under- standing learning representations: To what extent do different neural networks learn the same represen- (alion. Advances in neural information processing systems, 31, 2018.
Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78 (10):1550-1560) 15600 1990.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Yujie Wu, Tei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural net- works: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311 1318, 2019.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Scibert, and James J DiCarlo. Performance-oplimiized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academyy of sciences, 111((3):8619 062499624,
8624, 2014.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. arXiv preprint ar Xix:20002. 10085, 2020.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trainet larger spiking neural networks. arXic preprint orXim:2011.052 2020.
15

---

Published in Transactions on Machine Learning Research (04/2023)
A Additional CKA Results
A.1 Results on VGG Networks
30
80
Artificial VGG
Artficial VGG-19
40
Artificaal VGG-25
Artificial VOG-31
Artricial VIG6-45
100
60
20
30
00
40
20
20
10
20
28
02
Spiking VGG-13
Spiking VGG-19
Spiking VGG-25
Spiking VGG-31
Spiking VGG-43
1.0
M x2
Artifciial VGG-13 x4
Artifriial VGG-133 x8
Artificial VGG 13 x11
Artifical VSG-33
VGG-1V
20
20
20
20
20
10
10
10
10
10
10
20
30
10
20
30
10
20
30
10
20
30
10
20
30
Spiking VGG-13
Spiking VGG-13 x2
Spiking
VGG-13 x4
Spiking VGG-13 x8
Spiking VGG-13 x10
1.0
VGG-13
19
0.9
0.8
0.7
VGG-25
10
90
66
76
0.5
VGG.31
VGG-13
VGG-13 x8
VGG-13 x2
VGG-13 x10
VGG-13 x4
0.3
VGG-43
Layers
Layers
Figure A.1: CKA heatmap between spiking VGGs and artificial VGGs with different depth and width CIFAR-10 dataset. Top: the CKA cross-layer heatmap across different depth from 13 layers to 43 layers. Middle: the CKA cross-layer heatmap across different width from original channel number to 8 times. Bottom: visualizing only the corresponding layer, which the diagonal of the CKA heatmap.
A.1.1 Scaling up Width or Depth
In this section, we study the representation similarity between ANNs and SNNs based VGG-series net- works Simonyan & Zisserman (2015). Since VGG-Networks do not employ residual connections, they may bring different representation heatmaps when compared to ResNets. We first evaluate if the spatial scaling observation in ResNets can also be found in VGG Networks. Starting from a VGG-13, we either increase its channel size to times as before or its number of layers to 43, (detailed network configuration provided in Table D.2). Since VGG networks do not contain residual connections, we could scale less depth in VGG networks than ResNets.
The results are illustrated in Fig, A.1. We can find that for deeper networks, the heatmaps tend to exhibit hicrarchical structure, which means the shallow layers and deoper layers have different representations. Increasing the number of layers in VGG networks to 19 or 25, the shallower and deeper layers only have 0.3 CKA similarity (purple). More seriously, when the network depth reaches 31 or 43, the similarity becomes 0.4 even across cach other deeper layers, indicating the representations are diverging. Another important discovery for deep VGG networks is the disappearance of periodical CKA curve, which likely due to the lack of residual connections.
16

---

Published in Transactions on Machine Learning Research (04/2023)
As for the wider networks, the observations are consistent with ResNet families. The wider networks have both brighter heatmaps and higher CKA A curves. These results confirm the similar representation in extremely wide networks.
14
Antra 12
0.9
64
62
Similarity
V0G-13315 14
Syling 10--33330
0.6
0.5
6.1
0.3
VGG-13
VGG-13
VGG-13
VGG-13 T32
spaing
13
20
60
10
20
25
30
Layers
Figure A.2: The effect of time steps in spiking VGG networks. Left: CKA healmaps between ANNs and
SNNs with the different number time steps (from to 32). Right: The CKA curve corresponding layers (diagonal values as in left).
features.7
10
fesitures.0
feattures.7
features.24
08
2
features.0
features.7
testures.22
Teatures.24
res.27
00
25
20
20
22
20
20
20
20
Figure A.3: The similarity across times in Spiking VGG-13. Each heatmap shows the CKA among different time steps in the output of convolutional layers. The top/middle/bottenn rows stand for spiking ResNet-20 with 4/8/32 time steps.
A.1.2 Scaling up Time Steps
We next study whether the time dimension in spiking VGG networks has similar effect to that in spiking ResNets. As can be found in Fig. A.2, the spiking VGG-13s with 1/8/16/32 time sleps do not show a significant difference CKA heatmaps as well as CKA curves, which is the same with Fig. 5. Fig. A.3 shows the CKA across different time steps in the spiking VGGs. In the first layer, the convolution does not have dynamic representation through time. As the layer goes deeper, the dissimilarity across time steps continues to increase, similar to Fig. 6.
17

---

Published in Transactions on Machine Learning Research (04/2023)
A.2 Results CIFAR-100
The results we reported the main context are majorly based on the CTFAR-10 dataset. Here, we provide the visualizations on the CIFAR-100 dataset to further strengthen our findings in the main context.
We first report the spatial dimension results, Le., scaling up the width and depth of the network. Starting from the ResNet--20, we either increase its width to 164 layers or increase its width to 8 times as before. The visualizations are shown below. We find extremely deep networks, e.g., ResNet-164, has a very dark heatmap compared to the ResNet-20 heatmap. The wider network shows somewhat irregular results. The extremely wide network ResNet-200x6 yet has even lowest similarity in the last several blocks. However, it has the highest similarity in the output layer.
1.0
Artificial ResNet-20
60
Artificial ResNet-56
Artificial ResNet-110
300
Artificial ResNet-166
400
40
100
200
300
0.4
50
100
100
20
40
60
0
50
100
150
0
100
200
300
0
200
400
0.0
Spiking ResNet-20
Spiking ResNet-56
Spiking ResNet-110
Spiking ResNet-166
Artificial ResNet-20 X2
Artificial ResNet-20 x4
Artificial ResNett-20 x8
Artffcial ResNet-22
60
60
60
60
40
40
40
40
0.5
20
20
20
20
40
20
0.0
Spiking ResNet-20
Spiking ResNet-20 x2
Spiking ResNet-20 x4
Spiking ResNet-20 x8
1.0
BBA
1.0
0.7
Siminere
380
125
150
375
309
0.6
MW
0.5
ResNet20
ResNet20 ResNet20 XB
0.3
ResNet20 X2
30
Layers
Layers
Figure A.4: CKA heatmap between SNNs and ANNs with different depth and width on CIFAR-100. Top: the CKA cross-layer healmap different depth from 20 layers 164 layers. Middle: the CKA cross-layer heatmap across different width from original channel number to 16 times. Bottom: visualizing only the correspondiug layer, which the diagonal of the CKA heatmap.
Next, we visualize the dotails inside a residual block. In Fig. A.5, we sub-sample the 10-th and the 34-th residual block in a ResNet-110, which shows the same phenomenon. The LIF and convolutional layers cause decrease in similarity, while the residual addition operation restores the similarity. We also provide the CKA heatmap of the partial residual network. As done in Fig. 4, we train 3 spiking ResNet-56 on the CIFAR-100 dataset with several blocks disabling the residual connections. Moreover, we train a linear probing layer the fully-connected classifier on top of each block to see if it contributes to the overall performance of the whole network. The visualization is shown in Fig. A.6, where we find similar observations.
We also rum experiments to test the time dimension of SNNs on CIFAR-100. In Fig. A.7, we train 4 spiking ResNet-20 with 1//116/3 time steps and compute their representations with artificial ResNet-20. Both
18

---

Published in Transactions on Machine Learning Research (04/2023)
3.0
10-th Residual Block
34-th Residual Block
1.00
1.00
0.75
0.75
Res. Block
0.50
0.50
LIF/ReLU Conv (Ld)BN
Figure A.5: Emergence of periodic jagged CKA curve on CIFAR-100. We subplot the 10-th and the 34 th residual blocks in ResNet- 110, which forms periodic jagged curve.
2.0
Similarity
0.25
0.25
1.5
0.00
68
246
0.5
100
150
200 Layers
250
300
350
Spiking Moz06 Null Rasidual
Sakkigg ho55 Part (
Sprking Ruz5d funtal Residua (2)
Spicing Nuz6 Parsia fia dual (3)
Figure A.6: The effect of residual connections in the SNN. We selectivel dis- able residual connections of three stages in the ResNet-56. Top: the CKA heatmap of SNN üsself, con- taining networks with dif- ferent types of non-residual blocks. Bottom: The lin- probing accuracy of each block.
150
81
100
100
100
100
50
50
0
50
100
150
0
50
100
150
50
100
150
0
50
100
150
60
60
60
Accursty
40
9
0
00
Blocks
Blocks
the CKA heatmap and the CKA curve show little variations by changing the number of time steps. This confirms the results CIFAR-10. In addition, visualize the CKA similarities across time steps in each residual block. Fig. A.8 demonstrates that the first stage still produces temporal static features while the last stage has lower similarity across time steps.
1.0
0.9
M
00
30
0.7
Similarty
Spldng Raclet-2074
Soldrg Merkeb-22
0.6
0.4
88 6
0.3
ResNet-20 ResNet-20
ResNet-20
ResNiet-20 T32
20
30
40
60
70
Sokino 2p T14
5340gg Borken 20 T12
Layers
Figure A.7: The effect of time steps in SNNs. Left: CKA heatmaps between ANNs and SNNs with the different number of time steps. Right: The CKA curve of corresponding layers (diagonal values as in left).
19

---

Published in Transactions on Machine Learning Research (04/2023)
2
10
06
2
00
25
20
20
20
20
20
times in SNN CIFAR-100. Each heatmap shows the CKA among different time steps in the output of residual block. "s" means stage, and "b" means block. The top/middle/bbotom rows stand for spiking ResNet-20 with 4/8/32 time steps.
Figure A.8: The similarity
Finally, we rerun the adversarial robustness experiments on CIFAR-100. Here, we train two ResNet-20 with times more channels and use PGD attack to mcasure the robustness against adversarial attack. Also, we plot the CKA curve between the feature of clean images and the feature of corrupted images. The results are shown in Fig. A.9.
Robustness of ANN and SNN
Similarity of ANN
Similarity of SNN
1.0
1.0
25.25
Accuracy
50
Similarity
Similarity
0.6
0.6
0.4
0.4
E=0,001
E=0.02
e=0,001
E=0.02
10
0.2
0=0,005
E=0.0
0.2
E=0,005
0=0.
ANN
SNN
0=0.01
e=0.01
0.0
0.0
0.0
0.001
0.005
0.01
0.02
0.05
0
10
20
30
40
50
60
70
0
10
20
30
40
50
60
70
Layers
Layers
Figure A.9: The robustness against adversarial attack on CIFAR-100. Left: The accuracy SNN and ANN after attack under different e. Right: The CKA curve between clean images and adversarial images of ANN and SNN, respectively.
A.3 Results on CIFAR10-DVS
In this section, we conduct representation similarity analysis on an event-based dataset. CIFARIO-DVS Li et al. (2017). As aforementioned in Fig. 7, the event dataset may generate different CKA heatmaps and curves when compared to the static dataset. Here, we first scale up the spatial dinensions in SNNs and ANNs for the CIFARIO-DVS dataset. As shown in Fig. A.10, we gradually increase either the depth to 110 layers or the width to 8 times as before.
Inercasing the depth of ResNets on CIFARIO-DVS demonstrates a similar effect when compared static CIFAR-10. Interestingly, the deep ResNet, for example, ResNet-110, emerges similar periodical pattern that static CIFAR-10. However,
find the peak CKA value is only around 0.75, (recall that the
20

---

Published in Transactions on Machine Learning Research (04/2023)
peak CKA value the CIFAR10 dataset is nearly 0.95, cf. Fig. 2), indicating the CIFARIO-DVS creates
higher ANN-SNN difference in representations.
Artificial ResNet-22
60
Artificial ResNet-38
Artificial R55Net-55
150
Artificial ResNett-110 110
0.8
75
0.5
100
20
25
0.2
20
50
100
100
100
200
300
Spiking ResNet-20
Spiking ResNet-38
Spiking ResNet-56
Spiking
ResNet-110
Artificial ResNet-20 x2
Artificial ResNet-20 x8
Artificial ResNet-2 tet
60
Artificial ResNet-20
60
40
40
0.4
20
20
20
20
20
40
60
20
40
60
20
40
60
20
40
60
Spiking
Net-20
Spiking
Spiking
x4
Spiking
ResNet-20
0.9
30-
0.7
Similartt
Aeloyass Sm m
23
25 383
150
375
300
0.4
ResNett
ResNet20 x4
ResNet20
ResNet20
0.2
10
20
3p
Layers
Layers
Figure A.10: CKA heatmap between spiking and artificial ResNets with different depth and width on CIFAR1O-DVS dataset. Top: the CKA cross-layer heatmap across different depths from 20 layers to 110 layers.
Middle: the CKA cross-layer heatmap across different widths from the original channel number to 8 times, Bottom: visualizing only the corresponding layer, which is the diagonal of the CKA heatmap.
s1.bl.convl
cl.b1.comv2 s1.b2.conv1
<l.b2.con22 <2.b0.comvl 2.h02.omv2
<2.60.65.1
s2.b1.comv1
1.0
33.b0.convl
04
02
0.0
Figure A.11: The similarity across times in SNN CIFAR1O-DVS. Each heatmap shows the CKA among different time steps in the output residual block. "s" means stage, and "b" means block. The model is trained with 10 time steps, ie., 10 frames integrated for each CIFARIO-DVS data input.
For the extremely wide neural network, the trend holds for ResNet-20, ResNet-20 (x2), and ResNet-20 (x4). However, we find the ResNet-20 (x8) exhibits a significantly low CKA curve than other models. The difference may come from the artificial ResNet-20 (x8). According to the CKA heatmap, the 35th-60th layers in artificial ResNet-20 (x8) become much darker than other heatmaps. We hypothesize that, with
21

---

We next study the internal CKA of each convolution in ResNet-220, which reveals the temporal dynamics of each convolution. As illustrated in Fig. A.11, the first convolution demonstrates dynamic features across time steps, which shows different characteristics with static dataset (cf. Fig. A.12). Another notable difference is that the first and the second time step show very low similarity with other time steps. In Published in Transactions on Machine Learning Research (04/2023)
larger capacity of the neural network the layer dynamics in the imput data, the representation may change significantly.
summary, the rich temporal information dataset may increase the dynamics in SNNs across time steps and bring more differences when compared to ANNs.
A.4 Similarity Across Time
In addition to the residual block, also visualize the CKA heatmaps of convolutional layers and ReLU//LI /LI layers by comparing the similarity among different time steps. As can be seen in Fig. A.12, different from residual blocks, the similarity in convolutional and activation layers is more dynamic. Even in the first stage, the convolutional layers show different outputs across different time steps. This result further confirms our observations residual blocks, where we found the convolutional and activation layers always decrease the ANN-SNN similarity while the residual block restores the similarity. Therefore, suggests that a more temporal dynamic CKA heatmap may produce distinct features.
B Numerical Results
Here, we provide the clean accuracy of our trained models, both on CIFAR-10 and CIFAR-100. All models are trained with 300 epochs of stochastic gradient descent. The learning rate is set 0.1 followed a cosine annealing decay. The weight decay is set to 0.0001 for all models. The original ResNet-20 is 3-stage model, each stage contains residual blocks. The first stage contains 16 channels and the channels are doubled every time when entering the next stage. ResNet-33/56/1ID /160/11 contains 6/9/18/ 19///// residual blocks in each stage. The wider networks just simply multiply all the channels by fixed factor. We provide their top-1 accuracy in Table B.1.
Table B.1: The top-1 accuracy SNNs ANNs on CIFAR-10 (aka C10) and CIFAR-100 (aka C100) datasets.
Layers
Width Factor
Time Steps
ANN (C10)
SNN (C10)
ANN (C100)
SNN (C100)
20
1
91.06
89.67
64.23
61.51
38
1
4
92.34
91.14
N/A
N/A
56
1
92.98
91.94
68.39
66.63
110
1
92.37
91.83
69.20
66.95
164
I
93.00
92.05
70.27
67.09
20
2
93.36
92.00
70.14
68.65
20
4
4
94.52
93.96
74.12
72.39
20
8
4
94.78
94.48
76.57
75.81
20
16
4
94.78
94.73
N/A
N/A
20
1
8
90.44
63.03
20
1
16
90.98
64.34
91.06
64.23
20
1
90.99
64.33
20
1
91.08
N/A
C
The Effect of Network Initialization
In this section, we study the effect network initialization and verify if our findings are invariant to different initialization. Specifically, we train spiking and artificial ResNet-20s with two random soeds and measure the CKA similarity on these models. Fig. C.1 shows the case of CKA heatmaps on the same model: the first columns demonstrate the CKA heatmaps of the same initialization and the third column demonstrates the
22

---

Published in Transactions on Machine Learning Research (04/2023)
CKA heatmaps of different initialization. We can find that SNNs ANNs trained with different initialization share high CKA similarity. Moreover, in Fig. C.2, compare the CKA between SNNs and ANNs by permuting two different initialization. It can be seen that all four permutations have a similar distribution
of CKA values. By comparing Fig. C.2 and Fig. C.I, we also find that the CKA heatmaps between ANNs and SNNs are also similar to the CKA healmaps on the same model with two initializations. This indicates that the similarity between ANN and SNN quite high.
Network Architecture Details
Table D.1: The architecture details of ResNets.
20 layers
38 layers
56 layers
110 layers
164 layers
convl
3 x 3, 16, s1
3 x 3, 16, sl
3, 16,
x 3, 16, s1
3 x 3, 16, s1
3 x 3, 16)
3 x 3, 16
3 x 3, 16
3 x 3, 16
3 16
block1
3
6
9
18
x 27
3 x 3,
3 x 3, 16
3 x 3, 16
3 x 3, 16
3 3, 16
33 x 3,32
(3 x 3,32
3 x 3,32
3 x 3,32
3,32
block2
x
18
x 27
x 3,32
3 3,32
3 x 3,32
3 x 3,32
3 3,
3 x 3,64
3 x 3,64
3 x 3, 64
3 x 3, 64
3 x 3, 64
block3
x 3
x 6
x 9
x 18
x 27
x 3,64
3 x 3,64
3 x 3, 64
3 x 3,64
3 3, 64
pooling
Global average pooling
classifier
10-d fully connected layer, softmax
Table D.2: The architecture details of VGG networks.
13 layers
19 layers
25 layers
31 layers
43 layers
block1
(3 x 3, 64) x
(3 x 3, 64) 2
(3 x 3, 64)
(3 x 3, 64) 2
(3 x 3, 64) 2
poolingl
Average pooling, 82
block2
(3 x 3, 128) x
(3 x 3, 128) 2
(3 3, 128) 3 (3 x 3, 128) 4
(3 x 3, 128) x 5
pooling2
Average pooling, s2
block3
(3 x 3, 256)
(3 3, 256) 3
(3 256) 4 (3 3, 256)
5
(3 x 3, 256) x 6
pooling3
Average pooling, s2
block4
(3 x 33 512) x
(3 512) 8
(3 3, 512) 12 (3 512)
16
(3 512) 24
pooling
Global average pooling
classifier
10-d fully connected layer, softmax
23

---

Published in Transactions on Machine Learning Research (04/2023)
s2.b0.comv2
08
66
<2. bb Lonv2
s2. 62 convl
52.b2.comme
c..b0.coml
33.b0.conve
:3.b0.ds.1
s3.bl.coml
s3.b1.comv2
s3.b2.comv1
s3.b2.comv2
ft
04
06
<2. bbl conv2
s2. 62 conv1
c2.w2.comw
:3 comwllco
s3.b0.conv2
33.b0.ds.1
s3.bl.comvl
s3.bl.comv2
s0.b2.comv1
<3.b2.comv2
fc
04
02
s1. c1.b0.comw2
s1
51.b2.comv1
b1.b2.comw2
62.b0.canv]
62.b0.conv2
$2.60.65)
62.b1.canvl
06
12. -2.b11. conv2
42. .cnv1
42. <2.b2.comw2
s3.b0.convl
63.50 conv2
s3.b0.ds..1
s3.b1.comw1
s3.b1.conv2
63.b2.conv1
63.b2.conv2
fc
6.4
30
100
02
3
25
r..b0l
s1.b0.rehu relu2
>1 .l.bbl.reeul
sl.bl.relu2
s1.b2.relu1
relu2
s2. b0 relu
s2. r2.b0.
s2. #2.bl. relul
f..elurelu2
<2.b2.relul
12.b2.relu2
r3.b0.relul
63.b0.relu2
63.b1 zelut
63.b1. reluu
63.b2.relu)
63.b2. relu2
0.0
3.0
<2.b1.relu2
12.b2.
relu2
r3.b0.relul
63.b0.relu2
s3.bl zelu1
63.b1. reluu
63.b2.relul
$3.b2. relu2
0.0
S1.61.
3.0
13
10
10
10
10
23
30
20
<2. f2.b1.relu2
12.b2 12.b2.relu1
#2.82. relu2
#3.b0.relu1
#3.b0.relu2
$3.bb nelu1
$3.b1 relu2
53.b2.reluu
$3.b2 reluz
30
90
Figure A.12: The effect of time steps in convolutional and activation layers of SNNs.
24

---

Published in Transactions on Machine Learning Research (04/2023)
1.0
Artificial ResNet20 inito
Artificial ResNet20 init1
Artificial ResNet20 init.
0.0
40
40
40
20
20
20
20
40
60
20
40
60
20
40
60
Artificial ResNet20 initO
Artificial ResNet20 initl
Artificial ResNet20 initO
1.0
Spiking ResNet20 initO
Spiking ResNet20 init1
Spiking ResNet20 initl
60
60
60
0.8
0.6
40
40
40
0.4
20
20
20
20
60
20
40
60
20
40
60
Spiking ResNet20 inito
Spiking ResNet20 initl
Spiking ResNet20 inito
Figure C.1: The similarity across different initialization on SNNs and ANNs. We compare the CKA
heatmap the model (optionally initialized with different random seed).
1.
Artificial ResNet20 Inito
Artificaal ResNet20 Initl
Artificaal ResNet20 initl
Artificial ResNet20 inito
40
0.4
20
20
20
20
01
20
40
60
20
40
60
Spiking
ResNet20 initO
Spiking ResNet20 initl
Spiking ResNet20 initO
Spiking ResNet20 initl
Figure C.2: The similarity between ANN and SNNs across different initialization.
25


Reviews:
[{"review_id": "NZpmdsaVcs", "reviewer": "Reviewer_LSrQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is written well, though the language/phrasing can be made clearer in some regions. The analyses are described sufficiently in detail. Analyses seem technically correct.\n\nOne of the main weakness of the paper is that the analysis is limited to Resnet, and image classificiation task. With event-based task, the results are very different. This reduces my confidence generality of the results.\n\nAxes labels missing in some places (ex. Fig 4). \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a similarity analysis between spiking neural networks (ANN) trained with surrogate gradient method and artificial neural networks (ANN). Using extensive analysis with different versions of the Resnet network, the authors show that Residual connections improve similarity of SNN and ANNs. \n\nAdditionally, they show that temporal dimension in SNN is not likely necessary of image classification tasks, the representation is likely very different based on the task and SNNs are more robust to adversarial attacks than ANNs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "• Add more details on the accuracy of SNN, how it was evaluated, and relative performance of each architecture with respect to ANN. \n\n• Add a figure showing the correlation between accuracy of SNN and similarity of SNN to ANN. If there is the tight correspondence, it would \nbe help the main point of the paper. \n\n• Similarity curves for each block in Figure 3 could be plotted on top of each other, and averaged to see if effect of different layers is consistent in each block. For example, the second LIF stage gives a big drop in similarity in 34th residual block, whereas both the LIF stages give similar drop in 10th residual block. Averaging across multiple blocks might be insightful.\n\n* All curves and similarity matrices should be averaged across multiple networks, trained with different initializations. Or, show some evidence of consistency across initializations.\n", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ED67zDXJpD", "reviewer": "Reviewer_91Rq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n--\nThe biggest strength of this manuscript is the detailed experimentation, including how several different changes to spiking and artificial neural networks (SNN/ANN) impact their relational similarity.  This is also performed over many different networks and has detailed information on how blocks and residual connections impact similarity in these neural networks.  This detailed analysis can help add to the understanding of how these networks are trained.\n\nWeaknesses:\n--\nThe biggest weakness is the lack of discussion on what the representational similarity really means, and how dissimilar two networks can become.  For example, if a SNN and an ANN both have accurate predictions, this all but necessitates that the representation at the final layer is quite high.  As such, I would expect representational similarity to stay high throughout the neural networks.  If the final layers were dissimilar, it seems like there would be vastly different performance.  Thus, I struggle with how to interpret this final claim:\n> Unfortunately, our results may not fully support the opinion that SNNs learn effective and distinct spatial- temporal representation compared to the spatial representation in ANNs. Current SNN learning relies on the residual connection and wider neural networks (for example, Zheng et al. (2020) use ResNet-19 which is similar to our ResNet-20 8×) to obtain decent task performance. However, our study suggests that this task performance is highly credited to the similar representation with ANN.\n\nI would encourage the authors to discuss how representational similarity could be different.  For example, if the blocks are dissimilar in SNN and ANN, does that mean that the paths to the final representation are different even if the final representation is the same?\n\n\nAdditionally, several claims are a bit stronger than the empirical evidence, which can be handled with mild language edits (see requested changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript takes a careful look at the representational difference between spiking neural networks and artificial neural networks.  Through many different experiments, they identify that residual networks help maintain the predictive performance and representational similarity to artificial neural networks.  Conversely, the time dimension of spiking neural networks does not have a big effect.  The many different tests over many networks and detailed layer-level analysis adds to the broader set of literature exploring the relationships between spiking neural networks and artificial neural networks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major:\n--\nIt is necessary to add description on how similarity should be interpreted, and discuss how dissimilar final layers in a neural network could be if those networks are predicting similar outcomes (see weakness above).  It is necessary to add more discussion and context to the discussion and conclusion under this viewpoint.   \n\nPlease add a discussion on how much this framework could detect the scenario where the ANN and SNN follow different paths to a common final representation.\n\nMinor:\n--\n\n\n> 3. The time dimension does not provide much additional representation power in SNNs. We also demonstrate that the shallow layers learn completely static representation along the time dimension. Even reducing the number of time steps to 1 in shallow layers does not significantly affect the performance of SNNs.\n\nIt does have a non-trivial impact in your own analysis, Table 2.  This is downplaying the effect too much.\n\n\n> In the 3rd and 4th heatmaps, we can see that blocks without residual connections exhibit significantly different representations when compared to their former stage. Therefore, as long as ANN and SNN learn similar representations in the first layer, the similarity can propagate to very deep layers due to residual connections.\n\nIn Figure 4, I don't see how the visualizations support this conclusion.  Please add some clarifying comments and more detailed analysis.\n\n\n> These results substantiate that the convolutional layers and LIF layers in SNNs are able to learn different representations than ANNs. However, the representation in the residual branch still dominates the representation in post-residual layers and leads to the junction of ANN’s and SNN’s representation.\n\nThis is an interesting claim.  Does this imply that the convolutional representations are much smaller than the residual?  At what point would this mean than an SNN follows a different path to a common representation with an ANN? \n\n\n> This visualization is more specific and may accurately reveal the similarity between SNNs and ANNs because it only measures the CKA value of layers at the same positions.\n\nI disagree with this claim as similarity could easily match to different layers if the networks are learned/defined differently.  If you want to keep this claim in there, please justify.\n\n\n> Notably, we find the last 2/3 layers in artificial ResNet-164 exhibit significantly different representations than spiking ResNet-164, which demonstrates that deeper layers tend to learn disparate features.\n\nI cannot visually discern this from the figures.  I would suggest quantifying for clarity.\n\n\nI do not see a definition for PGD.\n\n\n>Figure 6:...The top/middle/bottom rows stand for spiking ResNet-20 with 4/8/32 time steps.\n\nThis should state they are those networks, not stand for.\n\n\nThere are several grammar errors, please proofread again.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d5cdisq1jr", "reviewer": "Reviewer_3q9t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It would be very interesting to study the representation differences between ANN and SNN but this definition of SNN here is a narrow view on what are spiking neural networks: here an SNN is simply a replacement of the ANN activation function with \"LIF-activation\" but there are many other implementation which would yield much more different representation (recurrent SNN, spike-time encoded SNN, etc...).\n\nThis also raises an important conceptual limitation because this type of LIF activation function are expected to yield a precise equivalence with quantized Relu network [1] under rate-coding hypothesis. Therefore using this type of conversion results, it means that we are basically studing here the quantization error or the difference in representation with an ANN and a quantized-relu ANN.\n\nThe last problem for me is that the work is not acknowledging that most the results would be identifical if studing a quantized relu. The robustness of quantized networks is infact well known. Is it not acceptable for SNN research to ignore (and not forget to cite) the ideas and progress from quantized ANN research.\n\n\n[1] Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks\nDeng and Gu 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper describes a method to analyze the similarity between ANN and SNN where the definition of SNN is here restricted to an deep neural network (identifical to the correpsonding ANN) with replacement of the Relu activation with a \"LIF-activation-function\". The similarity is studied Central Kernal Alignment measure defined previously.\n\nThe authors find that\n- within each block of a ResNet the network similarity is rather constant and fairly high. But between each residual connection, the difference rapidly increases before being brought back to high similarity right after each residual connection.\n- In the lower layers the representation seems very time-invariant.\n- The SNN seems to be a bit more robust to adversarial attacks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1) In Section related work, please acknowledge the following sentence (or anything similar):\n\"there exists many other types of SNN: e.g. recurrent SNN or spike-timing SNN but this study focuses the type of feedforward SNN which has been shown to be equivalent to quantized ANN under a rate based hypothesis [ref 1 above]\".\n\n(2) It would be great to shown on similiraty matrix ANN Rest vs. another ANN ResNet of same size, this would provide a baseline for what is high/low similarity.\n\n(3) In section of robustness to adversarial attacks or discussion, please acknowledge the following sentence (or anything similar):\n\"Quantized ANN are robust to adversarial attack, therefore it was expected that SNN which are roughly equivalent to quantized relu ANN [1] inherit this property.\".\n\n[2] Defensive Quantization: When Efficiency Meets Robustness\nJi Lin, Chuang Gan, Song Han", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper had an additional round of revision before the decision was reached.  The authors addressed all concerns that were raised.  I now endorse it to be accepted as is.
Decision: Accept (poster)