Efficient Surrogate Gradients for Training Spiking Neural Networks

Hao Lin 1 * Shikuang Deng 1 2 * Shi Gu 1 2

Abstract

gradient in order to control the effective domain for thesurrogate gradients. We introduce an index k to denote theproportion of membrane potential with non-zero gradientsin backpropagation and present a technique to control the
proportion of non-zero gradients (CPNG) in the network.
The CPNG technique modifies the shape of surrogate gradients during network training, progressively approachingthe δ-function while maintaining the index k steady within
an effective range to ensure training stability. Finally, eachlayer succeeds in finding a surrogate gradient shape thatmakes a better balance between the approximation error tothe δ-function with the size of effective domain than the
fixed-shape surrogate gradients. It’s worth mentioning that
our strategy only incurs minor additional costs during thetraining phase and has no effect on the inference phase. We
verify the compatibility of CPNG to the existing mainstream
SNN infrastructures such as VGG (Simonyan & Zisserman,2014), ResNet (He et al., 2016), and Sew-ResNet (Fanget al., 2021). In all reported comparative experiments, training with CPNG gives more accurate models than training
with vanilla surrogate gradients. Our main contributions canbe summarized as follows:

To guide the shape optimization in applying surrogate gradients for training SNN, we propose
an indicator k, which represents the proportionof membrane potential with non-zero gradientsin backpropagation. Further we present a novel
k-based training pipeline that adaptively makes
trade-offs between the surrogate gradients’ shapes
and its effective domain, followed by a seriesof ablation experiments for verification. Our algorithm achieves 68.93% accuracy on the ImageNet dataset using SEW-ResNet34. Moreover,
our method only requires extremely low external
cost and can be simply integrated into the existing
training procedure.

1. Introduction
Spiking Neural Networks (SNN) have gained increasing
attention in recent years due to their biological rationale
and potential energy efficiency as compared to the common real-value based Artificial Neural Networks (ANN).
SNN communicates across layers by spiking signals. On
the one hand, this spiking mechanism turns multiplicative
operations to additive operations, increasing the inference
procedure’s efficiency. On the other hand, it introduces an intrinsic issue of differentiability, which makes training SNNs
more challenging. At present, the method for obtaining
practical SNNs can be roughly divided into three categories:
converting a pretrained ANN to SNN (Sengupta et al., 2019;
Deng & Gu, 2020; Li et al., 2021a; Bu et al., 2021), training with biological heuristics methods (Hao et al., 2020;
Shrestha et al., 2017; Lee et al., 2018), and training with
BP-like methods (Wu et al., 2018; Zheng et al., 2020; Liet al., 2021b; Yang et al., 2021).

• We identify and investigate the impact of the shapeof surrogate gradients on SNN training. Our finding
characterizes a special representative power for SNN
that can be utilized to improve its performance.
• We propose a statistical indicator k for the domain
efficiency of surrogate gradients and a k-based training method CPNG that adjusts the shape of surrogate
gradients through the training process, driving the surrogate gradients close to the theoretical δ-function withensured trainability on sufficiently large domains.
• Our CPNG method improves classification accuracy
on both static image datasets including CIFAR10, CIFAR100 and ImageNet, as well as event-based image
datasets such as CIFAR10-DVS. We achieve an accuracy of 68.93% in the experiment that trains ImageNet
on Sew-ResNet34.

The purpose of this work is to optimize the SNN training pipeline by adaptively altering the shape of surrogate*

Equal contribution 1 University of Electronic Science and Technology of China 2 Shenzhen Institute for Advanced Study, UESTC.
Correspondence to: Shi Gu <gus@uestc.edu.cn>.

2. Preliminary

Published at the Differentiable Almost Everything Workshop of the
40 th International Conference on Machine Learning, Honolulu,
Hawaii, USA. July 2023. Copyright 2023 by the author(s).

Through out the paper, we use bold letters to denote matrices and vectors, superscripts to identify specific layers,1Efficient Surrogate Gradients for Training Spiking Neural Networks

subscripts to denote specific neurons, and indexes to identify
specific moments.

70
60

2.1. Leaky Integrate-and-Fire Model50(l)ci [t + 1] =

(l−1)
NX

Test Accuracy

We use the Leaky Integrate-and-Fire (LIF) module for spiking neurons. Formally, given the pre-synaptic input (denoted(l)by ci [t + 1]) of the ith neuron in the lth layer at time stept + 1, we can model the iterative process in LIF as

40
=0.25
=0.5
=1.0
=1.5
=2.0

30
20
10

(l) (l−1)
wij sj[t + 1],

0

(1)

0

50

100

j

(l)

(l)

(l)

(l)

ui [t + 1] = τ ui [t](1 − si [t]) + ci [t + 1],(2)

si [t + 1] = H(ui [t + 1] − Vth ).

(3)

(l)

(l)

150
Epochs

200

250

300

Figure 1. Test accuracy of different β when threshold is 1.0.

Here, N (l−1) is the number of neurons in the (l − 1)th layer,(l−1)
sj[t + 1] is the output spike of the j th neuron in the(l)(l − 1)th layer at time t + 1, wij is the weight between j th(l)neuron in (l − 1)th layer and ith neuron in lth layer, ui [t]is the membrane potential of the ith neuron in the lth layer
at time t, τ is the membrane potential attenuation factor,
H(·) is the step function, and Vth is the activation threshold. When the membrane potential of a neuron exceeds theactivation threshold, a spike is released and the membranepotential of the current neuron is set to zero.

Figure 2. Membrane distribution of each layer in experiment training ResNet19 on CIFAR10.
0.40
1

1

β

β

0.32

2.2. Surrogate Gradient Function
0.24

There are various surrogate gradient shapes adopted by previous work (Wu et al., 2018; Neftci et al., 2019). In thiswork, we used triangle-like function, rectangular-like function and arctan-like function to verify the effectiveness of
CPNG. These functions are described below:β(1 − β|x|) if |x| < 1/βφtriangle (x) =,(4)0otherwise
β if |x| < 1/(2β),(5)φrectangular (x) =0otherwise
β,(6)φarctan (x) =1 + (πβx)2

0.16

0.08

0.00
-3

-2

-1

0

1

2

3

Figure 3. The proportion of neurons with non-zero gradient undera surrogate gradient with a certain β.

3.1. Shape Parameters and Effective Domain Indicatorto obtain SNN for a good performance. We first examine whether this intuitive approach is possible. We trained
VGG16-structured SNN (replace ReLU with LIF, and use
average pooling) on CIFAR100 using triangle-like surrogate
gradient with β set from 0.25 to 2 respectively. As shownin Fig. 1, the test accuracy increases when β varies from0.25 to 1.0 but remains at 1.0% when β is set to 1.5 and
2.0, indicating that properly increasing β may benefit thetraining but arbitrarily increasing β will drive the training
collapse.

Shape Parameters. Intuitively, increasing the shape parameter β of surrogate gradients would drive it closer tothe δ-function. One might expect to adopt a very high β

In fact, these results unveil that efficient training of SNNs
requires not only the approximation to the δ-function but
also the insurance for the surrogate gradients to work. Thus

where β represents the maximum gradient value of current
surrogate gradient function. Notably, the surrogate gradientR +∞satisfy −∞ f (x) = 1, which is also the property of theδ-function.

3. Method2

Efficient Surrogate Gradients for Training Spiking Neural Networks

Algorithm 1 Control the Proportion of Non-Zero Gradient
Input: SNN model with L layer, current iterator epoch e,
klimit , and vector krecorder : store each layer’s smallest k
Output: Each layer’s surrogate gradient parameter β
if e == 0 thenfor l = 1, 2, · · · L do
calculate current k by Eqn.7 for layer-l and store at
krecorder [l]end forelse
for l = 1, 2, · · · L do
calculate current kcur by Eqn.7 for layer-l
if krecorder [l] <klimit thenkrecorder [l] = klimit
else if kcur <krecorder [l] thenkrecorder [l] = kcur
end ifkmin = krecorder [l]if kmin ̸= kcur thenuse kmin to update β using binary search method.
end ifend forend ifreturn β for each layer

it is necessary to employ a dynamic shape-changing strategy
rather than using a fixed-shape surrogate gradient. Thisissue is also covered by (Li et al., 2021b) as well and was
owed to the lack of adaption to the dataset variation.
Effective Domain Indicator.
To quantitatively guide the choice of β, we propose a statistical indicator to denote the percentage of membrane potentials that fall into the domain of surrogate gradients. As
illustrated in Fig. 2, the distribution of membrane potentials on each layer takes the normal shape. Thus, for thesimplicity of calculation, we regard the membrane potential distribution of all neurons within the same layer as a
Gaussian one. By calculating the mean µ and the standard
deviation σ of the membrane potential before this layer releases spikes, we can obtain the proportion of the neuron
with a non-zero gradient during a certain iteration (the area
between the red lines in Fig. 3). For a given β, the effective gradient domain of triangle-like surrogate gradient is[Vth − 1/β, Vth + 1/β], we can obtain the definite integralof the current normal distribution in this effective gradientdomain, which is the Effective Domain Indicator k:
Z Vth +1/β(x−µ)21√e− 2σ2 dx.
k=
(7)
2πσ
Vth −1/βFor each layer, we record the membrane potential of all neurons in every time step (a tensor shaped like batchsize-bytimestep-by-channels-by-H-by-W) and calculate the mean
and variance. Based on this indicator k, we can then effectively determine to what extent we can tune the β while
ensuring that there are enough membrane potentials locatedwithin the effective range of surrogate gradients to make thetraining progress.
3.2. CPNG Method

we expect the network parameters to have been reasonably
updated, that is, the network has traversed the whole dataset
and has minimal training loss at the current moment. Thiscan reduce the misleading of data randomness to CPNG.
For example, if we use CPNG once per batch, the network
parameters are mostly affected by the first few batches in theearly stages of network training, and the statistical indicatork obtained by using such network parameters will have a
lot of randomnesses.

In this section, we will cover how to combiningly optimize
β and k to maximize the effectiveness of surrogate gradients.
To train the network successfully, there must be sufficient
membrane potential values in the effective domain of thesurrogate gradient, i.e., k must be large enough. The mostextreme case is β → 0, which gives k → 1. Obviously,
this is not an optimal solution as it introduces substantial
error for the gradients. If we reasonably restrict the effective
interval of the surrogate gradients, it is possible to drive the
SNN to select those more essential membrane potentials forbackpropagation.

CPNG computes the smallest k of each layer during theiteration process as kmin and records it. If the k value of acertain layer rises after an epoch, CPNG adjusts the k valueof the current layer to kmin by increasing the β, otherwise,
keep the current β fixed and update kmin . Since the kmin ofeach layer of neurons may be different, different layers may
have different surrogate gradient shapes. In addition, we
set a safe lower bound klimit . When k falls below the lower
bound, CPNG will pause adjusting the current neuron until
the k of the current neuron exceeds the lower bound. The
CPNG algorithm is detailed in Algo.1.

We also need to ensure that the new k does not make the
network difficult to train, for this, CPNG sets the target k ofeach layer to the smallest k that has occurred in the current
layer during the training iteration, rather than an artificial
goal. If the network can be trained when the smallest kappears, then the network should still be trained after weadjust the k and maintain the smallest k. When using CPNG,

3.3. The Cost of CPNG Method
The extra cost of CPNG occurs in two steps: (1) collecting
the mean and variance of the membrane potential before
releasing the spikes of each layer; (2) altering the β using
the indicator k. In our experiment, we only use the mean
and variance of one batch to calculate the indicator k, which
3Efficient Surrogate Gradients for Training Spiking Neural Networks

makes the cost of the first step in the same order of magnitude as the batch normalization (Ioffe & Szegedy, 2015)
operation. For the latter step, we provide a binary searchmethod that solves the problem very fast, and further optimization algorithms can further improve the solution speed.
Quantitatively, in the VGG16+CIFAR100 experiment, it
takes an average of 1.7GFLOPs to obtain the output corresponding to an input without CPNG, and the first step ofCPNG will only add 5.59 × 10−3 additional GFLOPs. Using CPNG once per epoch takes an average of 3.3 secondsof overhead (1.57% of total training time).

Table 2. Examine CPNG under various conditions.
DatasetCIFAR100

4. Experiment

CIFAR10To verify the effectiveness of the CPNG method, we provide
groups of comparative experiments (Sec. 4.2) and compare
CPNG with existing works in Sec. 4.3. More experimental
results are presented in the appendix.

CIFAR1004.1. Implementation Details
CIFAR10-DVS

All the SNN architectures include the tdBN layer (Zheng
et al., 2020) with the average-pooling layer, and comparedto their ANN versions, we replace the activation function
ReLU with LIF. Our experiment settings, such as optimizer,
learning rate, are detailed in Appendix. A. Except for applying the CPNG method at the end of each epoch, all otherconditions, such as learning rate, batch size, etc., are consistent. We used TIT (Deng et al., 2022) when training CIFAR,
which reduces the training time by initializing the SNN witha smaller simulation length.

ImageNet

0.1
74.66%

0.2
74.66%

0.3
74.66%

0.4
73.96%

0.5
74.24%

0.6
74.02%

Time Step
2
2
2
2
2
2
2
2

Accuracy
73.71±0.005%
74.38±0.005%
72.35±0.002%
73.63±0.026%
71.46±0.080%
72.43±0.035%
75.36±0.049%
75.90±0.001%

MethodSTBP-tdBN (Zheng et al., 2020)
ANN-to-SNN (Li et al., 2021a)
ANN-to-SNN (Bu et al., 2021)
Dspike (Li et al., 2021b)
TET (Deng et al., 2022)
CPNG
CPNG
CPNG
Diet-SNN (Rathi & Roy, 2020)
ANN-to-SNN (Li et al., 2021a)
ANN-to-SNN (Bu et al., 2021)
Dspike (Li et al., 2021b)
TET (Deng et al., 2022)
CPNG
CPNG
CPNG
CPNG
Dspike (Li et al., 2021b)
TET(Deng et al., 2022)
CPNG
CPNG + TET
ANN-to-SNN (Li et al., 2021a)
ANN-to-SNN (Bu et al., 2021)
Sew-ResNet (Fang et al., 2021)
TET (Deng et al., 2022)
TET (Deng et al., 2022)
CPNG

Architecture
ResNet19
ResNet20
ResNet20
ResNet18
ResNet19
ResNet19
ResNet19
ResNet19
VGG16
VGG16
VGG16
ResNet18
ResNet19
VGG16
ResNet19
ResNet19
ResNet19
ResNet18
VGGSNN
ResNet18
VGGSNN
ResNet34
ResNet34
Sew-ResNet34
Spiking-ResNet34
Sew-ResNet34
Sew-ResNet34

Time Step
6
32 64 128
8 16 32
246
246
2
4
6
5
32 64 128
8 16 32
246
246
5
2
4
6
10
10
10
10
32 64 128
16 32 644
6
4
4

Accuracy
93.16%
94.78% 95.30% 95.42%
89.55% 91.62% 92.24%
93.13% 93.66% 94.25%
94.16% 94.44% 94.50%
93.79±0.002%
94.14±0.009%
94.10±0.005%
69.67%
73.55% 76.64% 77.40%
73.96% 76.24% 77.01%
71.68% 73.35% 74.24%
72.87% 74.47% 74.72%
71.32±0.20%
74.40±0.005%
75.29±0.001%
75.37±0.056%
75.40%
83.17%
76.5±0.007%
83.27±0.162%
64.54% 71.12% 73.45%
59.35% 69.37% 72.35%
67.04%
64.79%
68.00%
68.93%

4.3. Comparison to Existing Works
In this section, the experimental results we report all use
triangle-like surrogate gradient. On some datasets, the current SOTA conversion method performs better than direct
training, but they need lengthy simulation time steps, especially on the ImageNet dataset. Our main purpose is not tochase SOTA, but to demonstrate the effectiveness of CPNG
itself and the compatibility of CPNG with other methods.
The results of the experiment are shown in Tab. 3.

Table 1. klimit Experiments on ResNet19+CIFAR100.
—
74.05%

Architecture
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19

Table 3. Result of training spiking neural network.
DatasetklimitAccuracy

MethodTriangular
Triangular+CPNG
Rectangular
Rectangular+CPNG
ArcTanArcTan+CPNG
Triangular+TET
Triangular+TET+CPNG

0.7
73.89%

4.2. Performance improvement with CPNGFirst we explored the effect of different klimit on CPNG, Tab.
1 shows that we may don’t need to focus too much on klimit .
When klimit is 0.1-0.3, it can bring about a 0.61% improvement. In our experiments, we set klimit to 0.2 uniformly.

5. Conclusion
This work proposes a new perspective for directing the shapechange of the surrogate gradient, we propose a statisticalindicator k that guides the shape change of the surrogate
gradient, and propose the CPNG method for modifying
the shape of the surrogate gradient during training while
guaranteeing the proportion of membrane potential with
non-zero gradients. It’s possible that the failure to produce
satisfactory results when pulling surrogate gradient directlyto δ-function is due to a failure to meet the premise thatthe network can be trained normally. In other words, there
may exists a trade-off between the approximation to theδ-function and the effective domain of gradients under the
given dataset, and CPNG helps us approach the equilibrium
point.

Additionally, we tested the applicability of CPNG to various
surrogate gradient functions. For surrogate gradients with
non-zero gradient everywhere, such as arctan, we directlyuse [Vth − 1/β, Vth + 1/β] as the integration interval tocalculate the indicator k, then use Algo. 1 to solve new β.
Finally, we verify the compatibility of CPNG with existing
direct training methods TET (Deng et al., 2022)(Tab. 2). In
all experiments, CPNG can achieve better results. It is worth
mentioning that the improvement brought by CPNG is notdue to bad counterpart. For example, in the experimentof using TET to train CIFAR100, we can achieve 75.36%
accuracy without using CPNG in our experimental setting.
4

Efficient Surrogate Gradients for Training Spiking Neural Networks

References

Sengupta, A., Ye, Y., Wang, R., Liu, C., and Roy, K. Going
deeper in spiking neural networks: Vgg and residualarchitectures. Frontiers in neuroscience, 13:95, 2019.

Bu, T., Fang, W., Ding, J., Dai, P., Yu, Z., and Huang, T.
Optimal ann-snn conversion for high-accuracy and ultralow-latency spiking neural networks. In International
Conference on Learning Representations, 2021.

Shrestha, A., Ahmed, K., Wang, Y., and Qiu, Q. Stablespike-timing dependent plasticity rule for multilayer unsupervised and supervised learning. In 2017 international
joint conference on neural networks (IJCNN), pp. 1999–
2006. IEEE, 2017.

Deng, S. and Gu, S. Optimal conversion of conventional
artificial neural networks to spiking neural networks. In
International Conference on Learning Representations,2020.

Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv
preprint arXiv:1409.1556, 2014.

Deng, S., Li, Y., Zhang, S., and Gu, S. Temporal efficient training of spiking neural network via gradient reweighting. arXiv preprint arXiv:2202.11946, 2022.

Wu, Y., Deng, L., Li, G., Zhu, J., and Shi, L. Spatiotemporal backpropagation for training high-performance
spiking neural networks. Frontiers in neuroscience, 12:
331, 2018.

Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., andTian, Y. Deep residual learning in spiking neural networks. arXiv preprint arXiv:2102.04159, 2021.

Yang, Y., Zhang, W., and Li, P. Backpropagated neighborhood aggregation for accurate training of spiking neuralnetworks. In International Conference on Machine Learning, pp. 11852–11862. PMLR, 2021.

Hao, Y., Huang, X., Dong, M., and Xu, B. A biologicallyplausible supervised learning method for spiking neuralnetworks using the symmetric stdp rule. Neural Networks,
121:387–395, 2020.

Zheng, H., Wu, Y., Deng, L., Hu, Y., and Li, G. Going
deeper with directly-trained larger spiking neural networks. arXiv preprint arXiv:2011.05280, 2020.

He, K., Zhang, X., Ren, S., and Sun, J. Identity mappingsin deep residual networks. In European conference on
computer vision, pp. 630–645. Springer, 2016.
Ioffe, S. and Szegedy, C. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
In International conference on machine learning, pp. 448–
456. PMLR, 2015.
Lee, C., Panda, P., Srinivasan, G., and Roy, K. Training
deep spiking convolutional neural networks with stdpbased unsupervised pre-training followed by supervised
fine-tuning. Frontiers in neuroscience, 12:435, 2018.
Li, Y., Deng, S., Dong, X., Gong, R., and Gu, S. A free lunch
from ann: Towards efficient, accurate spiking neuralnetworks calibration. arXiv preprint arXiv:2106.06984,
2021a.
Li, Y., Guo, Y., Zhang, S., Deng, S., Hai, Y., and Gu, S.
Differentiable spike: Rethinking gradient-descent fortraining spiking neural networks. Advances in Neural
Information Processing Systems, 34, 2021b.
Neftci, E. O., Mostafa, H., and Zenke, F. Surrogate gradientlearning in spiking neural networks: Bringing the powerof gradient-based optimization to spiking neural networks.
IEEE Signal Processing Magazine, 36(6):51–63, 2019.
Rathi, N. and Roy, K. Diet-snn: Direct input encoding withleakage and threshold optimization in deep spiking neuralnetworks. arXiv preprint arXiv:2008.03658, 2020.
5

Efficient Surrogate Gradients for Training Spiking Neural Networks

A. More Experiment Details
In this section, we provide more experimental details. We use cosine decay to gradually reduce the learning rate to 0 and
the initial β for all experiments is 1.0. In the last batch (a fixed batch randomly selected at the beginning), the mean andvariance of the membrane potential are additionally stored in the forward process, then the stored mean and variance areused to obtain a new β according to Algo. 1. Hyperparameters are shown in the table 4.
Table 4. Experiment Setting

Experimentlearning rate
weight decay
momentum
optimizer
warm-upbatch size

CIFAR100
0.1
1e-4
0.9
sgd
False
256

CIFAR10
0.1
1e-4
0.9
sgd
False
256

CIFAR-DVS (ResNet18)
0.01
4e-5
0.9
sgd
False
72

CIFAR-DVS (VGG11)
0.001
4e-5
—adam
False
72

ImageNet
0.03
4e-5
0.9
sgd
False
256

B. Results of Comparative Experiment
Only the results of CPNG on the CIFAR100 dataset are shown in Sec. 4.2, while the results on more datasets are presented
in Tab. 5. For all comparative experiments, using CPNG can yield better results than not using CPNG.

C. Surrogate Gradient Shapes of Different Layers
Fig. 4 describes the change of k during the training process. CPNG does control the rise of k during the training process andmakes it stabilize at the end. Fig. 4 also shows that we don’t need to pay too much attention to klimit . klimit only provides aguarantee in extreme cases, but extreme cases do not necessarily appear during the network training process.

Figure 4. In the experiment of ResNet19+CIFAR100, the proportion of non-zero gradient membrane potentials of neurons in different
layers without CPNG (a) and with CPNG (b).

We show the final β of some experiments in Fig. 5. Various layers’ β are different, which demonstrates that various layers
match distinct surrogate gradient shapes as a result of their varying membrane potential distributions. CPNG eventually
increases the β of most layers. Compare to CPNG, randomly increasing the β can make the network difficult to train (Fig.
1). Even with β set to 1.5, which most of the neuron layers shown in Fig. 5a can approach or reach, the network is still
difficult to train. This demonstrates that it is safe to increase β using CPNG, while it is unsafe to increase β arbitrarily.

6Efficient Surrogate Gradients for Training Spiking Neural Networks

Table 5. Examine CPNG on various surrogate gradients.

DatasetCIFAR10CIFAR100

CIFAR10-DVS

MethodTriangular
Triangular+CPNG
Triangular
Triangular+CPNG
Triangular
Triangular+CPNG
Rectangular
Rectangular+CPNG
ArcTanArcTan+CPNG
Triangular+TET
Triangular+TET+CPNG
Triangular
Triangular+CPNG
Triangular
Triangular+CPNG
Triangular
Triangular+CPNG
Rectangular
Rectangular+CPNG
ArcTanArcTan+CPNG
Triangular+TET
Triangular+TET+CPNG
Triangular
Triangular+CPNG
Triangular+TET
Triangular+TET+CPNG
Triangular
Triangular+CPNG
Rectangular
Rectangular+CPNG
ArcTanArcTan+CPNG
Triangular+TET
Triangular+TET+CPNG

Architecture
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
ResNet19
VGG16
VGG16
VGG11
VGG11
ResNet18
ResNet18
ResNet18
ResNet18
ResNet18
ResNet18
ResNet18
ResNet18

Time Step
2
2
4
4
6
6
2
2
2
2
2
2
2
2
4
4
6
6
2
2
2
2
2
2
2
2
10
10
10
10
10
10
10
10
10
10

Accuracy
93.73±0.015%
93.77±0.003%
94.07±0.002%
94.14±0.009%
93.99±0.002%
94.10±0.005%
92.51±0.038%
92.92±0.014%
92.12±0.016%
92.84±0.024%
93.76±0.006%
93.81±0.006%
73.71±0.005%
74.38±0.005%
75.06±0.005%
75.37±0.010%
75.00±0.016%
75.40±0.056%
72.35±0.002%
73.63±0.026%
71.46±0.080%
72.43±0.035%
75.36±0.049%
75.90±0.001%
69.13±0.110%
71.14±0.098%
82.93±0.116%
83.27±0.162%
75.8±1.787%
76.5±0.007 %74.3±0.053%
75.0±0.772 %70.8±0.345%
70.87±0.002 %80.37±0.616%
80.43±0.309‘ %Figure 5. a: Final β of each layer in VGG16 experiment, the initial β is 1.0. b: Final β of each layer in Sew-ResNet34 experiment, theinitial β is 1.0

7