Synaptic Interaction Penalty: Appropriate Penalty Term for Energy-Efficient Spiking Neural Networks
Authors: Kazuma Suetake
Takuya Ushimaru
Ryuji Saiin
Yoshihide Sawada
Venue: Accepted by TMLR ()

Abstract: Spiking neural networks (SNNs) are energy-efficient neural networks because of their spiking nature. However, as the spike firing rate of SNNs increases, the energy consumption does as well, and thus, the advantage of SNNs diminishes. Here, we tackle this problem by introducing a novel penalty term for the spiking activity into the objective function in the training phase. Our method is designed so as to optimize the energy consumption metric directly without modifying the network architecture. Therefore, the proposed method can reduce the energy consumption more than other methods while maintaining the accuracy. We conducted experiments for image classification tasks, and the results indicate the effectiveness of the proposed method, which mitigates the dilemma of the energy--accuracy trade-off.

Published in Transactions on Machine Learning Research (12/2023)
Synaptic Interaction Penalty: Appropriate Penalty Term for Energy-Effcieent Spiking Neural Networks
Kazuma Suetake"
AISIN SOFTWARE, Aichi, Japen
Takuya Ushimaru"
AISIN SOFTWARE, Aichi, Japan
Ryuji Saiin
AISIN SOFTWARE, Aichi, Japan
Yoshihide Sawada
yostahiide.suvuddeeiisiiicc
Tokyo Research Center, AISIN, Tokyo, Japan
Reviewed on OpenReview: https: // openreview. net/ forum? id- 422KKTTTTqa
Abstract
Spiking neural networks (SNNs) energy-efficient neural networks because of their spik- ing nature. However, as the spike firing rate of SNNs increases, the energy consumption does as well, and thus, the advantage of SNNs diminishes. Here, we lackle this problem by introducing a novel penalty term for the spiking activity into the objective function the training phase. Our method is designed so as to optimize the energy consumption metric directly without modifying the network architecture. Therefore, the proposed method can reduce the energy consumption We conducted experiments for image classification tasks, and the results indicate the ef- fectiveness of the proposed method, which mitigates the dilemma of the energy-accuracy trade-off.
than other methods while maintaining the accuracy.
1 Introduction
With the rapid growth and spread of neural networks, realizing energy-efficient neural networks is an ur- gent mission for sustainable development. One such model the spiking neural network (SNN), which is also known be more biologically plausible than ordinary artificial neural networks (ANNs). SNNs are energy-efficietlly driven on neuromorphic chips (Akopyan et al., 2015; Davies et al., 2018) certain field- programmable gate arrays (FPGAs) (Maguire et al., 2007; Misra & Saha, 2010) by asynchronously processing spike signals. However, the spike firing rate of an SNN increases, the energy consumption does as well, and thus, the advantage of the SNN diminishes. Therefore, in addition to the shift from ANNs to SNNs, it is advantageous to adopt training methods that reduce energy consumption in the inference phase. At the same time, such a training method should be independent of the network architecture to avoid limitations in the application. That is, our goal is to develop a training method that realizes energy-efficient SNNs without any constraint on the network architecture.
There various approaches toward energy-cfficient SNNs, such pruning, quantization, and knowledge distillation (Kundu et al., 2021; Chowdhury et al., 2021a; Lee et al., 2021), which are widely-used approaches also in ANNs. Further, there are SNN-specific approaches sparsifying the spiking activity related to the energy consumption (Lee et al., 2020; Kim & Panda, 2021; Naya et al., 2021), to which our method belongs. particular, the methods that penalize the spike firing rate the training phase are close to our aforementioned
"Equal contribution.

---

Published in Transactions on Machine Learning Research (12/2023)
goal (Esser et al., 2016; Sorbaro et al., 2020; Pellegrini et al., 2021). However, they indirectly reduce the energy consumption by arbitrarily reducing the spike firing rate, where there is no strict proportionality between them. Hence, reducing energy consumption while maintaining accuracy is difficult.
Principle and Idea Our principle is that we should optimize the metric as it is in the training phase. In this spirit, we propose to introduce proper penalty term for the spiking activity- synaptic interaction penally into the objective function. It derived so that its expected value is precisely proportional to the energy consumption metric for the SNN. Although the difference between the proposed and existing methods is only at the scaling factor for each spike, demonstrate that this minor correction causes significant improvement.
Main Contributions
We derived a novel penalty term that can directly optimize the metric for the total energy consump- tion an SNN without modifying the network architecture.
We demonstrated that the proposed method can reduce the energy consumption more than other methods while maintaining the accuracy for image classification tasks, which mitigates the dilemna of the energy-accuracy trade-off.
We also demonstrated that the proposed method is compatible with the weight decay, which imposes implicit sparsity on the nelwork (Yaguchi et al., 2018), and that the proposed method creates a higher sparsification effect than the weight decay.
2
Related Work
2.1 Spike Sparsification in Direct SNN Training
The most relevant approaches our proposal introduce the penalty term for the spike firing rate and directly train SNNs by the surrogate gradient method (Esser et al., 2016; Pellegrini et al., 2021). It is a straightforward idea to penalize the spike firing rate to obtain energy-efficient SNNs because the spike firing rate appears in the SNN energy consumption metric (Lee et al., 2020; Kim & Panda, 2021). We refer to the reduction in spike firing rate as spike sparsification. Although the spike firing rate cannot be optimized by the ordinal backpropagation method owing to non-differentiabili,, gradient method, which is the same technique as training spiking neurons SNNs (Zenke Ganguli, 2018; Shrestha Orchard, 2018). However, neither of these penalty terms (Esser et al. (2016); Pellegrini et al. (2021)) precisely matches the energy consumption metric. As opposed them, our synaptic interaction penalty resolves this limitation.
can be optimized by the surrogate
2.2 Spike Sparsification via Conversion from ANN
Other approaches introduce the penalty term for corresponding ReLU networks (ANNs with ReLU acti- vations) and convert them to SNNs (Sorbaro et al., 2020; Narduzzi ct al., 2022). Although there is no guarantee that the penalty terms for ReLU networks contribute to the reduction of the energy consumption for converted SNNs, ReLU networks can be optimized by the ordinal backpropagation method. Note that the same synaptic scaling factor for the penalty term as ours is proposed to reduce the energy consumption for SNNs in Sorbaro et al. (2020). However, they failed to provide evidence to support their claim, as they mentioned. As opposed to them, we provide theoretical and experimental proof in the setting of the direct SNN training by the surrogate gradient.
2.3 Neuron Sparsification
Neuron sparsification means increasing the number of permanently zero-valued activations for all data-deaa neurons. It stronger condition than spike sparsification, which does not force neurons to be permanently inactive. In ReLU networks, the training with the Adam optimizer and weight decay regularization implicitly
2

---

Published in Transactions on Machine Learning Research (12/2023)
induce neuron sparsification (Yaguchi et al., 2018) because ReLU activations have an inactive state. However, this claim has yet to be demonstrated in the context of SNNs, where spiking neurons also have an inactive state, even though weight decay is usually adopted in SNNs. Therefore, to detect the effect of our method correctly, we shall also focus on the weight decay.
3 Method
In this section, propose the synaplic interaction penalty. First, we describe the spiking neuron model with surrogate gradient mechanism. Next, describe the metric for energy consumption, which can be represented by the spiking activity. Note that need to optimize both the accuracy and energy efficiency in the training phase. Finally, we state that the synaptic interaction penalty is the proper penalty term to optimize the energy consumption metric.
3.1 Neuron Model and Surrogate Gradient
In this study, we use SNNs constructed by single-step spiking neurons, which are superior the multi-time step SNNs in terms of training and inference costs for static tasks (Suetake et al., 2023). Note that the single-step spiking neurons are the same setup that in previous study of the penalty term (Esser et al., 2016).
Let us denote I E (1 E Z P
L} as the layer, de as the number of neurons in the 1-th layer, 80 = x E R as the input data, and the subscript i of any vector as its i-th component. Then, the single-step spiking neuron defined as follows (Suetake et al., 2023).
Definition 3.1. The forward mode of single-step spiking neuron consists of two ingredients: the membrane potential 2 E Rd and spikes emitted by neurons 81 10.1 They are defined using the Heaviside step function H as follows:
U1 == W181-1,
(1)
s1 (w,,) = H (un unh) = = {
(u, 2 unn), (ui < uth),
(2)
where Wi Rd:xd=1 is the strength of the synapse connections, also called the weight matrix, and uth E R is the spike firing threshold (Eq. 1 for 1 - 1 corresponds to the direct encoding (Rucckaucr et al., 2017)). In this context, we classify the i-th neuron in the 1-th layer as dead neuron if $1,i = for all input data within a given dataset.
A backward mode of the single-step neuron as it is does not work in the standard backpropagation algorithm because the derivative of Eq. 2 vanishes almost everywhere. Therefore, we adopt the technique called surrogate gradient, i.e., we formally replace the derivative function with some reasonable function, for example, the following one (Suetake et al., 2023):
03L.i (ui,) = oun
(ur, 2 Mch) (ur, < war)
wi.
(3)
daa Jhy
(u1,s)
where 5 and a are hyperparameters and oa the scaled sigmoid function expressed as follows:
0a (u,,) = 1 exp ((-u,i + ucn) /a)
(4)
don (w,,) oa (w1,) 10 aa (u,,))
1
(5)
Ougi a
Note that the choice of function for the surrogate function is irrelevant to proposal.
3

---

Published in Transactions on Machine Learning Research (12/2023)
3.2 Metric for Energy Consumption
We prepare the symbol Vh,s for the number of synapses outgoing from the i-th neuron in the 1-th layer, te., the number of matrix elements in (Wi+1),, R that is not forced to vanish in terms of network architecture. Let us denote W and Hy as the width and height of the feature map, respectively, C as the channel size in the I-th layer, and k--1 as the kernel size associated with Wi+1. We restrict both the kernel width and height to be identical to k++1 for the sake of simplicity. Then, explicit forms of tht, are as follows, e,g,, the standard fully connected (fc) and two-dimensional convolutional (conv) layers,
th = 4/2 C++1
(fc),
(6)
W4nHH4 C411444 (conv),
11 = 11-
(7)
WiHl
where the convolutional layer assumes appropriate padding with respect to k-- to satisly Eq. 7. Additionally, Eq. 7 provides the average value in the case of downsampling layers (Wi+1H1-1 < W;Hf) because the procisc value varies depending on the neuron's position and the downsampling method. Note that th, may also vary depending on the neuron's position within the convolutional layers, e.g., corners, edges, and other positions, contingent upon hardware implementation. For instance, the hardware design could involve computing all kernels tied to spike firing at edge positions including padding and then discarding redundant computation results (Kang et al., 2020). Alternatively, the hardware could be engineered to evade unnecessary computations in the initial phase through a conditional branch (Bamberg expression presented in Eq. 7 remains consistent with the approach adopted by Kang et al. (2020), thereby providing a sound measure that accurately accounts for hardware realization. In addition, it is worth noting that more complex scenarios involving nontrivial padding, stride, and dilatation might introduce additional i-dependency in t/t.;- However, such specific cases are beyond the scope of this study.
al., 2023). The
Using 17,6, we can express the number of floating point operations (FIOOss), which is often used as metric to measure the computational complexity in ANNs, as follows:
FLOPs(I) = Σ Wh.
and the layer-wise and balanced spike firing rates, which are also important metrics to measure the sparsity of spiking activity in SNNs, as follows:
di
R(I)
ii=
81,
(9)
ZEX
di
R(I),
(10)
where the operation Erex means taking the empirical expectation in the dataset X. Then, the energy consumption metric that we should optimize is defined follows.
Definition 3.2. Let us denote T as the size time steps and EAC [pJ] as the energy consumption per accumulate operation. Then, the laser-wise and total energy consumption metrics for the SNN are defined as follows:
FSNNN (1)= = TEAC E ZEX
Σ 11,,81,4
(11)
ESNN
ESNN(1).
(12)
Note that T is equal to one for the single-step neuron model.
4

---

Published in Transactions on Machine Learning Research (12/2023)
This total energy consumption metric holds practical validity, it aligns with the achievable energy con- sumption levels when implementing SNNs through specific neuromorphic chips. For example, in Kim & Panda (2021); Esser et al. (2016), the implementation of SNNs on the TrueNorth platform was demon- strated (Akopyan et al., 2015), where the energy proportional to tt, exclusively consumed upon the occurrence of the corresponding spike firing. However, nole that we consider the ideal setting for SNN infer- ence, where peripheral energy consumption (Lemaire et al., 2022) does not contribute except for the spiking interaction.
If t is independent of i (3tr, Vi, 11, wh)), by combining Eqs. 8 and 9, Eq. 11 is rewritten follows:
Σ i=
ESNx(1) TAC Σ
$1
DE
dp
-TEAcFLOPs(()RU),
(13)
which is the same metric that used in Kim & Panda (2021).
Note that dead neurons (Def. 3.1) for a given dataset never contribule to the energy consumption metric (Def. 3.2) for that dataset. Moreover, these dead neurons never influence model outputs for the same dataset. Therefore, pruning dead neurons after training can help save memory capacity, provided that data distributions do not differ significantly between before and after training.
3.3 Synaptic Interaction Penalty
To optimize the energy consumption ESNN (Eq, 12), we propose the following penalty terms.
Definition 3.3. The layer-wise and total synaptic interaction penalty terms are defined as follows:
di
Psyu (1) Paya
(14)
p
7
Oayn = Payn (8)) =
Dayn (1, 81),
(15)
1=1
where s s:= {8t)/1 and 2 1.
The equivalency between the total energy consumption metric and total synaptic interaction penalty imme- diately follows from their definitions and the equation sf, = =t,i, which is derived from Eq. 2.
Theorem 3.4. The expected value of the layer-wise and total synaptic interaction penalties are precisely proportional to the laser-wise and total energy consumption metrics of SNNs:
ETEAC E sex
[Sayn (1) EsNN(I),
(16)
ET E. [Sayal ESNN. wEX
(17)
for arbitrary p 2 1.
This fact means that optimizing Eq. 15 leads to optimizing Eq. 12. Hence, we strongly propose to use Eq. 15 as the penalty term to optimize the energy consumption metric. In the following, we indicate the total synaptic interaction penalty when simply referred to as the synaptic interaction penalty. The spike firing rate and energy consumption metric represent the balanced spike firing rate and total energy consumption metric, respectively, well as the synaptic interaction penalty.
5

---

Published in Transactions on Machine Learning Research (12/2023)
Table 1: Comparison among total penalty terms. Our penalty term Qaya is precisely equal the ground truth ESNN/ EAC-
Model
EsNx/ / BAC
Sgyn
Qtotl
Nbalance
CNN7
98895888
98895888
59688
6
VGG11
2526060544
2526060544
249856
10
ResNet18
553730048
553730048
671744
20
Remark 3.5. The proposed penalty term can be optimized in the manner of the surrogate gradient as Sec. 3.1, and using p + options controls the backward signal when the spike does not fire as follows:
1 0oh (my,)) st,7
p Oui,
Sun,
SEL.
(u1, Wet))
Dot.
(18)
(ur, < Unh)
where we used Eq. 2. From Eqs. 14 and 18, there is no intrinsic difference when p 1; hence, we do not consider p > options except for p 2, which is commonly used in several studies (Esser et al., 2016; Pellegrini et al., 2021). However, the open problem still remains, ie., it cannot be theoretically decided which choice is better, p 1 or p >>> 1. We experimentally examined it for 1, 2 described in Sec. 4.
3.3.1 Differences from Other Penalty Terms
The other candidates for the penalty term follows:
d
Pental
(19)
1,57
p
1=1
i=1
1
Pbalance
dg $
(20)
p
where, for p= 2, Drotal and Dalance are the same as those in Esser et al. (2016) and Pellegrini et al. (2021). respectively. However, we claim that neither can directly optimize the energy consumption because they do not have the proportional nature (Eq. 17), although they sparsify the spiking activity to some exlent.
Fig. and Table show the discrepancy between the energy consumption metric and penalty terms of the model used in the following experiment. In these figures and table, assumed that all the spiking neurons fired, i.e., 81, 1(v1, for the sake of simplicity. In this assumplion, the ground truth is proportional to FLOPs without loss of generality. These figures and table indicate that the synaptic interaction penalty is preciscly proportional to the energy consumption metric, but other penaltics are not. In the next section, we will experimentally verify how this claim affects performance.
3.3.2 Normalization of Penalty Terms
Penalty terms are inchuded in the objective function with their intensity parameter À as the coupling AΩ., where the symbol denotes "syn", "total", or "balance". For tractable treatment of the intensity parameter between various penalty terms or among models of various scales, we recommend normalizing the penalty terms by (1), where indicates 1, i.e., 81 1V,, Note that replacing Ω. with (z)S.(1) is equivalent to replacing À with
X = X11.(1)).
(21)
Hence, sometimes adopt the normalized notation X' instead of X.

---

Published in Transactions on Machine Learning Research (12/2023)
Esxe
Rayz (ours)
Boxd
Reiance
Essss
Rayn (ours)
Qual
Qallane
10
12/0/2 (000
10
(200)20000)
10
10
100
6
Layer I
Layer
(A) CNN7
(B) VGG11
ESNN
Paya (ours)
Qulaase
10"
2000000
10
10
2
3
6
9
10
12
13
14
15
16
18
20
Layer
(C) ResNet18
Figure 1: Comparison among layer-wise penalty terms. The z-axis represents the layer mumber, and the axis represents (L)/ for data of EsNN or (1)/ for data of Ω. (* denotes syn, total, or balance). The network architectures are (A) CNN7, (B) VGG11, and (C) ResNet18 (App. A.1). Our penalty term (blue) is precisely proportional to the ground truth (gray).
4 Experiment
In this section, we evaluate the effectiveness of the proposed synaptic interaction penalty. First, we describe the setup for experiments. Next, we show that the proposed method can decrease energy consumption. Finally, show that the proposed method reduce the energy consumption more than other methods while maintaining the accuracy. In particular, surrogate gradient functions and outperforms the conversion approach (Sorbaro et al., 2020). Overall, the main objective is to analyze the behavior of our method rather than to achieve state-of-the-ard accuracy.
show that the proposed method work under distinct
4.1 Experimental Setup
As the single-step spiking neuron is developed for static tasks (Suetake et al., 2023). we experimented for the Fashion-MNIST (Xiao et al., 2017), CIFAR-10, and CIFAR-100 (Krizhevsky, 2009) datasets widely used in SNN experiments (Essor et al., 2016; Zhang & Li, 2020; Chowdhury et al., 2021b) with some network architectures, CNN7, VGG11, and ResNet18 (refer App. A.1 for details). For these experiments, we implemented the program by the PyTorch framework and used one GPU, an NVTDIA GeForce RTX 3090, with 24 GB (refer Table B.1 in the appendix for the difference in the training time).
In these experiments, used the following objective function:
N
L (CE(f(z)):t)) Mlyyn (s(x,)))
n n=1
AWD (1WW2,, + WaNllWNNll?
(22)
where (In, (n) denotes the pair of input data and its label, f denotes some spiking neural network, CE denotes the cross-cntropy function, I|,?, denotes the L2 penalty for the weights, i.e., weight decay, W l|? denotes the L2 penalty for the trainable parameters of batch normalization layers, and XWD denote the intensity of penalties, and BaN e {0, 1}. Note that we included the weight decay into the objective function verify its sparsifying effect (Yaguchi et al., 2018) in the context of SNNs. In addition, we explicitly specified the weight docay for batch normalization layers because it was included in the default setting of the Torch framework (Paszke et al., 2019) but not in Yaguchi et al. (2018). The training of f was done by

---

Published in Transactions on Machine Learning Research (12/2023)
Table 2: Performance with respect to varying X, ÄWD, and BBN. The network architecture VGG11, the dataset is CIFAR-10, and the optimizer is Adam. ESNN denotes the energy consumption metric for the SNN (Eq. 12), Enaseline denotes the EsNN for the model with À AWD BBN 0, dead rate denotes the ratio of the number of dead neurons to the number of total neurons, and R denotes the spike firing rate (Eq. 10).
All the metrics were calculated using the test dataset.
A
AWD
BBN
Accuracy [%]
ESNN/ Ebaceline [%]
Dead rate [%]]
R [%]]
0
89.09 0.271
100.0 1.530
3.367 0.221
12.11 0.114
1e-08
0
0
89.23 + 0.141
73.92 + 0.247
5.935 1 0.822
10.34 1 0.050
1e-07
0
0
87.81 + 0.458
34.07 + 0.397
22.42 + 1.005
6.046 0.061
1e-06
0
0
79.75 + 0.694
11.85 + 0.492
59.01 + 1.174
2.176 + 0.040
1e-05
0
0
21.81 + 14.46
1.053 + 1.246
92.43 + 3.149
0.174 0.198
0
1e-04
0
89.15 + 0.493
81.07 1.497
9.567 + 1.005
8.474 + 0.771
0
1e-03
0
89.00 + 0.832
68.64 + 6.714
18.72 + 1.942
6.802 + 0.492
1e-02
84.98 1.455
44.39 3.540
34.13 0.683
4.992 0.448
0
1e-01
50.51 = 36.24
17.90 + 16.08
70.93 + 25.66
2.988 + 2.642
1e+-00
10.00 1 0.000
0.000 0.000
100.0 1 0.000
0.000 0.000
1e-08
1e-03
0
89.61 + 0.026
60.28 + 1.384
20.20 + 2.268
6.244 + 0.155
1e-07
1e-03
0
88.06 + 0.653
28.72 + 0.448
35.58 + 2.255
4.076 0.064
1e-06
1e-03
76.00 + 0.962
7.829 + 0.076
68.02 + 0.504
1.378 + 0.016
1e-05
1e-03
10.02 + 0.029
0.007 + 0.012
99.84 + 0.283
0.001 0.002
0
le-05
88.99 + 0.626
96.29 + 2.003
6.268 + 0.786
9.753 + 0.246
0
1e-04
89.87 1 0.526
85.64 + 3.412
10.24 + 1.735
7.457 + 0.293
0
1e-03
86.91 0.315
51.06 1.277
32.79 + 1.257
4.376 0.138
0
1e-02
1
10.00 - 0.000
0.000 + 0.000
100.0 + 0.000
0.000 + 0.000
1c-08
1e-04
I
89.41 + 0.147
45.87 0.202
18.38 0.631
5.643 0.019
1e-07
le-04
86.76 + 0.957
19.21 + 0.610
41.97 + 2.169
2.911 + 0.043
1e-06
1e-04
1
74.38 + 0.465
6.582 + 0.264
71.21 + 0.608
1.111 0.028
1e-05
1e-01
1
10.00 + 0.000
0.003 + 0.005
99.92 + 0.140
0.001 + 0.001
the backpropagation algorithm with the surrogate gradient of Eq. unless otherwise stated. The optimizer was selected from the momentum SGD (mSGD) Adam to confirm the claims in Yaguchi et al. (2018).
Refer to App. A for further details of the experimental setup such as hyperparameters.
4.2 Energy Reduction by Synaptic Interaction Penalty
We investigated whether optimizing the synaptic interaction penalty (Eq. 15) led to optimizing the enorgy consumption metric (Eq. 12) and whether there was any conflict with other terms the objective func- tion (Eq. 22). The setting was as follows. The baseline model was trained for Eq. 22 with = AWD 0. The other models were trained from scratch with
combinations of the weight decay (AwD 0). L2 penalty for batch normalization layers (BBN = 1), and p 1 synaptic interaction penalty (A > 0). The results are presented in Tables 2 and 3. Note that the values in this table were taken for X and AWD from point where the accuracy was very low (approximately 20%) until the accuracy reached the upper bound and stopped changing.
From the result Table 2, we can observe the following. First, as the intensity the penalty term increases, the energy consumption metric decreases; the inference accuracy also decreases. Therefore, the intensity parameter À controls the trade-off between them. Second, the combination of the synaptic interaction penalty and weight decay further reduces the energy consumption metric. Therefore, them simultancously. In addition, we found that the combination of the weight decay and Adam optimizer induces neuron sparsification even without the synaplic interaction penalty, though its contribution to the energy reduction less than the synaptic interaction penalty. Furthermore, neuron sparsification proceeds more strongly, maintaining higher accuracy for the Adam optimizer than the mSGD optimizer (compare the dead rate for X= BBN 0 and AWD 0 1e-02 in Table 2 with that in Table 3), consistent with the findings in Yaguchi et al. (2018). Note that we cannot observe a remarkable difference between Ban 0
propose to adopt both of
8

---

Published in Transactions on Machine Learning Research (12/2023)
Table 3: Performance with respect to varying X, AWD, and BBN. The optimizer is mSGD. The remaining descriptions consistent with those detailed in Table 2.
AWD
BBN
Accuracy [%]
ESNN/ Ebaseline [%]
Dead rate [%]
R [%]
0
0
89.14 = 0.158
100.0 + 2.548
1.678 + 0.167
14.19 + 0.277
1e-09
89.36 = 0.159
78.84 + 1.166
2.428 1 0.188
13.35 1 0.214
1e-08
0
0
88.81 = 0.224
40.00 + 0.335
11.04 + 0.139
10.07 = 0.097
le-07
0
86.80 = 0.348
18.58 + 0.384
33.62 + 0.875
4.476 = 0.143
le-06
0
24.32 7.158
73.27 + 98.19
78.72 - 21.06
4.207 = 4.165
le-04
88.59 = 0.413
84.40 + 6.843
2.705 + 0.754
12.73 = 0.344
1e-03
88.60 = 2.134
70.96 = 8.041
3.968 = 1.326
11.97 0.582
0
1e-02
70.80 = 2.795
45.18 + 3.008
19.57 = 2.636
10.85 = 0.651
1e-01
20.23 = 17.72
18.52 = 13.99
92.18 = 10.94
3.322 4.056
1e-09
1e-03
90.96 = 0.202
64.62 1 0.802
3.976 1 0.196
11.82 + 0.085
1e-08
1e-03
90.06 = 0.387
35.65 + 0.076
13.51 + 0.195
9.043 0.068
1e-07
1e-03
87.74 = 0.257
16.32 + 0.166
37.68 + 0.598
4.163 + 0.040
1e-06
1e-03
21.42 = 9.467
26.13 + 24.14
87.16 + 4.910
3.255 + 1.576
1e-07
89.30 0.263
99.75 + 2.576
1.798 + 0.260
14.17 0.270
1e-06
88.85 = 0.277
94.59 + 2.186
1.838 + 0.117
13.63 + 0.108
1e-05
88.80 = 0.235
95.85 + 1.806
1.834 = 0.134
13.43 0.299
1e-04
88.51 = 0.295
87.16 + 1.614
2.657 + 0.105
10.12 = 0.151
le-03
84.47 = 0.757
51.21 1 2.576
16.90 1 0.388
5.005 0.310
1e-02
19.53 = 16.50
1.786 + 3.093
97.64 + 4.088
0.164 - 0.284
1e-09
1c-07
89.32 = 0.417
78.35 + 1.694
2.606 + 0.078
13.24 0.143
1e-08
1e-07
88.77 0.181
39.70 + 0.844
11.29 + 0.428
10.04 + 0.173
1e-07
1e-07
86.95 = 0.236
18.60 + 0.285
33.49 + 0.418
4.478 + 0.130
le-06
1e-07
28.05 = 1.125
78.52 + 105.4
81.75 + 13.47
4.500 = 4.411
and 1. Therefore, we adopt the weight decay with BBN in further experiments to simplify our objective function. Finally, all the above results hold for not only VGG11 but also CNN7 and ResNet18 (see Tables 8- 11 in the appendix).
4.3 Trade-off between Accuracy and Energy Efficiency
4.3.1
Comparison Between Penalties
To examine the impact of distinet penalty terms on the energy consumption of trained models, we conducted a comparative experiment. The setting was as follows. For fair comparison, we used the X' notation for the intensity parameter of penalties rather than the raw À (see Eq. 21). The baseline model was trained for Eq. 22 with X' BBN O, and we tuned ÅWD 0 obtain the highest accuracy. Then, the others were trained by varying X' > 0 and by replacing Ωayn in Eq. 22 with Ptotal or Paalance from scratch. The results are shown in Fig. 2 (A) as X-parameterized curves of the consumption rate was produced
trade-off, where the energy the energy consumption of each model normalized by that of the baseline model. Note that it is better for data to be located at the upper left corner in the figure. Refer to App. A.3 for the sampling of X. In addition, the quantitative analysis is presented in Table 4, where higher scores are better for all the metrics: area under the curve (AUC), Spearman's rank correlation coefficient. (Spearman), and the mutual information (MT). Note that the argument of cach metric represents cutoff parameter, where data with lower accuracy than are omitted. We introduced the cutoff parameter because training tended to break the intensity parameter was increased for all the methods. Refer to App. A.4 for details of quantitative metrics.
From the result in Fig. (A) and Table 4, we can observe the following. First, for each Ω, the = 1 option is apparently better than the p = 2 option. Therefore, we propose to adopt the p = 1 option. Note that this difference arises from the backward control as Eq. 18. We expect that the p = option would substantially diminish the membrane potential below the spike firing threshold, even in cases where spike firing does not occur. Consequently, the likelihood of the membrane potential stay below the spike firing
9

---

Published in Transactions on Machine Learning Research (12/2023)
Say (y= (Ours)
Duxe
Dustaxe (p (1=))
(p (Ours)
(p - Qusto (p
Oen 2) (Ours)
(p= (p
(px(==2) (Ours)
Perce
(p
1.0
1.0
0.9
0.9
Acnuacy
0.8
Accuracy
0.8
0.7
0.7
0.6
0.6
0.50 0.0
0.5
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Energy comsumption rate
Energy consumption rate
(A) Eq. 15
(B) Eq. 23
Pay (p= 1) (Ours)
Dueal (==
Dutane (p=1)
fg (Ours) - Saerr (T=)) --ge- (7 (1
Ray (p 2) (Ours)
- Doezn (p-
Quteecs (p 2)
(p= (Ours) - Querr (T=))
1.0
1.0
0.9
0.9
Accurcy
0.8
Accuracy
0.8
0.7
0.7
0.6
0.6
0.0
0.2
0.4
0.6
0.8
1.0
0.50
0.5
1.0
1.5
2.0
2.3
Emergy consumption rate
Energy consumption rate
(C) Eq. 24
(D) Conversion
Figure 2: Energy- accuracy tradc-off curves. The network architecture is CNN7, the dataset is Fashion- MNIST, and the optimizer is Adamn The energy consumption rate is the energy consumption of each model normalized by that of the baseline model. Each model is trained using the penalty term Ω, indicated in the legend. From (A) to (C): each model is trained by the indicated surrogate gradient below each figure. (D): comparison with the conversion approach (Sorbaro et al., 2020). ΩANN denotes SNNs converted from the QReLU network. The r-axis exceeds one because converted SNNs have higher energy consumption than the SNN baseline for Dayn.
Table 4: Quantitative comparison corresponding to Fig. 2 (A). Higher scores are better. The best and the second-best best results are highlighted bold and underlined, respectively, Refer App. A.4 for details of the quantitative metrics.
Method
AUC(70)%%
AUC(50) [%]
Spearman(70)
Spearman(50)
MI(70)
MI(50)
Dayn (p = 1) (Our)
68.02
79.60
0.9861
0.9865
3.465
3.610
Daya (p = 2) (Ours)
61.62
72.69
0.9474
0.9709
3.233
3.476
Ototal (p
63.30
76.05
0.9766
0.9767
3.244
3.319
Qustal (p = 2)
55.16
64.63
0.9831
0.9701
3.218
3.295
Qualance (p 1)
54.23
67.16
0.9412
0.9412
2.978
2.978
Qpalance (p= 2)
31.47
42.94
0.8500
0.8946
2.708
2.833
threshold following weight updates due to data fitting. However, the precise reason why the benefits of lowered energy consumption outweigh the drawbacks of potential inference accuracy deterioration remains unresolved, constituting a subject for our future investigations. Second, for p = 1, the trado-off curve of Payn is the best, followed in order by Piotal and Poalence- Therefore, we experimentally clarified the advantage of the coefficient vhy for Eq. 14, which had remained issue in the method proposed by Sorbaro et al.
10

---

Published in Transactions on Machine Learning Research (12/2023)
(2020). Finally, all the above results hold for not only CNN7 but also VGG11 and ResNet18 (see Fig. 5 and Tables 12-17 in the appendix).
4.3.2 Robustness to Distinct Surrogate Gradient Functions
To examine the impact of distinct functions for the surrogate gradient on the energy consumption of trained models, we conducted the same experiment as that in Sec. 4.3.1 except for the choice of a function for the surrogate gradient. Instead of Eq. 3, we adopted the piece-wise linear function (Esser et al., 201 and scaled sigmoid (Pollegrini ct al., 2021) function for the surrogate gradient as follows:
Ös
~
max (1 - |u -uu. O),
(23)
du
Ōs doa
(24)
Ou Ou
where da is the same as Eq. The results are presented in Figs. 2 (B) and (C) (and Tables 19 and 20 the appendix).
From the result in Figs. 2 (B) and (C), the same observations those in Sec. 4.3.1 hold. That is, the p 1 option is apparently better than the p = 2 option; the trade-off curve of Saya is the best, followed in order by Rotal and Qualance- Therefore, the synaplic interaction penalty works under distinct surrogate gradient functions.
4.3.3 Superiority to Conversion Approach
To examine the impact of distinct training methods on the energy consumption of trained models, we also produced the trade-off curve for the conversion approach (Sorbaro et al., 2020). We trained a single QReLU network (an ANN with quantized ReLU activations) increasing the intensity of the penalty and evaluated the converted SNNs for each intensity in different time steps: T 1, and 10 (refer the original paper (Sorbaro et al., 2020) for details). The results presented in Fig. 2 (D) (and Table 21 in the appendix), where the energy consumption
normalized by that of the baseline for Ωayn.
From the result in Fig. 2 (D), we can observe that both the energy consumption and accuracy for the conversion approach are worse than those for the surrogate gradient approach. This is because the conversion process degrades the accuracy, and the penalty term for the QReLU network cannot directly optimize the energy consumption metric for the SNN. Hence, we should dircctly train SNNs by the surrogate gradicnt and synaptic interaction penalty to avoid such degradation.
4.3.4 Additional Trade-Off Curves
To examine the impact of distinct penalty terms on metrics beyond the energy consumption of trained models, we have included additional trade-off curves in Fig. 3 (and Tables 22 24 in the appendix). These curves depict the relationship between accuracy and specific metrics other than the energy consumption metric presented in Fig. 2 (A), while maintaining the same training procedure adopted in Fig. 2 (A).
From the results in Fig. 3 (A, B), we can observe that training with a specific penalty term leads to a reduction the associated metric, particularly for the p = option compared to the p = 2 option. These findings underscore the suitability of the p = option when applying penalty term aligned with the targeted metric optimization. This choice aligns with our principle of directly optimizing the desired metric. Additionally, focusing on the results for dead neurons displayed in Fig. 3 (C), the trade-off curve for advantageous outcome that
Qgyn emerges as the most favorable. Importantly, this observation serves complements our guiding principle. It suggests a stronger correlation between energy consumption and dead neurons compared to other metrics. Detailed analysis will be further explored in our future work.
5
Conclusion
We studied the training method to obtain energy-efficient SNNs in terms of the surrogate gradient. Based our principle that should optimize the metric as it is, derived the synaptic interaction penalty
11

---

Published in Transactions on Machine Learning Research (12/2023)
Ske (P2 (1km))
Puy (dam))
2
(em1) (bura)
Ryp 2 (1(ns)
Reaal 21
Reseao 2
1,0
1.0
0.8
0.8
6.7
0.7
0.00
0.2
4.50
03
0.5 0.00
0.33
0.10
0:15
0123
Ratio of Slusa for each model to that for base zodel
Rdlo of Ruate for eech modello that for bese model
Deed rate
(A)
(B)
(C)
Figure 3: Additional trade-off curves corresponding to (CNN7/ Fashion-MNIST Adam). (A), (B), and (C) indicate the trade-off curves between accuracy and the three metrics: Auotal, Amllance and the ratio of the number of dead neurons to the number of total neurons, respectively.
to optimize the energy consumption metric. Then, experimentally showed that the synaptic interaction penalty (especially for 1) is superior to the existing penalties and conversion approach. Furthermore, its offectiveness remains consistent across different network architectures and choices of surrogate gradient functions. We conchude that our principle has worked well.
An apparent limitation is that the definition of the synaptic interaction penalty depends on that of the energy consumption metric. However, if the larget metric becomes deformed, the penalty should be accordingly deformed in accordance with our principle-eve though it is metric irrelevant to the energy consumption. Another limitation is that although the target metric is directly included in the objective function, it is just indirectly optimized by the surrogate gradient.
We further list some outstanding issues. First, it unclear why there was a difference between the training result for p- 1 and of the synaptic interaction penalty. Elucidating the mechanism of this difference could help us understand the surrogate gradient. Second, we did not focus on the synergy between the spike sparsification and pruning. A pruning-aware sparsification training will help obtain SNNs. Finally, the high availability of the synaptie interaction penalty should be verified on neuromorphic chips, for example, in the case of real datasets, large networks, and other tasks. By solving these issues, we can contribute to the realization of genuinely eco-friendly SNNs.
energy-efficient
References
Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-loaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, et al. TrueNorth: Design and tool flow of a 65 mW 1 million neuron programmable neurosynaptic chip. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 34(10):15377 1557, 2015.
Lennart Bamberg, Arash Pourtaherian, Lue Waeijen, Anupam Chahar, and Orlando Moreira. Synapse compression for event-based hwwpoouutiooal--euaaaaeeeeeeeeee accelerators. IEEE Transactions on Parallel and Distributed Systems, 34(4):1227-1240, 2023.
Sayeed Shafayet Chowdhury, Isha Garg, and Kaushik Roy. Spatio-temporal pruning and quantization for low-latency spiking neural networks. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-9. IEEE, 2021a.
Sayeed Shafayet Chowdhury, Nitin Rathi, and Kaushik Roy. One timestep is all you need: Training spiking neural networks with ultra low latency. arXiv preprint arXiv:21 10. 05929, 2021b.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Lc. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 102-703, 2020.
12

---

Published in Transactions on Machine Learning Research (12/2023)
Mike Davies, Narayan Srinivasa, -Sung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82 99, 2018.
Steven K Esser, Paul A Merolla, John Arthur, Andrew S Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David Berg, Jeffrey McKinstry, Timothy Melano, Davis R. Barch, et al. Convolu- tional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences of the United Stotes of America, 113(41) 2016.
Ziyang Kang, Lei Wang, Shasha Guo, Rui Gong, Shiming Li, Yu Deng, and Weixia Xu. ASIE: An asyn- chronous SNN inference engine for AER events processing. ACM Journal on Emerging Technologies in Computing Systems (JETC), (6(4):1-22, 2020.
Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in Neuroscience, 15:1638, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Souvik Kundu, Gourav Datta, Massoud Pedram, and Peter A Becrel. Spikc-thrift: Towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression. In Proceedings of the TEEE/CVF Winter Conference on Applications of Computer Vision, pp-
2021.
Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike- based backpropagation for training deep neural network architectures. Frontiers in Neu- roscience, 14:119, 2020.
Dongjin Lee, Seongsik Park, Jongwan Kim, Wuhyeong Doh, and Sungroh Yoon. Energy-efficient knowledge distillation for spiking neural networks. arXiv preprint vxXiv:2106.07177 2021.
Edgar Lemaire, Loic Cordone, Andrea Castagnetti, Pvere-EEmannee Novac, Jonathan Courtois, and Benoit Miramond. An analytical estimation of spiking neural networks energy efficiency. In International Con- ference on Neural Information Processing, pp. 574-587, Springer, 2022.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Liam P Maguire, T Martin McGinnity, Brendan Glackin, Arfan Ghani, Ammar Belatreche, and Jim Harkin. Challenges for large-scale implementations spiking neural networks on FPGAs. Neurocomputing, 71 (1-3):13 29, 2007.
Janardan Misra and Indranil Saha. Artificial neural networks in hardware: A survey of two decades of progress. Neurocomputing, 74(1-3):239-25) 255, 2010.
Simon Narduzzi, Siavash A Bigdeli, Shih-Chii Liu, and L, Andrea Dunbar. Optimizing the consumption of spiking neural networks with activity regularization. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 61-65. IEFE, 2022.
Katsumi Naya, Kyo Kutsuzawa, Dai Owaki, and Mitsuhiro Hayashibe. Spiking neural network discovers energy-efficient hexapod motion in deep reinforcement learning. IEEE Access, 9:150345 150354, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32:8024- 8035, 2019, https:
//pytorch.ort org/
Thomas Pellogrini, Romain Zimmer, and Timothée Masquelier. Low-activity supervised convolutional spiking neural networks applied to speech commands recognition. In 2021 IEEE Spoken Language Technology Workshop (SL.T) pp. 903. IEFF, 2021.
13

---

Published in Transactions on Machine Learning Research (12/2023)
Bodo Rueckauer, Inlia-Alexandra Lungu, Yuhuang Hu, Michael Pfciffer, and Shih-Chii Liu. Conversion of continuous-valuud deep networks to efficient event-driven networks for image classification. Fronliers in Newroscience, 11:682, 2017.
Sumit B Shrestha and Garrick Orchard. SLAYER: Spike layer error reassignment in time. Advances in Newral Information Processing Systems, 31:1419-1428, 2018.
Martino Sorbaro, Qian Liu, Massimo Bortone, and Sadique Sheik. Optimizing the energy consumption of spiking neural networks for neuromorphic applications. Frontiers in Neuroscience, 14:662, 2020.
Speaamann The proof and measurement of association between two things. American Journal of Psychology, 15:72-100 1904.
Kazuma Suetake, Shin-ichi Ikegawa, Ryuji Saiin, and Yoshihide Sawada. S3NN: Time step reduction of spik- ing surrogate gradients for training energy elficient single-step spiking neural networks. Neural Nelworks, 159:208 219, 2023.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST; A novel image dataset for benchmarking machine learning algorithms. arXiv preprint orXiv:1708.0774, 2017.
Atsushi Yaguchi, Taiji Suzuki, Wataru Asano, Shuhei Nitta, Yukinobu Sakata, and Akiyuki Tanizawa. Adam induces implicit weight sparsity in rectifier neural networks. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 318-325. IEEE, 2018
Friedemann Zenke and Surya Ganguli. SuperSpike: Supervised learning in multilayer spiking neural networks. Neural Computation, 30(6):1514-1544 2018
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Newral Information Processing Systems, 33:12022-1200 12033, 2020.
14

---

Published in Transactions on Machine Learning Research (12/2023)
Table 5: Network architectures. C(i ((1ss))) is a two-dimensional convolutional layer with channel size i1, kernel size = i2, stride size = ls, and padding size 14, FC(x) a fully connected layer with channel size x, S is spiking activation function (Eq. 2), BN is batch normalization layer, DO(x) is dropout layer with dropout rate I, Avg is 2 2 average pooling layer, and GAP is the global average pooling layer. Both ResA and ResB are the certain residual modules-Rossee S (())(1)) BN S ((()a())(())p))) and the identity function; ResB(x) consists of two paths, [BN C(x) ((3)21)))))) BN-S ((T))(3)(())(() and [BN-S, S C(F))(3)=(()(())
consists of two paths, [BN-
Name
Network architecture
CNN7
(( -N------------------------ 1I----DOOO.---C00xxxaaaaaaa BNN S-CL0))))a GAP
((61))0)))))) BN- -B
S-Avg
VGG11
0(512)(3))(1111) BN S DO(0.2) ((512) BN
FC(4096) BN DO(0.2) FC(4096) BN S DO(0.2) FC(10)
((64)))))(())(11)
(64)A(64) (os)))ResA(()))))) (Res)BRRReB888)) (128)
ResNet18
(ResB(256)-BB ReelA ) (512))))1))))--------)) (en))RReAAA (e5-(R))))
BN DN-S-C(10)unl (B-C(10)) GAP
A Details of Experimental Setup
A.1 Network architecture
The network architectures that we adopted in the experiments are described in Table Note that thi is independent of in those network architectures, and all the two-dimensional convolutional and fully connected layers have no bias terms.
The batch normalization layer affects the membrane potential (Eq. 1) as follows:
(Wis 14) + 7
(25)
where 14 and 0 E Rd denote the running average and standard deviation value for the post-synaptic current, W18t-1, respectively, and on and y E Rd are the trainable affine parameters for the batch normalization layer.
A.2 Dataset
We used three datasets as benchmarks and divided each datasel into three datasets train, validation, and test datasets as follows: the (#train dataset, #validation dataset, #test dataset) for each dataset is (54000, 6000, 10000) for Fashion-MNIST, (45000, 5000, 10000) for CIFAR-10, and (45000, 5000, 10000) for CIFAR-100. We fitted trainable parameters to the training dataset, optimized hyperparameters the validation dataset, and calculated all the metrics using the test dataset. We used random augmentation the data augmentation technique (Cubuk et al., 2020).
A.3 Hyperparameter
We used the following hyperparamcters. The weights were initialized by He initialization for the ReLU fune- tion (although we adopted the spiking neuron) and optimized using the Adam (8 = (0.9, 0.999), = 10-5) or mSGD (momentum = 0.9) optimizer. The mini-batch size was 100, the epoch size was 150, the spike firing threshold was uhh = 1, the learning rate and (a,7) for the surrogate gradient in Eq. 3 are summa- rized in Table 6, and the learning rate was scheduled by the cosine annealing (Tmax 150. Mmnn = 0.0) (Toshchilov & Hutter, 2017). Note that the penalty terms Ω were linearly scheduled, ie. À
15

---

Published in Transactions on Machine Learning Research (12/2023)
Table 6: Hyperparameters. The following hyperparameterss were used in the experiment unless otherwise
stated. The learning rate, ÄWD, a, and y were grid scarched.
Model
Optimizer
Surrogate gradient
Learning rate
AWD
BBN
CNN7
Adam
Eq. 3
le-3
le-4
0
0.25
0.6
CNN7
Adam
Eq. 23
le-3
1e-6
CNNT
Adam
Eq. 24
le-2
le-7
0
0.45
CNN7
mSGD
Eq.
1o-2
lc-4
0.35
0.6
VGG11
Adam
Eq. 3
1e-3
1e-3
0
0.25
0.6
VGG11
mSGD
Eq. 3
le-2
1e-3
0.35
0.8
R8sNet18
Adam
Eq. 3
1e-3
le-1
0
0.35
1.0
ResNet18
mSGD
Eq. 3
le-2
1e-3
0.35
1.0
Algorithm AUC(P)
Input: data X = {(1 v. M C R2 cutoff p' P/100 (0 S P < 100).
X + Sort(X,2;-1 x;-1 S 2;);
X+ + {(Ti, w:) E 34 1}];
+ (Io = z1, = 0) UX:
(xx V/c = max {yr E R. (It, E x))));
X ((11,W)) E X yj S (j < i)}; linearly interpolated curve for X; A(P) + Area under
y p' for range E [0, 1];
Return A(P)/(1-P))
multiplied by the ratio of the current epoch to the full epoch size. In the experiment for trade- off curve (Sec. 4.3), the normalized intensity parameter X' (Eq. 21) was selected from 14 patterns (1,2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192]. All of the experiments were conducted with three random seeds.
A.4 Metrics
For the quantitative analysis in Sec. 4.3, we used three quantitative scores: the area under the curve (AUC), Spearman's 's rank correlation coefficient (Spearman), and mutual information (MT). The argument P of the scores means that the score is calculated for data whose accuracy is over P. By AUC(P), we mean the normalized area under the and [P/100, 1), respectively. Therefore, the higher the AUC is, the lower the energy consumption metric maintaining the higher accuracy. Refer Alg. 1 and Fig. for detailed calculation of AUC(P). Note that our algorithm of AUC overestimates the non-monotonic curves, i.e., other methods than ours tend to be overestimated. Spearman's 's rank correlation coefficient p describes how well two ingredients by the monotonic function (Spearman, 1904). Therefore, the higher the Spearman's 's is, the higher was the aptitude of the intensity parameter as a trade-off controller. The mutual information is also suitable for the metric for a trade-off controller as it describes the mutual dependence between two ingredients, and higher score is better.
trade-off curve, where the range of the z- and y-axis is [0,1]
represented
B Details of Experimental Result
B.1 Difference in training time
Table presents the training time ratio of cach method to the bascline (A = 0). As indicated in this table, we cannot observe the significant difference in training time. Note that similar trend was observed in other settings.
16

---

Published in Transactions on Machine Learning Research (12/2023)
-K--
data
A(0)
7ZZ
A(70)
1.0
0.8
Accuracy
0.4
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Energy consumption rate
Figure 4: Example AUC(P) calculation (ResNet18 CIFAR-10 mSGD). A(P) denotes an area under a curve before normalization, and e denotes a linearly interpolated curve. Refer to Alg. for detailed definitions of them.
Table 7: Difference in training time. The network architecture is ResNet18, the dataset is CIFAR-10, the optimizer Adam, and p 1.
Penalty term
None
Sgyn 0.998
Potal
balance
Training time rate 1.000
0.968
0.999
B.2 Verification of Synaptic Interaction Penalty
The results of Sec. 4.2 are also presented in Tables 3 11. Note that the values in these tables were taken for and AWD from a point where the accuracy was very low (approximately 20%) until the accuracy reached the upper bound and stopped changing.
B.3 Energy-Accuracy Trade-Off Curve
The results of Sec. 4.3 are also presented in Fig. and Tables 12-21. For all the methods, the training results the mSGD optimizer tends to be more sensitive to the intensity of the penalty than those of the Adam optimizer. Additionally, the advantage of the proposed method over existing methods in VGG11 and ResNet18 appears to be smaller than that in CNN7. This discrepancy can be attributed the reduced variability of [[) Ω, (T)Il across layers in VGG11 and ResNet18 when compared to in CNN7 (Fig. 1). A detailed analysis of this observation will be part of our future work.
B.4 Additional Trade-Off Curves
The results from Sec. 4.3.4 also presented in Tables 22 23-3.
17

---

Published in Transactions on Machine Learning Research (12/2023)
The network architecture is CNN7, the dataset is Fashion-MNIST, and the optimizer is Adam. The remaining descriptions are consistent with those
Table 8: Performance with respect to varying X, AWD. and BBN.
detailed in Table 2.
A
AWD
BBN
Accuracy [%]
ESNN/ Passeline [%]
Dead rate [%
R [%]
0
92.25 = 0.143
100.0 + 0.530
6.350 + 0.055
14.19 + 0.436
1e-08
92.26 0.098
88.71 + 1.472
6.971 + 0.157
13.90 + 0.442
le-07
0
91.90 = 0.120
49.53 + 0.478
14.25 + 0.266
12.33 = 0.472
le-06
90.45 = 0.517
16.01 + 0.152
41.91 + 0.632
9.055 0.272
1e-05
0
80.60 = 4.078
5.289 + 0.717
76.00 + 0.992
4.014 + 0.303
le-04
92.35 = 0.034
91.96 + 0.749
7.484 + 0.095
14.57 + 0.571
1e-03
92.05 = 0.132
76.01 = 0.889
12.25 = 0.160
15.34 = 0.560
le-02
90.21 = 0.035
54.68 + 0.743
25.81 + 0.812
15.26 = 0.372
1e-01
69.26 10.32
35.26 + 1.263
51.27 + 0.807
13.67 = 0.980
1e-08
1e-04
92.31 = 0.165
82.88 1 0.417
8.169 + 0.084
14.34 + 0.706
1e-07
1e-04
92.01 = 0.059
48.29 + 0.268
15.24 1 0.379
13.09 1 0.571
le-06
1e-04
90.42 = 0.320
15.80 + 0.426
43.55 + 0.802
10.09 = 0.481
le-05
1e-04
80.59 1.119
4.133 + 0.533
78.53 + 0.325
4.468 0.141
1e-05
92.35 = 0.161
98.77 + 0.450
6.410 + 0.298
14.18 + 0.578
1e-04
92.39 = 0.138
89.38 + 0.876
7.771 + 0.102
14.70 + 0.524
1e-03
91.68 = 0.045
61.78 = 1.470
19.44 = 0.540
13.74 = 0.391
le-02
81.55 = 0.215
22.13 + 0.868
68.90 = 0.260
8.833 = 0.712
1e-01
10.00 0.000
0.000 1 0.000
100.0 1 0.000
0.000 0.000
1e-08
le-04
92.52 = 0.025
79.23 1 0.457
8.763 1 0.074
14.27 + 0.468
1e-07
1e-04
91.86 0.068
44.18 + 0.395
17.35 + 0.476
12.97 0.461
1e-06
1e-04
90.35 = 0.187
14.15 + 0.315
46.94 + 0.790
9.987 + 0.200
1e-05
1e-04
77.68 1.547
4.128 + 0.824
78.89 + 0.918
4.592 1.018
Table 9: Performance with respect to varying X, AWD, and BBN. The network architecture is CNN7, the dataset is Fashion-MNIST, and the optimizer is mSGD. The remaining descriptions are consistent with those
detailed in Table 2.
AWD
BBN
Accuracy [%]
ESNN/ Ebsseline [%]
Dead rate [%]
R [%]
0
0
92.02 = 0.142
100.0 + 1.288
11.17 + 0.749
14.34 + 0.643
1e-08
91.96 0.072
94.65 + 0.996
11.98 1 0.602
14.28 0.755
1e-07
0
91.62 0.119
61.72 + 2.295
19.16 + 1.284
13.12 + 0.677
1e-06
90.16 0.225
21.72 + 0.324
45.37 + 1.028
10.26 + 0.781
le-05
0
77.14 0.217
4.651 = 0.270
79.80 = 0.902
4.370 = 0.257
1e-05
92.11 = 0.114
98.84 + 2.827
11.41 + 0.633
14444 + 0.763
1e-04
92.01 = 0.130
98.50 = 0.888
11.90 = 0.786
14.76 0.774
1e-03
91.84 = 0.229
85.42 + 2.323
14.92 + 0.359
15.75 = 0.979
1e-02
89.70 = 0.605
73.98 + 0.855
22.17 + 0.477
15.02 0.647
1e-01
0
62.45 = 1.900
19.16 + 1.442
62.08 + 1.776
13.46 + 0.259
1e-08
1c-04
92.13 = 0.
91.49 + 2.061
12.72 + 0.450
14.62 0.856
1e-07
1e-04
92.01 = 0.135
61.60 + 1.440
19.45 + 0.833
13.56 + 0.793
1e-06
1e-04
90.48 = 0.420
21.20 + 0.151
45.99 + 1.018
10.81 = 0.665
1e-05
le-04
73.40 7.835
4.413 = 0.958
80.46 + 3.399
5.385 = 1.278
le-05
92.15 = 0.154
100.6 1 1.111
11.38 1 0.742
14.44 +:: 0.712
1e-04
92.24 = 0.021
103.0 + 1.589
10.89 + 0.781
14.77 0.877
1e-03
91.64 = 0.161
83.64 + 1.326
16.90 + 1.336
13.89 - 1.016
1e-02
79.16 = 3.171
30.91 + 1.941
70.40 + 1.285
8.008 0.381
le-01
10.00 = 0.000
0.000 + 0.000
100.0 + 0.000
0.000 = 0.000
1e-08
le-05
= 0.077
93.33 + 1.838
12.43 + 0.928
14.26 0.624
1e-07
1e-05
91.86 0.165
61.27 + 1.850
19.44 + 0.982
13.21 + 0.631
le-06
le-05
90.20 = 0.170
21.53 +
45.55 + 1.673
10.45 0.608
le-05
le-05
73.70 2.893
4.297 + 0.546
81.37 - 2.846
4.293 = 0.441
18

---

Published in Transactions on Machine Learning Research (12/2023)
Table 10: Performance with respect to varying X, AWD. and BBN.
The network architecture ResNet18,
the dataset is CIFAR-10, and the optimizer Adam. The remaining descriptions are consistent with those detailed in Table 2.
A
AWD
BBN
Accuracy [%]
ESNN Ebaseline [%]
Dead rate [%]]
R [%]]
0
91.32 + 0.209
100.0 0.293
1.074 0.033
17.74 0.067
1e-09
0
0
91.27 = 0.175
93.06 + 0.486
1.325 + 0.020
16.75 + 0.023
le-08
0
0
91.40 = 0.293
58.03 + 0.222
3.846 + 0.067
11.69 0.055
1e-07
0
0
89.46 + 0.215
19.13 + 0.247
25.36 + 0.825
4.936 + 0.010
le-06
0
78.76 + 0.239
4.328 + 0.192
65.38 + 0.283
1.629 0.015
le-05
0
0
19.53 + 2.170
0.010 0.004
99.36 0.094
0.040 + 0.008
0
le-05
0
92.15 = 0.032
92.79 + 0.547
2,088 + 0.077
16.55 + 0.086
le-04
92.21 0.196
79.67 0.721
5.377 0.084
14.22 0.097
0
1e-03
91.64 + 0.135
70.58 + 0.114
12.08 + 0.185
12.38 + 0.064
1e-02
86.88 1 0.466
50.96 1.071
32.84 1.270
10.59 0.099
0
1e-01
0
80.94 = 0.264
26.48 + 0.815
58.03 + 1.090
7.163 + 0.073
0
le++00
0
60.29 + 2.386
13.99 + 0.264
76.32 + 1.462
5.239 0.136
1e-09
1e-04
0
92.44 + 0.093
74.74 0.310
5.851 0.116
13.57 + 0.078
le-08
le-04
0
92.12 + 0.177
52.82 + 0.206
8.275 + 0.161
10.20 0.012
1e-07
1e-04
0
90.18 = 0.135
15.27 0.075
31.96 0.398
3.895 0.063
1e-06
1e-01
0
75.38 + 0.766
2.385 + 0.449
74.48 + 1.881
1.320 + 0.054
1e-05
1e-04
0
15.95 3.741
0.002 0.001
99.72 0.046
0.032 0.014
0
le-05
1
91.87 + 0.065
89.63 + 0.447
2.467 + 0.022
15.55 + 0.043
le-04
91.97 0.183
66.36 0.600
10.98 0.076
10.32 0.079
0
1e-03
85.93 + 0.244
23.50 + 0.551
54.07 + 0.387
3.321 + 0.039
0
1e-02
19.22 + 6.452
0.574 + 0.040
99.19 + 0.492
0.114 0.011
1e-09
1e-04
1
91.79 + 0.215
61.58 0.537
11.45 0.088
9.699 + 0.034
le-08
le-04
1
91.85 + 0.187
38.43 + 0.512
15.40 + 0.274
6.616 0.097
1e-07
1e-01
88.97 + 0.227
12.29 0.292
40.56 0.295
2.735 0.039
1e-06
1e-04
1
73.40 + 0.533
2.153 + 0.115
77.12 + 0.622
0.996 0.038
1e-05
1e-04
15.79 + 3.494 1
0.001 0.001
99.74 + 0.071
0.027 0.016
19

---

Published in Transactions on Machine Learning Research (12/2023)
Table 11: Performance with respect to varying X, AwD, and BBN- the dataset is CIFAR-10, and the optimizer is mSGD. The remaining descriptions are consistent with those detailed in Table 2.
The network architecture ResNet18,
A
AWD
BaN
Accuracy [%]
BaNN/ Ehaseline [%]
Dead rate [%
R [%]]
90.15 0.291
100.0 0.583
1.188 0.060
17.40
0.074
1e-09
0
0
90.31 + 0.294
95.21 + 0.552
1.275 + 0.032
16.70 +: 0.086
le-08
0
89.83 1 0.092
60.41 0.201
3.120 0.072
11.57 0.040
1c-07
0
0
87.86 1 0.274
16.01 + 0.067
26.68 1 0.865
4.135 1: 0.022
1e-06
0
54.23 + 3.671
2.911 2.568
90.89 + 0.931
1.389 0.410
1e-05
0
0
15.65 + 3.972
0.613 + 0.490
99.04 + 0.247
0.176 + 0.034
0
1e-04
0
90.56 + 0.098
98.25 + 0.253
1.157 + 0.134
17.26 + 0.038
0
1e-03
92.59 + 0.150
86.46 0.818
1.380 0.092
15.96 0.113
0
1e-02
0
91.86 = 0.036
74.17 + 0.965
3.932 + 0.277
14.19 + 0.213
1e-01
88.24 0.432
61.39 2.329
19.10 1.422
12.66 0.274
0
le++00
14.65 + 1.956
17.60 1 1.014
54.54 1 2.851
4.290 1 0.438
1c-09
1e-03
92.44 + 0.268
81.41 0.328
1.588 0.028
15.23 0.026
1e-08
1e-03
92.48 + 0.124
48.00 + 0.160
4.646 + 0.111
10.37 + 0.056
1e-07
1e-03
89.05 + 0.245
11.51 + 0.320
43.32 + 0.177
3.997 0.050
1e-06
1e-03
65.07 + 1.377
0.794 + 0.100
91.15 + 0.777
1.105 + 0.044
le-05
le-03
0
12.85 = 2.377
0.036 + 0.051
99.63 + 0.335
0.065 0.066
0
le-05
89.98 + 0.142
98.41 0.691
1.163 0.027
17.06 0.091
0
1e-04
90.24 = 0.211
80.90 + 0.597
1.886 + 0.058
13.80 + 0.071
0
1e-03
89.84 0.162
32.15 0.172
26.02 0.286
5.147 0.030
0
1e-02
10.04 + 0.064
0.563 + 0.523
99.68 1 0.279
0.111 + 0.099
1c-09
le-04
90.17 + 0.192
74.70 0.739
2.154 0.056
12.91 0.114
1e-08
1e-0d
90.08 + 0.071
44.94 + 0.286
5.625 + 0.230
8.558 + 0.045
1e-07
1e-04
88.17 + 0.151
13.89 + 0.066
31.53 + 0.324
3.586 0.016
1e-06
1e-01
1
59.17 + 3,370
1.119 + 0.522
91.11 + 0.966
0.992 + 0.092
le-05
le-04
16.01 = 0.654 1
1.087 + 1.670
99.05 + 1.117
0.236 0.257
Table 12: Quantitative comparison corresponding to (CNN7
Pashion-MNIST mSGD). The descriptions
are consistent. with those detailed in Table 4.
Method
[UC
AUC(50) [%)]
Spearman(70)
Spearman(50)
MI(70)
MI(50)
Daya (p = 1) (Ours)
65.67
77.18
0.9652
0.7486
3.355
3.454
Oaya 2) (Ours)
58.19
70.15
0.9387
0.7794
3.332
3.434
Pustal (p 1)
60.90
74.22
0.9557
0.7705
3.178
3.295
Diotal (p 2)
52.73
66.27
0.9313
0.9562
3.062
3.233
Obalance (p 1)
45.42
59.26
0.9100
0.9229
2.871
2.926
Opalance (p = 2)
28.53
38.09
0.8694
0.8694
2.813
2.813
Table 13: Quantitative comparison corresponding to (VGG11 consistent with those detailed in Table
CIFAR-10 / Adam).
The descriptions are
Method
AUC(70)%%)
AUC(50) [%)
Spearman(70)
Spearman(50)
MI(70)
MI(50)
Qayn (p = 1) (Our)
52.56
68.63
0.9365
0.9497
3.610
3.688
Qayn 2) (Ours)
46.85
59.73
0.7758
0.8445
3.536
3.671
Protal (p = 1)
52.21
68.69
0.8883
0.9121
3.583
3.663
Diotal (p 2)
47.38
60.58
0.7400
0.7953
3.506
3.592
Abalance (p 1p=10
41.21
55.66
0.8770
0.8770
3.344
3.344
Dbalance (p 2)
40.68
53.07
0.8633
0.9006
3.436
3.556
20

---

Published in Transactions on Machine Learning Research (12/2023)
1.0
Accuracy
0.8
(p= 1) (Ours)
Dexu (p=1)
Qudeess (p=
(p==)=2) (Ours)
Qansd (p=2)
- x
Quieee (p=
0.7
Legend for the curves
0.6
0.5 0.0
0.2
0.4
0.6
0.8
1.0
Energy consumption
(CNN7 Mishion-MNIST / mSGD)
0.9
0.9
Accurncy
0.8
Accuracy
0.8
0.7
0.7
0.6
0.6
0.50
0.5
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Energy consumption rate
Energy consumption rute
(VGG11 / CLEAR-10 / Adam)
(VGG11 / CIEAR-10 / MSGD)
1.0
1.0
0.9
0.9
Accuracy
0.8
Accurncy
0.8
0.7
0.7
0.6
0.50.0
0.2
0.4
0.6
1.0
0.5
0.8
1.0
Energy consumption
Energy consumption
(ResNet18 CIFAR-10 Adam)
(ResNet18 CIFAR-10 / mSGD)
1.0
1.0
0.9
0.9
Accuracy
0.8
Accuracy
0.8
0.7
0.6
0.6
0.5
0.5
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Energy consumption rate
Energy consumption rate
(ResNet18) / CIFAR-100 / Adam)
(ResNet18 CTFAR-100 mSGD)
Figure 5: Energy-accutt trade-off curve. The energy consumption rate is the energy consumption of each model normalized by that of the baseline model. The network architecture/ dataset / optimizer are referred to below each figurc.
21

---

Published in Transactions on Machine Learning Research (12/2023)
Table 14: Quantitative comparison corresponding to (VGG11/ CIFAR-10 mSGD).
The descriptions are
consistent with those dctailed in Table
Method
AUC(50) [%]
Spearman (70)
Spearman(50)
M1(70)
M1(50)
Qayn (p = 1) (Ours)
58.82
72.98
0.9576
0.9613
3.454
3.485
Gayn = 2) (Ours)
45.73
60.29
0.9348
0.9348
3.308
3.308
Stotal (p = 1)
56.62
69.23
0.9631
0.9631
3.401
3.401
Qsoaal (p 2)
44.82
59.09
0.9373
0.9373
3.401
3.401
Aualance (p = 1)
35.94
48.02
0.8525
0.4420
3.075
3.163
Analance (p 2)
32.15
46.92
0.9591
0.9633
3.295
3.332
Table 15: Quantitative comparison corresponding to (ResNet18 / CIFAR-10 Adam). consistent with those detailed in Table 4.
The descriptions
Method
AUC
AUC(50) [%]
Spearman(70)
Spearman(50)
MI(70)
MI(50)
Payn 1) (Ours)
67.68
80.07
0.9021
0.9261
3.308
3.412
Qayn (p = 2) (Ouss)
54.60
62.74
0.8169
0.7917
3.454
3.573
Quntal (p
66.51
78.67
0.9321
0.9475
3.296
3.400
Quotal (p = 2)
55.08
64.29
0.8751
0.8846
3.412
3.536
balance (p
65.19
76.63
0.9484
0.9572
3.434
3.526
Obalance (p = 2)
51.07
58.70
0.7931
0.7441
3.412
3.506
Table 16: Quantitative comparison corresponding to (ResNet18 consistent with those detailed in Table
CIFAR-10 /SGD). The descriptions are
Method
AUC(70)[%)
AUC(50) [%]]
Spearman(70)
Spearman(50)
MI(70)
MI(50)
Qayn (p = 1) (Ous)
68. 68.85
80.56
0.9009
0.9274
3.244
3.355
Qayn = 2) (Ours)
62.15
75.04
0.9660
0.9730
3.454
3.545
Qotall (p = 1)
67.47
79.15
0.9573
0.9573
3.308
3.308
Ptotal (p 2)
59.02
72.34
0.9469
0.9552
3.496
3.555
Abalance (p = 1)
67.57
78.56
0.9389
0,9350
3.295
3.401
Pualance (p 2) 2)
56.18
68.44
0.9408
0.9201
3.319
3.389
Table 17: Quantitative comparison corresponding (ResNet18 / CIFAR-100 / /Adam). The descriptions consistent with those detailed in Table 4. Note that the metrics with P = 70 are excluded because the maximum accuracy is approximately 70%.
Method
AUC(50)[%)
Spearman(50)
MI(50)
Gsya (p = 1) (Ours)
38.98
0.9091
3.496
Payn (p (Ours)
33.86
0.8162
3.496
Pootal (p = 1)
38,05
0.9520
3.454
Piotal (p 2)
33.15
0.8143
3.355
Abalance (p=10 = 1)
38.00
0.8536
3.412
Pbalance (p=
32.69
0.7170
3.401
Table 18: Quantitative comparison corresponding (ResNet18 CIFAR-100 mSGD). The descriptions consistent with those detailed in Table 4. Note that the metrics associated with P 70 have been excluded because the maximum accuracy is approximately 70%.
Method
AUC(50)
Spearman(50)
MI(50)
Psya (p = 1) (Ours)
39.74
0.9879
2.303
Oayn (p 2) (Ours)
33.98
0.9704
2.272
Ooou (p = 1)
38,57
1.0000
2.303
Ptotal (p = 2)
33.74
0.9758
2.303
Obalance (p (p=1) 1)
38.32
0.9636
2.303
Qbalance (p - 2)
31.47
0.9152
2.303
22

---

Published in Transactions on Machine Learning Research (12/2023)
Table 19: Quantitative comparison corresponding to (CNN7/
/
Fashion-MNIST / Adam) using Eq. 23. The
descriptions are consistent with those detailed in Table
Method
AUC(70)[%)
AUC(50) [%]
Spearman(70)
Spearman(50)
MT(70)
MT(50)
Qayn 1) (Ours)
68.67
79.98
0.9831
0.9863
3.091
3.258
Dayn (p = 2) (Our)
60.19
71.01
0.9608
0.9496
2.833
3.135
Ototal (p
64.78
77.19
0.9765
0.9835
2.772
2.890
Rrotll (p = 2)
52.99
63.03
0.9492
0.9609
3.091
3.178
Dalance (p
46.25
65.31
0.7549
0.8211
2.833
2.944
Ahalance (p = 2)
40.83
49.20
0.9588
0.9789
2.772
2.995
Table 20: Quantitative comparison corresponding to (CNN7/ / Fashion-MNIST / Adam) using Eq. 24. The descriptions are consistent with those detailed in Table 4.
Method
AUC(50) [UC(70)[%)
Spearman(70)
Spearman(50)
MT(70)
MT(50)
Qayn (p = 1) (Ours)
68.66
80.05
0.9935
0.9948
3.044
3.178
Dayn (p 2) (Ours)
60.01
71.01
0.9676
0.9797
2.772
3.091
Qtotal (p = 1)
64.03
76.76
0.9365
0.9460
2.813
2.871
Ototal (p 2)
54.16
62.96
0.9123
0.9123
3.075
3.075
Apalance (p = 1)
49.28
66.39
0.9236
0.9390
2,890
3.044
Analance (p 2) 2)
39.97
48.82
0.9500
0.9628
2.772
2.890
Table 21: Quantitative comparison with Sorbaro al. (2020) corresponding to (CNN7 / Fashion-MNIST
Adam). The descriptions consistent with those detailed in Table 4.
Method
AUC(70)[%)
AUC(50)
Spearman
Spearman(50)
MT(70)
MT(50)
Dayn (p 1) (Ours)
68.02
79.60
0.9861
0.9865
3.465
3.610
Dayn (p = 2) (Ours)
61.62
72.69
0.9474
0.9709
3.233
3.476
ANN2SNN (T 1)
53.51
67.80
0.6863
0.6275
4.316
4.355
ANN2SNN (T = 5)
47.40
63.17
0.8548
0.7015
4.030
4.385
ANN2SNN (T 10)
37.93
56.30
0.8602
0.6044
3.784
4.406
Table 22: Quantitative comparison corresponding to Fig. 3 (A) (Drota)). The doscriptions are consistent with
those detailed in Table 4.
Method
[
AUC(50) [%]
Spearman(70)
Spearman(50)
MI(70)
MI(50)
Paya 1) (Ours)
61.12
74.00
0.9879
0.9930
2.303
2.485
Qayn (p = 2) (Ous)
54.24
65.93
1.000
1.000
2.197
2,398
Quntal (p =
61.81
74.84
1.000
1,0000
2.197
2.303
Protal (p = 2)
53.48
63.06
1.000
0.9833
2.079
2.197
Obalance (p
55.99
68.73
1.000
1.000
1.946
1.946
Abalance (p = 2)
31.92
43.19
0.9000
0.9429
1.609
1.792
Table 23: Quantitative comparison corresponding to Fig. 3 (B) (Abalance). The descriptions are consistent with those detailed in Table
Method
AUC(70
AUC(50)
Spearman(70)
Spearman(50)
MI(70)
M1(50)
Doyn (p = 1) (Ours)
44.18
59.91
0.9879
0.9930
2.303
2.485
Qayn (p 2) (Ours)
40.14
54.67
1.000
1.000
2.197
2.398
Protal (p 1)
57.01
71.21
1.000
1,0000
2.197
2.303
Quatal (p = 2)
51.18
62.91
1.000
1.000
2.079
2.197
Pralance (p = 1)
62.79
74.80
1.000
1.000
1.946
1.946
Abalance (p= 2)
41.15
53.19
0.9000
0.9429
1.609
1.792
23

---

Published in Transactions on Machine Learning Research (12/2023)
Table 24: Quantitative comparison corresponding to Fig. 3 (C) (dead neurons). The descriptions are con-
sistent with those detailed in Table 4. Note that the AUC scores have been computed based the inverted curve relative to = 0.5.
Method
AUC
AUC(50) [%]
Spearman(70)
Spearman(50)
MI(70)
MI(50)
Dayn 1) (Ours)
6.856
8.742
-0.9879
-0.9930
2.303
2.485
Qayn (p mp=22 2) (Ours)
4.919
6.581
-1.000
-1.000
2.197
2.398
Qustal (p
6.550
8.615
-1.000
-1.000
2.197
2.303
Orotal (p= = 2)
4.361
5.432
-1.000
-1.000
2.079
2.197
Oalance (p =
5.122
6.883
-1.000
-1.000
1.946
1.946
Dalance (p = 2)
2.042
2.939
-0.0000
-0.9429
1.609
1.792
24


Reviews:
[{"review_id": "HDfXkCGGgt", "reviewer": "Reviewer_M4zp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the topic of optimization of SNN for reduced energy consumption is of interest for this venue\n- the manuscript is, for the most part, easy to follow and understand; the theoretical derivation is especially clear\n- while the use of this penalty term was previously introduced (as acknowledged in the manuscript), herein the authors justify its usage and comparatively analyze its impact on energy/accuracy trade offs\n- reported results show that, subject to hyperparameter tuning, this penalty term compares favorably against alternative choices for penalty and it is effective in obtaining SSN with improved energy efficiency (according to the selected energy consumption metric) while suffering limited to no accuracy degradation\n- extensive results on hyperparameter sweeps are reported and discussed\n- overall, the paper presents a valuable technique to train SNN with reduced energy consumption\n\nWeaknesses:\n\n- The reader is immediately presented with a figure (Fig. 1) of hard interpretation. Its caption stating \"the y-axis represents the ratio of some layer-wise metric or penalty term to some total metric or penalty term\" is also unclear. I understand the desire to present one of the main results upfront (in this case, the proportionality between the proposed penalty term and the energy consumption) but at this early stage it is just confusing and the significance cannot be grasped until much later (Section 3.3).\n- I am not sure what motivated the choice of highlighting *CCN7* results in Fig. 1, and later showing similar outcomes on VGG11 and ResNet18 in Fig. 3, while highlighting *VGG11* accuracy (Table 2 and 3), leaving CNN7 and ResNet18 results to the Appendix. Seems inconsistent\n- The authors state that their objective is to discuss trends not present state-of-the-art accuracy. Although I do not disagree, starting from baselines that are 2-3% below the best available on the given networks and tasks, somewhat detracts from the strength of their conclusions.\n- the proposed method requires manual selection of penalty intensity (lambda), which is model and task dependent\n- Section 4.3.4 and corresponding Fig. 4 are unclear. My understanding is that the same training of previous section (Fig. 3A) is utilized, but alternative metrics are monitored and plotted against accuracy. A re-writing of the first paragraph could help the reader. In addition, when stating \"training with a specific penalty term leads to a reduction in the associated metric, especially for the p=1 option\", it is not clear what are these plots being compared to.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates various trade-offs between energy consumption and classification accuracy in Spiking Neural Networks (SNN), when using a loss penalty term that reduces the spike firing rate. The expected value of the chosen penalty is proportional to the SNN total energy consumption metric. The selection of the penalty term is justified theoretically. The impact of tuning the penalty term contribution to the loss is investigated, and compared against competing choices for penalty and backpropagation gradient functions. Results are reported on 3 networks (CNN7, VGG11, ResNet18) and 3 datasets (Fashion-MNIST, CIFAR-10, CIFAR-100).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Some minor clarifications, restructuring, and re-writing as requested under \"Weaknesses\"", "broader_impact_concerns": "Broader impact concerns are not discussed in the manuscript. No specific concern on my end.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l9yZd9MPwr", "reviewer": "Reviewer_kTgm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1.\tThe proposed loss function is more hardware-accurate than traditional firing-rate-based loss functions. In addition, the authors used experimental results to show the proposed loss function works as expected and gets higher energy reduction compared to firing-rate-based methods.\n\n2.\tThe authors performed very detailed experiments showing many aspects of the result. The experiment includes 3 vision-based datasets and 3 convolutional neural networks. The scale of the experiment is large enough to support the claim.\n\nWeakness\n\n1.\tThe paper didn’t deliver a clear message on the dead neurons. There are mainly two important points that are not very clear:\n\na.\tHow to decide if a neuron is dead or not? Do the authors use the training set to make the decision or use validation or test set to make the decision?\n\nb.\tHow does the dead neuron influence the computational cost of the network? Do the authors prune the dead neurons after training?\n\n2.\tThere are some inaccurate claims in the paper:\n\na.\tThe energy cost defined in Equations 10 & 11 is not the total energy cost of SNN inference. The total energy cost of SNN inference on a typical neuromorphic processor usually consists of three components: event pre-processing cost, synaptic integration cost, and firing cost. Equation 10 & 11 only defines the synaptic integration cost. The authors need to be clear on the claim of this part.\n\nb.\tThe hardware model used by the authors is under the assumption that the synaptic integration cost will dominate the energy cost of SNN inference. This is true with two conditions: first, the post-synaptic dimension needs to be high enough to balance the event pre-processing overhead; second, the activity sparsity cannot be too low. I think the model here satisfies the two conditions. However, the authors should explain these explicitly to make it clearer.\n\nc.\tThe paper claimed multiple times that the proposed method is independent to surrogate gradient function. However, I think this claim is not accurate since the method needs a surrogate gradient function to work. I think it’s better to claim the method is independent to the selection of surrogate gradient function, or the method can work under different surrogate gradient functions.\n\nQuestions:\n\n1.\tThe state-of-the-art ANN sparsity-aware training works normally fine-tuned trained model with sparsity loss. Why do the authors choose to train the models with sparsity loss from the beginning?\n\n2.\tFrom Figure 6, the reviewer observed a much smaller advantage of the proposed method than the traditional firing-rate-based method. Can the authors discuss in more detail why this happens?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a loss function to sparsify the activity of a single-step spiking neural network for energy-efficient inference. Different from traditional firing-rate-based loss functions, the proposed loss function is proportional to the energy cost of synaptic integrations in the SNN. The authors performed extensive experiments and showed that the proposed loss function could achieve better accuracy compared to firing-rate-based loss functions when having the same synaptic operation reduction. In addition, the authors also demonstrate the effect of dead neurons when using weight decay together with the sparsity-aware loss function, which further increases the energy efficiency of the network.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please address the concerns in the weaknesses and questions.\n\n2.\tCan the authors explain the Conv computation in Equation 6 in more detail? Does this also consider downsample operations?\n\n3.\tDoes the R in Equation 9 represent the overall firing rate? It looks like R is defined as the addition of layer firing rates, but not the overall firing rate of the network. \n\n4.\tThe symbols used in Figure 2 are used before definition in the main text.\n\n5.\tIn the last line of Page 7, the author claimed the neuron sparsification is stronger with Adam than mSGD. However, the review cannot get the same conclusion from Tables 2 & 3. Can the authors provide more detailed interpretations of the results to support this claim?\n\n6.\tIn the discussion section, the authors claim it’s possible to use equilibrium training for the proposed network. However, it’s not very clear to the reviewer how the proposed one-step network can use equilibrium training which requires the network to have a dynamical state. Can the authors give a more extended discussion on this?", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gk5smSHT3o", "reviewer": "Reviewer_V1jZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper has a good balance of math derivations and numerical experiments, although most of the math is quite straightforward. I don't find major weaknesses in this paper whereas I think the presentation can be improved (see Requested Changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study proposes a new penalty term to optimize the energy consumption in spiking neural networks. The author performed experiments to show the proposed penalty is better than existing penalties in terms of better accuracy given the same energy consumption.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Major\nI strongly suggest the author change the wording \"spiking synaptic penalty\". This is because the __synapse__ refers to the connection weight between neurons, then penalizing the synapse can mean some penalty on weight, e.g., the L2 penalty on weights. A potential name could be _synaptic interaction penalty_ because Eq. 13 shows the penalty is proportional to the non-negative output of the multiplication of the weight matrix with the spiking output. Some other names are OK as long as avoid this confusion.\n\n### Minor\n- Eq. 15 and 16: It looks like they don't hold but differ by the factor $T$, i.e., the size of the time step.\n- The 2nd paragraph after Eq. 19: an undefined reference is shown as ??.\n- A better way to illustrate the spiking synaptic penalty reflects the energy consumption metric than others is by generating a scatter plot where each dot shows the penalty and energy consumption in a layer. Then we can directly see whether the dots are in a linear line.\n- it is better not to present Fig 1 on the first page because the notations are undefined at this moment and thus it is not easy to understand.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers note that the proposed method to reduce the energy consumption of SNNs is of interest. The authors propose a loss function that is more accurate than traditional firing-rate-based loss functions. The methods is well-justified and extensive simulations support the main claims. All reviewers therefore propose acceptance of the manuscript.
Decision: Accept (poster)