Can we get the best of both Binary Neural Networks and Spiking Neural Networks for Efficient Computer Vision?
Authors: Gourav Datta
Zeyu Liu
Peter Anthony Beerel
Venue: ICLR 2024 poster (2024)
Keywords: BNN
Hoyer regularizer
gradient descent
FLOPs
object detection
Primary area: general machine learning (i.e., none of the above)
TL;DR: We present a novel training framework for sparse binary activation neural networks (BANN) that achieves the state-of-the-art in the trade-off between performance and compute efficiency.

Abstract: Binary Neural networks (BNN) have emerged as an attractive computing paradigm for a wide range of low-power vision tasks. However, state-of-the-art (SOTA) BNNs do not yield any sparsity, and induce a significant number of non-binary operations. On the other hand, activation sparsity can be provided by spiking neural networks (SNN), that too have gained significant traction in recent times. Thanks to this sparsity, SNNs when implemented on neuromorphic hardware, have the potential to be significantly more power-efficient compared to traditional artifical neural networks (ANN). However, SNNs incur multiple time steps to achieve close to SOTA accuracy. Ironically, this increases latency and energy---costs that SNNs were proposed to reduce---and presents itself as a major hurdle in realizing SNNs’ theoretical gains in practice. This raises an intriguing question: *Can we obtain SNN-like sparsity and BNN-like accuracy and enjoy the energy-efficiency benefits of both?* To answer this question, in this paper, we present a training framework for sparse binary activation neural networks (BANN) using a novel variant of the Hoyer regularizer. We estimate the threshold of each BANN layer as the Hoyer extremum of a clipped version of its activation map, where the clipping value is trained using gradient descent with our Hoyer regularizer. 
This approach shifts the activation values away from the threshold, thereby mitigating the effect of noise that can otherwise degrade the BANN accuracy. Our approach outperforms existing BNNs, SNNs, and adder neural networks (that also avoid energy-expensive multiplication operations similar to BNNs and SNNs) in terms of the accuracy-FLOPs trade-off for complex image recognition tasks. Downstream experiments on object detection further demonstrate the efficacy of our approach. Lastly, we demonstrate the portability of our approach to SNNs with multiple time steps. Codes are publicly available [here](https://github.com/godatta/Ultra-Low-Latency-SNN).

Published as a conference paper at ICLR 2024
CAN WE GET THE BEST OF BOTH BINARY NEURAL NETWORKS AND SPIKING NEURAL NETWORKS FOR EFFICIENT COMPUTER VISION?
Gourav Datta"* Zeyu Liu", Peter A. Beerel University of Southern California, Los Angeles, CA, USA {gdatta, liuzeyu, fabeerel]@use edu "Equally contributing authors *Currently employed at Amazon Inc.
ABSTRACT
Binary Neural networks (BNN) have emerged as an attractive computing paradigm for wide range of low-power vision tasks. However, state-of-the- art (SOTA) BNNs do not yield any sparsity, and induce a significant number of non-binary operations. On the other hand, activation sparsity can be provided by spiking neural networks (SNN), that too have gained significant traction in re- cent times. Thanks to this sparsity, SNNs when implemented on neuromorphic hardware, have the potential to be significantly more power-efficient compared traditional artifical neural networks (ANN). However, SNNs incur multiple time steps to achieve close to SOTA accuracy. Ironically, this increases latency and energy-costs that SNNs were proposed to reduce-and presents itself as a major hurdle in realizing SNNs' theoretical gains in practice. This raises an in- triguing question: Can we obtain SNN-like sparsity and BNN-like accuracy and enjoy the eneryy-effciennc benefits of both? To answer this question, in this pa- per, we present a training framework for sparse binary activation neural networks (BANN) using novel variant of the Hoyer regularizer. We estimate the threshold of each BANN layer as the Hoyer extremum of clipped version of its activation map, where the clipping value is trained using gradient descent with our Hoyer regularizer. This approach shifts the activation values away from the threshold, thereby mitigating the effect of noise that can otherwise degrade the BANN accu- racy. Our approach outperforms existing BNNs, SNNs, and adder neural networks (that also avoid emergy-expensic SNNs) in terms of the accuracy-FL.O -LOOss trade-off for complex image recognition tasks. Downstream experiments on object detection further demonstrate the effi- cacy of our approach. Lastly, we demonstrate the portability of our approach SNNs with multiple time steps. Codes are publicly available here
multiplication operations similar to BNNs and
INTRODUCTION
Due to its low memory footprint and use of cheaper pop-count operations instead of energy- expensive muliiplyaaddaccuuullass (MAC), BNNs have emerged a promising low-power al- ternative to compute- and memory-expensive deep neural networks (DNN) (Rastegari et al., 2016 Liu et al. 2020a; 2018a). Recent works have proposed novel network architectures (Bethge al.
2021
Zhang et al. 2022 Shi et al. 2022] Hu et al.) 2022) and training algorithms (Zhijun Tu & Wang 2022a Xu al.
202Tb; Rastegari et al.) 2016 Bulat & Tzimiropoulos; 2019 Geng al. Chen et al. 2001
2023
Xu et al., 2021a Kim et al., 2021 Wang et al., 2023 Lee et al. 2022]
approximate the full- full-precision representation of the weights and activations with 1-bit bi-polar
values without significant drop in accuracy.
However, most of these efforts induce significant number of non-binary operations which degrade the computational efficiency. For example, ReactNet-based BNNs (Liu et al.) 2020al Zhijun Tu Wang 2022b) incur custom non-linear functions, including RPReLU that are significantly more complex compared to threshold or ReLU operations, duplicated basic blocks that significantly in- crease the total number of floating point operations (FI.OPs) and parameter count. Moreover, all

---

Published as a conference paper at ICLR 2024
these BNNs employ the sign quantization function to generate the bi-polar weights and activation values. However, bi-polar activations do not yield any sparsity and thus can not benefit any com- putational efficiency from hardware that can leverage sparsity. Moreover, shown in (Wang al.) 2020b Lin et al. 2023; Falkena et al. 2023), it is also unclear whether sign is the optimal binariza- tion function.
In contrast, SNNs can provide significant sparsity, even up to 80% for ultra-low number of time steps (Chowdhury et al. 2021). However, this comes at the cost of significant accuracy drop (Datta al.) 2021j without any specialized neuron model (Datta et al. 2023b). In an attempt to push the frontier of sparsity-accuracy trade-off, we propose class of uni-polar binary activation neural networks (BANNs) that can enjoy sparsity similar to SNNs while getting rid of the temporal dimension, and achieve BNN-like accuracies. Our BANNs are more compute-efficiens than existing bi-polar BNNs. This efficiency stems from the fact that we can skip the memory access of the weight and the eventual accumulate/bit-count operation when the corresponding activation value is O.
Our Contributions. Our training framework for BANNs is based on a novel application of the Hoyer regularizer and a novel Hoyer thresholding layer. More specifically, our threshold is training- input-dependent and is set to be equal to the Hoyer extremum a clipped version of the activation tensor, where the clipping value is trained using gradient descent with our Hoyer regularizer. In this way, compared to traditional uni-polar BNNs and SNNs with ultra-low number of time steps, our threshold increases the rate of weight updates and our Hoyer regularizer shifts the activation distribution away from this threshold, improving convergence. We consistently surpass the accura- cies obtained by SOTA uni-polar BNNs (Wang et al., 2020b] Sakr et al.) 2018) diverse image recognition datasets with different convolutional architectures. Compared to SNNs, and adder neu- ral network (AddNN) models that are also compute- compute-efficient, our BANN models yield higher test accuracy with a ~5.55x reduction in floating point operations (FLOPs), thanks to the extreme spar- sity enabled by our training framework. Incorporating the training framework of BANNs can also improve the test accuracy in advanced bi-polar BNNs. Downstream tasks on object detection also demonstrate that our approach surpasses the test mAp of existing BNNs and SNNs. Our approach can also be incorporated to SNNs with multiple time steps, thereby leading to small but significant accuracy increase at the cost of significant increase in memory and compute cost. Thus, our pro- posed approach acts as a continuum between one time-step sparse BNNs and low time-step SNNs, and can help bridge both the BNN and SNN communities for low- low-power vision tasks.
2
PRELIMINARIES
2.1 HOYER REGULARIZER
Based on the interplay between é, and E2 norms, a new measure of sparsity was first introduced in (Hoyer) 2004). based on which, (Yang et al.) 2020) proposed a new regularizer, termed the Hoyer regularizer for the trainable weights that was incorporated into the loss term to train DNNs. We adopt the same form of Hoyer regularizer for the activation to train our BANN models as Huw)
[|u/lla] |
(Kurtz et al. 2020). Here, || ||: represents the E, norm of the activation tensor uy and the superscript ē for the time step omitted for simplicity. Compared to the & and l regularizers, the Hoyer regularizer has ccaleinvvrriane (similar to the to regularizer). It is also differentiable almost everywhere (see equation] I where |ui] represents the element-wise absolute of the tensor u,
(H 21|uu111 Du
|uill
(iin(u)) 1
|uill2
24
|u
Letting the gradient 8H(m) -0 and making all the ui positive, the value of the Hoyer extremum
Du.
becomes E(u)= Iw
12 This extremum is the minimum, because the second derivative is greater |
than zero for any value of the output element. Training with the Hoyer regularizer can effectively help push the activation values that are larger than the extremum (u,> F(u)) even larger and those that are smaller than the extremum (u F(:)) even smaller.
2

---

Published as a conference paper at ICLR 2024
2.2 BINARY NEURAL NETWORKS
Given a input 0t-1 E Rex and weight w E R=xxxxxx of a layer I, we can get the output Wt E Rexh xwo by convolution operation as o -1 wt To accelerate the inference process, bi-polar BNNs (Liu al. 2020a] Zhijun Tu Wang 2022b) partition the input and weight into two clusters, -I and +I with sign function as Eq. 2.
1, {
if 2 0;
sign (I)
1-,, otherwise
To yield sparsity similar ReLU activation, uni-polar BNNs (Wang et al. 2020b; Sakr et al. 2018) parition the weights to 11 and +1, while the activations to O and as illustrated below in Eq. 3.
1, if 2 1;
O
0, otherwise
where denotes the normalized activation output, and with denotes a trainable threshold parame- ter. While our BANNs primarily focus on sparse uni-polar activations with full-precision weights (activations are significantly more difficult to binarize compared to weights (Lin al.) [201bb] Ding et al. 20199), we also demonstrate the efficacy BANNs with binary weights (-1 and +1) for fair comparison with BNNs.
3 PROPOSED TRAINING FRAMEWORK
Our approach is inspired by the fact that Hoyer regularizers can shift the pre-activation distributions away from the Hoyer extremum in DNN (Yang et al. 20200) Our principal insight that setting our activation threshold to this extremum shifts the distribution away from the threshold value, reducing noise and improving convergence. To obtain our BANNs, we present a novel Hoyer threshold layer that sets the threshold based upon Hoyer regularized training process, as described below.
3.1 HOYER THRESHOLD LAYER
As illustrated above in Eq. 3, a binary neuron with unit step tivation function is difficult to op- timize with straight through es- timator (STE) based approaches (Bengio al. 2013), which ei- ther approximates the binary neu- functionality with continu- ous differentiable model or sur- rogate gradient approaches used in SNNs (Panda & Roy] [0066] Lee al. (016)) This is because the number of ones in the acti- vations becomes too low ad- just the weights sufficiently using gradient descent. If pre-synaptic neuron does not output 1, the synaptic weight connected it can- not be updated because its gradient from neuron is calculated as Juy XO,, where Pay is the gradient of the activation uy and o is the output of neuron i.
1.0
101A twvstoak
scale
Jown-scaled gradient
clip
pm
(a)
Hoper thyichadd
(b)
zitkotior
dz
H,(clip
Mk
1.0
14
1.0 2.0
Figure I: (a) Comparison of our Hoyer threshold function with existing uni-polar binary activation functions where the blue distribution denotes the shifting of pre-activation values away from the threshold using Hoyer regularized training, (b) Pro- posed derivative of our Hoyer threshold function.
Therefore, it crucial to reduce the value of the threshold to generate enough Is for better network convergence. However, a sufficiently low value of threshold can generate but that would yield random outputs in the final classifier layer. Hence. it is challenging to yield the optimal threshold for each layer. Previous works (Wang et al.) (2020b) show that this problem can be partially mitigating by training the threshold term otth using gradient descent. However, that still leads to a large accuracy gap with the SOTA bi-polar BNNs, especially for ImageNet-level tasks.
for every neuron,
In contrast, we propose to dynamically down-scale the threshold (see Fig. aa)) based on the ac- tivation using our proposed form of the Hoyer regularizer. In particular, clip the activation
3

---

Published as a conference paper at ICLR 2024
corresponding to each convolutional layer to the trainable threshold etth obtained from the gradient descent with our Hoyer loss, as detailed later in Eq. [2] Unlike existing approaches (Rathi et al. 2020a; Chowdhury et al. 2021) that require 1 to be initialized from pre-trate pre-trained full-precision model. our approach can be used to train BANNs from scratch with a Kaiming uniform initializa- tion (He et al. 20155 for both the weights and thresholds. In particular, the normalized down-scaled threshold value, with which we compare the normalized activation 2,, for each layer is computed as the Hoyer extremum of the clipped activation tensor shown in Fig. aa) and below.
1, if 2(
clip 21
21, 21,if05a151
o1-h((Zz)=
(4)
0, otherwise
0,
Note that our normalized threshold F(= is less than the normalized threshold whose value is
for any output (proof in supplementary materials). Hence, actual threshold value E clip)
2(2)-) x
is indeed less than the trainable threshold E used in carlier works (Datta & Beerel 2022 Rathi
et al. 2020a). We also observe that the Hoyer extremum in each layer changes only slightly during the later stages of training, which indicates that it is most likely inherent attribute of the dataset and model architecture. Hence, to estimate the threshold during inference, we calculate the expo- nential average of the Hoyer extremums during training, similar to batch normalization (BN) layers, and use the same during inference.
3.2 HOYER REGULARIZED TRAINING
The loss function (Lrotal) of our proposed approach is shown below in Eq. 5
L-1
Liotal = LCE + AHL# = LCE + λи H clip
(5)
where LOE denotes the cross-entropy loss calculated on the softmax output of the last layer L, and Lи represents the Hoyer regularizer calculated on the input of our Hoyer threshold layer after dividing the threshold term of and clipping. The weight update for the layer L-1 computed as
aLCE +AH owi,-1
dLH
aLCE d7L-1 duL-1 +AH ou,-1 ow
0LH duL-1
AWL-1=
Owi-11
do,-1 Tug- iw
OLCE do1-11
(H(2)-1)
cip
(6)
001-1 dui-1
+λH duL-1
OL-2
OLCE òLog Ou,
(s y)WL
dOL-1 dur do1-1
where 8 denotes the output softmax tensor, i.e., Σ of where ui and wř denote the 7th and
t elements of the activation tensor of the last layer L, and N denotes the number of classes. Note that y denotes the one-hot encoded tensor of the true label, and (H(u)) computed using Eq. dug The last layer does have any threshold and yields full-precision outputs.
The weight and threshold update computations for the hidden layers are in Appendix A.I.
Note that all the derivatives to update the trainable parameters can be computed by Pytorch auto- grad, except the derivative do: dsy whose gradient is zero almost everywhere and undefined at 2y=0. We extend the existing idea of surrogate gradient descent (SGD) (Neftei et al. 2019 in SNNs to compute this derivative for our BANNs with Hoyer threshold layers, as illustrated in Fig. Tbb) and mathematically defined as follows.
do:
scalex1
i0 < 21 < otherwise
θzi
where scale denotes a hyperparameter that controls the dampening the gradient. Note that o jumps from 0 to 1 at =1 We assume the surrogate gradient is non-zero spanning a width of 1 in both directions around 1

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of the test accuracy of our BANNs with full-precision DNNs for object recog-
nition. Model* indicates that we remove the first max pooling layer, and Sp. denotes sparsity.
Network
dataset
DNN (%)
BANN (%)
Sp. (%)
VGGI6
CIFARIO
94.10
93.44
78.13
ResNet18
CIFARTO
93.34
92.50
74.17
ResNet18
CIFARIO
94.28
93.88
83,88
ResNet20
CIFARIO
93.18
92.39
76.31
ResNet34
CIFARIO
94.68
93.53
83.96
ResNe150-
CIFARIO
94.90
94.07
82.21
VGGT6
ImageNet
70.08
68.00
75.52
ResNet50
ImageNet
73.12
66.98
76.11
1.4
(Se8 43)
(Sm0
Tw2 (5-0 381
Tod (5-0.33 35)
T-1 (5-0.171
T-4 (S-0.27)
T-1 15-0.121
T-3 05-0.30
T-1 ours (S-0.20
spiking activity
1.0
T-3 (S-0.23)
T-1 ours (5-0.16)
Log, Normalized Energy
CIFAR10
0.8
ImageNet
0.8
(e)
(b)
(d)
0.4
0.2
0.0
10
33
34
8
10
ImageNNc
Layer
Layer
Figure 2: Layerwise spiking activities for a VGG16 across time steps ranging from 5 to (average spiking activity denoted as S in parenthesis) representing existing low-latency SNNs including our work (which represents one time step) on (a) CIFAR1O, (b) ImageNet, (c) Comparison of the total energy consumption between SNNs with different time steps and full-precision DNNs.
EXPERIMENTAL RESULTS
Datasets Models: Similar to existing BNN and SNN works (Rathi et al. 2020b a Liu et al. 2020a), we perform object recognition experiments on CIFAR10 (Krizhevsky 2009) and ImageNet
2009) dataset using VGG16 (Simonyan Zisserman, 2014) and several variants of ResNet (He et al. 2016) architectures. For object detection, we use the MMDetection framework (Chen al. 2019) with PASCAL VOC2007 and VOC2014 (Everingham et al., 2010) as training dataset, and benchmark our BANN models and the baselines on the VOC2007 test dataset. We use the Faster R-CNN (Ren et al. 2015) and RetinaNet (Lin et al. (017a) framework, and substitute the original backbone with our BANN models pretrained on ImageNet.
(Deng et al.
Object Recognition Results: For training the recognition models, we use Adam (Kingma & Ba 2014) optimizer for VGG16, and use SGD optimizer for ResNet models. As shown in Table with FP weights, we obtain the SOTA accuracy 93.44% on CIFARIO with VGG16; the accuracy of our ResNet-based BANNs ImageNet also surpasses the existing works. Note that our ResNet models are based on the spike-element-wiss (SEW) architecture (Fang et al., 2020) as detailed in Appendix A.2. On ImageNet, we obtain a 68.00% top-1 accuracy with VGGT6 which is only ~2% lower compared to the iso-architecture full-precision counterpart. All our BANN models yield a sparsity of 75% or higher both CIFARI0 10 and ImageNet, which is significantly higher com- pared to existing SNNs/BNNs as shown in Fig. 2
Accuracy Comparison with SNNs: We compare
results with various SOTA ultra low-latency Our BANNs yield comparable or better test accuracy
SNNs for image recognition tasks in Table 2 compared to all the existing works both CIFARIO and ImageNet, with significantly lower infer- ence latency (due to the existence of multiple time steps in SOTA SNNs). The only exception for the latency reduction is the one-time-step SNN proposed in (Chowdhury al. 2021), however, increases the training time significantly as illustrated later in Fig. 3.
Energy Efficiency Comparison with SNNs and full-precision DNNs: We compare the energy- efficiency our BANNS with full-precision DNNs and existing multi-time-seep SNNs in Fig. The compute-efficiency of BANNS stems from two factors: 1) sparsity, that reduces the number of synaptic operations in convolutional and linear layers compared to full-precision DNNs according to SNN = S₁ x DNN/Llops (Chowdhury et al. 2021), where S) denotes the average number of spikes per neuron per inference over all timesteps in layer 7. Note that the sparsity induces small
5

---

Published as a conference paper at ICLR 2024
Comparison of our BANN models existing SNNs. SGD and hybrid denote surrogate gradient descent and pre-trained DNN followed by SNN fine-tuning respectively. (qC, dL) denotes an architecture with q convolutional and d linear layers. T.S denotes the number of time steps. *Results reported without autoaugmentation and cutout for fair comparison with our work.
Table 2:
Ref.
Training
Architecture
Acc. (%)
T.S
Dataset CIFARIO
Meng et al.
2022)
SGD
PreAet-ResNet19
95.40
Deng al. 2021)
DNN-SNN conversion
VGGI6
92.29
16
Bu et a 2022a)
DNN-SNN coonversion
VGG16
90.96
8
Zhang & Lt| 2020)
SGD
5C, 2L
91.41
Rathi et al. 2020aa
Hybrid
VGGI6
92.70
5
Zheng et al.) (2021)
STBP-tdBN
ResNet19
93.16
Datta | 2022)
Hybrid
VGGT6
91.79
Bu al. 2022b)
DNN-SNN conversion
VGGI6
91.18
2
Fang et al.
20200)
SGD
5C, 2L
93.50
Deng et al.
(0225)*
SGD
ResNet18
94.58
2
Xiao et al.
2022)
SGD
VGG (sWS)
93.73
6
Deng et al.
(022))
SGD
ResNet19
94.16
Li et al.
202
SGD
ResNer18
93.13
2
Chowdhury ct al.
2021
Hybrid
VGGI6
93.05
Chowdhury et al.
2021
Hybrid
ResNer20
91.10
Ours
Adam+Hoyer Reg.
VGGI6
93.44
Dataset : ImageNet
(Bu et al.
2022b
DNN-SNN conversion
ResNet34
59.35
16
Li al.
2021b)
SGD
ResNet18
71.24
Rathi et al.
(2020a)
Hybrid
VGGI6
69.00
5
Fang et al.
2021
SGD
ResNet34
67.04
rang et
202T
SGD
5esNet152
69.26
4
Deng et al. 2023)*
SGD
ResNet34
65.77
2
Duan et al.
2022)
SGD
ResNet34
68.28
Deng al.
2022)
SGD
ResNet19
68.00
4
Rathi et al.
2020a)
Hybrid
VGGTT
69.00
5
Zheng et al. al..|2222
(2021)
STBP-1dBN
ResNet34
67.05
Chowdhury et al.
2021)
Hybrid
VGGI6
67.71
Ours
Adam+Hoyer Reg.
VGG16
68.00
overhead of checking whether the 1-bit activation zero, which consumes 0.05pJ in 28nm Kintex-7 FPGA platform according to our post place-and-route simulations. 2) Use of only AC (1.8pJ) op- erations that consume 7.4x lower compared to each MAC (13.32pJ) operation in our FPGA setup for floating point (FP) representation. Note that the binary activations cations with logical operations, i.e., conditional assignment to with a bank of AND gates. These replacements may be realized using existing hardware depending the compiler and the details of their data paths. Building a custom accelerator that can efficiently implement these reduced opera- tions is also possible (Wang et al. 2020a Frenkel et al., 2019 Lee & Li 2020). Additionally, BANNs also enjoy superior
replace the FP multipli-
compared to existing multi-ime-step SNNs since the latter requires the membrane potentials and weights be fetched from and read the on-/off- chip memory for each time step. Our BANNs can avoid these repetitive read/write operations as it does involve any state and lead to Tx reduction in the number of memory accesses compared to a T-time-step SNN model. Compared to traditional bi-polar BNNs, our BANNs can also reduce the number of memory accesses with the support of zero gating logic leveraging the high activation sparsity. This can be achieved by skipping the reading of the weights when the activation associated to it is zero. However, the exact savings in memory energy will depend the weight reuse scheme and the underlying hardware. We have provided the detailed energy model of traditional BNNs, multi-time-step SNNs, BANNs, and full-precision DNNs, considering the sparsity overhead and memory accesses without any weight reuse scheme, in Appendix A.9. We compare the layer-wise spiking activities S, for time steps ranging from 5 to 1 in Fig. a-b) that are computed based on the

---

Published as a conference paper at ICLR 2024
Table Accuracies from different strategies to train BANNS on CIFARIO
Pretrained DNN(%)
Training Strategies
BANN (%)
Sparsity (%)
93.15
91.39
23.56
Iterative training (N=10)
93.25
92.68
10.22
Iterative Training (N=20)
92.68
92.24
9.54
Proposed Training
93.13
22.57
Table Accuracies and energies of weight quantized BANN models based on VGG16 on CIFARIO where FP is 32-bit floating point.
Spiking Activity (%)
Dataset
Bits
Acc. (%)
Energy (mJ)
CIFARIO
FP
93.44
22.57
18.67
CIFARIO
6
93.11
22.46
7.46
CIFARIO
2
92.94
21.39
5.66
CIFARIO
1
92.80
22.68
5.02
ImageNet
FP
68.00
24.48
424.3
ImageNet
2
66.86
23.82
231.3
ImageNet
I
64.62
23.99
221.8
replicated implementation of existing low-latency SNN (Rathi et al. 2020a), against our work (where the spiking activity denotes the number of 1s) that represents a time step 1. Note, the spike rates decrease significantly with time step reduction from 5 to I, leading to considerably lower FLOPs in our one-time-step SNNs. These lower FLOPs, coupled with the 7.4x reduction for AC operations and the sparsity overhead, lead to 22.7x and 31.9x reduction in compute energy CIFARIO and ImageNet respectively with VGG16. Considering the memory accesses as provided in our energy models in A.9, the total energy reduces by 70.28x and 81.13x.
Comparison with different training strategies: Based on existing BNN and SNN literature, we hypothesize that two training strategies that can be effectively used train BANNs, other than our proposed approach.
Pre-trained DNN, followed by BANN fine-tuning. Similar to the hybrid training proposed in (Rathi ct al. 2020b), we pre-train a full-precision DNN model, and copy its weights to the BANN model. Initialized with these weights, train our BANN with normal cross-entropy loss.
Iteratively convert ReLU neurons to binary neurons. First, we train a DNN model which uses the ReLU activation with threshold, then iteratively reduce the number of the ReLU neurons whose output activations are multi-bit. Specifically, we first force the activation values in the top N per- centile to output 1, and those in bottom N percentile percent to output O, and gradually increase N until the accuracy drops beyond a certain threshold or all neuron outputs are either I or 0. Proposed training from scratch. With training, we train a BANN model from scratch. Our results with these training strategies are shown in Table [ which indicates that it is difficult for training strategies that involve pre-training and fine- tuning to approach the accuracy of full- preccisonn models with BANNs. One possible reason for this might be the difference in the distribution of the aeeaciivationn values between the DNN and BANN models (Datta & Beerel| 2022). Our Hoyer threshold layer and our Hoyer regularizer, can help train BANN model with SOTA accuracy from scratch.
proposed Hoyer threshold layer and Hoyer regularized
order to further improve the energy efficiency of our BANNs, we per- form quantization-awant training of the weights in models to 1-6 bits for both CIFAR10 and ImageNet. This transforms the FP ACS to 1-6 bit ACs, thereby leading to a 4.8- reduction in compute energy as obtained from our FPGA simulations. The reduced weight precision also reduces the memory access cost by ~5-32: with efficient bit-packing (bit) 2021) for 1-6 bits according to FPGA simulations. Note that only quantize the convolutional layers, as quantizing the linear layers lead to a noticeable drop in accuracy. As shown in Table 4 when quantized 6 bits, our VGG-based BANN incur negligible accuracy drop of only 0.02% CIFAR10, while reducing the total energy by 3.3x (energy computed from the models developed in A.9). Even with 1-bit quantization, our model can yield an accuracy of 92.80% without any special modification, while still yielding a sparsity of 788%
Effect on Quantization:
Comparison with AddNNs & BNNs: We compare the accuracy and energy of BANN
7

---

Published as a conference paper at ICLR 2024
Table 5: Accuracy Energy Comparison of our BANNS to AddNNs and BNNs
Reference
Dataset
(%)
Energy (mJ)
Uni-polar BNNs
Sakr al. 2018
CIFARTO
89.6
4.94
Wang et al.
2020b)
ImageNet
59.7
213.0
Bi-polar BNNs
Diffenderfer & Kailkhura
2021)
CIFARIO
91.9
14.72
AddNNs
Chen ct
2020) (FP weights)
CIFARTO
93.72
95.22
Chen al.
2020)
(2-bit weights)
CIFARIO
92.08
24.57
Chenn ct al.
2020) (FP weights)
ImageNet
67.0
2291.2 .2
Our BANNS
This work (FP weights)
CIFARIO
93.44
18.67
This work (1-bit weights)
CIFARTO
92.80
5.02
This work (FP weights)
ImageNet
68.00
424.3
This work (1-bit weights)
ImageNet
64.62
221.8
Table 7: Test accuracy obtained by our approach with multiple time steps for SNNs on CIFARIO.
Architecture
Time steps
Acc. (%)
Spiking activity (%)
VGGI6
93,44
21.87
VGG16
2
93.71
44,06
VGG16
6
94.14
101.22
ResNet18
1
91.48
25.83
RseNet18
91.93
33.24
ResNet18
6
92.01
83.82
models with recently proposed AddNN models (Chen al. 2020) that also removes multi- plications for increased energy-efficiency in Table [ With the VGGI6 architecture, on CI- FARIO, we obtain 0.6% lower accuracy, while on ImageNet, we obtain 1.0% higher accu- racy. Moreover, unlike our BANNs and SNNs, AddNNs do not yield any sparsity, and con- sume ~5.1x more compute energy (with the sparsity overhead incorporated) compared to our BANNs on average across both CIFAR1O and ImageNet (see Table 55) We also com- pare our BANN models with SOTA BNNs in Table 7 that replaces the costly MAC op- crations with cheaper pop-count counterparts, thanks to the binary weights and activations. Our BANNs with 2-bit quantized weights con- sume 2.6x lower energy compared to the bi-polar BNNs (see (Diffenderfer & Kailkhura 2021) in Table 5). This is due to the improved trade-off between the high sparsity activity (~78% as shown in Table 1) provided by our BANN models, and less energy due XOR operations and 1-bit memory access compared to weight-quantized ACs and 2-bit memory access. On the other hand, our BANNs with 1-bit weights i.e., sparse uni-polar BNNs consume similar energy compared to existing unipolar BNNs (see (Sakr et al. 2018 Wang et al. 2020b) in Table while yielding 2.6% higher accuracy (92.80% vs 90.2%) iso-architecture. Lastly, it is worth mentioning that though our sparse BNNs can surpass test accuracies obtained by few existing BNNs, it fails yield accuracies similar to the SOTA bi-polar BNNs (Zhang et al.) 2022] Bethge et al.] 202T; Zhijun Tu & Wang 2022a Xu et al., 20216). This is due their higher expressivity (further explained in A.11) and network modifications (e.g. ReactNet with PReLU compared to our simple VGG and ResNets) that increases the FLOPs
Table 6: Impact of our proposed Hoyer regularized training frame- work on the test accuracy with advanced BNNs
Work Poke 1.0x
Acc. (%)
Acc. w/ Hoyer (%)
73.4
73.6 (+0.2)
MeliusNet59
71.0
72.3 (+1.3)
ReCU (Resnet18)
66.4
67.8 (+1.4)
CIFAR1O and 4.9% higher accuracy (64.62% 59.7%) ImageNet
by more than 2x. However, applying our proposed training technique these BNNs results in
8

---

Published as a conference paper at ICLR 2024
0.2-1.3% increase in test accuracy shown in Table This is done using the same way using the Hoyer extremum of the normalized and clipped activation as the threshold, where the clipping is done at -1 and +1 (instead of 0 and +1 for our BANNs). This demonstrate the efficacy of our proposed method.
Extension to SNNs: We extend our proposed approach to multi-time-step SNN models by adopting the standard LIF model (Sengupta et al. 2019) for the neurons but computing the threshold based on the Hoyer extremum defined in Eq. and unrolling the gradients derived in Section 3.2 using traditional backpropagation through time (Rathi et al. 2020a). As shown in Table 7 as time step increases from to the accuracy of our model also increases from 93.44% to 94.14%, which validates the effectiveness of our method. However, this accuracy increases comes at the cost of a significant increase in spiking activity (see Table / where the spiking activity is computed across the total number of time steps similar to (Sengupta et al.] 2019)), increasing the compute energy. The memory cost also increases due to the repetitive potential and weight accesses across time steps.
5 RELATIONSHIP BETWEEN SNN, BNN, AND BANN
Our BANNs are identical to uni-polar sparse BNNs (Sakr et al. 2018 Wang et al. 2020b) when the weight precision is quantized down to I-bit. Our BANNs are also identical to one-time-step SNNs, and can be readily extended to traditional multi-time-step SNNs with the incorporation the membrane potential, using the traditional LIF model for the neurons where the membrane potential integrates the weight modulated input spikes and leaks over time. We use the soft reset mechanism that reduces the membrane potential by the threshold value when output spike generated (Datta) jet al., 2023a). It has been shown that soft reset minimizes the information loss by allowing the spiking neuron to carry forward the surplus potential above the firing threshold to the subsequent time step O. We use our proposed combination of Hoyer regularized training and Hoyer spike layer to train the per layer threshold, while we train the weights and leak term using SGL. The dynamics of our LIF model for our multi-time-step SNNs are shown below.
U (t) (,)
((()==ifU?"*
else O, (100)
Here (t) denotes the membrane potential of the 7th neuron at timer step 6, (t) denotes the binary output of the 7th neuron time step t, yth denotes the threshold, À denotes the leak, and Wi denotes the weight connecting the pre-synaptic neuron j and the neuron i.
The sparse activation in BANNS bears resemblance to SNNs while the difference is that there is no temporal dimension. On the other hand, the 1-bit sparse activation in BANNs bears resemblance to BNNs. in particular uni-polar BNNs, while the difference is that the weight precision is multi-bit. Thus, BANNs can bridge the gap between BNNs and SNNs by bringing the activation sparsity of SNNs to BNNs, and removing the temporal dimension of SNNs for equivalence with BNNs.
CONCLUSIONS
In this work, we propose BANNs trained using a novel variant of Hoyer Regularizer and Hoyer threshold layer that jointly optimizes the distribution of the BANN pre-activations and the placement of the BANN threshold in order to improve the -cuursc-FLOPss trade-off compared to existing networks. Our Hoyer regularizer introduces loss term that penalizes the normalized and clipped activation values around their Hoyer extremum (which depends on the SGD trained threshold). Our proposed Hoyer threshold layer takes advantage of this distribution shift by setting the IF threshold to the Hoyer extremum. Our BANN models surpass test accuracies obtained by existing SNN, uni- polar BNN, and AddNN models with similar or better energy- efficiency. Our training framework can also improve the accuracy of a number of advanced bi bolpo BNNs. Our SNNs can further improve the inference energy and lateney efficiency with recently proposed in in-sensor computing systems (Datta et al. [202d] a c where the bandwidth between the sensor that implements the first few CNN layers and the back-end processing unit that implements the remaining layers can be significantly reduced, thanks to the binary activations for only one time step (Datta et al. (023c).
7 ACKNOWLEDGEMENT
This work is supported by gift funding from Intel Labs.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Bit-Packing Compressor in FPGA-Hardware.
2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 330.3433,8
Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, and Christoph Meinel. Meliusnet: An improved network architecture for binary neural networks. In 2021 IEEE Winter Conference on
Applications of Computer Vision (WACV), pp. 1438- 1447, 2021. 2021.00144
doi: 10.1109/WACV446630.
Tong Bu, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Optimized potential initialization for low- latency spiking neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1):11 1111:111-2,,, Jun. 2022a. doi: 10.1609/aaii. w.t.................. URL bttps://ojss....... aaai. org
Tong Bu, Wei Fang. Jianhao Ding, PENGLIN DAI, Zhaofei Yu, and Tiejun Huang. Optimal ANN- SNN conversion for high-accuracy and ultra-low-lateng, spiking neural networks. In Interna- tional Conference on Learning Representations, 2022b. URL http:::t///// httpss:///ppeeeieereeee n
forum?id=7B31JWA
Adrian Bulat and Georgios Trimiropoulos. Xnor-net++: Improved binary neural networks, 2019.
Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open MMLab detection toolbox and benchmark. arXiv preprint arXiv: / 990600016555 2019.
Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, and Zhangyang Wang. "bnn- bn ?": Training binary neural networks without batch normalization. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4619-4629, June 2021.
Sayeed Shafayet Chowdhury et al. One timestep all you need: Training spiking neural networks with ultra low latency. arXiv preprint arXiv:2110.05929, 2021.
G. Datta, S. Kundu, Z. Yin, R. T. Lakkireddy, J. Mathai, A. P. Jacob, P. A. Beerel, and A. R. Jaiswal. A processing in- su--pxxeliin--ee -n---oo--mmyy paradigm www.orrce-oooootrriiiieeeee tinyml applications. Scientific Reports, 12, 2022a.
Gourav Datta and Peter A. Beerel, Can deep neural networks be converted to ultra low-latency spiking neural networks? In 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE), volume 1, pp. 718-723, 2022. doi: 1222919/DDTTES4114220022.2
Gourav Datta, Souvik Kundu, Akhilesh R. Jaiswal, and Peter A. Beerel. ACE-SNN: Algorithm- Hardware co-design of energy-efficient & low-latency deep spiking neural networks for 3D image recognition. Frontiers in Neuroscience, 16. 2022b. URL https: /wwww/wwwww..mm front iersin..
org/articles/ 3tt.eiieeeeee...9///... 3338//fninn...... miniiinnn02.22... 022..112252288
Gourav Datta, Souvik Kundu, Zihan Yin, Joe Mathai, Zeyu Liu, Zixu Wang. Mulin Tian, Shunlin Lu, Ravi T. Lakkireddy, Andrew Schmidt, Wael Almagede Ajey P. Jacob, Akhilesh R. Jaiswal, and Peter A. Beerel. P2M-DeTrack real-time multi-object detection and tracking. arXiv preprins arXiv:2205. /4285, 2022c.
for energy-efficient and
Gourav Datta, Zihan Yin, Ajey Jacob, Akhilesh R. Jaiswal, and Peter A. Beerel. Toward efficient hyperspectral image processing inside camera pixels. arXiv preprint arXiv:2203.05996 2022d.
10

---

Published as a conference paper at ICLR 2024
Gourav Datta, Haoqin Deng, Robert Aviles, Zeyu Liu, and Peter A. Beerel. Bridging the gap be- tween spiking neural networks & Istms for latency & energy efficiency. 2023 IEEE/ACM International Symposium Low Power Electronics and Design (ISLPED), pp. 1-6, 2023a. doi: 10.1109/1LLPDDS842 202..
Gourav Datta, Zeyu Liu, James Diftenderfer, Bhavya Kailkhura, and Peter A. Becrel. When bio- inspired computing meets deep learning: Low-latency, accurate, & energy-efficient spiking neural networks from artificial neural networks. arXiv preprint arXiv:2312.06900 2023b.
Gourav Datta, Zeyu Liu, Md Abdullah-AA Kaiser, Souvik Kundu, Joe Mathai, Zihan Yin, Ajey P. Jacob, Akhilesh R. Jaiswal, and Peter A. Beerel. In-sensor & neuromorphic computing are all you need for energy efficient computer vision. ICASSP 2023 2023 IEEE Interna- lional Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023c. doi: 10. 1A110099ASS599955..2. 9000
Gourav Datta et al. Training energy-efficient deep spiking neural networks with single-spike hybrid input encoding. In 202/ International Joint Conference on Neural Networks (IJCNN), volume 1, pp. 1-8, 2021. doi: 10.1 1LL100//CCNN5533722222111.2 9534306.
Deng. Dong, R. Socher, L.-J. Li, Li, and L. Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Shikuang Deng. Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient weightin.. International Conference on Learning Representa- lions, 2022. URL
www. met/forummid= wtw.rrmmmid_NNiiisiI...
Shikuang Deng, Hao Lin, Yuhang Li, and Shi Gu. Surrogate module learning: Reduce the gradient error accumulation in training spiking neural networks. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 7645-7657. PMLR, 23-29 Jul 2023. URL https: /ttp:////rcceeddes... mlr. wtt.peess/v2222eeg23332222 html
Shikuang Deng et al. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations, 2021.
James Diffenderfer and Bhavya Kailkhura. Multi-prize lottery ticket hypothesis: Finding accu- rate binary neural networks by pruning randomly weighted network. In International Confer- ence on Learning Representations, 2021. URL htttps://pperevvee..o nww.:w..e/eeeee.ee www.eeeee/ooiiooooo
mat 0b9iv
Ruizhou Ding, Ting-Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11400-11409, 2019. doi: 10. 10.11999CV
Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Bel- grave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL https: openreview http:ptp///peeeeeiiieee// net www.//rutttttt id- fLIgyyQijqz
M. Everingham, L. Van Gool, K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338 June 2010.
Sieger Falkena, Hadi Jamali-Rad, and Jan van Gemert. Lab: Learnable activation binarizer for binary neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6425-6434, January 2023.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothee Masquelier, Tiejun Huang, and Yonghong Tian. In- corporating learnable membrane time constant to enhance learning of spiking neural networks. arXiv preprint arXiv:2007.05788, 2020.
11

---

Published as a conference paper at ICLR 2024
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In M. Ranzato, A. Beygelz- imer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural
Information Processing Systems, volume 34, pp. 21056-21069. Curran Associates,
Inc., 2021. URL htttps://proceeddngg wewpeedddinnn..euiissssses wttp://pppp///////1112111 1LE23183898220044422222522-----2
Charlotte Frenkel, Martin Lefebvre, Jean-Didier Legat, and David Bol. A 0.086-mm² 12.7 7ppppp/spp 64k-synapse 256-neuron online-learning digital spiking neuromorphic processor in 28-nm emos. IEEE Transactions on Biomedical Circuits and Systems, 13(1):145-158, 2019. doi: 10.1109/ TBCAS.2008.2288
Xue Geng, Jiawei Hu, Jie Lin, Zhe Wang, Shaohua Li, Min Wu, and Mohamed M. Sabry Aly. MABA-net: Masked additive binary activation network, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, arXiv preprini urXiv::502.0000 018222 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pailern recognition, pp. 770-778, 2016.
Patrik O Hoyer.
Non-negative matrix factorization with sparseness constraints. Journal of machine learning research, 5(9), 2004.
Jie Hu, Ziheng Wu, Vince Tan, Zhilin Lu, Mengze Zeng. and Enhua Wu. Elastic-link for binarized neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1):942 :942-950., Jun. 2022. doi: (wt66///aaii.366.......
(RRtss:s::///f aaai. orq index. php/
Hyungjun Kim, Jihoon Park, Changhun Lee, and Jae-Joon Kim. Improving accuracy of binary neural networks using unbalanced activation distribution. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 7862-788 1676,7871, June 2021.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection, 2019.
Youngeun Kim and Priyadarshini Panda. Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks, 144:686-69% 2021. ISSN 0893-6080. doi: https://doi.ooo org/
10.1016/j.eummet2221..ggo 022.0.00....22 URL https: //www//wwww////// www.://wweeeeeett...coocceceeeee
atticlee/s11/00833882220200
Youngeun Kim, Joshua Chough, and Priyadarshini Panda. Beyond classification: Directly train- ing spiking neural networks for semantic segmentation. Neuromorphic Computing and En- gineering, 2022. URL http::/iopscience 1og.oorg/attille/11.118888
4386/ac9b86]
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arX1v:1412.6980, 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https:
// //www. cs. toronto. wwwooto.eedo-oo.......o.... z/lea rni leaiiin--feaaeeee 2009 TR. p
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting acti- vation sparsity for fast inference on deep neural networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5533-5543. PMLR, 13-18 Jul 2020. URI. fttts://proceedi
ml wtl..prees/vvl99
html
Changhun Lee, Hyungjun Kim, Bunhyeok Park, and Jae-Joon Kim. Insta-bnn: Binary neural net- work with instance-aware threshold, 2022.
12

---

Published as a conference paper at ICLR 2024
Jeong-Jun Lee and Peng Li. Reconfigurable dataflow optimization for spatiotemporal spiking ral computation on systolic array accelerators. In 2020 IEEE 38th International Conference on Computer Design (ICCD), volume 1. pp. 57-64, 2020. doi: (0.1099/CCDD0033.2200000000
Jun Haeng Lee et al. Training deep spiking neural networks using backpropagation. Frontiers in Neuroxcience, 10, 2016.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Dif- ferentiable spike: Rethinking gradient-descent for training spiking neural networks. In
Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 23426-23439. Curran Asso- ciates, Inc., 2021a. URL c4ca 0b 00
neur iaurrips... ps. ccc/paper/2021/f11 .ddf]
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng. Yongqing Hai, and Shi Gu. Differen- tiable spike: Rethinking gradient-descen for training spiking neural networks. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021b. httptttst///peeeeeeeeeee
wwt.:ooumm:.o-teeeeeee. net/ forum? 1d= 7 7mBnC9F0
Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuromor- phic data augmentation for training spiking neural networks. arXiv preprint arXiv:2002. 100::4002..0044,, 2022.
Minghao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Fei Chao, Chia-Wen Lin, and Ling Shao. Siman: Sign-to-magnitee network binarization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):6277-6288 2023. doi: 10. 1000/PPAM22022.322266555
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, arXiv preprint arXiv: (0:::/00000000 201
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017b. URL
wtwp::/wwweeddn.....needod...oee neur hcppp...ccccpcpppppppr.r files//appr//0// fil fil//155001fcc3000ecebeeeeee133.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang. Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018a.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 722-737, 2018b.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards pre- cise binary neural network with generalized activation functions. In European Conference on Computer Vision (ECCV), 2020a.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: binary neural network with generalized activation functions. European conference on computer vision, pp. 143-159. Springer, 2020b.
Towards precise
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Train- ing high-performance low-latency spiking neural networks by differentiation on spike representa- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12444-12453, June 2022.
E. O. Neftei, H. Mostafa, and F. Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Pro- cessing Magazine, 36(6):51-63. 2019.
13

---

Published as a conference paper at ICLR 2024
Fabrizio Ottati, Chang Gao, Qinyu Chen, Giovanni Brignone, Mario R. Casu, Jason K. Eshraghian, and Luciano Lavagno. To spike or not to spike: digital hardware perspective on deep learning acceleration. arXiv preprint arXiv:2306.15774 15749, 2023.
Priyadarshini Panda and Kaushik Roy. Unsupervised regenerative learning of hierarchical features in spiking deep networks for object recognition. arXiv preprint arXiv:1602.01. 00000000011, 2016.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European conference on computer vision, pp. 525-542. Springer, 2016.
Nitin Rathi et al. DIET-SNN: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008. 03658, 2020a.
Nitin Rathi et al. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.0180) 0000000000.......0 2020b.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Charbel Sakr, Jungwook Choi, Zhuo Wang, Kailash Gopalakrishnan, and Naresh Shanbhag. True gradient- based training of deep binary activated neural networks via continuous binarization. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), vol- ume 1, pp. 2346-2350, 2018. doi: 10. 10/11CASSS...001... 846444 8461456.
Abhronil Sengupta et al. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers Neuroscience, 13:95, 2019.
Xulong Shi, Zhi Qi, Jiaxuan Cai, Keqi Fu, Yaru Zhao, Zan Li, Xuanyu Liu, and Hao Liu. Repbnn: towards a precise binary neural network with enhanced feature map via repeating. arXiv preprint wwXiv:2207.09044 09049, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 2014.
Dewei Wang, Pavan Kumar Chundi, Sung Justin Kim, Minhao Yang, Joao Pedro Cerqueira, Joon- sung Kang, Seungchul Jung, Sangjoon Kim, and Mingoo Seok. Always-on, sub-300-nw, event- driven spiking neural network based on spike-driven clock-generation and clock- and power- gating for an ultra-low-powee intelligent device. In 2020 JEEE Asian Solid-State Circuits Confer- ence (A-SSCC), volume 1, pp. 1-4, 2020a. doi: 10.1
Peisong Wang, Xiangyu He, Gang Li. Tianli Zhao, and Jian Cheng. Sparsity-inducing bina- rized neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 34 (07):12192-12199, Apr. 2020b. URL. sttps://ojs.aasi. org/ htee/indexx... wnw.iixeeppppp...m AAAT/ axticle/view/5000
Yikai Wang, Wenbing Huang, Yinpeng Dong, Fuchun Sun, and Anbang Yao. Compacting binary neural networks by sparse kernel selection. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24374-24383, June 2023.
Ziwei Wang, Ziyi Wu, Jiwen Lu, and Jie Zhou. Bidet: An efficient binarized object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020c.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, and Zhouchen Lin. On- line training through time for spiking neural networks. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 20717-20730. Curran Associates, Inc., 2022. URL httpss///ppooceeeennnneeeeeerree
file/ /2/4
14

---

Published as a conference paper at ICLR 2024
Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing XU, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.). Advances in Neural In- formation Processing Systems, volume 34, pp. 35553-25565. Curran Associates, Inc., files/paper/20211
2021a. URL htt.tp:://ppoccedigg.s... www.:diiinnnnn.eninnooeee www.wneuuiii.oo File/ F11//6G552233seaaaacceeeeeeeeee2
Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao. Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5198-5208, October 2021b.
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differen- tiable scale-invariant sparsity measures. In International Conference Learning Representa- tions, 2020.
Ruokai Yin, Abhishek Moitra, Abhiroop Bhattacharjee, Youngeun Kim, and Priyadarshini Panda. Sata: Sparsity-aware training accelerator for spiking neural networks. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 1(1):1-1, 2022. doi: 10.1109/TCAD. 2022.3213211.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spik- ing neural networks. In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12022-12033. Curran As- sociates, Inc., 2020. URL httL:ttt: /pppottdd.//rocc........ newriis.....ipssp. cc wwtp:/ccpppppc////eeeeeee 8bdb50583 611337664338 0
2ttp1/66668--PPap-aaa-e-- pdf
Yichi Zhang, Zhiru Zhang, and Lukasz Lew. Pokebnn: A binary pursuit of lightweight accuracy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12475-12485, June 2022.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35 (12):1102--11070 May 2021. doi: 10. htt.0/9aaai.3512...... org/indes http:///....aaa..aeeeeeeeee/////
URL httpt::/tp////t aaai.
Pengju Ren Zhijun Tu, Xinghao Chen and Yunhe Wang. Adabin: Improving binary neural networks with adaptive binary sets. In European Conference on Computer Vision (ECCV), 2022a.
Pengju Ren Zhijun Tu, Xinghao Chen and Yunhe Wang. Adabin: Improving binary neural networks with adaptive binary sets. In European Conference on Computer Vision (ECCV), 2022b.
A APPENDIX
A.I THRESHOLD & WEIGHT UPDATES DUE TO HOYER Loss
For hidden layer the weight update is computed as
AW aLCE +YH
aL ÔLCE do dzi Ou:
Owi
owi do Oz ow, ow
OLH du OLCE do 0 +Ao
ØLH
+A ou, Own do Ozi ofth
Du
O
where ØLn can be computed as Ou:
aLh aLH dutt1 do: dzi
0H (H(2 (z
+
Ou 0utt1
do
dz Oui
Ou
(10)
öL
do 1
0H (2)
clip
0z
clip 1
wi+1 0z ofth
Jui+1
2z clip
dz v
15

---

Published as a conference paper at ICLR 2024
Proposed block
Max Pool
Conv
Max Pool
Hoyer thr.
Dropout
Conv
BN
Block
Hayer thr.
BN
Block
Downsample
Existing block
Hoyer
Conv
Binary
Avg
Pool
Dropout
Conv
Comv
Block
(a)
(b)
Figure 3: BANN architectures corresponding to (a) VGG and (b) ResNet based models.
This is because LH sum of the Hoyer regularizer of the first 1-1 layers, the /th layer, and the layers after the layer. While computing ôLæ the gradient of the 1st part is O, and the gradient Thn of the 2nd (3rd) part is captured in the second (first) term in Eq. 8. Note that 0LH is the gradient 0u111
backpropagated from the + 1)th layer, that iteratively computed from the last layer L (see Eqs. 7 Jand I1 Also, note that for any hidden layer I, there are two gradients that contribute to the Hoyer loss with respect the activation uy; one is from the subsequent layer (I+1) and the other is directly from its Hoyer regularizer. Similarly, aLCE is computed iteratively, starting from the penultimate 20:
layer (L-1) defined in Eq. 7 as follows.
ôLCE ÅLCE do1+1 0zl+1 0ur+1 ôLCE dol+1 wi+1
(11)
do: 00:-1 0zi+1 0ui+1 do: 00 0z1+1 of
Finally, the threshold update for the hidden layer is computed
Auth OLCE
ŌLH
ÔLCE do: dzi
ŌLH duth
+AH
+N
Oush
Out
do Oz duth
(12)
alCE do:
-u
ŌLn 0utt1
+A 0ut+1 Soth th
do: 0zi (uth)2
0ui+1 Oui+1 8o:
do: u1
auth
do:
auth
wi+1
dzi
(uth)²
(13)
Note that we use this of which is updated in each iteration, to estimate the threshold in our BANN model using Eq. 4
A.2 NETWORK ARCHITECTURE
We adopt a series of network architectural tricks for our BANNs (Datta & Beerel 2022 Chowdhury et al. 2021 Rathi et al. 2020a). As shown in Fig. a)) for the VGG variant, we use the max pooling layer immediately after the convolutional layer that is common in many BNN architectures (Raste- gari et al.) 2016), and introduce the BN layer after max pooling. Similar to recently developed
multi-time-step SNN models (Zheng al. 2021 Li et al. 2021a Deng al. 2022 Meng al.) 2022), we observe that BN helps increase the test accuracy with one time step. contrast, for the ResNet variants, inspired by (Liu et al. (2018)), we observe models with shortcuts that bypass every block can also further improve the performance of the SNN. We also observe that the sequence of BN layer, Hoyer spike layer, and convolution layer outperforms the original bottleneck in ResNet. More details are shown in Fig. ())).
A.3 ABLATION STUDIES
We conduct ablation studies analyze the contribution of each technique in proposed approach. For fairness, we train all the ablated models on CIFAR dataset for 400 epochs, and use Adam as the optimizer, with 0.0001 the initial learning rate. Our results
shown in Table 8, where the model without Hoyer threshold layer indicates that we set the threshold as of similar to existing works (Datta & Beerel 2022 Rathi al. 2020a) rather than our proposed Hoyer extremum.
16

---

Published as a conference paper at ICLR 2024
Table 8: Ablation study of the different methods in our proposed training framework on CIFARIO.
Hoyer Reg.
Hoyer Spike
Arch.
Acc. (%)
Sparsity (%)
VGG16
X
x
88.42
84.38
VGG16
V
90.45
79.52
VGG16
X
V
92.90
78.30
VGG16
V
v
93.13
77.43
ResNet34
X
x
87,41
77.22
ResNet18
X
90.95
79.50
ResNet18
X
V
91.17
74.13
ResNet18
91.48
74.17
Table 9: Comparison of our BANNs with full-precision DNNs & BNNs on VOC2007 dataset.
Framework
Backbone
mAP(%)
Faster R-CNN
Original ResNet50
79.5
Faster R-CNN
Bi-Real (Liu et 2018b)
65.7
Faster R-CNN
ReActNet (Luu et al. 2020b)
73.1
Faster R-CNN
BANN ResNet5U
73.7
Retinanet
Original ResNet50
77.3
Retinanet
BANN ResNet50 (ours)
70.5
YOLO
BANN DarkNet (Kim et al.
2019
53.01
SSD
BNN VGG16 (Wang et al.
2020c
66.0
With VGG16, our Hoyer regularizer approach leads to 2. 2.03% increase in accuracy. Together, with our Hoyer threshold layer, the accuracy improves by 2.68% 93.13% while also yielding 2.09% reduction in sparsity.
We observe similar trend for our network modifications and Hoyer threshold layer with ResNet18. However, Hoyer regularizer in this case also increases the sparsity from 72.38% to 77.22% to 79.50%, while also negligibly reducing the accuracy. In summary, the combination of our Hoyer regularizer and Hoyer threshold layer yield the SOTA BNN/SNN performance.
A.4 OBJECT DETECTION RESULTS
For object detection VOC2007, compare the performance ob- tained by our BANN models with full-precision DNNs and BNNs in Table 9 For two-stage ar- chitectures, such Faster R-CNN, the mAP of our BANN models surpass the existing BNNs by ~0.6%. For one-stage architectures, such as RetinaNet (chosen because its SOTA performance), BANN models with ResNet50 backbone yields mAP of 70.5% (highest among existing BNN, SNN, AddNNs). Note that our BANN-based VGG and ResNet-based backbones lead to a signifi- cant drop in mAP with the YOLO framework that is more compatible with the DarkNet backbone (even existing DarkNet-back SNNs lead to very low mAP with YOLO as shown in Table
Object Detection Results:
A.5 PROOF OF TIRESHIOLD DOWNSCALING HOYER EXTREMUM
In order to prove that Hoyer regularized threshold is always less than equal to the trainable threshold tht we first prove that the Hoyer extremum of 2, clip is less than equal 1. Let us C: to represent zkkpp so Vj, < of <
Ext(c)) = |cl12 E)(
|all Σg
(14)
< Eg(d -ma((())
< max (c)) <
Ejd d
17

---

Published as a conference paper at ICLR 2024
Table 10: Comparison of our one- and multi-time-step SNN models existing SNN models on DVS-CIFARIO and IBM-Gesture datasets, *Results are based on our replicated implementation.
Dataset
Reference
Trinning
Architecture
Acc.
T.S.
83.17
10
Deng et al
2022
TET
VGGSNN
75.20*
4
et al
2022
(dBN-NDA
VGGI1
81.7
10
DVS-CIFARIO
wAithSwiched
VGGT6
67.1
This work
Hoyer reg.
VGGSNN
K3.68
10
76.17
Litt 2022
LdBN-NDA
VGGII
78.6
N-CalTech101
Kim & Fanta 2021
4ALTcSwichhed BN
VOGT6
35.0
This
Hoyer rey.
VGGIT
79.92
So the Hoyer extremum of zi always less than or equal one, and thus, our Hoyer regularized threshold, which is the product of of and Ext(z((ip) is always less than or equal to the trainable threshold ofth
A.6 DATASETS & HYPERPARAMETERS
For training VGGI6 models, we using Adam optimizer with initial learning rate of 0.0001, weight decay of 0,0001, dropout of 0.1 and batch size of 128 in CIFAR10 for 600 epochs, and Adam optimizer with weight decay of 5e-6 and with batch size 64 in ImageNet for 180 epochs. For training ResNet models, we using SGD optimizer with initial learning rate of 0.1, weight decay of 0.0001 and batch size of 128 in CIFARIO for 400 epochs, and Adam optimizer with weight decay of 5e-6 and with batch size 64 in ImageNet for 120 epochs. We divide the learning rate by 5 at 60%, 80%, and 90% of the total number of epochs. Note that A 1e-8 for all our experiments.
When calculating the Hoyer extremum we implement two versions, one that calculates the Hoyer extremum for the whole batch, while another that calculates it channel-wise. Our experiments show that using the channel-win version can bring 0.1-0.3% increase in accuracy. All the experimental results reported this paper use this channel-wil -wie version.
For Faster R-CNN, we use SGD optimizer with initial learning rate of 0.01 for 50 epochs, and divide the learning rate by 10 after 25 and 40 epochs each. For Retinanet, we use SGD optimizer with initial learning rate of 0.001 with the same learning rate scheduler as Faster R-CNN.
A.7 EXTENSION TO SNNS FOR DYNAMIC VISION SENSOR (DVS) TASKS
The inherent temporal dynamics in SNNs may better leveraged DVS or event-based tasks (Deng et al. 2022 Li ct al. 2022 Kim & Panda 2021 Kim et al. 2022) compared to standard static vision tasks. Hence, we also evaluate our framework on two DVS datasets, namely DVS-CIFARIO and --estuueee. As illustrated in Table IO we surpass the test accuracy of existing works (Li al.) 2022
Kim & Panda 202T) by 0.93% on average at iso-time-step and architecture. Note that the architecture VGGSNN employed in work and (Deng et al. (022) is based VGG11 with two fully connected layers removed as (Deng et al. 2022) found that additional fully connected layers were unnecessary for neuromorphic datasets. To compensate for the loss of momentum in SGD and improve the accuracy, we adopt the temporal efficient training (TET) method in our framework (Deng et al. 2022). In fact, our accuracy gain is more significant at low time steps, thereby implying the portability of our approach to DVS tasks. Note that similar to static datasets, a large number of time steps increase the temporal overhead in SNNs, resulting in large memory footprint and spiking activity.
A.8 ENERGY EVALUATION FRAMEWORK
A.8.1 COMPUTE ENERGY
The total compute energy (CE) of BANN (BANNcE) can be estimated as
BANNcE=DNN? Emac (SDDNN Eac + Esg + Σ DNN Ecom
(15)
18

---

Published as a conference paper at ICLR 2024
because BANN receives full pell-precision input in the first layer (1=1) without any sparsity (Chowdhury et al. 2021; Datta et al. 2022b; Rathi et al. 2020a). Note that D N denotes the total number of comparison (thresholding) operations in the layer with each operation consuming 1.64pJ energy in our FPGA platform for FP reperesentation. Also, note that DNN' denote the total number of floating point (MAC or AC) operations in layer where each FP MAC operation consumes 13. 13.32pJ energy. while each FP AC operation consumes 1.8pJ energy. Lastly, St denote the spiking activity in the layer and Esp 0.05pJ denotes the energy overhead due to sparsity, that is incurred in checking whether the binary activation is zero.
The CE of the full-precision DNN (DNNcE) and bi-polar BNN (BNNcE) is estimated
DNNCE Σ DNN,PER Finacy BNNCE DNNT Einac + DNNN Fizor
(16)
where we ignore the energy consumed by the ReLU operation (significantly lower compared to thresholding operation) since that includes only checking the sign bit. Note that Ezor denotes the XOR energy incurred in the binary convolution/linear operations in bi-polar BNNs.
The compute energy of an SNN with total of T time steps is denoted as
SNNCE= Σ DNN'' Emac
Σ (SS Eac + END,NN + 1=2
DNN from
Ecom
(17)
where St denotes the activation sparsity of layer I at time step and replicate the BANN energy for all T time steps. Note that we ignore the potential reset energy since the number of reset opera- tions is negligible compared to the FLOPs, and the energy consumed per reset operation is similar to that of an AC operation.
Intuitively. BANNs with low-bit (<6-bits) weights, consume lower compute energy compared to BNNs since the energy savings due to the activation sparsity even with its overhead outweighs the energy savings due to XOR operations compared to low-precision AC operations. BANNs also consume lower compute energy compared to SNNs and DNNs due to the higher activation sparsity, leading to less number of FP operations.
A.8.2 MEMORY ENERGY
Since memory accesses often dominate the total energy consumption in DNNs, we incorporate the same in our energy model using (Ottati et al. (002)..
The memory energy of BANN is estimated as
BANN Σ
CI-CHRH Ei
1=1
(18)
Σ
CCRHHWSSS Σ H.W.C
where Cj. Cf. hi, Hy, and Wi denote the number of input channels, number of output channels, kernel size, output feature map height, and output feature map width respectively. Note that E and ESP denote the energy incurred in reading a spike input and writing a spike output respectively from/to the on-chip FPGA memory. On the other hand, EteP denotes the read energy of each 8-bit input pixel value, and End denotes the read energy of the weights from the memory.
The memory energy of traditional bi-polar BNN is estimated as
BNNME- E G.G.E CCC E
I CORREEEE
E H.WiC Eef
(19)
where Fof denotes the energy incurred in writing the FP output feature map to the on-chip memory. While Fuo is lower for BNNs compared multi-bit-weighhee BANNs, BNNs ineur higher output
19

---

Published as a conference paper at ICLR 2024
Table 1I: Comparison of the different compute and memory energy numbers for FP weights and membrane potentials obtained from our post place-and-route FPGA simulations on 28nm Kintex7 device.
Parameter
Energy (pJ)
Parameter
Energy (pJ)
Emac
13.2
Eu
33.4
Eac
1.8
E E
8.6
Eeeon 1.4
E
1.7
Eisp
0.05
E
3.8
Egor
0.08
Eurem 40.1
feature map write energy due to multi-bit outputs, and higher input feature map read energy due to absence of spike sparsity.
In summary, BANN can hep reduce both the compute and memory energy compared to BNNs, SNNs, and non-spiking DNNs, provided the hardware supports the activation sparsity.
The memory energy of an SNN with time steps is estimated
SNNME EH FiNP
F.
(20)
Σ Σ CICK E Σ H,W.CT + Erd
Emeny
where ETa and E denote the energies incurred in read and write operations of the membrane potential respectively, that happens for every time step. Lastly, the memory energy of FP DNN is computed as
DNNME Σ
CAC I H.W.C
(21)
where E%d denotes the read energy of the full-precision input feature map, that higher than the spike read energy incurred in the BNN/BANN/SNN, Note that all the memory access energy num- bers, Ey E. Esp, Ejap, yd with a 42MB of on-chip (BRAM and URAM) capacity. Note that each of these numbers depend on the bit-precision of the value read/written. For example, assuming that 32 spikes are encoded to an 32b memory word, the energy associated with spike read memory operation is almost same as Ex 32 for 32-bit FP weights. All these numbers, along with those for the compute operations, tabulated in Table 11, where the weights, membrane potentials, input and output feature maps have FP representation.
Elf Eof. E Emem are obtained from our FPGA simulation setup
Note that for SNNs, the memory footprint is primarily dominated by the read and write accesses of the post-synaptic potential at each time step (Yin et al.) 2022). This is because these memory accesses are not influenced by the SNN sparsity since each post-synaptic potential is the sum of k2cin weight-modulated spikes. For typical convolutional layer, k = 3, Cin. 128, and so it is almost impossible that all the cn spike values are zero for the membrane potential to be kept unchanged at a particular time step. Note that the number of input (output) feature map read (write) accesses can be reduced with spike sparsity, and typically do not dominate the memory footprint of SNNs. Since BANNs do not have the temporal membrane potential, they can significantly reduce the memory cost compared to SNNs. Compared to DNNs and BNNs, the memory cost, that is dominated by the feature map accesses (with weight re-use), be reduced in BANNs by the zero gating logic that leverages spike sparsity.
A.9 TRAINING & INFERENCE TIME REQUIREMENTS
Because SOTA SNNs require iteration over multiple time steps and storage of the membrane potentials for each neuron, their training and inference time can be sub- stantially higher than the DNN/BNN counterparts. Though reducing their latency
20

---

Published as a conference paper at ICLR 2024
The Hoyer Ext. for Each Layer (VGG16)
Hoyer Sach Reokeett
66-
54
04
250
53
560
150
(a) VGG16
(b) ResNet20
Figure 5: Evolvement of the learned threshold value for each layer of VGG16 and ResNet20 on CIFAR1O.
to time step bridge this gap significantly. icantly increases the training complexity due to the iterative time-step reduction. On average, our BANNs represent a 2.38 and 2.33x reduction in training and inference time per epoch respectively, compared to the multi- time-step training approaches (Datta & Beerel 2022 Rathi al., 2020a) with iso-batch and hardware conditions.
shown in Figure 4 it signif-
22
Compared to the existing one-time-step SNNs Chowdhhur et al., 2021), we yield a 19x and
ViNente
1.25x reduction in training and inference time. This inference time reduction is possibly due to more efficient PyTorch tensor operations used in our code which may be better optimized us- ing the underlying CUDA compiler. Such sig- nificant savings in training time, which trans- lates to power savings in big data centers, can potentially reduce Al's environmental impact.
Figure 4: Normalized training and inference time/epoch with iso-batch (256) and hardware (RTX 3090, 24 GB memory) conditions for (a) CIFARIO and (b) ImageNet with VGG16.
A.10 COMPARISON OF LATENCY FOOTPRINT
Using a hardware validated latency model on ResNet-18, our 6-bit weighted BANNs incur 1.34x higher latency compared to uni-polar BNNs, activation sparsity favors energy latency & pop- count incurs lower latency than accumulates. On the contrary, our binary weighted BANNs incur almost similar (0.96x) latency as a uni-polar BNN.
A.11 STABILITY OF HOYER REGULARIZED TRAINING
The effect of the Hoyer spike layer is a down-scaling of the thresholds obtained from SGD. How- ever, because these thresholds are fixed during inference, the mapping to neuromorphic hardware is no more complex than traditional SNN. Moreover, since our training method incorporates the learnable training threshold obtained via SGD, we report its evolvement during the training stage in Fig. (a) and (b) for VGG16 and ResNet20 respectively on 1..R10. As we can see, our thresh- old value stabilizes well during the later parts of the training for all the layers, demonstrating the robustness of our method.
21


Reviews:
[{"review_id": "FAWVMG0Tyz", "reviewer": "Reviewer_gkE7", "summary": "This paper proposes a new training method for binary activation neural networks. This method employs the Hoyer regularizer to each BANN layer to shift the activation values away from the threshold. As a result, the proposed BANN model can achieve better accuracy than other methods.", "questions": "* In section 2.2, “Note that this model is similar to leaky-integrate-and-fire (LIF) model”. This is not very direct. Why they are similar? In this model, there is no membrane potential, which is important in LIF.\n* Typo: in section 4, Accuracy Comparison with SNNs, “Table 7” should be “Table 2”", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method can improve the accuracy of both BANN and SNN models\n* Experiments show the energy efficiency comparisons to demonstrate the effectiveness of this method", "weaknesses": "* The proposed method is not closely related to SNN, and it mainly focuses on the BANN. This method can be used in SNNs for better model accuracy, but it can not bridge the gap between BNN and SNN as the title said. It would be preferable to discuss more about the relationship between BNN, SNN, and the proposed method to illustrate how this method bridges BNN and SNN.\n* The energy evaluation method in the experiments is inaccurate. Usually, memory accesses take a large proportion of energy consumption. However, in the experiments, it only estimates the computed energy. It is better to consider the energy of memory access as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kht4p8DHnV", "reviewer": "Reviewer_X4qq", "summary": "The presented method trains binary neural networks with the goal of sparse activations. The is achieved by borrowing concept from SNN training that use a threshold function in the neuron model to generate output spikes. In this method, the threshold value is determined for each layer using the Hoyer extremum. The method was tested on CIFAR-10 and ImageNet datasets and achieved superior accuracy and energy efficiency.", "questions": "During inference, the Hoyer extremum is used to do the binarisation of the activations. How does is affect the number of FLOPS in Equation 9? How can the Hoyer extremum be reduced to a simple comparison to make the inference match the equation?\n\nCould you take some actual energy measurements using your already existing FPGA simulator? It would also be interesting to see SNN’s energy consumption on hardware. This way, we do not need to rely on outdated energy numbers from 2014 and also account for the overhead induced by the sparsity.\n\nWhy is the ImageNet energy lower than that for CIFAR-10 in Table 4? Could you elaborate how you arrived at those numbers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "BNNs can certainly benefit from sparsity, just as SNNs do. Hence, the approach to transfer the SNN thresholding concept to BNNs with binary activations 0 and 1 is promising. With the right hardware, operations can be saved. The effect of sparsity becomes clear when compared with BNNs that use binary weights. The proposed method seems more energy efficient despite using floating-point weights.\nThe activations achieve a high sparsity. And by using the Hoyer regularisation during training, activations are resistant to noise and achieve higher accuracy than most SNNs, especially ones with only 1 time step.", "weaknesses": "The experiments executed in this paper are generally insightful. It is especially interesting to compare with both BNNs and SNNs. However, some evaluations are lacking detail. The energy consumption is estimated using power numbers of a paper from 2014 at an outdated technology node. Real values obtained from logic synthesis will be significantly different. Additionally, sparse operation induces overhead, such as checking if an activation is actually zero. Those factors are not accounted for in the estimations. Since there was an FPGA simulation done to evaluate the effect of quantisation, I would like to see a power estimation using this FPGA simulator. Alternatively, a GPU evaluation of the actual power consumption is interesting, since it is stated that the BNN operations are compatible with “existing hardware (standard GPUs)”.\n\nFor comparison of the energy consumption between SNNs and the proposed method, sparsity is used as a proxy. However, it is not clear how the sparsity numbers in Figure 3 were obtained except that they “represent existing low-latency SNN works”. Were those numbers given by the references or was it a replicated implementation by the authors?\n\nMore emphasis could be placed on the comparison with BNNs. It is stated that SOTA BNNs use network modifications that increase FLOPS but also expressivity. To which extend does calculating the Hoyer extremum during inference incur additional FLOPS? The power estimation in Equation 9 only accounts for one additional comparison.\n\nImportant parameters such as λ are not provided. At the same time, it was not indicated that training code and trained models will be provided. That means it is currently difficult to validate the results of this paper and derive further research. I would see open sourcing as a requirement of acceptance.\n\nThe formatting of the references in the text makes the paper very hard and slow to read. Figure 2 was prominently placed in the manuscript but never mentioned or further explained in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y2ac9OtAUj", "reviewer": "Reviewer_J4aA", "summary": "This paper proposed a spiking neural network-based training scheme that eliminates the iterative time step down to 1 while keeping the uni-polar activation scheme (0 and 1). The proposed algorithm is mainly constructed based on a layer-wise learnable threshold trained by the Hoyer regularization. The proposed method achieves comparable accuracy as the prior BNN work on both the CIFAR and ImageNet datasets.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The major contribution of the proposed method is reducing the multi-step, iterative spiking process down to one. I agree with the argument that increasing the number of 1s in the output spike can improve the activity of the training process (weight update) and convergence.\n\nRegarding the performance of the proposed method, reporting the accuracy of the classification and object detection datasets is acceptable.\n\nIt is helpful that the author can provide some theoretical insights and proofs.", "weaknesses": "**W1:** Most of the binary neural network research binarizes the weights weights as well. However, this paper does not report the weight precision and compares the precision scheme with BNN in a comprehensive manner.\n\n**W2:** The performance comparison against SNN is not comprehensive enough. E.g., The recent TET [R1], DSpike [R2], and DSR [R3] are excluded from Table 2.\n\n**W3:** The proposed method shows worse performance on ResNet-50 than VGG-16, which is controversial with the full-precision model performance, why is that?\n\n**W4:** The proposed method incorporates the learnable potential threshold, I think it is important to report the evolvement of the threshold value during the training process, especially the training stability of the threshold value.\n\n**W5:** Although the static image-based classification datasets (e.g., ImageNet) are good indicators of model performance, it is also important to evaluate the performance of SNN with DVS datasets due to the property of spatial-temporal events (e.g., DVS-CIFAR10, N-CalTech101, IBM-Gesture).\n\n[R1]: Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting \n\n[R2]: Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks, NeurIPS'21 \n\n[R3]: Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)