Differentiable hierarchical and surrogate gradient search for spiking neural networks
Authors: Kaiwei Che
Luziwei Leng
Kaixuan Zhang
Jianguo Zhang
Qinghu Meng
Jie Cheng
Qinghai Guo
Jianxing Liao
Venue: NeurIPS 2022 Accept (2022)
Keywords: spiking neural network
architecture search
event-based stereo
image classification
surrogate gradient

Abstract: Spiking neural network (SNN) has been viewed as a potential candidate for the next generation of artificial intelligence with appealing characteristics such as sparse computation and inherent temporal dynamics. By adopting architectures of deep artificial neural networks (ANNs), SNNs are achieving competitive performances in benchmark tasks such as image classification. However, successful architectures of ANNs are not necessary ideal for SNN and when tasks become more diverse effective architectural variations could be critical. To this end, we develop a spike-based differentiable hierarchical search (SpikeDHS) framework, where spike-based computation is realized on both the cell and the layer level search space. Based on this framework, we find effective SNN architectures under limited computation cost. During the training of SNN, a suboptimal surrogate gradient function could lead to poor approximations of true gradients, making the network enter certain local minima. To address this problem, we extend the differential approach to surrogate gradient search where the SG function is efficiently optimized locally. Our models achieve state-of-the-art performances on classification of CIFAR10/100 and ImageNet with accuracy of 95.50%, 76.25% and 68.64%. On event-based deep stereo, our method finds optimal layer variation and surpasses the accuracy of specially designed ANNs meanwhile with 26$\times$ lower energy cost ($6.7\mathrm{mJ}$), demonstrating the advantage of SNN in processing highly sparse and dynamic signals. Codes are available at \url{https://github.com/Huawei-BIC/SpikeDHS}.

Differentiable hierarchical and surrogate gradient search for spiking neural networks
Kaiwei Chel.2,, Luziwei Lengl.2. Kaixuan Zhang Jianguo Zhang', Max Q--.. Meng', Jie Cheng", Qinghai Guo², Jiangxing Liao²
Southern University of Science and Technology, China ACS Lab, Huawei Technologies, Shenzhen, China
Abstract
Spiking neural network (SNN) has been viewed as potential candidate for the next generation of artificial intelligence with appealing characteristics such sparse computation and inherent temporal dynamics. By adopting architectures deep artificial neural networks (ANNs). SNNs are achieving competitive performances in benchmark tasks such as image classification. However, successful architectures of ANNs are not necessary ideal for SNN and when tasks become effective architectural variations could be critical. To this end, we develop a spike- based differentiable hierarchical search (SpikeDHS) framework. where spike-based computation is realized on both the cell and the layer level search space. Based this framework, find effective SNN architectures under limited computation cost. During the training of SNN, a suboptimal surrogate gradient function could lead poor approximations of true gradients, making the network enter certain local minima. To address this problem, we extend the differential approach to surrogate gradient search where the SG function is efficiently optimized locally. Our models achieve state-of-the-e performances on classification of CIFAR 10/100 and ImageNet with accuracy of 95.50%. 76.25% and 68. 64%. On event-based deep stereo, our method finds optimal layer variation and surpasses the accuracy of specially designed ANNs meanwhile with 26x lower energy cost (6. (6.7m.), demonstrating the advantage of SNN in processing highly sparse and dynamic signals. Codes are available at htthtss: thub. cow/Huawei- BIC/SSiieDD/SS
diverse
1 Introduction
Inspired from biological neural networks, spiking neural network (SNN) [41] has been viewed as a potential candidate for the next generation of artificial intelligence, with appealing characteristics such as asynchronous computation, sparse activation and inherent temporal dynamics. However, the training of deep SNNs is challenging due to the binary spike which is incompatible with gradient- based backpropagation. To solve this problem, various surrogate gradient (SG) methods were proposed [ 65 48) where soft relaxed functions were used approximate the original discontinuous gradient. Based on these methods, SNNs have achieved high level performances on benchmark image classification tasks such CIFAR and ImageNet (58 66
54 76). However, the accuracy of SNN often drops when directly adopting ANN architectures such as ResNet [2]] and VGG networks [59]. With recent improved SNN training methods [35 13 [3] the performance gap is decreasing but remains. This gap is more evident in tasks where the network architecture requires more variation,
74
such dense image prediction [8] 20 25
Directly inheriting sophisticated ANN architectures might not be ideal for SNN and there is a lack of study of optimal architectures for spiking neurons given particular task. Intuitively, topology
'Equal Contributions. *Corresponding author. lengluziwei @ huawei.com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

in cortex should offer us some inspiration. Neuroscience has studied macro structures such as columnar organization 44 and miero dynamics such as interaction between lateral and feedback connections (36] However, on the network level the relation between connections of neural circuits and functionalities remains largely unknown. In this work, we develop spike-based differentiable hierarchical search (SpikeDHSS framework in order to find optimal task-specific network architectures under limited computation cost. Based on traditional differentiable architecture search (DARTS) framework [40], we redesign the search space and information flow in the principle of spike-based based computation. For both the cell and the layer search space, we study how to realize this principle towards a fully spiking network.SG provides approximation of the non-existing gradient in SNN and its selection is not unique. To explore optimal SG functions for the training of SNN, we propose a differentiable SG search method to efficiently adapt the function locally, and demonstrate its efficiency on both static and event-based benchmark machine learning tasks. In summary, our contributions are following:
We develop a differentiable hierarchical search framework for spiking neurons, realizing spike-hased computation both the cell and the layer level search space, based on which optimal SNN architectures can found under limited computation cost.
To improve gradient approximation of deep SNNs, we propose differentiable SG search method to efficiently optimize SG functions locally, which is easy to scale and also effective for binary networks.
Extensive experiments show that our methods outperform SNNs based on sophisticated ANN architectures on image classification of CIFARIO, CIFARI00 and ImageNet datasets.
On event- based deep sterco task, to the best of our knowledge we show the first time SNN surpasses specially designed ANNs on the Multi Vehicle Sterco Event Camera (MVSEC) [0] dataset in terms accuracy, network sparsity and computation cost, demonstrating its advantage in processing highly sparse and dynamic signals with extremely low power and latency.
2 Related work
2.1
Architecture search
Designing high performance network architectures for specific tasks often requires expert experience and trial-and-error experiments. Neural architecture search (NAS) [14] aims automate this manual process and has recently achieved highly competitive performance in tasks such as image classification
etc. However, searching over a discrete set of candidate architectures often results in a massive number of potential combinations, leading to explosive computation cost. The recently proposed differentiable architecture search (DARTS) method [40] and its variations [67 7 I0) address this problem using a continuous relaxation of the search space which enables learning a set of architecture coefficients by gradient descent, and has achieved competitive performances with the state-of-the-as using orders of magnitude fewer computation resources 40 39 9. Recently, (46) studied pooling operations for downsampling in SNNs and applied NAS to reduce the the overall number of spikes. (26) applied NAS to improve SNN initialization and explore backward connections. However, both works only searched for different SNN cells or combinations of them under fixed network backbone and their application limited to image classification.
82
83
38
55
52 object detection [83 g 64 [9] and semantic segmentation (3
75
49
37],
2.2
Training of SNN
The success of deep ANNs in solving benchmark machine learning tasks has motivated efforts to make SNNs realize similar capabilities, either based on bio-inspired mechanisms [5T] 47
33 45]
32
27
or approximating ANN learning algorithms [5 65
48 3 Currently, two approaches have
demonstrated their efficiency, showing the ability solve hard problems similar levels as their
artificial counterparts, namely, ANN-to-SNN conversion [57 6 34] and directly training SNNs with SG
15 13). The theoretical soundness of the SG approach for training binary Experiments show that the training of SNN robust to the shape of SG function as long as it meets certain criteria, such as the overall scale [71.
58 66
74
35
activation networks has been studied and justified 4 69
(20) shows that suitable SG function is critical when the SNN goes deeper. (35] further improved

---

the performance of SNN by optimizing the width (or temperature) of continuous SG function, through the guidance of an approximated gradient measured with finite difference gradient (FDG). However, a suitable hyperparameter setting of FDG is largely empirical. In terms of computation, it iterates over each element of the weight and is sequential across layers, making a global application of this method computationally impractical.
2.3 Event- task with SNN
Inspired by biological retina, event camera [8] captures instantaneous changes of pixel intensity microsecond resolution. Compared to traditional frame-based cameras, it covers a higher dynamical range (120dB) and offers a low power solution for vision tasks in high-speed scenario. Further combining it with neuromorphic processors (50 56
30 can create a fully neuromorphic system, realizing extremely low power and low latency sensing. However, learning from highly sparse and asynchronous events is challenging. Given its inherited asynchronous dynamics, SNN an ideal candidate for such task and recently number of works have applied it for event-based problems such as classification (29 34], tracking [68], detection [24], semantic segmentation (25) and optical flow estimation 31 etc. Multi-view event-based deep stereo solves the problem of 3D scene reconstruction based on pixel differences of the same physical point from event streams obtained by multiple views. Given the problem's complexity, it has been addressed by specially designed deep ANNs, Several works use additional information such as camera motion to produce sparse depth maps 79 78 Estimating dense disparity images from sparse event inputs more challenging. A recent work [62] addresses this problem by an event queue method which encodes events into event images through 3D convolution, followed by an hourglass network estimate disparity. Based on [2]. 1 further enhances local contours of the estimated disparity using image reconstruction. 44 creates feature pyramids with multi-scale correlation learned from a cycle of gray-scale images and event inputs. Inspired from biological neuron models, (73] developed discrete time convolution to encode events with temporal dynamic feature maps and proposed a dual-path encoder with spatially adaptive modulation to strengthen events representation. A very recent work (53) applies SNN to this problem with a handcrafted U-net structure. Nevertheless, compared with ANN models using geometric volumes its performance is suboptimal.
3 Method
3.1 Preliminary
We adopt the iterative leaky integrate-and-fire (LIF) neuron model [66] described by
u'." -ul-1.n( y'= + f
(1)
where superscripts and denote layer index and time step, respectively. T is the membrane time constant, u the membrane potential, y denotes the spike output and denotes the synaptic input with - E, wyy, f- where w is the weight. The neuron will fire spike y when um exceeds a threshold Vea, otherwise y = 0. In this work, we set r = 0.2 and VAA = 0.5. Given loss T., using chain rule the weight update of SNN can be expressed
ØL dw By du' ar Ow
ŌL dy' Ou' ar
Σ
(2)
where Je is the gradient of the spiking function, which is zero everywhere except at u = Vin- The SG approach uses continuous functions to approximate the real gradients, such rectangular (76). triangular (2). Superspike 770) ArcTan [16] and exponential curves (58). We adopt Dspike function from 355 Dsppkke()))
a (anh(b(æ - c)) d, which can cover a large range of smoothness by changing the temperature parameter b, with Dspike (z) = 1 or 0 for x > 1 or x < O. We set Vi, 0.5 and determine a and d by setting Dspike (0) 0, Dspike 1.
3.2
Differentiable hierarchical search for SNN
3.2.1
Cell level search
Similar to traditional DARTS 40). a cell is defined as a repeated and searchable unit, which is a directed acyelic graph with N nodes, {z+)x= depicted in Fig, Each cell receives input from two
3

---

1/3
1/6
G
Syp
1/12
cell to node node to node
node
1/24
(b)
Figure 1: Hierarchical search space of SpikeDHS. (a) Cell level search space, with 3 nodes example. Within one node, operations summed (2 operations here) and filtered by sign function. BN denotes batch normalization. (b) Layer search space for event- event-based deep stereo.
previous cells and forms its output by concatenating all outputs of its nodes. In SpikeDHS, each node a spiking neuron described by:
Zy f( f(E (xx)) iCj
(3)
where f sign function (or spiking neuron) taking the sum of all operations as input, off-) the operation associated with the directed edge connecting node and During search, each edge is represented by a weighted average of candidate operations, the information flow connecting node and node j becomes:
exp(or (1.1)
(x)
E OEO(U)) Eoco exp(a (8.)) o(x)
(4)
where O613 denotes the operation space on edge (i,j) and ds (1.s) the weight of operation o, which a trainable continuous variable. At the end of search, discrete architecture is selected by replacing each mixed operation g(6d) with the most likely operation o(+-) max at (4.f). For spiking neurons,
oEO(t-)
we can either mix the operation at the spike activation or at the input of the membrane potential, ie. y =
I ō. The former allows a search over operations with different SG functions, while the latter transfers more accurate learning signals for and leads to a concise node with less spiking filters (see supplement material for deduction). We choose the former for search phase and apply the latter particularly for SG search during retraining when operation type freeze (Section 3.4). Specifically, in the former choice, we distinguish between operations with the same forward computation but with different SG functions in candidate operation space. In addition, in original DARTS, a prepossessing step is required on operations between nodes and cells previous layers in order to align the dimension of feature maps. We merge this step with subsequent candidate operations to reduce model complexity and improve inference speed.
3.2.2 Layer level search
Task specific knowledge has been proved to be helpful in speeding up the search process and improving network performance. For classification task, we adopt a fixed downsampling structure (Fig. 2 in 40), with normal cells and reduction cells searched separately. For dense image prediction where high resolution output is needed and network architecture requires more variation, we implement differentiable search on the layer level as proposed in [39]. A of scalars (.) are trained to weight different potential layer resolutions and they are updated together with a. By the end of search, an optimal structure is decoded from a pre-defined L-layer trellis, as shown Fig. I For upsampling layers, we use nearest interpolation to maintain binary feature maps. For the stereo matching task, following volumetric approaches which embeds geometric knowledge of the binocular input, and search cell structure separately for the feature and matching subnetworks, similar to 9. At the end both subnetworks, the output of the last cell is upsampled to the initial resolution of the trellis using spike-based activation. We use batch normalization and ANN stem layers the search phase. the retraining phase, the ANN stem layers are converted to SNN by replacing the Relu function with spike function and retraining the
(3 77 we construct a feature volume

---

weights from scratch. The reason for using Relu activation during search is to ensure more stable updating of the supernet, since deep SNNs may suffer from gradient vanish problem when the SG functions is not chosen appropriately. In extended experiments we replace the Relu function in stem layers with Dspike function under appropriate hyperparameters, e-g h 3, and the search process is also stable. Parameters of batch normalization are converted into convolution weights and biases after retraining [22] 57], leading to full spike-based network for inference. We provide more details for spike-based layer search space in the supplement.
3.3 Loss function, estimator and optimization
For classification task, an auxiliary loss as in 40), with weight 0.4. For event-based stereo matching, our network produces matching cost tensor C of size dmax h w, based on which estimate a disparity map Î using a sub-pixel estimator [61
ò D(d)
D(d) softmin (Ca.y.z). with d argmin((Can (Cag.)) dald-d|<3
(5)
where 5 = 2 is an estimator support and ((d) 2d is disparity corresponding index d in the matching cost tensor. We use a sub-pixel cross entropy loss [61] to train the network, which is described by:
T())=
1
ΣΣ V.ar
Laplace( DGT b) log(softmin(Ca...))
wh
where Laplace(Dle DGT b) is a discretized Laplace distribution with the mean equal to the y.a ground truth disparity = DCT and diversity b = 2. Following bi-level optimization [40], we update weight and architecture parameters {a, 3} alternately based on two disjoint training sets A and
v.z
* Update network weights by V. 3) on A
Update architecture parameters a and by Va,aL(w, a, on B
We use first-order approximation to speed up the search process. After search, we decode the discrete cell structure by retaining two strongest afferent edges for each node, As to network structure, we decode it by finding the maximum probability path between different layers.
3.4 Differentiable surrogate gradient search
recent work 20 shows that suitable SG function is critical when the SNN goes deeper, and demonstrates that by optimizing the width (or temperature) of the SG function the performance of SNN can be improved. Continuous relaxation through gradient descent is an efficient approach to explore diverse operations on the same path, inspired from this idea, we propose a differentiable surrogate gradient search (DGS) method to parallelly optimize local SGs for SNN. In the retraining phase, with certain epoch intervals, candidate SG functions, {git N. based on which we update the weight (or N copies of the weight) separately, leading to [w These weights are then combined form a mixed operation weighted by a set of factors {ag. J through softmax function, described as:
35
associate each operation path with N
N
exp((x.))
Te, with I = Wgg Σラ exp(ar;))
We then update {ag,), } through the loss of the mixed operation output. This process is repeated for multiple batches and finally we update the original SG to {g.li = argmax, (ax)), with (-) denoting the average over batches. Note that Wgg can be obtained either by repeatedly calculate for each gi, or directly estimating from the gradient of the original SG, y.,w L. if { y are linear to The psuedo code of the algorithm is summarized in Algorithm. [ The intuition behind DGS is that the updated value of Og. indicates the contribution of Wg. in decreasing the loss. So {9s/t argmax, (az) which leads to the best updated weight, could be the most suitable SG function for the original local weight. Note that the difference between DGS and the SG search in Section 3.2.1 is that the former aims to optimize SG function for the local weight, while the latter essentially search of different operation types.
5

---

Algorithm 1: Differentiable surrogate gradient search (DGS)
Input: Training dataset, training epoch E, training iteration DGS iteration Ty. SG function y- candidate SG functions {9: x. SG weighting factors (og, ) and their initializing value e, epoch interval for DGS ep
for all e 1, ... E-th epoch do for all i 1, .... I-iteration do
L Collect training data and labels, update weights w based on SG function gi
4
if e/ee = Int. then
for all j =1, .... I..........geeeee. -tteraation do Initialize {a to c, update weights w with V a. an based on {9i} N obtain associated weights { N:
Combine (wg, & to form mixed operation, update {ag.tN with V agolwe) L.
Update g to {gsli argmax, {a:))
return trained network.
normal cell
reduction cell
feature cell
matching cell
7
Resolution
1/3
Resolution
Resolation
1/2
1/6
1:12
skip connection
conv3=3 bm3
1:4
1/12
(a)
(b)
Figure 2: (a) Architecture for classification: SpikeDHS-CLL CLL and (b) wvent-based stereo: SakkeDHS-Steroe
4
Experiments
4.1
Classification
The CIFAR1O and CIFAR100 datasets 28 have 50K/10K training/testing RGB images with a spatial resolution of 32 32. The ImageNet dataset [2 contains more than 1250k training images and 50k test images. We apply SpikeDHS on CIFARIO and then retrain on target datasets including CIFAR 10, CIFAR 100 and ImageNet. For ImageNet, we use larger variant of the searched network with one more stem layer and two more cells. We use standard pre-processing and augmentation for training as in 21 The test image directly centered cropped 224 224. More details about the model architecture and training are provided in the supplement material.
4.1.1 Architecture search and retrain
In the search phase, the training set equally split into two subsets for bi-level optimazation. For retraining, the standard training/testing split is used. We use 4 nodes (n4) within one cell and limited number of candidate operations reduce search time, which are (conv3x3 with g(b 3). conv3x3 with g( = 5), skip connection], ), with g Dspike. For the sign function we use g(b = 3). For the network architecture, we adopt an layer fixed downsampling architecture proposed in 40 The search phase takes 50 epochs with mini-batch size 50, the first 15 epochs are used to warm up convolution weights. We use SGD optimizer with momentum 0.9 and learning rate of 0.025. The architecture search takes about 1.4 GPU day on single NVIDIA Tesla V100 (32G) GPU. We term the searched architecture SpikeDHS-CC and plot Fig. 2 After search, we retrain the model on target datasets with channel expansion for 100 epochs with mini-batch size

---

50 for CIFAR and 160 for ImageNet, with cosine learning rate 0.025, We use SGD optimizer with weight decay 3e and momentum 0.9. The DGS method applied the first stem layer (s1), 0.001, = 5. Dspike with {0:) having equal temperature interval: {g(b (N-1) Ab), ..... g(b - ]b)} قb)} where Ab = 0.2 and 5. In extending experiments, we slightly increase the output channel of the first stem layer and use 3 nodes (n3) within cell. In
we set 1 = 100,
addition, we apply DGS the first node of the 5th cell (c5).
4.1.2 Results
The results are summarized in Table the values of other models are obtained from literature. On CIFAR datasets, SpikeDHS- -LA with DGS achieves the highest accuracy comparing with other directly trained SNNs under similar model capacity. Note that both Dspike [35] and TET [13] use advanced training algorithms while plain SpikeDHS- CLA models are trained with fixed SG function. Our model also outperforms the recent SNN works with NAS in terms of accuracy, inference steps and model size As a reference, we run the original ANN DARTS network with the same architecture and it achieves 95. 55.8% accuracy CIFARIO 10. On ImageNet, model with much smaller model capacity and rivals VGG-16 16 with less than half of its size. We conjecture that the large number of jump connections both within and across cells potentially help gradient propagation through deep layers, which improves training of the model.
models surpass the ResNet-34-large la-le
Table 1: Comparison image classification. P; DGS method. NoP: Number of parameters.
Dataset
Methods
Architecture
NoP
Accuracy[22
TPP--P
CIFARNet
5
91.41
D4]
ResNet-20
92.54
[T)PTBPtdBBN
Re
13M
6
93. 16
Dspike
ResNet-18
11M
6
94.25 0.07
13TTE
ResNet-19
13M
6
94.50 0.07
135 Dspike
ResNet-18
11M
6
94.25 = 0.07
CIFAR1O
26
SNASNet
SNASNct-Bw
8
94.12 = 0.25
46 AutoSNN
AutoSNN (C=128)
21M
8
93.15
SpikeDHS
SpikeDIIS-CLA (n4)
12M
6
94.34 1 0.06
SpikeDHS-CLA (n3)
14M
6
95.35 1
SpikeDHS
SpikeDIIS-CLA (n4s1)
12M
6
94.68 1 0.05
SpikeDIIS-LLA (n3s1)
14M
95.36 0.01
SpikeDHS- (n3c5)
14M
95.50 0.03
54 Diet-SNN
ResNet-20
5
64.07
STBP-idBN
ResNet-19
13M
6
71.12 = 0.57
35 Dspike
ResNet-18
11M
6
71.21 0.10
Re
13M
6
74.72 0.28
SNASNet
SNASNet-Bw
5
73.04 0.36
CIFAR 100
AutoSNN
AutoSNN (C=64)
SM
8
69.16
SpikeDHS
SpikeDHS-CIE (n4)
12M
6
75.70 + 0.14
SpikcDHS-CLA (n3)
14M
6
76.15 - 0.20
SpikeDHSD
SpikeDHS-CI. (n4s1)
12M
6
76.03 - 0.20
SpikeDHS-CC CLL (n3s1)
14M
6
76.25 - 0.10
STBP-tdNN
ResNet-34
22M
63.72
66 STBP-ddNN
ResNet-34-large
86M
6
67.05
ImageNet
D4ctietNNN
VGG-16 16
138M
5
69.00
Dpi
SpikeDHS-CLA-laee
58M
6
67.96
SpikeDHS
SpikcDHS-CL. MINSS-CLA-laagg
58M
6
68.64
4.2 Event-based deep stereo
We further apply our method to event-based deep stereo matching on the widely used benchmark MVSEC dataset [80]. The dataset contains depth information recorded by LIDAR sensors and event streams collected from pair of Davis346 cameras, with synchronized 20 Hz gray scale images at
Thee size of SNASNet is not given.

---

346x260 resolution. We split and preprocess the Indoor Flying dataset from the MVSEC following the same setting as [62 I 79]. We use the mean depth error (MDE), one-pixel-accuracy (IPA), median depth error, and mean disparity error as evaluation metrics. Learning from highly sparse raw events challenging and a prepossessing step often required to encode events. We use stacking based on time (SBT) [63] which merges events into temporally neighboring frames. During training, we use multiple consecutive stacks as one input, with an equal number of consecutive ground truth disparities as one label. More details can be found in the supplement material.
4.2.1 Architecture search and retrain
We use 3 nodes within one cell. For candidate operations, we use: x3 with g(b 3), conv3x3 with g(b = 5). skip connection) | for the feature net and 3onn333333 with g(b = 3), conv3 x x3x3 with g(b 5), skip connection) for the matching net, with . = Dspike. For the layer search space, we adopt a four-level trellis with downsampling rates of [3,2,2,2|. The number of layers is set to 2 and 4 for the feature and matching net, respectively. In addition, two stem layers applied in front of both subnetworks to reduce input spatial resolution and increase channel number. We search the architecture for 12 epochs with batch size 1. The first epochs are used to initiate the weight of the supernet. The rest 9 epochs are applied with bi-level optimization. We use SGD optimizer with momentum 0.9 and learning rate of 0.002. The architecture search takes about 0.4 GPU day on a single NVIDIA Tesla V100 (32G) GPU. We term the searched architecture as SpikeDHS-Sterns and plot it in Fig. 2 Extensive random seed experiments show that the architectures are gradually optimized during the search phase. After search, we retrain the model with channel expansion for 200 epochs with mini-batch size 2. We use Adam optimizer with initial learning rate 0.001 and momentum (0.9, 0.999), with learning rate decaying half at [50, 100, 150] epochs. The DGS method applied the first stem layer of the feature net with the same setup as classification. In extending experiments, we also retrain the network with operation mixed at the membrane potential (MM).
4.2.2 Results
We compare our method with other event-based stereo matching approaches (Section 2.3) on dense disparity estimation. The results are summarized in Table values of other models are obtained from literature. Among SNN methods, SpikeDHS significantly outperforms StereoSpike, exhibiting the effectiveness of the searched architecture. Among events-only approaches, SpikeDHS even surpasses ANN-based specially designed DDES network in all criteria with only one-third of its number of parameters. The performance of SpikeDHS- Sterro is further improved with DGS. As shown in Fig. 4b.b the temperature of the SG function is constantly optimized, which avoids vanishing gradient compared to training with fixed SG function, leading to more stable training of SNN. The MM approach also improves the performance of the network. A qualitative comparison of estimated disparities is shown in Fig. [ It can be seen that SpikeDHS with DGS predicts better disparities compares to other models, especially in local edges.
Streaming inference: In real-world scenario, events are generated consecutively by the sensor with flexible lengths. To test the real-time applicability of our model, we fed the entire test split contin- uously into the model, which evolves an equal length of steps and estimates sequential disparities. The inference speed of our model achieves 44 FPS (26 FPS for the DDES model, see supplement material for measurement details) while achieving similar accuracy as in standard testing (Fig. 3). where the model always receives a fixed length of events.
4.2.3 Ablation study
The inherent temporal dynamics of SNN is assumed to be helpful for it to learn temporal correlation of the data. To investigate this property, we fix the membrane time constant to 0 to create binary neuron (BN) and perform architecture search and retrain. As shown in Table 2 the SpikeDHS -BN model performs worse than SpikeDHS, which shows the benefit of temporal dynamics in this task. In addition, we evaluate different SG functions including Superspike 770.. Triangle 22, Arctan (16] SG functions with fixed hyperparameters during training and varying hyperparameters with DGS. The results demonstrate the robustness of DGS for different SG functions as it consistently improves network performance, as shown in Table 3 More details are provided in the supplement.
8

---

Table 2: We denote the best and second best results in bold and underscore. EO denotes events-only method. EITNet requires gray scale images for training but not inference. Symbols meaning: unavailability of value; (+), streaming tests; D training with DGS; estimated value; NoP, Number of parameters.
Median depth
Mean disparity
MDB [em] 4
IPA [%]]
error [em] 4
error [pix]
Method
EO
NoP
split1
split3
split1
split3
split1
split3
split1
split3
EIS
43
x
13.7
22.4
89.0
EITNet 1
>16M
14.2
19.4
5,9
10.4
0.55
0.75
92.1
89.6
2.33M
27.8
0.59
0.94
89.4
StereoSpike
58
18.5
25.4
SpikeDHS-BN
0.87M
17.5
20.3
10.8
0.59
0.74
88.7
SpikeDHS-BND
0.87M
17.0
19.8
6.8
10.2
0.58
0.73
89.3
88.5
SpikeDHS
0.87M
16.5(16.5)
19.4(19.5)
6.5(6.5)
10.6(10.6)
0.57(0.57)
0.73(0.74)
90.1(90.2)
88.5(88.4)
SpikeDHS
0.87M
15.9(15.8)
19.1(19.3)
6.3(6.)
10.4(10.5)
0.54(0.54)
0.72(0.74)
90.7(90.8)
88.9(88.8)
SpikeDHS (MM)
0.87M
15.7(15.7)
6.3(6.)
0.55(0.54)
91.0(91.1)
SpikeDI1S (MM)
0.87M
15.4(154)
6.0(6.0)
0,54(0.54)
91.3(91.))
(a) Events
(b) DDES
(c) SpikeDHS
(d) SpikeDHS"
(e) Ground truth
(f) Gray Image.
Figure 3: Qualitative comparison MVSEC. Disparity maps of different methods are on same frames, supplement material for details.
Table Different functions event-based stereo task on split
DGS.
SG function
1PA w/ DGS (w/o DGS) [1]]
Triangle 2
91.3 (90.9)
Arctan [16
90.1 (89.3)
Superspike 77
89.7 (89.6)
5 Sparsity and energy cost
The dense computation of deep ANNs came at a significant energy cost. In contrast, SNN performs sparse computing and multiplication-free inference. As Fig, shows, our models show an overall sparse activity with different degrees across layers. For event- based stereo, we plot network activity along with the density events for a short duration (Fig. ()- The activity of the first stem layer in the feature net highly correlates with the density of the event streams. This relation is weakened with the increase of layer depth. As activity propagates, the sparsity matching layers drops to create dense disparities. The operation number of ANN is collected by public available PyTorch package [6)] Following (35), the addition count of SNN is calculated by * T A, where the mean sparsity, T is the time step and A is the addition number. The operation number and

---

3.5
3.0
0.25
events
2.5
0.20
stemn
0.15
2.0
$ 0.100
cel0
1.5
0.05
50
125
175
50
100
200
250
Epoch
[50ms]
(a)
(C)
x10-1
10
SpicsDMS
0.35
SpikeDHS*
Mosin PLAW
SpikeDes
0.30
stam]
0.25
cell
cell
0.15
0
50
100 Epoch
150
200
50
100
150
200
50
100
150
200
250
300
Epoch
Time [50mm]
(b)
(d)
Figure 4: DGS training process and sparsity of SpikcDHS-Steree (a): Evolution SG temperatures during DGS training. (b): IPE (left) and first stem weight gradient (right) of DGS training and normal training. (c) and (d): Sparsity of the feature net and the matching net.
energy cost is in Table We can see that BakkeDHS-Steree has much lower operation number than DDES. Note that due to streaming inference, SNN realizes natural usage of its temporal dynamics with 7' 1. We measure the energy consumption following [54). In 45nm CMOS technology, the addition operation in SNN costs 0.9pJ while the multiply- accuuulaa (MAC) operation in ANN consumes 4.6pJ. SpikeDHS-Seee Stereoo costs only 6.7mJ for a single forward, with 26 lower energy than DDES.
0.3
0.3
Method
#Add.
#Mult.
Energy
0.
DARTS CIFARIO
29654M
29654M
136.4 mJ
0.1
0.1
SpikeDHS-CLA CLA (n4)
5973M
19M
5.5 m.J
0.0
0.0
DDES (ANN)
37917M
37917M
174.4 mJ
Layer
Layer number
Spik:DHS-Stereo
7414M
0M
6.7 mJJ
(a)
(b)
Figure Table Left: Network spansity for (a) classification and (b) event-based stereo. Right: The
operation number and energy cost.
Discussion
In this paper, develop the SpikeDHS framework which finds optimal architectures for SNNs. On image classification and event-based deep stereo tasks, our models outperform previous SNNs with ANN handcrafted architectures. The latter task demonstrates the advantage of SNNs in processing highly sparse and dynamic event streams with extremely low power. One potential reason of the high performance of architecture could be the large number of jump connections both within and across cells, which helps gradient propagation. Our method can be applied to other tasks where network requires more architectural variations such object detection and semantic segmentation. The DGS algorithm efficiently optimizes SGs locally, which is of general usage in improving training of deep SNNs binary networks. Currently only apply DGS on fixed layer, future works could develop a dynamical algorithm which applies it optimal layers. In addition, we only use limited candidate operations, other types of biological plausible connections such as recurrent and feedback connections, as well as excitatory/nnhiiiory synapses could be considered in the future.
Acknowledgements
This work is supported by the Science and Technology Innovation 2030-Major Project (Brain Science and Brain-Like Intelligence Technology) under Grant 2022ZD0208700.
10

---

References
[1] Soikat Hasan Ahmed, Hae Woong Jang, SM Nadim Uddin, and Yong Jung. Deep event stereo leveraged by event-to-image translation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 882-890, 2021.
[2] Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and Saaning-to-leant in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
[3] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Leg- enstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications, 11(1):1-15, 2020.
[4] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: Ie0:.30883333 3432, 2013.
[5] Sander M Bohte, Joost N Kok, and Johannes A La Poutré. Spikeprop: backpropagation for networks of spiking neurons. In ESANN, volume 48, pages 419-424. Bruges, 2000.
[6] Tong Bu, Wei Fang. Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal amn- snn conversion for high-accuue accuracy and aatra-low-latents spiking neural networks. In International Conference on Learning Representations, 2021.
[7] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 1294-1303, 2019.
[8] Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Xinyu Xiao, and Jian Sun. Detnas: Backbone search for object detection. Advances in Neural Information Processing Systems, 32, 2019.
[9] Xuelian Cheng, Yiran Zhong, Mehrtash Harandi, Yuchao Dai, Xiaojun Chang, Hongdong Li, Tom Drummond, and Zongyuan Ge. Hierarchical neural architecture search for deep stereo matching. Advances in Neural Information Processing Systems, 33:22158-22169, 2020.
[10] Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. Darts-: robustly stepping out of performance collapse without indicators. arXiv preprint arXiv:2099001027, 2020.
[11] Mike Davies, Andreas Wild, Garrick Orchard, Yulia Sandamirskaya, Gabriel A Fonseca Guerra, Prasad Joshi, Philipp Plank, and Sumedh R Risbud. Advancing neuromorphic computing with loihi: A survey of results and outlook. Proceedings of the IEEE, 109(5):911-99) 2021.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
[13] Shikuang Deng, Yuhang Li. Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv: / 2022.
[14] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997-2011 2019.
[15] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[16) Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2661- 2021.
[17] Steve Furber, Francesco Galluppi, Steve Temple, and Luis Plana. The spinnaker project. Proceedings of the IEEE, 10265)66526665 2014.

---

[18] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger. Andrew Davison, Jörg Conradt, Kostas Daniilidis, et al. Event- based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1): 184-180, 2020.
[19] Jianyuan Guo, Kai Han, Yunhe Wang, Chao Zhang, Zhaohui Yang, Han Wu, Xinghao Chen, and Chang Xu. Hit-detector: Hierarchical trinity architecture search for object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11405-11414, 2020.
[20] Jesse Hagenaars, Federico Paredes- Vallés, and Guido De Croon. Self-supervised learning of event-based optical flow with spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[22] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[23] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE International Conference on Computer Vision, pages 66-75, 2017.
[24] Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11270-11277, 2020.
[25] Youngeun Kim, Joshua Chough, and Priyadarshini Panda. Beyond classification: Directly training spiking neural networks for semantic segmentation. arXiv preprint arXiv:2110.07743 2021.
[26] Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks. arXiv preprint arXiv:2201. 2022.
[27] Agnes Korcsak- Korzsak-Gorzo, Michael G Müller, Andreas Baumbach, Luziwei Leng, Oliver Bre- itwieser, Sacha J van Albada, Walter Senn, Karlheinz Meier, Robert Legenstein, and Mihai A Petrovici. Cortical oscillations support sampling-based computations in spiking neural networks. PLoS computational biology, 1803):e1009753 2022.
[28] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- (canadian institute for advanced research). URLp
twwotto. httitttt.ddd///iiiiiiieii/e .htll
[29] Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in neuroscience, 14: 439, 2020.
[30] Akos F Kungl, Sebastian Schmitt, Johann Klähn, Paul Müller, Andreas Baumbach, Dominik Dold, Alexander Kugele, Eric Müller, Christoph Koke, Mitja Kleider, et al. Accelerated physical emulation of bayesian inference in spiking neural networks. Frontiers in neuroscience, page 1201, 2019.
[31] Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis, and Kaushik Roy. floweett event-based optical flow estimation with energy-efficien hybrid neural networks. In European Conference on Computer Vision, pages 366-382, Springer, 2020.
[32] Luziwei Leng. Solving Machine Learning Problems with Biological Principles. PhD thesis, 2020.
12

---

[33] Luziwei Leng, Roman Martel, Oliver Breitwieser, Iija Bytschok, Walter Senn, Johannes Schem- mel, Karlheinz Meier, and Mihai A Petrovici. Spiking neurons with short-term synaptic plasticity form superior generative networks. Scientific reports, 8(1):1-11, 2018.
[34] Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from amn: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning, pages 6325.. PMLR, 2021.
[35] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng. Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descen for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[36] Hualou Liang, Xiajing Gong, Minggui Chen, Yin Yan, Wu Li, and Charles D Gilbert. Interac- tions between feedback and lateral connections in the primary visual cortex. Proceedings of the National Academy of Sciences, 114(32):837-8642, 2017.
[37] Peiwen Lin, Peng Sun, Guangliang Cheng, Sirui Xie, Xi Li, and Jianping Shi. Graph-guided architecture search for real-time semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4203 4212, 2020.
[38] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pages 19 19-34, 2018.
[39] Chenxi Liu, Lang-Chiihh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 82-92, 2019.
[40] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 1806.090050. 2018
[41] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):659--16) 1997.
[42] Paul A Merolla, John V Arthur, Rodrigo Alvarez-lcaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, al. A million spiking- neuron integrated circuit with scalable communication network and interface. Science, 345 (6197):668-66) 673,, 2014.
[43] Mohammad Mostafavi, Kuk-Jin Yoon, and Jonghyun Choi. Event-intensity stereo: Estimating depth by the best of both worlds. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 4258-4267. 2021.
[44] Vernon B Mountcastle, The columnar organization of the neocortex. Brain: a journal of neurology. 120(4):701-722, 1997.
[45] Milad Mozafari, Saeed Reza Kheradpisheh, Timothée Masquelier, Abbas Nowzari-Daling and Mohammad Ganjtabesh. First-spike-based visual categorization using reward-modulated stdp. IEEE Iransactions on neural networks and learning systems, 29(12)66188 116666786690. 2018.
[46] Byunggook Na, Jisoo Mok, Seongsik Park, Dongjin Lee, Hyeokjun Choe, and Sungroh Yoon. Autosnn: Towards 2022.
spiking neural networks. arXiv preprint arXiv:2201. 12738.
[47] Emre Neftci, Srinjoy Das, Bruno Pedroni, Kenneth Freutz-Delgade and Gert Cauwenberghs. Event-driven contrastive divergence for spiking neuromorphic systems. Frontiers in neuro- science, 7:272, 2014.
[48] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-63 2019.
13

---

[49] Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid. Fast neural architecture search of compact semantic segmentation models via auxiliary cells. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9126-9135, 2019.
[50] J. Pei, L. Deng, S. Song, M. Zhao, and L. Shi. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):106, 2019.
[51] Mihai A Petrovici, Johannes Bill, Dija Bytschok, Johannes Schemmel, and Karlheinz Meier. Stochastic inference with spiking neurons in the high-conductance state. Physical Review E, 94 (4):042312, 2016.
[52] Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pages 4095- 4104, PMLR, 2018.
[53] Ulysse Rançon, Javier Guadrado-Anibaroe Benoit R Cottereau, and Timothée Masquelier. Stereospike: Depth learning with a spiking neural network. arXiv preprint 1xXiv:210.11375) 2021.
[54] Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[55] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages $780-4789 2019.
[56] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- machine intelligence with ncuromorphic computing. Nature, 575(77844::00--11,,2 2019.
[57] Bodo Rueckauer, lulia-Alexandaa Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of cantinuous-valuedd deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience, 11:682, 2017.
[58] Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time, arXiv preprint arXiv:/810.00666 080466 2018.
[59] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[60] Vladislav Sovrasov. Flops counter for convolutional networks in pytorch framework, 2019. pytorrc/
URL http:pp//ggggaaa...ooooo/rr////s
com/ cctmm///orrassrllgsss-oottteeeee
[61] Stepan Tulyakov, Anton Ivanov, and Francois Fleuret. Practical deep stereo (pds): Toward pepiicatioos-friendt deep stereo matching. arXiv preprint arXiv: 100.:1800..1177, 2018.
[62] Stepan Tulyakov, Francois Fleuret, Martin Kiefel, Peter Gehler, and Michael Hirsch. Learning an event sequence embedding for dense event-based deep stereo. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1527-1537, 2019.
[63] Lin Wang, Yo-Sung Ho, Kuk-Jin Yoon, al. based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10081-10090, 2019.
[64] Ning Wang, Yang Gao, Hao Chen, Peng Wang, Zhi Tian, Chunhua Shen, and Yanning Zhang Nas-fcos: Fast neural architecture search for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11945,11951, 2020.
[65] Yujic Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
[66] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1311-1318, 2019.
14

---

[67] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-efficien architecture search. arXiv preprint arXiv: 1907.05777 2019.
[68] Zheyu Yang, Yujie Wu, Guanrui Wang, Yukuan Yang, Guoqi Li, Lei Deng, Jun Zhu, and Luping Shi. Dashnet: A hybrid artificial and spiking neural network for high-speed object tracking. arXiv preprini arXiv: /909.
100009.2944 22942,9922222 2019.
[69] Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding straight-through estimator in training activation quantized neural nets. arXiv preprint 190..v::903.055600 2019.
[70] Friedemann Zenke and Surya Ganguli, Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 3066;:514-1541, 2018.
[71] Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 33(4):899-925,
2021.
[72] Feihu Zhang, Victor Prisacariu, Ruigang Yang, and Philip HS Torr. Ga-net: Guided aggregation net for end-to-end stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 185-194, 2019.
[73] Kaixuan Zhang, Kaiwei Che, Jianguo Zhang, Jie Cheng, Ziyang Zhang, Qinghai Guo, and Luziwei Leng. Discrete time convolution for fast event-based stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8676-8686, 2022.
[74] Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems, 33:12022 2020.
[75] Yiheng Zhang, Zhaofan Qiu, Jingen Liu, Ting Yao, Dong Liu, and Tao Mei. Customizable architecture search for semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11441-11550, 2019.
[76] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11062-1 11700, 2021.
[77] Yiran Zhong, Yuchao Dai, and Hongdong Li. Self-supervieed learning for stereo matching with self-improving ability. arXiv preprint arXiv: 170:: 11091170993393 2017.
[78] Yi Zhou, Guillermo Gallego, Henri Rebecq, Laurent Kneip, Hongdong Li, and Davide Scara- muzza. Semi-dense 3d reconstruction with a stereo event camera. In Proceedings of the European Conference on Computer Vision (ECCV), pages 235-251, 2018.
[79] Alex Zihao Zhu, Yibo Chen, and Kostas Daniilidis. Realtime time synchronized event-based stereo. In Proceedings of the European Conference on Computer Vision (ECCV), pages 433-447, 2018.
[80] Alex Zihao Zhu, Dinesh Thakur, Tolga Özaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Daniilidis. The multivehicle stereo event camera dataset: An event camera dataset for 3d perception. IEEE Robotics and Automation Letters, 3(3):2032-2339 2018.
[81] Lin Zhu, Xiao Wang, Yi Chang, Jianing Li, Tiejun Huang, and Yonghong Tian. Event- based video reconstruction via potential-assisted spiking neural network. arXiv preprint 2TX1v:2001.10943, 2022.
[82] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint 16110117 2016
[83] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Lc. Learning transferable architectures for scalable image recognition. In Proceedings the IEEE conference computer vision and pattern recognition, pages 8697-8710, 2018
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Abstract and Introduction.
(b) Did you describe the limitations of your work? [Yes] See Discussion.
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work is concerned with technical basic research and we do not foresee any such direct negative impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] It does.
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See both the main text and the supplemental material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See main text and supplemental material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See tables Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 1.1.1 and Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [Yes] Public resource.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "xINKCDvGgDo", "reviewer": "Reviewer_LhUf", "summary": "In this submission draft, the authors device a differentiable hierarchical search framework tailored for SNNs. In the meantime, this framework is able to search the surrogate gradient in a differentiable manner. Their methods are validated on the CIFAR dataset and an event-based deep stereo dataset. ", "questions": "1. Can authors elaborate on the search space in this work, especially when compared to the search space in DARTS and NAS-Bench-101?\n\n2. Can the authors provide more details on the difference in searching gradients with Dspike in Section 3.4? Currently, the only saying is *\" [31] demonstrates that by optimizing the width (or temperature) of the SG function the performance of SNN can be improved\"*, but there is no drawbacks analysis and to which degree that this DGS method can improve. ", "limitations": "1. Need to include two prior SNN NAS papers in the discussion or experiments. See references below.\n\n\n2. A critical problem is that there is no comparison between the searched architecture and the ResNets used in other works. What if the searched architecture has a higher capacity than ResNets?\n\n3. An ablation study on the DGS is recommended, the authors should compare static temperature gradient, DGS, and [31] on the same neural architecture and under the same training receipt.\n\n4. Better to have an ImageNet result. \n\n\n\n------\n\n**References**\n\nNa B, Mok J, Park S, et al. AutoSNN: Towards Energy-Efficient Spiking Neural Networks[J]. arXiv preprint arXiv:2201.12738, 2022.\n\nKim Y, Li Y, Park H, et al. Neural architecture search for spiking neural networks[J]. arXiv preprint arXiv:2201.10355, 2022.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall this is an interesting work. The authors come up with an end-to-end differentiable framework that solves two critical problems in SNN: the architecture and the surrogate gradient. \n\n\n1. Developing SNN-oriented architectures are novel and necessary. Even though this work is not the first trial in the community. \n2. Searching the SG is interesting and I am glad to see a learning-based method to address the issue. \n2. The results on the CIFAR10/100 dataset are promising.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YxxibRhimpE", "reviewer": "Reviewer_hUzC", "summary": "In this work, the authors propose a differentiable hierarchical search framework for spiking neurons, where spike-based computation is realized on both the cell and the layer level search space. Meanwhile, the authors find effective SNN architectures under limited computation cost. In order to avoid the standard SG approach that leads the network into suboptimal solutions, the authors propose a differentiable surrogate gradient search method where the SG function can be efficiently optimized locally in parallel. Finally, this work shows some interesting results on the image classification tasks.", "questions": "1. My core concern lies in the choice of the surrogate gradient (SG) function in the methods section. First, it looks like the SG function is from existing work [31]. If this is true, the reviewer suggests that the authors put this subsection in the background section. Second, what is the motivation for choosing such a SG function? Because, the motivation for the design of the SG function has been given in the existing studies [1-2]. It is clear that SG methods are a key part of the direct training of spiking neurons, and the choice of such methods also affects the search results, and the reviewers suggest that the authors explain the motivation clearly.\n\n[1]\"Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks,\" Nat. Mach. Intell. 3(10): 905-913 (2021)\n\n[2]\"A Hybrid Spiking Neurons Embedded LSTM Network for Multivariate Time Series Learning under Concept-drift Environment,\" in IEEE Transactions on Knowledge and Data Engineering, doi: 10.1109/TKDE.2022.3178176.\n\n2. The parallel optimization involved in the search process needs further explanation by the authors.\n\n3. There is a significant difference in performance improvement on the CIFAR-10 and CIFAR-100 datasets, and the authors need to explain the reason for this phenomenon.\n\n4. The ablation experiments concluded the important value of temporal dynamics in the spiking neuron, but why the MDE metric and Median depth error metric appear opposite on split3. i.e., DARTS-BN^D better than DARTS-SNN^D in terms of Median depth error metric-split3.\n\n5. In Table III, the experimental results show that the power consumption of the proposed method is significantly lower. However, the reviewers note that the SNN model is tested on a chip and whether it is the device that leads to a more recent reduction in power consumption.\n", "limitations": "The authors illustrate the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. A hierarchical differentiable surrogate gradient search framework is proposed to obtain better performance of the spiking model.\n2. Significant improvements in energy savings on deep stereo.\n\nWeakness：\n1. In terms of writing, some methods that were not proposed in the work were placed in the methods section. There are also some typos in terminology.\n2. The results of the ablation experiments and the analysis of some elements do not match.\n3. The font of the figure seems to be a small and not clear enough, which leads to a very careful reading to find valuable information.\n4. The percentage improvement of the proposed method varies greatly on the two image classification datasets. Even the improvement on CIFAR-10 is only 0.18.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GxGNgybmjru", "reviewer": "Reviewer_cjsQ", "summary": "This work is aimed to search for both the optimal SNN architecture and hyperparameters of surrogate gradient (SG) functions. In the architecture search phase, they use DARTS and refine the search to different granularities (layer-level and cell-level). The search for SG function (DGS) focuses on optimizing the temperature of the Dspike SG function. The results show that searched architecture achieve SOTA  performance on image classification and event-based stereo matching task.", "questions": "1. The comparison of previous NAS work on SNNs should be added.\n2. The training pipeline is unclear to me. Does the training pipeline include **both** ANN-to-SNN conversion and direct training using SG? I notice that L158 suggests that ANN layers are converted to SNN layers. If so, the comparison to some previous work using only directly trained SNNs is not fair.\n3. Why Dspike is the only tested SG function? As mentioned in ref. 64, the choice of SG function affects the viable range of hyperparameters. The range determines the difficulty level of search. As there has been no claimed best SG function till now and ref. 64 points out that some different SG functions actually have the same maximum performance, the author should try some popular SG functions like SuperSpike[1], triangle[2], ArcTan[3], etc., to show the versatility of this method.\n\n[1] Friedemann Zenke, and Surya Ganguli. \"Superspike: Supervised learning in multilayer spiking neural networks.\" *Neural computation* 30.6 (2018): 1514-1541.\n\n[2] Guillaume Bellec, et al. \"Long short-term memory and learning-to-learn in networks of spiking neurons.\" *NeurIPS* (2018).\n\n[3] Wei Fang, et al. \"Incorporating learnable membrane time constant to enhance learning of spiking neural networks.\" *ICCV*. 2021.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros**\n1. The search for the architecture alone significantly increases the performance of image classification tasks, which reveals the potential to be applied to various more complicated tasks.\n2. The idea of searching hyperparameter of SG function is novel, simple but effective.\n\n**Cons**\n1. The idea of applying NAS on SNNs is not novel till the deadline of NeurIPS submission. SNASNet[1] and AutoSNN[2] have proposed that NAS methods can be used for searching the structure of SNNs. The latter has been accepted at ICML2022.\n2. The articulation of the training pipeline is not highlighted and is somewhat unclear to me. See the **Questions** below.\n3. The trials of search on SG functions are confined to the Dspike function.\n\n[1] Youngeun Kim, et al. \"Neural architecture search for spiking neural networks.\" *arXiv preprint arXiv:2201.10355* (2022).\n\n[2] Byunggook Na, et al. \"AutoSNN: Towards Energy-Efficient Spiking Neural Networks.\" *arXiv preprint arXiv:2201.12738* (2022).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CVvSInpg_hh", "reviewer": "Reviewer_Y1Po", "summary": "The paper presents a NAS optimization algorithm for SNN search.", "questions": "See my above comments.", "limitations": "See weakness section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+The authors present interesting results with the differentiable NAS search.\n-There are two major works related to NAS for SNNs that has been recently out [5], [11]. The authors have not cited these works. It makes me wonder what is the author's contributiona s compared to these works. [5]  talks about the fact that training SNN using standard NAS methods might be too complex because SNNs need large training time, so they come up with a NAS without tarining technique. [11] talks about a differentiable NAS technique. Both works show good results on a avraiety of datasets, and talk about the intricacies of architecture search.\n-The authors have also compared their technique to select works in table 1.  There is a lot of work from Priya Panda's group at Yale, Emre Neftci's group, and many others with regard to SNN training that show SOTA results on DVS and static datasets. The authors have failed to acknowledge most recent works.\n\nBelow is a list of publications (not exhaustive) that the author should check:\n[1] Towards spike-based machine intelligence with neuromorphic computing K Roy, A Jaiswal, P Panda Nature 575 (7784), 607-617\n\n[2] Enabling spike-based backpropagation for training deep neural network architectures C Lee, SS Sarwar, P Panda, G Srinivasan, K Roy Frontiers in neuroscience, 119\n\n[3] Rate Coding Or Direct Coding: Which One Is Better For Accurate, Robust, And Energy-Efficient Spiking Neural Networks? Y Kim, H Park, A Moitra, A Bhattacharjee, Y Venkatesha, P Panda ICASSP 2022-2022\n\n[4] Neuromorphic Data Augmentation for Training Spiking Neural Networks Y Li, Y Kim, H Park, T Geller, P Panda arXiv preprint arXiv:2203.06145\n\n[5] Neural architecture search for spiking neural networks Y Kim, Y Li, H Park, Y Venkatesha, P Panda arXiv preprint arXiv:2201.10355\n\n[6] Optimizing deeper spiking neural networks for dynamic vision sensing Y Kim, P Panda Neural Networks 144, 686-698\n\n[7] Federated Learning with Spiking Neural Networks Y Venkatesha, Y Kim, L Tassiulas, P Pand IEEE Transactions on Signal Processing 2021\n\n[8] Beyond classification: directly training spiking neural networks for semantic segmentation Y Kim, J Chough, P Panda arXiv preprint arXiv:2110.07742\n\n[9] Revisiting batch normalization for training low-latency deep spiking neural networks from scratch Y Kim, P Panda Frontiers in neuroscience, 1638\n\n[10]Na, Byunggook, et al. \"AutoSNN: Towards Energy-Efficient Spiking Neural Networks.\" arXiv preprint arXiv:2201.12738 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept