Deep activity propagation via weight initialization in spiking neural networks

Authors: aurora  micheli
olaf  booij
jan  van gemert
nergis  tömen
Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-09 14:43:40

Abstract: Spiking Neural Networks (SNNs) offer advantages such as sparsity and ultra-low power consumption, making them a promising alternative to conventional neural networks (ANNs). However, training deep SNNs is challenging due to the quantization of membrane potentials into binary spikes, which can cause information loss and vanishing spikes in deeper layers. Traditional weight initialization methods from ANNs are often used in SNNs without accounting for their distinct computational properties. In this work, we derive an optimal weight initialization method tailored for SNNs, specifically taking into account the quantization operation. We demonstrate through theoretical analysis and simulations with up to 100 layers that our method enables the propagation of activity in deep SNNs without loss of spikes. Experiments on MNIST confirm that the proposed initialization scheme leads to higher accuracy, faster convergence, and robustness against variations in network and neuron hyperparameters.

Main paper: 
 introduction: 
 Spiking Neural Networks (SNNs) are a class of artificial neural networks inspired by the dynamics of biological brains, where information is encoded and transmitted through discrete spikes [1, 2, 3] . This unique mode of communication enables SNNs to perform fast computations with low power consumption [4, 5] , especially when combined with specialized neuromorphic hardware [6, 7, 8] . However, SNNs still underperform compared to conventional artificial neural networks (ANNs), mainly due to the additional challenges associated with their training. While ANNs are typically trained via gradient descent, the discrete nature of spikes in SNNs complicates the use of backpropagation. Different methods such as ANN-SNN conversion [9, 10, 11] and backpropagation with surrogate functions [12, 13] have been proposed to circumvent this problem. Nevertheless, the proposed solutions haven't been sufficient to fully bridge the performance gap between SNNs and ANNs, without compromising the efficiency advantages of SNNs. Similarly to ANNs, in SNNs a suboptimal weight initialization can hamper the training process [14, 15] , especially in deep networks [16] . While ANN weight initialization strategies tailored to specific activation functions and weight distributions have been widely explored [17, 18, 19] , these methods are also often inappropriately applied to SNNs. Unlike ANNs, SNNs feature temporal dynamics, resetting mechanisms, information quantization and their activation function differs from those examined in the ANN literature. Hence, ANN initalization schemes are inadequate for SNNs and often cause undesired effects such as vanishing or exploding spikes in deeper layers.In this paper, we analytically derive a weight initialization method that accounts for the specific activation function of Spiking Neural Networks (SNNs), building on the approach proposed in [18] for standard ANNs. We empirically demonstrate that, unlike the standard ReLU-based method, our initialization enables spiking activity to propagate through deep networks without dissipation or amplification. Additionally, we show that proper weight initialization leads to improved accuracy, faster convergence, and lower latency in a simple classification task. 
related work: 
 A proper initialization method should avoid reducing or amplifying the magnitudes of input signals.In ANNs, Glorot & Bengio in [17] address the issue of saturated units for the logistic sigmoid activation function and propose a new weight initialization scheme aimed at maintaining constant activations and gradient variance across layers. He et. al [18] extend this analysis to Rectified Linear Unit (ReLU) activations, introducing the widely adopted Kaiming initialization for deep ANNs. In contrast, research on initialization schemes for SNNs is scarce and the problem of information propagation in SNNs has been often indirectly addressed. In [20] and [21] the appropriate membrane leak factor and firing threshold are learnt during training. Similarly, in [22, 23] some learnable parameters are incorporated in the SNN to optimize the neuron firing rate, thus increasing the computational complexity. Additionally, approaches like global unsupervised firing rate normalization [24] , batch normalization adapted to SNNs [25, 26] and constraints on the membrane potential distributions [27, 28] help regulate spike response and information flow. Some studies regard ANN-SNN conversion as an initialization method [29] , but this is limited to rate-based encoding, restricting its applicability to networks with alternative encoding schemes. Only a few studies directly address what constitutes a good initial state for SNNs. Some works attempt to empirically determine a suitable weight scale in the case of SNNs, often lacking a solid theoretical foundation [16, 30, 31, 32] . In [33] the authors derive a new initialization strategy considering the asymptotic spiking response given a mean-driven input. [15] proposes a fluctuation-driven initialization scheme, but neglects both the spiking and the resetting mechanism. In [34] a specular approach similar to the one presented in [18] is studied, yet the theoretical insights lack empirical validation. 
the spiking neuron: 
 The Leaky-Integrate-and-Fire (LIF) neuron [1] is one of the most popular models used in SNNs [2, 35] and neuromorphic hardware [7, 6] to emulate the functionality of biological neurons. The state of a LIF neuron at time t is given by the membrane potential U (t) which evolves according toEQUATIONwhere τ is the membrane time constant, R is the resistance of the membrane and I(t) is the timevarying input current. Following previous works [26] we use the discretized equation with time step ∆t which gives membrane potential u at time step t as:EQUATIONwhere β ∝ (1 -∆t/τ ) is a leak factor ∈ [0, 1] governing the rate at which the membrane potential decays over time, j is the index of the pre-synaptic neuron, w j represents the weight of the connection between the pre-and post-synaptic neurons and x j is the binary spike activation. When the membrane potential u exceeds a firing threshold θ, the neuron emits a binary output spike x = 1. After firing, the membrane potential is reset by subtracting from its value the threshold θ (soft reset).Our derivation is inspired by He et al. [18] , which suggests that an effective weight initialization should enable information flow across many network layers by keeping the variance of the input to each layer constant. We examine the variance of responses within each layer of a fully-connected SNN initialized at time step t = 0.For a generic layer l with m neurons: When applied to pre-activation distribution u l-1 (left) the SNN thresholding mechanism (middle) generates binarized activations x l (right). The dark shaded areas of u l-1 correspond to the fraction of neurons which will be activated and provide non-zero input to the next layer. With identical input distributions, this fraction is considerably lower for SNNs. This highlights why weight initializations optimized for ReLU will lead to vanishing activity in deep SNNs.EQUATIONHere x l ∈ {0, 1} n is a binary vector representing the n input spikes, w l ∈ R m×n is the weight matrix and u l ∈ R m represents the membrane potentials of neurons in layer l. x l is obtained by applying the activation function f to the membrane potentials of layer l -1. In a conventional SNN f is defined as the Heaviside step function:EQUATIONwhere u l-1 are the elements of u l-1 and θ > 0 is the neurons firing threshold. We assume that the elements of w l are mutually independent and share the same distribution (i.i.d.). Following [18] and [17] , the elements of x l are also considered to be mutually independent and identically distributed (i.i.d.). Lastly, w l and x l are independent of each other. We can then write:EQUATIONHere u l , w l , and x l represent each random variable element in u l , w l and x l respectively. We choose w l to be symmetrically distributed around 0. Since w l and x l are independent of each other, we can rewrite the variance of their product as:EQUATIONwhere E[x 2 l ] is the expected value of x 2 l . It is worth noting that the expression E[x 2 l ] strongly depends on the network activation function. Here is where our derivations differ from He et al. [18] .By assuming that u l-1 is zero-centered and symmetric around its mean, for a ReLU activation function,x l = max(0, u l-1 ), one obtains E[x 2 l ] = 1 2 Var[u l-1 ]. This result stems from the fact that the ReLU function preserves exactly the positive half of the distribution it acts upon. As depicted in Figure 1 , this doesn't hold true for the activation function of SNNs where, by definition, θ > 0. This difference leads to considerably sparser activations in SNNs. In the case of an SNN, we can expressEQUATIONThe binary elements x j l ∈ {0, 1} represent spikes. Applying the SNN activation function (5) to Eq. 8, we find that E[x 2 l ] = P (u l-1 > θ). Equation 7 can then be rewritten as:EQUATIONAs commonly done in recent works [28, 20, 36, 23] , we consider a real-valued input I 0 encoded to binary spikes using the first layer of the SNN. When feeding I 0 to the membrane potentials u 0 of the initial layer, u 0 = I 0 and u 0 trivially follows the same distribution as the input. We let I 0 be standard normal distributedI 0 ∼ N (µ = 0, σ 2 = 1), thus Var[u 0 ] = 1, E[u 0 ] = 0.A proper initialization method should avoid reducing or amplifying the magnitudes of the input signals when propagated across the network layers. This condition can be met if Var[u l ] = 1 for every layer l, which lets us simplify Eq. 9 and leads to a zero-mean Gaussian weight distribution with variance:EQUATIONEquation 10 is our proposed weight initialization method for training deep SNNs. Note that in terms of architecture parameters, it only depends on the number of input neurons n.Because u l-1 is symmetric around 0 and θ > 0, then P (u l-1 > θ) < 1 2 . It is important to note that:EQUATIONWhere 2 n l is the standard initialization for a ReLU network [18] . Thus, initializing the weights of an SNN using a method designed for conventional ANNs with ReLU activation functions does not ensure the propagation of information from the input throughout the network. 
validation with numerical simulations: 
 Unless otherwise specified, we consider fully-connected SNNs with 100 layers and n = 1000 LIF neurons in each layer. The input I 0 is real-valued and randomly drawn from N (µ = 0, σ 2 = 1). Consistently with the derivation in 3.2, we encode the inputs to binary spikes by feeding them to the membrane potentials of the initial LIF layer u 0 . We investigate the behavior of activity propagation under different weight initialization schemes and compare our method against the prevailing choice for conventional ANNs: Kaiming initialization ( [18] ). The weights in the network are therefore randomly initialized respectively from N (0, 1 nP (u0>θ) ) (our method) and N (0, 2 n ) (Kaiming), where n is the layer width. Since u 0 ∼ N (0, 1) , P (u 0 > θ) is defined as:EQUATIONThe integral in Eq. 12 doesn't have a closed-form solution, but it can be numerically estimated using the error function [37] . We recall from 3.2 that, to retain the activity over depth, we aim to conserve Var[u l ] across the layers. We initialize the network at time t = 0, propagate the input across its layers, record the values of the membrane potentials u l in every layer l and compute their variance. Figure 2 shows how Var[u l ] evolves with depth for the 2 different initialization schemes and for 6 different values of the firing threshold θ. For every different value of θ, we run the simulation 20 times and plot the average. The shaded areas represent the standard deviation over the different runs.The results (Figure 2 , left) demonstrate that in an SNN initialized with our proposed method, the variance Var[u l ] of the neuron states stays constant across layers regardless of the threshold θ, as the theory predicts. Conversely, with Kaiming initialization, information dissipates across layers, especially as firing thresholds increase. The only effective way to preserve information with Kaiming is to set θ = 0, where the activation function becomes effectively equivalent to ReLU. Kaiming for six firing threshold values (θ). For all θ, our proposed initialization method enables information propagation across all 100 layers. In contrast, Kaiming initialization leads to information dissipation across layers, particularly evident with higher threshold values. Each simulation was repeated 20 times, and the shaded areas represent the standard deviation over these runs.SNNs excel at processing time-dependent input thanks to the intrinsic memory of their spiking neurons, represented by the membrane potential u. This makes them particularly suited for tasks involving dynamic temporal patterns, like speech recognition and video analysis [38, 39, 40] .In this section, we extend the analysis from Section 4 to examine how weight initialization impacts information propagation across both space and time in multiple time-step simulations of deep SNNs. We employ fully-connected SNNs of 100 layers with n = 1000 neurons in each layer. We consider LIF neurons with soft reset and numerically compute the discrete-time dynamics based on Eq. 2. The dynamics of the membrane potentials including the reset term is given by:EQUATION)for time step t > 0 and layer l. β ∈ [0, 1] is the leak factor. Network weights are randomly initialized either using our initialization scheme or Kaiming, and the inputs are randomly drawn from ∼ N (0, 1), same as in Section 4. Differently, in this section, we iteratively feed the input (constant over time) to the membrane potentials of the initial LIF layer u t 0 at every time step t. We compute the variance of the membrane potentials u t l and the total number of spikes at every layer l and time step t, for a total of T = 20 time steps. We repeat each simulation 10 times, and report their average. The effect of the leak and reset terms becomes more pronounced for high values of β and pushes the network into the dissipative regime. However, with our proposed initialization method, we can still successfully retrieve an output. The network initialized with our method succeeds in conserving the Var[u t l ] across both space and time, whereas the network initialized with Kaiming fails (Fig. 3a ). Conserving Var[u t l ] is crucial, as it means conserving the number of spikes, and therefore ensuring a consistent network output.Although our mathematical derivation does not explicitly take time into account, unlike methods derived for ANNs, it considers the specific SNN activation function, and by keeping the variance of the membrane potentials u l constant, it aims to indirectly keep the variance of the layer input w l x l constant (Eq. 3). This helps to effectively propagate information also across multiple time steps. Nevertheless, deviations from theory are expected due to the leak and reset terms (Eq. 13). In particular, the reset operation affects the membrane potential distributions, violating the assumption of a normal distribution symmetrically centered around 0 (see Appendix 7: Figure 5 ). However, how well the normal distribution still holds as an approximation depends on neuron hyperparameters. For example, deviations from theory are visible at higher values of β. A larger β leads to broader distributions of u t l , and thus to a more abrupt change in the distributions when neurons with u t l > θ are reset. As illustrated in Fig. 3b , when β = 0.9, the network dissipates energy over time. We attribute this dissipation to the shift in u t l distributions. Still, we note that with our proposed initialization method, we can still successfully retrieve an output, unlike with Kaiming. 
experiments on mnist: 
 To empirically evaluate our variance-conserving weight initialization, we conduct object classification experiments using the MNIST digits dataset [41] . The 28×28 pixel grey-scale images are normalized to have mean 0 and variance 1, in line with the assumptions used in the derivations. As in Section 4 the inputs are encoded to binary spikes using the first LIF layer. The final layer of the network outputs binary spikes, which are accumulated over time steps and passed to the cross-entropy loss function. Unless otherwise specified, we employ a fully-connected SNN consisting of 10 layers, each comprising n = 600 LIF neurons with soft reset. We set θ = 1 and β = 0.5. Commonly, SNNs performing spike-count based object classification use a large number of total time steps T . A typical range of T can be between 10 and a few thousand [42] . Here, we set the number of total time steps to T = 3. We hypothesize that initializations which enable constant information propagation across depth might also enable inference with low latency, where there is no need to wait for many time steps to accumulate the necessary number of output spikes. The network is trained for 150 epochs using backpropagation through time (BPTT) [16] and the arctan surrogate gradient function [23] . We utilize the Adam optimizer [43] with a learning rate of 1 × 10 -3 and employ cosine annealing scheduling. The runtime for each experiment is approximately 2 hours on a single GPU. Our weight initialization method is compared to Kaiming, as in previous sections. The results illustrated in Figure 4 show that our method generally allows an SNN to converge faster and to achieve better accuracy than Kaiming for different values of network depth and β. We attribute the poor convergence and lower task performance of Kaiming-initialized networks to inadequate activity propagation, as supported by both theoretical insights and empirical findings from previous sections. Specifically, training becomes increasingly challenging with deeper networks, lower values of β (leading to less information retention from the previous time step) and higher values of θ (resulting in fewer neurons emitting spikes) (see Appendix 7: Figure 6 ). 
conclusion and discussion: 
 In this paper, we address the problem of weight initialization in Spiking Neural Networks (SNNs) and show how the techniques developed for ANNs, such as Kaiming initialization, are inadequate for SNNs. We analytically derive and empirically test a novel weight initialization method which takes into account the specific activation function of SNNs. Our weight initialization depends only on the number of input neurons n to a layer and is therefore broadly applicable to all deep, spiking network architectures with fixed connectivity maps. We demonstrate that our proposed initialization is robust against variations in several network and neuron hyperparameters, which can enable deep activity propagation for diverse models and machine learning tasks.A limitation of our proposed initialization is that it does not account for the temporal dynamics of membrane potentials u l . Specifically, after neurons are reset, our assumption that u l is normally distributed around zero is violated. We observe that the extent to which the normal distribution remains a valid approximation depends on the neuron hyperparameters, although our initialization seems more robust than Kaiming and the theory can be expanded to explicitly take into account the temporal variations in u l . Another assumption of our derivation is that the activations x l are mutually independent. This assumption is typically violated in the case of real-world data, such as images. Empirically, in section 5 we illustrate how, for an SNN trained on MNIST, our variance-conserving initialization scheme still translates into accelerated training, improved accuracy and low latency compared to Kaiming. Nevertheless, we acknowledge the necessity of extending this analysis to more complex architectures and datasets, in order to evaluate its effectiveness in various settings. 
appendix: 
 The reset operation violates the assumption that u t l is always normally distributed and symmetrically centered around 0, especially for higher values of β. In Figure 5 we show the values of skewness and excess kurtosis for u t l across layers and time steps in the case of β = 0.9. Skewness measures the degree of asymmetry of the distribution, while excess kurtosis measures the degree of peakedness and flatness of a distribution. A normal distribution has 0 skewness and 0 excess kurtosis. We note how u t l tends to a left-skewed and heavy-tailed distribution. . We compare our proposed initialization method (solid lines) to Kaiming (dashed lines) and find that it achieves better training accuracy and faster convergence. 
 
 Appendix: 38th Second Workshop on Machine Learning with New Compute Paradigms at NeurIPS 2024(MLNCP 2024). 



Reviews: review: title: title: The authors derive a weight initialization method for an SNN based on its activity in one single time step and instantaneous spike propagation. They show on an MNIST task, that their method outperforms Kaiming initialization.
main_review: main_review: The manuscript provides a new weight initialization method for SNNs, but lacks both, a solid motivation on what are the exact gaps they address with their theory to improve over existing SNN initialization methods (which already address methods similar to Kaiminig initialization for the spiking non-linearity) and a simulation-based comparison showing how they improved over existing SNN initialization methods.
Hence, I don't see what the manuscript adds over existing initialization methods for SNNs.

Moreover, there are other points, that should be addressed to improve clarity and relevance of the manuscript.

1. Chapter 3.2: while it looks like a good start to consider initially only the first time step, this in fact reduces the SNN to a binary perceptron, not considering any of the neuron dynamics or the reset. Could you comment on why you chose on that approach and what are the advantage of this approach compared to other taken approaches? Moreover, it would be nice if you make an effort towards extending your theory to multiple time steps or at least discuss the theoretical implications in chapter 4.1 as well.

2. Fig. 1 provides a nice intuition, unfortunately, the captions and labels are not readable.

3. While you comment quickly at the end of chapter 3 on how your initialization compares to Kaiming, it would be nice to see how it compares to other initialization methods that were developed for SNNs, since they were specifically developed for the same spiking activation function.

4. Again, also for the empirical validation in chapter 4, it would be nice to see comparison to other SNN initialization methods, as you wanted to improve over them. 

4. Furthermore, I was wondering, why the threshold at 1 is so important, as this is a rather arbitrarily chosen value for an SNN and could as well be at another value.

5. In chapter 4.1, you wanted to extend your initialization method to multiple time steps, however, there is no theory on it and no discussions on the approximations that would be necessary for such an extension.

6. I would consider 20 time steps still rather short, if you want to check, how time, and hence dynamics and reset influence the network behaviour. While the spike propagation is somewhat convincing for Fig. 3a, they do so at the cost of biological plausibility or reduced dynamics, i.e. the exponential decay of the membrane potential cannot be captured properly, if the membrane time constant is only twice the timestep, as $\beta=0.5$ means. Moreover, for $\beta=0.9$, which I assume the more realistic setting, one can see a quite prominent decay in the number of spikes over time steps. How does this evolve over more time steps, where the full dynamics are captured?

7. From the left plot on Figure 4, it looks like Kaiming init with 5 layers performs best, i.e. better than your init. What is your take on that?
8. Again Fig. 4, did you average these results over multiple initializations with different seeds? It would be nice to see the mean and the standard deviation, which would make the results more reliable.
9. Again Fig. 4. it seems, that for the left plot, you chose $\beta=0.5$, which is good for your theory but worst for Kaiming (see right) and less biologically realistic than $\beta=0.9$. Again, for the right plot, you chose 10 hidden layers, where your init performs well, but Kaiming doesn't (while for 5 layers, it is best). Why did you make these choices? And why do you need so many layers to solve MNIST?
10. Also related MNIST training, is the variance still conserved over layers?
11. Again MNIST training, did you use regularizers?
12. Again MNIST, you say in line 175 that typical number of time steps are between 10 and a few thousands, where I feel, that around 100 it starts to become more biologically realistic. Why did you use only 3 timesteps and not more?

13. Importantly, you are missing a comparison to existing SNN initialization methods.

14. The colors in Fig. 6 don't match the legend.
title: title: Learning SNN
main_review: main_review: SNNs, as a promising alternative to ANNs, have the advantages of low power consumption and sparsity. However, in deep SNNs, due to the nature of binary spikes, the propagation process leads to information loss thus making their training and application challenging. The initialization of network weights is very important for the propagation of information in deep networks, and currently in the field of SNNs, it is common to use the initialization methods of ANNs directly. However, the initialization methods of ANNs often consider the role of continuous activation functions, and the discrete, special activation function with temporal properties used in SNNs make the ANNs initialization methods not sufficient to solve the problem of information vanishing and explosion in SNNs. To solve the above problem, this work proposes a new weight initialization method for SNNs. The proposed method takes into account the nature of using discrete spikes in SNNs, by using the probability of neurons being fired, to regulate the initialized weights. The experimental results show that in the cases of deep SNNs with multiple time steps, the proposed method allows the information to be propagated efficiently without too much loss, whereas the initialization method of ANNs leads to severe information vanishing problem when applied on SNNs.
I think the work is meaningful and the derivation process and experimental results are convincing. However I think the authors should have gone into more detail in the results section. Three are the main points I refer:
1. In the section “Validation with numerical simulations”, please describe in detail the settings of the SNNs used, such as the time step, and calculate the value of the P(u > \theta).
2. In the section “Experiments on MNIST”, the authors mention that T in the conventional case belongs to tens to thousands, but the study uses T=3 and assumes that the proposed method in the conventional case (10<T<1000) still works as well as there is no need to wait for long time steps. Please explain the reason for choosing T=3. How about the performance of conventional cases?
3. In the section “Experiments on MNIST”, the authors mention that the proposed method yields better accuracy and faster convergence rate. Please add tables to summarize the performances, so we can compare them clearly. From the current figure 4, when the number of layers is less than 8, the comparison of convergence speed and accuracy is not clear, please add specific values.
Another point is the theoretical derivation as well as the experimental proofs use the case where the input information is normalized to have a variance of 1 and a mean of 0. Does the derivation process of the method need to be adjusted when the distribution of the inputs is set to something else?
Minor points:
In fig.4 (right), what is the setting of the used model? The labels should be different with fig.4(left), it is a little confusing.
In no.188 line, “training becomes increasingly challenging with deeper networks ”, in the appendix, only results of number of nodes are shown. There is no description of number of layers.
In the appendix, fig.6 (right) has 6 lines, but only 4 labels, and what are their settings?