SPIKEGRAD: AN ANN-EQUIVALENT COMPUTA-
Year: 2020

Abstract:
Event-based neuromorphic systems promise to reduce the energy consumption of deep neural networks by replacing expensive floating point operations on dense matrices by low energy, sparse operations on spike events. While these systems can be trained increasingly well using approximations of the backpropagation algorithm, this usually requires high precision errors and is therefore incompatible with the typical communication infrastructure of neuromorphic circuits. In this work, we analyze how the gradient can be discretized into spike events when training a spiking neural network. To accelerate our simulation, we show that using a special implementation of the integrate-and-fire neuron allows us to describe the accumulated activations and errors of the spiking neural network in terms of an equivalent artificial neural network, allowing us to largely speed up training compared to an explicit simulation of all spike events. This way we are able to demonstrate that even for deep networks, the gradients can be discretized sufficiently well with spikes if the gradient is properly rescaled. This form of spike-based backpropagation enables us to achieve equivalent or better accuracies on the MNIST and CIFAR10 datasets than comparable state-of-the-art spiking neural networks trained with full precision gradients. The algorithm, which we call SpikeGrad, is based on accumulation and comparison operations and can naturally exploit sparsity in the gradient computation, which makes it an interesting choice for a spiking neuromorphic systems with on-chip learning capacities.

heading: 1 INTRODUCTION
text: Spiking neural networks (SNNs) are a new generation of artificial neural network models (Maass, 1997) that try to harness properties of biological neurons to build energy efficient spiking neuromorphic systems. Processing in traditional artificial neural networks (ANNs) is based on parallel processing of operations on dense tensors of fixed length. In contrast to this, spiking neuromorphic systems communicate with asynchronous events, which allows dynamic, data dependent computation that can exploit high temporal and spatial sparsity.
The recent years have seen a large number of approaches devoted to optimization of spiking neural networks with the backpropagation algorithm, either by converting ANNs to SNNs (Diehl et al., 2015; Esser et al., 2016; Rueckauer et al., 2017; Sengupta et al., 2019) or by simulating spikes explicitly in the forward pass and optimizing these dynamics with floating point gradients (Lee et al., 2016; Yin et al., 2017; Wu et al., 2018b;c; Severa et al., 2019; Jin et al., 2018; Bellec et al., 2018; Zenke & Ganguli, 2018; Shrestha & Orchard, 2018). These methods aim to optimize SNNs for efficient inference, and backpropagation is performed offline on a standard computing system. It would however be desirable to also enable on-chip learning in neuromorphic chips using the power of the backpropagation algorithm, and to maintain the advantages of spike-based processing also in the error propagation phase.
Previous work on the implementation of backpropagation with spikes is mostly concerned with biological plausibility. A non-spiking version of biologically inspired backpropagation is presented
by Sacramento et al. (2018). Guerguiev et al. (2017), Neftci et al. (2017) and Samadi et al. (2017) introduce spike-based versions of the backpropagation algorithm using variants of (direct) feedback alignment (Lillicrap et al., 2016; Nøkland, 2016). The exact backpropagation algorithm, which backpropagates through symmetric weights, might however be required to achieve good inference performance on large-scale deep neural networks (Baldi & Sadowski, 2016; Bartunov et al., 2018). O’Connor & Welling (2016) and Thiele et al. (2019) present implementations of standard backpropagation where the gradient is coded into spikes and propagated through symmetric weights. In the same spirit, our work is mostly concerned with exploiting spike based information encoding for energy efficient processing, which means that inference performance and operational simplicity will be preferred over biological plausibility and complex neuron models.
We demonstrate how backpropagation can be seamlessly integrated into the spiking neural network framework by using a second accumulation compartment that discretizes the error into spikes. By additionally weighting the activity counters by the learning rate, we obtain a system that is able to perform learning and inference based on accumulations and comparisons alone. As for the forward pass, this allows us to use the dynamic precision computation provided by the discretization of all operations into spike events, and to exploit the sparsity of the gradient. Using a similar reasoning as Binas et al. (2016) and Wu et al. (2019) have applied to forward propagation in SNNs, we show that the system obtained in this way can be mapped to an integer activation ANN whose activations are equivalent to the accumulated neuron responses for both the forward and the backward propagation phase. This allows us to simulate training of large-scale SNNs efficiently on graphics processing units (GPUs), using their equivalent ANN. Additionally, in contrast to conversion methods that approximate pre-trained ANNs with SNNs, this method guarantees that the inference precision of the SNN will be equivalent to the ANN. In contrast to O’Connor & Welling (2016), this is true for any number of spikes and arbitrary spike order. We demonstrate classification accuracies equivalent or superior to existing implementations of SNNs trained with full precision gradients, and comparable to the precision of standard ANNs using similar topologies. This is the first time competitive classification performances are reported on the CIFAR10 and CIFAR100 datasets using a large-scale SNN where both training and inference are fully implemented with spikes. To the best of our knowledge, our work provides for the first time a demonstration of how the sparsity of the gradient during backpropagation could be exploited within a large-scale SNN processing structure.
2 THE SpikeGrad ALGORITHM
We begin with the description of SpikeGrad, the spike-based backpropagation algorithm. The algorithm mainly consists of special implementations of the integrate-and-fire (IF) neuron model, that are used to encode and propagate information in the forward and backward pass.
For each training example/mini-batch, integration is performed from t = 0 to t = T for the forward pass and from t = T + ∆t to t = T in the backward pass. Since no explicit time is used in the algorithm, ∆t represents symbolically the (very short) time between the arrival of an incoming spike and the response of the neuron, which is only used here to describe causality.
Spike discretization of the forward pass For forward propagation, the architecture is described by multiple layers (labeled by l ∈ [0, L]) of IF neurons with integration variable V li (t) and threshold Θff :
V li (t+ ∆t) = V l i (t)−Θffsli(t) + ∑ j wlijs l−1 j (t), V l i (0) = b l i. (1)
The variable wlij is the weight and b l i a bias value. The spike activation function s l i(t) ∈ {−1, 0, 1} is a function which triggers a signed spike event depending on the internal variables of the neuron. It will be shown later that the specific choice of the activation function is fundamental for the mapping to an equivalent ANN. After a neuron has fired, its integration variable is decremented or incremented by the threshold value Θff , which is represented by the second term on the r.h.s. of equation 1.
As a representation of the neuron activity, we use a trace xli(t) which accumulates spike information over a single example:
xli(t+ ∆t) = x l i(t) + ηs l i(t). (2)
By weighting the activity with the learning rate η we avoid performing a multiplication when weighting the input with the learning rate for the weight update equation 8.
Implementation of implicit ReLU and surrogate activation function derivative It is possible to define an implicit activation function based on how the neuron variables affect the spike activation function sli(t). In our implementation, we use the following fully symmetric function to represent linear activation functions (used for instance in pooling layers):
sl,lini ( V li (t) ) :=  1 if V li (t) ≥ Θff −1 if V li (t) ≤ −Θff 0 otherwise . (3)
The following function corresponds to the rectified linear unit (ReLU) activation function:
sl,ReLUi ( V li (t), x l i(t) ) :=  1 if V li (t) ≥ Θff −1 if V li (t) ≤ −Θff and xli(t) > 0 0 otherwise . (4)
The pseudo-derivative of the activation function is denoted symbolically by S ′l i . We use S ′l,lin i (T ) = 1 for the linear case. For the ReLU, we use a surrogate of the form:
S ′l,ReLU i (T ) :=
{ 1 if V li (T ) > 0 or x l i(T ) > 0
0 otherwise . (5)
These choices will be motivated in the following sections. Note that the derivatives depend only on the final states of the neurons at time T .
Discretization of the error into spikes For gradient backpropagation, we introduce a second compartment with threshold Θbp in each neuron, which integrates error signals from higher layers. The process discretizes errors in the same fashion as the forward pass discretizes an input signal into a sequence of signed spike signals:
U li (t+ ∆t) = U l i (t)−Θbpzli(t) + ∑ k wl+1ki δ l+1 k (t). (6)
To this end, we introduce a ternary error spike activation function zli(t) ∈ {−1, 0, 1} which is defined in analogy to equation 3 using the error integration variable U li (t) and the backpropagation threshold Θbp. The error is then obtained by gating this ternarized variable zli(t) with one of the surrogate activation function derivatives of the previous section (linear or ReLU):
δli(t) = z l i(t)S ′l i (T ). (7)
This ternary spike signal is backpropagated through the weights to the lower layers and also applied in the update rule of the weight increment accumulator ωlij :
ωlij(t+ ∆t) = ω l ij(t)− δli(t)xl−1j (T ), (8)
which is triggered every time an error spike (equation 7) is backpropagated. The weight updates are accumulated during error propagation and are applied after propagation is finished to update each weight simultaneously. In this way, the backpropagation of errors and the weight update will, exactly as forward propagation, only involve additions and comparisons of floating point numbers.
The SpikeGrad algorithm can also be expressed in an event-based formulation, described in algorithms 1, 2 and 3. This formulation is closer to how the algorithm would be implemented in an actual SNN hardware implementation of the IF firing dynamics.
Loss function and error scale We use the cross entropy loss function in the final layer applied to the softmax of the total integration V Li (T ) (no spikes are triggered in the top layer during inference). This requires more complex operations than accumulations, but is negligible if the number of classes is small. To make sure that sufficient error spikes are triggered in the top layer, and that error spikes arrive even in the lowest layer of the network, we apply a scaling factor α to the error values before transferring them to ULi . This scaling factor also implicitly sets the precision of the gradient, since a higher number of spikes means that a large range of values can be represented. To counteract the relative increase of the gradient scale, the learning rates have to be rescaled by a factor 1/α.
Input encoding As pointed out in Rueckauer et al. (2017) and Wu et al. (2018c), it is crucial to maintain the full precision of the input image to obtain good performances on complex standard benchmarks with SNNs. One possibility is to encode the input in a large number of spikes (Sengupta et al., 2019). Another possibility, which has been shown to require a much lower number of spikes in the network, is to multiply the input values directly with the weights of the first layer (just like in a standard ANN). The drawback is that the first layer then requires multiplication operations. The additional cost of this procedure may however be negligible if all other layers can profit from spike-based computation. This problematic does not exist for stimuli which are natively encoded in spikes.
Algorithm 1 Forward function PROPAGATE([l, i, j], s)
V li ← V li + s · wlij sli ← sli(V li , xli) . spike activation function if sli 6= 0 then
V li ← V li − sli ·Θff xli ← xli + ηsli for k in l + 1 connected to i do
PROPAGATE([l + 1, k, i], sli)
Algorithm 2 Backward function BACKPROPAGATE([l, i, k], δ)
U li ← U li + δ · w l+1 ki zli ← zli(U li ) . error activation function δli ← zli · S ′l i if zli 6= 0 then U li ← U li − zli ·Θbp for j in layer l − 1 connected to i do
BACKPROPAGATE([l − 1, j, i], δli) ωlij ← ωlij − δli · x l−1 j
Algorithm 3 Training of single example/batch init: V ← b, U ← 0, x← 0, ω ← 0 . variables in bold describe all neurons in network/layer while input spikes sini do
for k in l = 0 receiving sini do . spikes corresponding to training input PROPAGATE([0, k, i], sini ) S′ ← S′(V ,x) . calculate surrogate derivatives UL ← α · ∂L/∂ V L . calculate classfication error for i in l = L do
while |ULi | ≥ Θbp do . backpropagate error spikes BACKPROPAGATE([L, i,−], 0) . last layer receives no error
w ← w + ω . update weights with weight update accumulator

heading: 3 FORMULATION OF THE EQUIVALENT ANN
text: The simulation of the temporal dynamics of spikes in SpikeGrad requires a large number of time steps or events if activation or error values are large. It would therefore be extremely beneficial if we were able to map the SNN to an equivalent ANN that can be trained much faster on standard hardware. In this section, we demonstrate that it is possible to find such an ANN using the forward and backward propagation dynamics of the SpikeGrad IF neuron model described in the previous section.
Spike discretization error We start our analysis with equation 1. We reorder the terms and sum over the increments ∆V li (t) = V l i (t+ ∆t)− V li (t) every time the integration variable is changed either by a spike that arrives at time tsj ∈ [0, T ] via connection j, or by a spike that is triggered at time tsi ∈ [0, T ]. With the initial conditions V li (0) = bli, sli(0) = 0, we obtain the final value V li (T ):
V li (T ) = ∑ tsj ,t s i ∆V li = −Θff ∑ tsi sli(t s i ) + ∑ j wlij ∑ tsj sl−1j (t s j) + b l i (9)
By defining the total transmitted output of a neuron as Sli := ∑ tsi sli(t s i ) we obtain:
1
Θff V li (T ) = Sli − Sli, Sli :=
1
Θff ∑ j wlijS l−1 j + b l i  (10)
The same reasoning can be applied to backpropagation of the gradient. We define the summed responses over error spikes times τsj ∈ [T + ∆t, T ] as Zli := ∑ τsi zli(τ s i ) to obtain:
1
Θbp U li (T ) = Zli − Zli , Zli :=
1
Θbp (∑ k wl+1ki E l+1 k ) (11)
El+1k = ∑ τsk δl+1k (τ s k) = ∑ τsk S ′l+1 k (T )z l+1 k (τ s k) = S ′l+1 k (T )Z l+1 k . (12)
In both equation 10 and equation 11, the terms Sli and Zli are equivalent to the output of an ANN with signed integer inputs Sl−1j and E l+1 k . 1/Θff and 1/Θbp are implicit scaling factors of activations and gradients. If gradients shall not be explicitly rescaled, backpropagation requires Θbp = Θff . The values of the residual integrations 1/ΘffV li (T ) and 1/ΘbpU l i (T ) therefore represent the spike discretization error SDEff := Sli − Sli or SDEbp := Zli − Zli between the ANN outputs Sli and Zli and the accumulated SNN outputs Sli and Zli . Since we know that V li (T ) ∈ (−Θff ,Θff) and U li (T ) ∈ (−Θbp,Θbp), this gives bounds of |SDEff | < 1 and |SDEbp| < 1. So far we can only represent linear functions. We now consider an implementation where the ANN applies a ReLU activation function instead. The SDE in this case is:
SDEReLUff := ReLU ( Sli ) − Sli. (13)
We can calculate the error by considering that equation 4 forces the neuron in one of two regimes (note that xli > 0 ⇔ Sli > 0): In one case, Sli = 0, V li (T ) < Θff (this includes V li (T ) ≤ −Θff ). This implies Sli = 1/ΘffV li (T ) and therefore |SDE ReLU ff | < 1 (or even |SDE ReLU ff | = 0 if V li (T ) ≤ 0). In the other case, Sli > 0, V l i (t) ∈ (−Θff ,Θff), where equation 4 is equivalent to equation 3.
This equivalence motivates the choice of equation 5 as a surrogate derivative for the SNN: the condition (V li (T ) > 0 or x l i(T ) > 0) can be seen to be equivalent to Sli(T ) > 0, which defines the derivative of a ReLU. Finally, for the total weight increment ∆wlij , it can be seen from equation 2 and equation 8 that:
xli(T ) = ∑ tsi ∆xli(t s i ) = ηS l i, ⇒ ∆wlij(T ) = ∑ τsi ∆ωlij(τ s i ) = −ηSl−1j E l i, (14)
which is exactly the weight update formula of an ANN defined on the accumulated variables. We have therefore demonstrated that the SNN can be represented by an ANN by replacing all S and Z by S and Z and applying the corresponding activation function directly on these variables. The error that will be caused by this substitution compared to using the accumulated variables S and Z of an SNN is described by the SDE. This ANN can now be used for training of the SNN on GPUs. The SpikeGrad algorithm formulated on the variables s, z, δ and x represents the algorithm that would be implemented on a dedicated, event-based spiking neural network hardware implementation of the IF neurons. We will now demonstrate how the SDE can be further reduced to obtain an ANN and SNN that are exactly equivalent.
Response equivalence For a large number of spikes, the SDE may be negligible compared to the activation of the ANN. However, in a framework whose objective it is to minimize the number of spikes emitted by each neuron, this error can have a potentially large impact.
One option to reduce the error between the ANN and the SNN output is to constrain the ANN during training to integer values. One possibility is to round the ANN outputs:
Sl,roundi := round[S l i] = round  1 Θff ∑ j wlijS l−1 j + b l i  , (15) The round function here rounds to the next integer value, with boundary cases rounded away from zero. This behavior can be implemented in the SNN by a modified spike activation function which is applied after the full stimulus has been propagated. To obtain the exact response as the ANN, we have to take into account the current value of Sli and modify the threshold values:
sl,resi ( V li (T ), S l i ) :=  1 if V li (T ) > Θff/2 or (S l i ≥ 0, V li (T ) = Θff/2)
−1 if V li (T ) < −Θff/2 or (Sli ≤ 0, V li (T ) = −Θff/2) 0 otherwise . (16)
Because this spike activation function is applied only to the residual values, we call it the residual spike activation function. The function is applied to a layer after all spikes have been propagated with the standard spike activation function (equation 3 or equation 4). We start with the lowest layer and propagate all residual spikes to the higher layers, which use the standard activation function. We then proceed with setting the next layer to residual mode and propagate the residual spikes. This is continued until we arrive at the last layer of the network.
By considering all possible rounding scenarios, it can be seen that equation 16 indeed implies:
Sli + s l,res i ( V li (T ), S l i ) = round[Sli + 1/ΘffV l i (T )] = round[Sli]. (17)
The same principle can be applied to obtain integer-rounded error propagation:
Zl,roundi := round [ Zli ] = round
[ 1
Θbp (∑ k wl+1ki E l+1 k )] . (18)
We have to apply the following modified spike activation function in the SNN after the full error has been propagated by the standard error spike activation function:
zl,resi ( U li (T , Zli ) :=  1 if U li (T )) > Θbp/2 or (Zli ≥ 0, U li (T ) = Θbp/2) −1 if U li (T ) < −Θbp/2 or (Zli ≤ 0, U li (T ) = −Θbp/2) 0 otherwise , (19)
which implies:
Zli + z l,res i ( U li (T ), Zli ) = round[Zli + 1/ΘbpU l i (T )] = round[Zli]. (20)
We have therefore shown that the SNN will after each propagation phase have exactly the same accumulated responses as the corresponding integer activation ANN. The same principle can be applied to obtain other forms of rounding (e.g. floor and ceil), if equation 16 and equation 19 are modified accordingly.
Computational complexity estimation Note that we have only demonstrated the equivalence of the accumulated neuron responses. However, for each of the response values, there is a large number of possible combinations of 1 and −1 values that lead to the same response. The computational complexity of the event-based algorithm depends therefore on the total number n of these events. The best possible case is when the accumulated response value Sli is represented by exactly |Sli| spikes. In the worst case, a large number of additional redundant spikes is emitted which sum up to 0. The maximal number of spikes in each layer is bounded by the largest possible integration value that can be obtained. This depends on the maximal absolute weight value wlmax, the number of connections N lin and the number of spike events n
l−1 each connection receives, which is given by the maximal value of the previous layer (or the input in the first layer):
nlmin = |Sli|, nlmax = ⌊ 1
Θff N linw l maxn l−1 max
⌋ . (21)
The same reasoning applies to backpropagation. Our experiments show that for input encodings where the input is provided in a continuous fashion, and weight values that are much smaller than the threshold value, the deviation from the best case scenario is rather small. This is because in this case the sub-threshold integration allows to average out the fluctuations in the signal. This way the firing rate stays rather close to its long term average and few redundant spikes are emitted. For the total number of spikes n in the full network on the CIFAR10 test set, we obtain empirically n−nmin/nmin < 0.035.

heading: 4 EXPERIMENTS
text: Classification performance Tables 1 and 2 compare the state-of-the-art results for SNNs on the MNIST and CIFAR10 datasets. It can be seen that in both cases, our results are competitive with respect to the state-of-the-art results of other SNNs trained with high precision gradients. Compared to results using the same topology, our algorithm performs at least equivalently.
The final classification performance of the network as a function of the error scaling term α in the final layer can be seen in figure 1. Previous work on low bitwidth gradients (Zhou et al., 2018) found that gradients usually require a higher precision than both weights and activations. Our results also indicate that a certain minimum number of error spikes is necessary to achieve convergence. This strongly depends on the depth of the network and if enough spikes are triggered to provide sufficient gradient signal in the bottom layers. For the CIFAR10 network, convergence becomes unstable for approximately α < 300. If the number of operations is large enough for convergence, the required precision for the gradient does not seem to be extremely high. On the MNIST task, the difference in test performance between a gradient rescaled by a factor of 50 and a gradient rescaled by a factor of 100 becomes insignificant. In the CIFAR10 task, this is true for a rescaling by 400 or 500. Also the results obtained with the float precision gradients in tables 1 and 2 demonstrate the same performance, given the range of the error.
To investigate the performance of SpikeGrad on databases with a larger number of classes, we also ran an additional experiment on the CIFAR100 dataset. Using the exactly same architecture as for CIFAR10 with α = 500 (besides the final layer that is increased to 100 classes), we obtain a maximal classification score of 64.40%. Running the same experiment with only the forward pass encoded in spikes, but floating point gradients, we obtain 64.69%. This result could probably be improved by adapting the architecture better to the dataset. However, it demonstrates that coding the gradient into spikes does not lead to a large precision loss even in this scenario.
Sparsity in backpropagated gradient To evaluate the potential efficiency of the spike coding scheme relative to an ANN, we use the metric of relative synaptic operations. A synaptic operation corresponds to a multiply-accumulate (MAC) in the case of an ANN, and a simple accumulation (ACC) in the case of an SNN. This metric allows us to compare networks based on their fundamental operation. The advantage of this metric is the fact that it does not depend on the exact implementation of the operations (for instance the number of bits used to represent each number). Since an ACC is however generally cheaper and easier to implement than a MAC, we can be sure that an SNN is more efficient in terms of its operations than the corresponding ANN if the number of ACCs is smaller than the number of MACs (without considering potential hardware overheads of event-based computing).
Numbers were obtained with the integer activations of the equivalent ANN to keep simulation times tractable. As previously explained, the integer response of the equivalent ANN represents the best case scenario for the SNN, i.e. the activation encoding with the lowest number of spikes. The actual
number of events and synaptic operations in an SNN may therefore slightly deviate from these numbers.
In figure 1, it can be seen that the number of operations decreases with increasing inference precision of the network. This is a result of the decrease of error in the classification layer, which leads to the emission of a smaller number of error spikes. Figure 2 demonstrates how the number of operations during the backpropagation phase is distributed in the layers of the network (the float input bottom layer and average pooling layers were omitted). While propagating deeper down the network, the relative number of operations decreases and the error becomes increasingly sparse. This tendency is consistent during the whole training process for different epochs.

heading: 5 DISCUSSION AND CONCLUSION
text: Using spike-based propagation of the gradient, we demonstrated that the paradigm of event-based information propagation can be easily translated to the backpropagation algorithm. We have not only shown that competitive inference performance can be achieved, but also that gradient propagation seems particularly suitable to leverage spike-based processing by exploiting high signal sparsity. For both forward and backward propagation, SpikeGrad requires a similar event-based communication infrastructure between neurons, which simplifies a possible spiking hardware implementation. One restriction of our algorithm is the need for negative spikes, which could be problematic for some neuromorphic hardware platforms.
In particular the topology used for CIFAR10 classification is rather large for the given task. We decided to use the same topologies as other state-of-the-art SNNs to allow for better comparison. The relatively large number of parameters of these topologies may to a certain extent explain the very low number of relative synaptic operations we observe during backpropagation. It has been demonstrated that the number of parameters in many ANNs can be strongly reduced without a significant impact on performance (Han et al., 2016). It remains to show that the spike coding approach provides sufficient gains also in topologies that are optimized for minimal computation and memory requirements. This seems at least possible, since the reduction of operation count in SNNs is mainly caused by two factors: the first factor is actual sparsity, that means the number of zeros that are produced by thresholding and the ReLU activation function. The second factor is the more fine grained distribution of activation and error values. If most values are represented by small integers, only a few spikes are necessary to encode them. The consequence is that even topologies that are compressed and therefore have lower sparsity may be able to profit from spike coding if most of the activation or error values are small. This is typically the case for networks quantized to low precision integer values (see for instance (Zhou et al., 2018) or Wu et al. (2018a)) and could lead to interesting synergies between our spike-based computation model and ANN quantization methods.
Our analysis of the potential energy efficiency of SpikeGrad in a dedicated SNN architecture is based on counting the number of basic operations that are performed at the synapses of a neuron. This does not take into account the potential hardware overhead that is introduced by the implementation of the IF neuron model and the event-based processing scheme. If the low operation count can in the end be translated into higher processing speed or increased energy efficiency, compared to a normal ANN, depends on the particular SNN hardware implementation. Such dedicated neuromorphic hardware is the subject of current research efforts (see for instance Merolla et al. (2014) and Davies et al. (2018)). We assumed in this work that such a system might be built and used operation count as our principal metric, since it enables us to easily approximate the energy consumption of a spiking neuromorphic chip if the average cost of such an operation is known. Our work shows that a hardware implementation of SpikeGrad might profit sufficiently from the low operation count that energy or speed improvements could be possible at least in certain cases.

heading: A APPENDIX

heading: B ADDITIONAL EXPLANATIONS

heading: B.1 ROUND FUNCTION
text: It can be seen that the residual spike activation function covers indeed all possible cases. In particular, the boundary cases are rounded correctly: for Vi = Θff , we obtain S = Si + 0.5. For Si ≥ 0, this should be rounded to round[S] = Si + 1 and for Si < 0, we should obtain round[S] = Si. Similarly,
for Vi = −Θff , we obtain S = Si − 0.5. For Si ≤ 0, this should be rounded to round[S] = Si − 1 and for Si > 0, we should obtain round[S] = Si.
The same reasoning applies to error propagation.

heading: C ADDITIONAL ARCHITECTURE DETAILS

heading: C.1 AVERAGE POOLING
text: All pooling layers are average pooling layers, since these are much easier to implement in a spiking neural network. Average pooling can simply be implemented as an IF neuron with constant weights:
wlij = 1
pli , (22)
where pli is the number of neurons in the pooling window of the neuron.

heading: C.2 DROPOUT
text: Dropout can be implemented by an additional gating variable which randomly removes some neurons during forward and backward propagation. During learning, the activations of the other neurons have to be rescaled by the inverse of the dropout probability 1/(1− pdrop). This can be implemented in the SNN framework by rescaling the threshold values by (1− pdrop).

heading: C.3 MOMENTUM
text: While momentum is not required for the basic algorithm, we used it to speed up training. In a SNN implementation, this could be implemented by adding additional accumulators that save past gradient information.

heading: C.4 PARAMETER PRECISION
text: All real valued variables are coded with 32 bit floating point variables.

heading: D EXPERIMENTS

heading: D.1 COMPUTING FRAMEWORK
text: All experiments are performed with custom CUDA/cuDNN accelerated C++ code. Training is performed on RTX 2080 Ti graphic cards.

heading: D.2 ERROR BARS
text: For all experiments, the means, errors and maximal values are calculated over 20 simulation runs.

heading: D.3 PREPROCESSING
text: No preprocessing is used on the MNIST dataset. We separate the training set of size 60000 into 50000 training and 10000 validation examples, which are used to monitor convergence. Testing is performed on the test set of 10000 examples.
For CIFAR10, the values of all color channels are divided by 255 and then rescaled by a factor of 20 to trigger sufficient activation in the network. The usual preprocessing and data augmentation is applied. For data augmentation images are padded with the image mean value by two pixels on each side and random slices of 32× 32 are extracted. Additionally, images are flipped randomly along the vertical axis. We separate the training set of size 50000 into 40000 training and 10000 validation examples, which are used to monitor convergence. Testing is performed on the test set of 10000 examples.
Final scores were obtained without retraining on the validation set.

heading: D.4 HYPERPARAMETERS
text: The hyperparameters for training can be seen in tables 3 and 4.
The maximal inference performances in the results were achieved with α = 100 for MNIST and α = 400 for CIFAR10.

Review summaries:
This paper shows how SNNs can integrate backpropagation with a second accumulation module that discretizes errors into spikes. In other words, the authors show how to translate event-based information propagation (used by SNNs) into a backpropagation method, which is the main contribution. The description of establishing the equivalence between an ANN and SNN in Section 3 is mostly well done. They perform empirical studies on MNIST and CIFAR-10 to demonstrate the effectiveness of SpikeGrad.