Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck
Authors: Nicolas Skatchkovsky
Osvaldo Simeone
Hyeryung Jang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Spiking Neural Networks
Variational Learning
Autoencoders

Abstract: One of the key challenges in training Spiking Neural Networks (SNNs) is that target outputs typically come in the form of natural signals, such as labels for classification or images for generative models, and need to be encoded into spikes. This is done by handcrafting target spiking signals, which in turn implicitly fixes the mechanisms used to decode spikes into natural signals, e.g., rate decoding. The arbitrary choice of target signals and decoding rule generally impairs the capacity of the SNN to encode and process information in the timing of spikes. To address this problem, this work introduces a hybrid variational autoencoder architecture, consisting of an encoding SNN and a decoding Artificial Neural Network (ANN). The role of the decoding ANN is to learn how to best convert the spiking signals output by the SNN into the target natural signal. A novel end-to-end learning rule is introduced that optimizes a directed information bottleneck training criterion via surrogate gradients. We demonstrate the applicability of the technique in an experimental settings on various tasks, including real-life datasets. 

Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck
Nicolas Skatchkovsky KCLIP lab, Dept. of Eng. King's s College London London, UK
Osvaldo Simcone
Hyeryung Jang ION group, Dept. of A.I. Dongguk University
KCLIP lab, Dept. of Eng. King's College London London, UK
Scoul, Korea
nicolas. www.tccchweeewkkkkcllccccco uc.
osvaldo. aimeonefkcl.cc..
hyeryung jangedgu. ac..k
Abstract
One of the key challenges in training Spiking Neural Networks (SNNs) is that target outputs typically come in the form of natural signals, such as labels for classification or images for generative models, and need to be encoded into spikes. This is done by handcrafting target spiking signals, which in turn implicitly fixes the mechanisms used to decode spikes into natural signals, e.g., rate decoding. The arbitrary choice of target signals and decoding rule generally impairs the capacity of the SNN to encode and process information in the timing of spikes. To address this problem, this work introduces hybrid variational autoencoder architecture, consisting of an encoding SNN and decoding Artificial Neural Network (ANN). The role of the decoding ANN is to learn how to best convert the spiking signals output by the SNN into the target natural signal. A novel end-to-end learning rulc is introduced that optimizes directed information bottleneck training criterion via surrogate gradients. We demonstrate the applicability of the technique in an experimental settings on various tasks, including real-life datasets.
1 Introduction
While traditional Artificial Neural Networks (ANNs) implement static rate-based neurons, Spiking Neural Networks (SNNs) incorporate dynamic spike-based neuronal models that are closer to their biological counterparts. In different communities. SNNs are used either as models for the operation of biological brains implemented on dedicated hardware, hinges on the high capacity, low latency, and high noise tolerance of information encoding in the timing of spikes [1]. Recent results leveraging novel prototype chips confirm the potential order-of -frer-it-a gains in terms of latency- and for several optimization and inference problems [2]. In contrast, the landscape of training algorithms for SNNs still quite fractured [2), and a key open question is how to derive learning rules that can optimally leverage the time encoding capabilities of spiking neurons. This is the focus of this work.
efficient solutions for cognitive tasks. The efficiency of SNNs, when
metrics
Learning how decode. Consider the problem training an SNN - nerwork spiking neurons. To fix the ideas, as illustrated in Fig. ((a)-bb), say that we wish to train an SNN to classify inputs from neuromorphic sensor, such as a DVS camera [5], which are the form
collection spiking signals. We hence have spiking signal as input and natural signal (the label) outpul. As another example in Fig. 1(c), we may wish to "naturalize" the output of a DVS camera to produce the corresponding natural image [6]. The direct training of SNN hinges the definition of loss function operating on the output of the SNN. This is typically done by converting the target natural signals into spiking signals that serve the desired output of the SNN. As seen in Fig. (aa--b)), one could for instance assign a large-rate spiking signal or an earlier spike to the readout neuron representing the correct class. The loss function then measures the discrepancy between actual output spiking signals and the corresponding spiking targets.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

a.
c.
d.
8
b.
T
etTor
eye
wu + B Rie
natural target signal
0.75
3,62
0.99
eys
073
his
Figure 1: (a) Illustration of an SNN trained for image classification with first-to-spike decoding. (b) Illustration of an SNN trained for image classification with rate decoding. (c) Illustration of an SG scheme [3]: At each time-step, updates for neuron i are computed using local eligibility traces e and per-neuron learning signals Lip, obtained from errors computed based handcrafted target signals (gray). (d) Illustration of MI. learning for SRM with stochastic threshold [4]: Updates for readout neurons (green) are obtained with local eligibility traces based on handcrafted target spiking signal (gray), while updates for hidden neurons (red) make use of a common learning signal & based on the same target signal. (e) Illustration of the proposed hybrid variational autoencoder: An encoding SNN processes inputs from spiking image source, such as DVS camera, to produce spiking latent representation. decoding ANN learns decode this representation with the aim of reconstructing natural target signal, such as an image, corresponding to the spiking input of the SNN. SNN and ANN are jointly trained under directed information bottleneck criterion. Accordingly, training the ANN is based on backpropagation, while training of the SNN uses per-neuron learning signals and common feedback from both networks.
The approach outlined in the previous paragraph has important problem: By selecting handcrafted target spiking signals, one is effectively choosing a priori the type of information encoding into spikes that the SNN is allowed produce. This constrains the capacity of the SNN fully leverage the time encoding capabilities of spiking signals. For instance, the target signals outlined in Fig. ((a) correspond assuming first-to-spike decoding, while those in Fig. ((b) rate decoding [7]. This paper proposes a method that allows the SNN to automatically learn how to best encode the input into spiking signals, without constraining priori the decoding strategy.
To this end, we introduce the hybrid SNN-ANN autoencoder architecture shown in Fig. 1(e) that consists of an encoding SNN to be trained alongside decoding ANN. The role of the decoding ANN is to learn how to best convert the spiking signals output by the SNN into the target natural signal. Encoding SNN and decoding ANN are trained end-to-end informative spike-based representations that can be efficiently decoded into the desired natural signals.
to ensure that the SNN produces
Directed information bottleneck. How to jointly train the hybrid SNN-ANN autoencoder? On the one hand, we wish the spiking output of the SNN to be informative about the target natural signal - which we will refer to as reference signal and, on the other, we would like the spiking output of the SNN to remove extraneous information present in spiking input that not informative about the reference signal. To capture this dual requirement, formulation [8], and view the ANN as the inference network in a variational autoencoder system [9].
adopt information bottleneck (IB)
In applying the variational IB methodology to the hybrid SNN-ANN autoencoder in Fig. 1(c), we make two key contributions. 1) Given the inherently dynamic operation of the SNN, we generalize the
2

---

IB criterion to apply to directed information metrics [10), in lieu of conventional mutual information. We refer to the resulting approach as variational directed IB (VDIB). 2) In order to enable the metries, such the IB, we introduce an SNN model that
optimization of information-theoree encompasses deterministic hidden neurons and probabilistic readout neurons. In so doing, we combine two strands of research on training SNNs: The first line focuses purely on deterministic models, viewing an SNN as a Recurrent Neural Network (RNN) and deriving approximations of petpprppggaiionttrouug---eeeeeee
(BPTT) through surrogate gradient (SG) techniques [3, 11 15]; while the second assumes probabilistic neurons only, deriving approximations of maximum likelihood (ML) learning. Additional discussion on related work can be found in the next section.
2 Related works
Training SNNs. In recent years, SG techniques for training SNNs have gained significant momen- tum, becoming the state-of-the-art and de-facto eclipsing other techniques. As reviewed in [12], these techniques rely on surrogate derivatives to replace the ill-defined derivative the step activation function of spiking neurons. The SGs are combined with truncated and approximated versions of BPTT, including the use of per-layer local error signals [13] and per-ncuron learning signals defined using direct feedback alignment [3, 15]. Learning rules for probabilistic SNNs typically derived by tackling the ML problem, and result in updates that rely on random sampling and common rather than per-neuron learning signals. The approach was shown to generalize to networks of winner-take-all neurons [16]. As illustrated in Fig. I, the proposed approach builds on both models to enable the use of SG methods for the optimization of probabilistic learning criterion.
Hybrid SNN-ANN architectures. A hybrid SNN-ANN architecture has been recently proposed in [17] for gesture similarity analysis, based on the variational autoencoder (VAE) framework. Although yielding a similar objective, the encoder in [17] is not fully spiking since the ANN receives the membrane potentials of the spiking ncurons in the readout layer of the SNN. The model proposed in our work circumvents this limitation.
Information bottleneck. The IB technique originated two decades ago [8] in the context of a rate-distortion problem formation assuming discrete variables. The approach has regained interest recent years, first as a means to explain the performance of ANNs [18], and then as a training criterion in [9]. We extend the IB by considering temporal signals under causality constraints, hereby replacing the mutual information with directed information [10]. The IB principle has been previously used
in the SNN literature in the case of a single neuron [19, 20]. Finally, it has been used in thcoretical ncuroscience to explain processing in sensory neurons [21] and unifying principle for the predictive and efficient coding theories. Our model encompasses the framework introduced in [21].
Problem definition
Notations. Consider three jointly distributed random vectors x, y. and with distribution p(x, y, 2). The conditional mutual information (MI) between æ and y given z is defined as
[x; g =)) Ep(x,,.)) log p( p(=())p(y))) where Ep[:] represents the expectation with respect
p(a.y|z)
to the distribution The Kullback-Leiller (KL) divergence between two probability distributions [(ar)
p(z) and ((x) defined on the same probability space is KL(p(z) (x)) = Ep(=) log
((())
For any two jointly distributed random vectors a (a) (a....., ar) and b (bi,.... . br) from time to T, the 7-causally conditioned distribution of a given is defined as
(a I I b(,)),
where we denote as a:; = (at,+1, a:22) the overall random vector from time 4 + to 12, and write a' = af: This distribution (I) captures the causal dependence of sequence a on the last r samples of b. I T, we recover the causally conditioned distribution [10].
We also define the d-directed information with T 2 as
1 + b)= Σ (oll,;bblb'b
(2)
3

---

This metric quantifies the causal statistical dependence of the last samples from sequence to b. With T = T, this recovers the standard directed information [10]. We denote as the convolution operator fe 9:- E820 fagt- 5.
Problem definition. We study the hybrid SNN-ANN autocncoder architecture in Fig. with the aim of training an SNN encoder to produce time-cncoded spiking representation y of the spiking input æ that is informative about a natural reference signal r. Throughout, a spiking signal is a binary vector sequence, with "1" representing spike, and natural signal is real-valued vector sequence. In the prototypical application in Fig. I(e), the input x is obtained from a neuromorphic sensor, such as DVS camera [5], and the reference signal represents the corresponding target natural signal, such as a label or the original natural image. The spiking representation y produced by the SNN is fed an ANN, whose role is to decode y into the reference signal r. Importantly, we do not priori constrain the decoding strategy. In contrast, existing works typically handcraft target spiking signals y corresponding some fixed decoding rule, such as rate decoding (Fig. ((a)- Our main contribution is a method to train encoding SNN and decoding ANN jointly, with the goal of discovering automatically efficient time-decoding rules from spiking signals produced by the encoding SNN.
To optimize SNN encoder and ANN decoder, we consider data set comprising N pairs (æ, r) of exogeneous spiking inpuls a and natural reference signals generated from an unknown population distribution p(xx,7 r). The exogeneous inputs are in the form of arbitrary time vector sequence a (#1...., TT) with I; E {0, 1] while the reference signals r = (ri..... TT), with r. E RNA, define general target signals. In the image naturalization task illustrated in Fig. 1, the exogenous inputs z are the spiking signals encoding the input from a DVS camera, with Nx being the number of pixels; and the reference signals r encode the pixel intensities of the natural image. This can be done for instant by setting r,- ONR for = 1, .,... T 1, and rT equal to the grayscale image of NR pixels; or setting r; to equal the grayscale image of all I,...,T.
We wish to train the encoding SNN to output spiking representation y that is maximally informative about the natural target signal r. As we will detail below, the SNN implements a causal stochastic mapping between latent signal y and input æ that depends on a vector of parameters w as
T
(Te) Pw
[)] = II ply yf a
(3)
1=1
where the integer Ta represents the memory of the encoding SNN and w" the synaptic weights of the SNN. As starting point, consider maximizing the Ta-directed information between and as
max W
((Ta) (y r):
(4)
for decoding window Ta 2 O, over the vector 2f of the encoding SNN parameters. The use of the Ta-directed information, in lieu of the mutual information (yyr), reflects the practical requirement that reference signal r, be well represented by limited window yf of past outputs of the SNN in
causal manner. The directed information f(Tr) (y r) is evaluated with respect to the marginal p ply, r) of the joint distribution
Per (x. y. r) p(x, r W
(Te) (y]l=z)
(5)
combining population distribution and SNN mapping via the chain rule of probability.
The maximization (4) does not impose any constraint on the complexity of the spiking representation y. To address this problem, we adopt the directed information bottleneck (DIB) objective
RDIE ( ((5a) (y + r) - ((Te) (x+ y).
(6)
where 3 >> 0 a hyperparameter. The penalty term ((-=) (x + y) the directed information between the input x and the spiking representation y, which captures the complexity of the causal encoding
done by the SNN. This term is evaluated with respect to the marginal p(æ, y) = p(x) (T( (y/|()) of the joint distribution (5), and it reflects the limited memory of the SNN encoding mapping (3). It can be interpreted as the amount of information about input æ that is (causally) preserved by the representation y. To the best of our knowledge, the DIB objective is considered in this paper for the first time.

---

4 Learning to decode through the IB
In order to address the DIB problem (6), adopt variational formulation that relies the introduction of a decoding network, as in the architecture illustrated in Fig. (d). The decoding network implements the causally conditional distribution
aa (rlly)
= II Two (Falr yrr t=1
between spiking representation y and natural reference signal т. The decoder (7) is causal, with memory given by integer Ta and is parametrized by vector w The decoder can be implemented in several ways. A simple approach, that we follow in the remainder of this paper, is to use a model Iw
y Ta) = I (rr =.), where Iwd (relyf +2) is an ANN with inputs given by a window yru of Td samples from the spiking representation y. Alternatively, one could use an RNN to directly model the kernel awd (r/|rr=1,, y Note that the introduction the decoder network (7) is consistent with the use of the ra-directed information in (6). We emphasize that the use of an ANN, or RNN, for decoding, is essential in order to allow the reference sequence r to be a natural signal.
With such network, using a standard variational inequality [9], we bound the two terms (6) obtain the variational DIB (VDIB) loss LVDIB (w", w upper bound on the negative DIB objective (6)
LVDIB (w, wn)) Ex (uir)
log o (rlle)
+B Ep(ce)
(L (y|( (
(8)
information theoretic regularization
average decoder's 's log-loss
in which we have introduced an arbitrary "prior" distribution ((v) II ( y on the spiking representation. As in the implementation in [9] for the standard IB criterion, we will consider q(y) to be fixed, although it can potentially optimized on. To enforce sparsity of the encoder's outputs, the "prior" q(y) can be chosen as a sparse Bernoulli distribution [4], i.e., ((y) = II Bern for some small probability p. The derivation of the bound can be found in Appendix A.I.
The learning criterion (8) enables the joint training of encoding SNN and decoding network via the problem
min LVDIB (w"" w wo" wdd
(9)
which we address via SGD. To this end, we now give general expressions for the gradicnts the VDIB criterion (8) with respect to the encoder and decoder weights by leveraging the REINFORCE Monte Carlo gradients. In the next section, we will then detail how the gradient with respect to the encoding SNN weights wf be approximated by integrating SG [3] with the probabilistic approach [4]. To start, the VDIB loss in (8) can equivalently be stated as
Pw (y
LVDI (w", w² = E p(a,r) r)
E (xe) Purt
log Aund (rlly) + log
(10)
(g)l(z)
((V)
pepulation dienisation
SNN
Raw (2)r)
Ews (M.ae)
where we have defined the decoder loss ( r) and the encoder loss eape (y w) as shown in (10). The gradient with respect the encoder weights wt can be obtained via the REINFORCE gradient technique [22, Ch. 8] [23] as
V Vw LVDII(WW w - E (wl)))))))
((w) ( + 3 k (g., Vw- Vw-
log (Ta) (y x
while the gradient with respect to the decoder weights d can be directly computed as
un LVDII (w" wn) Ep(z.r) ) (re) (ulla) Vane Vonelaneeys
(12)
5

---

Unbiased estimates of the gradients (11)-(12) can be evaluated by using one sample (æ,r) r) p(x, r)
from the data set, along with a randomly generated SNN output y ~ plue (y]|æz) We obtain
V w LVDIB (w², ww)) (End r) 3 lav (w) -ww log pi (y]|
(13)
u LVIIB ( (w" w= 2 V (w r).
(14)
While the gradient V wellud w (y, r) can be computed using standard backpropagation on the decoding ANN, the gradient V V log
( (y/|x) for the encoder weights depends on the SNN model, and is
detailed in the next section.
As we will describe in the next section, and as illustrated in Fig. (c),, the learning rule for the SNN involves the common learning signal (B, + B L (y, æ)) which includes two feedback signals, one from the ANN decoder - the loss (1,, r) and one from the SNN encoder, namely (y, The latter imposes a penalty for the distribution of the represenlation y that depends on its "distance", in terms of log-likelihood ratio, with respect to the reference distribution q(y). Apart from the described common learning signals, which a hallmark of probabilistic learning rules [4, 24], the update also involves per-neuron terms derived as in SG-based rules. The proposed algorithm is detailed in Appendix A. and derived next.
5
SNN Model
The encoding SNN is a directed network N of spiking neurons for which at each time-step = 1...., T, cach neuron E N outputs binary value 2,, 10. The set of neurons can be partitioned as N = H U where H denotes the set of Nu hidden neurons, and y the set of Ny readout neurons that output spiking signal y over T time instants. The SNN defines the stochastic (Te)
mapping Pw= (y|[æz) in (3) between exogenous input sequence x and the output of the readout neurons. For each neuron i .V, we denote as P. the set of pre-synaptic neurons, i.e., the set of neurons having directed synaptic links to neuron i. With a slight abuse of notation, we include the Nx exogenous inputs in this set. In order to enable the use of nnformation-theoree stochastic mapping (3), as well as the efficiency of SG-based training, we model the readout neurons in Y using the (probabilistic) Spike Response Model (SRM) [25] with stochastic threshold [4], while the hidden neurons in 7 follow the standard (deterministic) SRM. Accordingly, the spiking mechanism of every neuron C A in the SNN depends on the spiking history of pre-synaptic neurons and of the post-synaptic neuron itself through the neuron's membrane potential [4, 24, 25]
criteria via the
U4,1
-E w (at 27,1) + 1 (Se 25,1-1) + w.
(15)
JEP:
In (15), the contribution each pre-synaptic neuron j is given by the synaptic trace Ory 2j,t, where the sequence oe is the synaptic spike response and zj.z is the pre-synaptic neuron's output, multiplied by the synaptic weight wfy- The membrane potential is also affected by the post-synaptic trace 3, 24,-11, where 3, is the feedback response and ži,t is the post-synaptic neuron's 's output, which is weighted by wf The spike response a and feedback response 3, can be different for visible and hidden neurons (see Appendix A.2), and they have memory limited for T. samples, i.c., a = Br 0 for t > Te- Finally, uzf is a learnable bias parameter. Choices for these functions and details on the neuron models are provided in Appendix A.2.
Deterministic SRM. For each neuron E N, the binary output 2i,3 E {0,, 1] at time t depends on its membrane potential Ug,e
2, = (u,,)),
(16)
where (()) is the Heaviside step function: A spike is emitted when the membrane potential u, is positive.
SRM with stochastic threshold. For each neuron E y in the readout layer, the spiking probability, when conditioned on the overall spiking history, depends on the membranc potential
mws (yi, = 1/u,,,) (u,,))
(17)
where (z) = 1/(1 + exp (-z)) is the sigmoid function, and w = {we wf, wos is the per-neuron set of parameters. This yiclds the encoding stochastic mapping (3) as
T
(Te) Pw
(u||
III V 1-
awr (U..ellut.t).
(18)
6

---

Gradients. From Sec. 4, given (13)-(14), need to compute the gradients Ven log (Te) (y)lz)
For the readout neurons, this can be directly done as [4]
wf, log Pws (Vi.e 1 (ar 25,1) (vr,, - ou(,))) Cij.b.
(19)
where recall that 2j is the spiking signal of pre-synaptic neuron j. In (19), we have defined the pre-synaptic eligibility trace Cij,t- Learning rules can similarly derived for weights wf and wf, with corresponding eligibility traces Ci,t and 6i,, and we define per-neuron eligibility trace ei, = re ei,t, J, J. For neurons i E H in the hidden layer, we rely on e-prop [3] an SG-based technique obtain the approximation
V " log (y|læ)
Σ
(20)
with the learning signal
Lift Σ Bie (Jece (uu,)))
(21)
KEY
characterized by random, but fixed, weights Bi following direct feedback alignment [26], and eligibility trace
Cjjt = 6 (a, 2,,a)
(22)
The approximation (20) comes from: (i) truncating the BPTT by ignoring the dependence of the log-loss - log Pw; (Vi.,) (u,,)) at time on the weight of due the previous time instants 1' < t; (ii) replacing backpropagation through neurons, via feedback alignment; and (iii) using the derivative of surrogate function ('(-) [12]. For example, with a sigmoid surrogate gradient, we have O'(-) d')-
The overall algorithm, referred to as VDIB, is summarized in Appendix A. 1. As anticipated in the previous sections, the updates (31) and (32) contain elements from both probabilistic and SG-based learning rules. In particular, the global lcarning signal (fad r) B lw- (y. x)) relates to the global signal used in probabilistic learning [4, 24], whilst per-neuron signal (21) is feature of SG schemes [3, 12].
Experiments
We now evaluate the proposed VDIB solution in an experimental setting. We specifically focus on two tasks: predictive coding and naturalization of spiking signals. The former task highlights the aspect of causal, online, encoding and decoding, and relates closely with work in theoretical neuroscience [21]. In contrast, the latter task emphasizes the role of the proposed approach for neuromorphic computing [6]. Specifically, the naturalization of spiking inputs, e.g., obtained from a DVS camera [5], enables the integration of neuromorphic sensors with conventional digital sensors and processors [6].
As the proposed hybrid architecture gives flexibility in the choice of the decoding ANN, we have considered three options: (i) a simple logistic regression model, (ii) a multilayer perceptron (MLP) with one hidden layer, and (ii) convolutional neural network (CNN) with two convolutional layers and one fully connected layer. As discussed in Sec. 3, at each time t, the decoding ANN is given the sequence vf-r, of Ta time samples produced by the encoding SNN. As a benchmark, we also consider conventional rate decoding, whereby time samples within the sequence yi- are summed over time, yielding Et= ye, before being fed to the ANN.
By considering encoding networks with layered architecture, our implementation of both ANN and SNN makes use of automatic differentiation and general-purposs deep learning libraries [27] and Tesla V100 GPUs.
6.1 Efficient predictive coding
In this first experiment, we explore the use of the proposed hybrid variational autoencoder for a predictive coding problem. Following the experiment proposed in [21], we consider exogenous signals consisting of 20 correlated spiking signals that represent two independent auto-regressive "drifting Gaussian blobs". At each time t, the position pt E exogenous inputs follows a (wrapped) Gaussian path A((,, r), with 6, evolving according to an
20) of each "blob" across the

---

ex
0.4
inputs
encoder spile rate
encoding window T.
SNN outputs
0.2
decoding window Ta
lag
ANN outputs
0.0
0.15
0.15
MSE
MSE
0.10
0.10
0.05
filtering
prediction
0.05
10
10
10 8
10
10
lag
Figure 2: Left: Illustration of the exogenous inputs (top), and accuracy as function of the target lag 5 (bottom). Right: Encoding network spike rates with lag ō 2 versus regularization constant 3 (top), and corresponding accuracy (bottom). Shaded areas represent standard deviations.
AR(2) process 0, 0 vt with U/ av/-1 Pε,, E, N 1), 0.45, 0.9 and b = 0.14. A realization of the exogenous inputs can be found in Fig. 2(left, top). The reference signal r, at time-step consists of the one-hot encoding of the positions of the two blobs at time some time lag 5 E Z. A one layer encoder compresses the 20 exogenous spiking signals into 10 spiking signals, which are decoded by softmax regressor (playing the role of the decoding ANN). We choose the reference distribution ( to be i.i.d. Bern(0.25 (0.2) to enforce temporal sparsity. We set Te =5, and Ta and, where not mentioned otherwise, = 1. Training done over 50, 000 randomly generated examples, of length T' 100. Testing is done on single example of length T= 1, .0.. Results are presented in terms mean squared сиror (MSE), and averaged over trials with random initializations. In the right panels, it can be seen that by modulating the value of 3, one can explore the trade-off between temporal sparsity and accuracy. Overall, the approach seen to yield compact spiking representations that enable the reconstruction of the input signal via online decoding. Finally, we provide study of the effect of the window sizes T, and 7, in Appendix A.3.
for
6.2 Naturalizing MNIST images
We now consider image naturalization. To start, we obtain exogenous input spiking signals x by encoding MNIST images using either Poisson encoding (see [25]) or time-to-first- firks--siikke encoding (as in [7]). The relevance signal is defined as r. = ON for f. 1, 1, T - 1, and rr is the original image. Results are provided in terms of the accuracy of the convolutional LeNet ANN trained the original MNIST dataset [28]. Specifically, we feed the decoded natural images to the LeNet ANN, and report its accuracy. The hybrid variational autoencoder compresses the 784 exogenous spiking signals into 256 spiking signals at the readout layer of the SNN. We set тe Ta T = 30, and 0.001. Training carried over 200, 000 examples, and testing done on the 10, 000 images from the test dataset. Other experimental details can be found in Appendix A.3.
Table 1: Comparison of classification accuracies for an MLP ANN decoder under various encoding strategies (for the MNIST digits at the input of the SNN) and decoding schemes (for the output of the SNN).
Encoding Decoding
Rate
VDIB
Rate
60.50 + 0.32%
91.32 + 0.27%
Time-to- -orrt-tS-oirtt-ikee
86.43 + 0.84%
92.82 0.25%

---

OX imputs
6
3
Y
5
TecOnI 11niae
2
3
9
rate docoding
VDIB
Figure 3: Top:
Exogenous spiking inputs from the MNIST-DVS dataset summed over time samples (first row), and reconstructed natural images via VDIB (sccond row). Bottom: T-SNE representation of the spiking signals produced by the SNN with rate decoding (left), and with VDIB (right).
We explore the impact of various encoding and decoding strategies in Table 1, when using a decoding MLP. From Table 1, note that, although rate decoding provides clear benefits in reducing the size of inputs to the decoder, this is at the cost of large drop in accuracy. The clear performance advantages for time decoding validate our working hypothesis regarding the importance move away from handcrafted decoding strategies.
6.3 Naturalizing MNIST-DVS images
Finally, we evaluate VDIB on the MNIST-DVS dataset [29]. Following [16, 30-32], we crop images to 26 x 26 pixels, and to 2 seconds. We also sampling period of 10 ms, which yields T 200, and evaluate the impact of coarser sampling rates on the proposed method. To obtain natural images, the reference signal r used during training is given by single image of the corresponding class from the MNIST dataset. In this experiment, the 676 exogenous signals from MNIST-DVS images are compressed by the encoder into 256 spiking signals, which are fed convolutional decoder. We set Te Td T, and / = 0.001. Training is carried over 100, 000 examples, and testing is done on the 10, 000 images from the test dataset.
Fig. 3top row) illustrates with some examples that VDIB can effectively naturalize MNIST-DVS images. In the bottom row, we show T-SNE [33] representations of encoded spiking signals produced by the SNN when using VDIB, and compare them with those produced when considering rate decoding. The latter is seen be unable to produce representations from which class patterns can be identified, since all the representations are superimposed.
7 Conclusion
In this work, we have introduced VDIB, variational learning algorithm for a hybrid SNN-ANN autoencoder that is based on a directed, causal, variant of the information bottleneck. The proposed system enables shift from conventional handerafted decoding rules to optimized learned decoding. VDIB can find applications in neuromorphic sensing through naturalization of spiking signals, and generally as broad framework for the design of training algorithms for SNNs. This includes applications in mobile edge computing, offering opportunities in, e.g., personal healthcare. Limitations of this work include the choice of ANNs for decoders, where the usage of RNNs may result improvements. Another line for future work consists in learning the reference distribution g(y) instead of fixing it.

---

Acknowledgments and Disclosure of Funding
This work was supported by the European Research Council (ERC) under the European Union's s Horizon 2020 research and innovation programme (grant agreement No. 725731), by Intel Labs via the Intel Neuromorphic Research Community (INRC), and by a National Research Foundation Korea (NRF) grant from the Government of Korea (MSIT) (No. 2021 AFFIA163188))
The authors acknowledge use of the Joint Academic Data science Endcavour (JADE) HPC Facility (http://www.jade. ac uk/),, and use of the research computing facility at King's s College London, Rosalind (https://rosalind.kk) kel.d.kk...... uk.)
References
[1] Mark Humphries, The Spike: An Epic Journey Through the Brain in 2 Seconds, Princeton University Press, 2021.
[2] Mike Davies, Andreas Wild, Garrick Orchard, Yulia Sandamirskaya, Gabriel A. Fonseca Guerra, Prasad Joshi, Philipp Plank, and Sumedh R. Risbud, "Advancing Neuromorphic Computing with Loihi: A Survey of Results and Outlook," Proceedings of the IEEE, vol. 109, no. 5, pp. 2021.
[3] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Robert Salaj, Darjan Legen- stein, and Wolfgang Maass, "A Solution to the Learning Dilemma for Recurrent Networks of Spiking Neurons," Nature Communications, vol. 11, no. 3625, 2020.
[4] Hycryung Jang, Osvaldo Simeone, Brian Gardner, and André Grüning, "An Introduction to Probabilistic Spiking Neural Networks: Probabilistic Models. Learning Rules, and Applications," IEEE Stg. Proc. Magazine, vol. 36, no. 6, pp. 64-77,
[5] Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck, "A 128 x 128 120db 30mw Asyn- chronous Vision Sensor that Responds to Relative Intensity Change," in Proc. Int. Solid State Circuits Conf., 2006, pp. 2060-2069.
[6] Dennis Robey, Wesley Thio, Herbert H. C. Iu, and Jason Kamran Eshraghian, "Naturalizing Neuromorphic Vision Event Streams Using GANs," " CoRR, vol. abs/ http/2122.000 abs/2102.072433, 2021.
[7] Alireza Bagheri, Osvaldo Simeone, and Bipin Rajendran. "Training Probabilistic Spiking Neural Networks with First-To-Spice Decoding," in 2018 IEEE Int. Conf. Acoustics, Speech and Sig. Processing (ICASSP), 2018, pp. 2986-2990.
[8] Naftali Tishby, Fernando C. Pereira, and William Bialek, "The Information Bottleneck Method," in The 37th annual Allerton Conf. Communication, Control, and Computing. 1999, pp.
368-377.
[9] Alex Alemi, Ian Fischer, Josh Dillon, and Kevin Murphy, "Deep Variational Information Bottleneck." in ICLR, 2017.
[10] Gerhard Kramer, Directed Information for Channels with Feedback, Hartung-Gorre, 1998.
[11] Emre Neftci, Charles Augustine, Somnath Paul, and Georgios Detorakis, "Event-Driven Random Back- propaatrpaattiinn:: Enabling Neuromorphic Deep Learning Machines," Frontiers in neuroscience, vol. 11, pp. 2017.
[12] Emre Neftci, Hesham Mostafa, and Friedemann Zenke, "Surrogate Gradient Learning Spiking Neural Networks: Bringing the Power of Gradient- Based Optimization to Spiking Neural Networks," " IEEE Sig. Proc. Magazine, vol. 36, no. 6, pp. 51-63, 2019.
[13] Jacques Kaiser, Hesham Mostafa, and Emre Neftci, "Synaptic plasticity dynamics for deep continuous local lcarning (DECOLLE))" Frontiers in Neuroscience, vol. 14, pp. 424, 2020.
[14] Sumit Bam Shrestha and Garrick Orchard, "SLAYER: Spike Layer Error Reassignment in Time," in Advances in Neural Information Processing Systems, 2018, vol. 31.
[15] Friedemann Zenke and Surya Ganguli, "SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks." Neural Computation, vol. 30, no. pp. 1514-1541, 2018.
[16] Hyeryung Jang, Nicolas Skatchkovsky, and Osvaldo Simeone, "VOWEL: A Local Online
Learning Rule for Recurrent Networks of Probabilistic Spiking Take-Al Circuits," arXiv preprint arXiv:2004.00 194116, 2020.
10

---

[17] Kenneth Stewart, Andreea Danielescu, Lazar Supic, Timothy Shea, and Emre Neftci, "Ges- ture Similarity Analysis on Event Data Using a Hybrid Guided Variational Auto Encoder," arXiv:2104. 104.0,, 2021.
[18] Naftali Tishby and Noga Zaslavsky, "Deep Learning and the Information Bottleneck Principle," 2015 IEEE Information Theory Workshop (ITW), 2015, pp. 1-5.
[19] Stefan Klampfl, Robert Albin Legenstein, and Wolfgang Maass, "Spiking Neurons Can Learn to Solve Information Bottleneck Problems and to Extract Independent Components," Neural computation, vol. 21, no. 4, pp. 911-959, 2009.
[20] Lars Buesing and Wolfgang Maass. "A Spiking Neuron Information Bottleneck," Neural Computation, vol. 22, no. 8, pp. 1961-1992, 08 2010.
[21] Matthew Chalk, Olivier Marre, and Gašper Tkačik, "Toward Unified Theory of Efficient, Predictive, and Sparse Coding." Proceedings of the National Academy of Sciences, vol. 115, no.
1. pp. 186-191, 2018
[22] Osvaldo Simeone, "A Brief Introduction to Machine Learning for Engineers," Foundations and Trends@ in Signal Processing, vol. 12, no. 3-4, pp. 200-431, 2018.
[23] Andriy Mnih and Danilo Rezende, "Variational Inference for Monte Carlo Objectives," in International Conference Machine Learning, 2016, pp. 2188-2196.
[24] Hyeryung Jang, Nicolas Skatchkovsky, and Osvaldo Simeone, "Spiking Neural Networks 14208, 2020.
Parts I, II, and III,"
[25] Wulfram Gerstner and Werner M Kistler, Spiking Neuron Models: Single Neurons, Populations, Plasticity, Cambridge University Press, 2002.
[26] Arild Nøkland and Lars Hiller Fidnes, "Training Neural Networks with Local Error Signals," in Proceedings of the 36th International Conference on Machine Learning. 2019, vol. 97, pp. 4839-4850, PMLR.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetioo, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala, "Pytorch: An Imperative Style, High-Performance Deep Learning Library," in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett, Eds., pp. 8024-8035. Curran Associates, Inc., 2019.
[28] Y. LeCun, B. Boser. J. S. Denker, D. Henderson. R. E. Howard, W. Hubbard, and L. D. Jackel, "Backpropagation Applied to Handwritten Zip Code Recognition," Neural Computation, vol. 1, no. 4, pp. 541-551, 1989.
[29] Teresa Gerrano-Gotarredona and Bernabé Linares- Barranco, "Poker-DVS and MNIST-DVS. Their History, How They Were Made, and Other Details," Frontiers in neurxccience, vol. 9, pp. 481, 2015.
[30] Bo Zhao, Ruoxi Ding, Shoushun Chen, Bernabe Linares-Barranco and Huajin Tang, "Feedfor- ward Categorization on AER Motion Events Using Cortex-Lik Features in Spiking Neural Network." " IEEE Trans. on Neural Net. and Learning Systems, vol. 26, no. 9, pp. 1963-1978, 2014.
[31] James A Henderson, TingTing A Gibson, and Janet Wiles, "Spike event based learning in neural networks," arXiv preprint 150i 05777, 2015.
[32] N. Skatchkovsky, H. Jang, and O. Simcone, "Federated neuromorphic learning of spiking neural networks for low-power edge intelligence." in Proc. of International Conference on Acoustics, Speech and Signal Processing. IEEE, 2020.
[33] Laurens van der Maaten and Geoffrey Hinton, "Visualizing Data using t-SNE," Journal of Machine Learning Research, vol. 9, no. 86, pp. 2579-2605, 2008
[34] Jonathan W. Pillow, Jonathon Shlens, Liam Paninski. Alexander Sher, Alan M. Litke, E.J. Chichilnisky, and Eero P. Simoncelli, "Spatio-Temporal Correlations and Visual Signalling in a Complete Neuronal Population," Nature, vol. 454, no. 7207, pp. 995 999, 2008
11


Reviews:
[{"review_id": "Qy0h_1GLtsY", "reviewer": "Reviewer_9EcL", "summary": "Proposes a way to effectively define spike codings for target labels, to optimize (or at least not inadvertently cripple) an SNN's ability to train and classify.\nThis combines a novel training regime applied to an SSN (as front end) with an ANN (as back end) in a VAE-type structure.", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting, original and well-written paper, with potentially high value, to both neuroscience applications (spike coding) and ML applications (opening up the option of using SNNs for ML tasks).\n\nLimitation: While I have confidence in the value of the sections that I could follow, sections 4 and 5 blew over my head without even ruffling my hair (too technical). So I have no opinion about those sections, and I'm happy to defer final assessment of the paper to better informed reviewers.\n\nMisc comments:\nFigures: excellent, especially the comprehensive caption.\n\n62: SNNs (plural)\n\n68, 69: introduce acronyms eg SG, BPTT \n\neqn 1: Notation difficulty: a^{t-1} is not defined and is unclear. I'm guessing the superscript should be a subscript; or is it missing a subscript? Is the intent to condition on tau samples of b and also on the previous sample of a? \neqn 2: same issue, with b^{t-1}, also eqn 3, 7, etc \n\n103, 185: typos\n\n136 - 137, eqn 5: Reason/motivation for this?   \n\nTable 1: The content was unclear to me on first examination. A more informative caption would help, and a clearer top left entry. \"time\" -> \"time-to-first spike\". Also, moving Table 1 to later in the paper (after it is mentioned) would smooth things for the reader. \n\n297: It sounds like r was not the corresponding MNIST sample, but a different sample from the same digit class. Why this choice for r?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N.A.", "review_text": "", "extra_scores": {"ethical_concerns": "N.A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "DdHNy-iSVnJ", "reviewer": "Reviewer_Y1vr", "summary": "Summary.\\\\\nThis work proposes a hybrid model for the naturalization and classification of spiking signals in an unsupervised manner. The proposed hybrid model consists of an SNN-based encoder and an ANN-based decoder and is constrained by the DIB, including the encoding and decoding discrepancies. This idea is interesting.\n\nGeneral Overview in My Mind.\\\\\nDespite the interest of the overall goal of the research, this paper suffers from several limitations,  which make it improper for acceptance.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Summary.\n\nThis work proposes a hybrid model for the naturalization and classification of spiking signals in an unsupervised manner. The proposed hybrid model consists of an SNN-based encoder and an ANN-based decoder and is constrained by the DIB, including the encoding and decoding discrepancies. This idea is interesting.\n\nReadability.\n\nThe soundness and objective of the method, particularly its formalization and theoretical derivation, are hard to understand and follow.\n\n[1] The use of some symbols is confusing, e.g., \";\" in Eq. (2).\n\n[2] As seen in Line 179, why can the unbiased estimation of gradients be achieved with only one sample?\n\n[3] What's the binary output $h_{i,t}$?\n\n[4] Figure 1 is hard to follow. Firstly, what's the pre-neuron learning signal $L_{i,t}$? Is it the same as $u_{i,t}$ in Eq. (15)? Secondly, what's the local eligibility trace $\\boldsymbol{e}_{i,t}$? How to calculate it? Thirdly, it's better to mark the crucial symbols, such as $\\boldsymbol{x}$, $\\boldsymbol{r}$, and $\\boldsymbol{r}$, on the roadmap of Figure 1(e). Finally, it's common sense that one can obtain the spiking inputs $\\boldsymbol{x}_t$ from a static image using DVS or Poisson encoding, whereas how to reconstruct the picture, namely natural target signal, from the sequence $\\boldsymbol{t}$?\n\nClarity and Quality.\n\n[1] For each spiking neuron in the readout layer, the signals are generated by SRM with stochastic thresholds. So the symbols $\\tau_e$ in Eq. (3) and $\\tau_d$ in Eq. (4) are not fixed, right? Furthermore, how to ensure that there is an unbiased estimation of gradients in Eqs. (11) and (12)?\n\n[2]  The experiments confused me since I lack some precision and information to conclude from the experiments. (1) How to evaluate the performance of the \"naturalization\" task? It's insufficient not to visualize some specific digits in Figure 3. (2) How to conclude the sentence \"The latter is seen to be unable to produce representations from which class patterns can be identified.\" in Line 205?\n\n[3] The comparative results in Table 1 are not promising since there lack of more contenders.\n\nMinor Points.\n\n[1]  There is a lot of blank space at the end of Page 6.\n\n[2]  The format of the reference is also extremely irregular. For example, it is \"Neural Computation\" rather than \"Neural computation\" at first glance, and further, the formats of [19] and [20] are not consistent.\n\nAnonymity.\n\nAnonymity is not difficult to break since the authors uploaded an arxiv version of this manuscript on June 2nd, well after the deadline of NeurIPS 2021. I do not take this as an argument for rejection, but the authors should pay more attention to this.\n\nGeneral Overview in My Mind.\n\nDespite the interest of the overall goal of the research, this paper suffers from several limitations,  which make it improper for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Mentioned above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "A3nShbx4uf", "reviewer": "Reviewer_H28A", "summary": "The authors proposed an end-to-end learning framework for learning spiking signal representations and corresponding decoding ANN. The framework optimizes a directed information bottleneck training criterion via SG and is demonstrated by experiments on various tasks on MNIST data. The contribution includes automatically generating spiking latent representations which are handcrafted before, generalizing standard information bottleneck principle to directed information bottleneck variation, developing the variational approximation of DIB objective, successfully applying the proposed framework on various tasks on MNIST.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "minor problem:\n  1. definition is missing for random vector a and b with only superscript (i.e. a^{t-1} := a^{t-1}_{0})\n  2. some papers are cited twice in the same bracket.\n\n  This paper focuses on spiking neural networks. Its SNN part is assembled with existing SNN works, and its ANN part is simple LR/MLP/CNNs. The main contribution of this paper is lying on adapting information bottleneck (IB) to the spiking signal encoding problem, and afterwards to train the SNN inside an autoencoder framework. The authors discover automatically efficient time-decoding rules from spiking signals produced by the encoding SNN. This claim somehow can be found in most existing papers on encoding-decoding architectures. The technical challenge itself is limited (for the DIB generalization from standard IB), although derivation steps are well written.\n\n  The insight of applying IB in this problem is reasonable. The authors also claimed their proposed directed information bottleneck (DIB) objective is considered first time for spiking signal encoding. The difference w.r.t IB is to replace the mutual information with \\tau-directed information. Generalized from directed information concept, \\tau-directed information is designed to improve the flexibility of quantifying casual statistical dependencies within a limited window. Since the authors mentioned a lot of causality, we hope there could be more discussion on how to tune \\tau_d and \\tau_e in SNN and ANN part, and what could be inferred or observed from these critical parameters.\n\n  This paper is relatively good but there is still space to improve. If the authors are trying to look deeper in the DIB aspect, targeting tasks compatible with this framework could be pretty much expanded so that you do not have to only play with MNIST data. If focusing on the SNN, it will be beneficial to give more detailed interpretations of your model and effects in the experiment, as what the cited paper [21, Matthew Chalk et al. PNAS 2018] does.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are briefly addressed in the conclusion section.\nNo negative social impacts are found.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concern was found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "52gbhXFG4uZ", "reviewer": "Reviewer_Hpuu", "summary": "In the paper \"Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck\" the author propose a hybrid variational autoencoder architecture with an encoding spiking neural network and a decoding artificial neural network. The authors evaluate the method on predictive coding and naturalization of spiking signals tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe authors devise a novel hybrid architecture that lets an ANN learn how to decode spikes and train it together with an SNN encoder, thereby not fixing the coding a priori. I have not seen this idea before and find it intriguing. To this end, the authors introduce a novel directed information bottleneck objective. Overall, the original contributions of this work are excellent.\n\nQuality:\nThe quality of the work is outstanding. The derivations of the variational directed information bottleneck algorithm and associated learning rules are rigorous. The predictive coding and naturalization tasks are sensible and demonstrate well the capabilities of the method.\nA minor point: in lines 40-41 the authors confused panels a and b.\n\nClarity:\nThe manuscript is very well written. Figures can be understood relatively easily. There is a notations section which is helpful.\n\nSignificance:\nThe authors mention important practical applications in neuromorphic sensing. I believe learning optimal decoding strategies for SNN encoders could also lead to new predictions in systems neuroscience.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations with respect to choice of decoders and the reference distribution are very briefly mentioned in the Conclusion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I want to thank the authors and reviewers for engaging with this paper, as well as the reviewers for participating in an internal discussion.

We have some disagreement between the reviewers with several positive reviews and one leaning rejection.

After reading the paper, and all of the related discussion I'm going to recommend this paper for acceptance.  I feel as though work has sufficient Quality, Originality, Clarity and Significance for acceptance.

I believe that many of the remaining issues reviewers have are largely stylistic and are not enough to block acceptance.
Decision: Accept (Poster)