Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In the paper "Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck" the author propose a hybrid variational autoencoder architecture with an encoding spiking neural network and a decoding artificial neural network. The authors evaluate the method on predictive coding and naturalization of spiking signals tasks.
main_review: Originality:
The authors devise a novel hybrid architecture that lets an ANN learn how to decode spikes and train it together with an SNN encoder, thereby not fixing the coding a priori. I have not seen this idea before and find it intriguing. To this end, the authors introduce a novel directed information bottleneck objective. Overall, the original contributions of this work are excellent.

Quality:
The quality of the work is outstanding. The derivations of the variational directed information bottleneck algorithm and associated learning rules are rigorous. The predictive coding and naturalization tasks are sensible and demonstrate well the capabilities of the method.
A minor point: in lines 40-41 the authors confused panels a and b.

Clarity:
The manuscript is very well written. Figures can be understood relatively easily. There is a notations section which is helpful.

Significance:
The authors mention important practical applications in neuromorphic sensing. I believe learning optimal decoding strategies for SNN encoders could also lead to new predictions in systems neuroscience.
limitations_and_societal_impact: Limitations with respect to choice of decoders and the reference distribution are very briefly mentioned in the Conclusion section.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Proposes a way to effectively define spike codings for target labels, to optimize (or at least not inadvertently cripple) an SNN's ability to train and classify.
This combines a novel training regime applied to an SSN (as front end) with an ANN (as back end) in a VAE-type structure.
main_review: This is an interesting, original and well-written paper, with potentially high value, to both neuroscience applications (spike coding) and ML applications (opening up the option of using SNNs for ML tasks).

Limitation: While I have confidence in the value of the sections that I could follow, sections 4 and 5 blew over my head without even ruffling my hair (too technical). So I have no opinion about those sections, and I'm happy to defer final assessment of the paper to better informed reviewers.

Misc comments:
Figures: excellent, especially the comprehensive caption.

62: SNNs (plural)

68, 69: introduce acronyms eg SG, BPTT 

eqn 1: Notation difficulty: a^{t-1} is not defined and is unclear. I'm guessing the superscript should be a subscript; or is it missing a subscript? Is the intent to condition on tau samples of b and also on the previous sample of a? 
eqn 2: same issue, with b^{t-1}, also eqn 3, 7, etc 

103, 185: typos

136 - 137, eqn 5: Reason/motivation for this?   

Table 1: The content was unclear to me on first examination. A more informative caption would help, and a clearer top left entry. "time" -> "time-to-first spike". Also, moving Table 1 to later in the paper (after it is mentioned) would smooth things for the reader. 

297: It sounds like r was not the corresponding MNIST sample, but a different sample from the same digit class. Why this choice for r?
limitations_and_societal_impact: N.A.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed an end-to-end learning framework for learning spiking signal representations and corresponding decoding ANN. The framework optimizes a directed information bottleneck training criterion via SG and is demonstrated by experiments on various tasks on MNIST data. The contribution includes automatically generating spiking latent representations which are handcrafted before, generalizing standard information bottleneck principle to directed information bottleneck variation, developing the variational approximation of DIB objective, successfully applying the proposed framework on various tasks on MNIST.
main_review: minor problem:
  1. definition is missing for random vector a and b with only superscript (i.e. a^{t-1} := a^{t-1}_{0})
  2. some papers are cited twice in the same bracket.

  This paper focuses on spiking neural networks. Its SNN part is assembled with existing SNN works, and its ANN part is simple LR/MLP/CNNs. The main contribution of this paper is lying on adapting information bottleneck (IB) to the spiking signal encoding problem, and afterwards to train the SNN inside an autoencoder framework. The authors discover automatically efficient time-decoding rules from spiking signals produced by the encoding SNN. This claim somehow can be found in most existing papers on encoding-decoding architectures. The technical challenge itself is limited (for the DIB generalization from standard IB), although derivation steps are well written.

  The insight of applying IB in this problem is reasonable. The authors also claimed their proposed directed information bottleneck (DIB) objective is considered first time for spiking signal encoding. The difference w.r.t IB is to replace the mutual information with \tau-directed information. Generalized from directed information concept, \tau-directed information is designed to improve the flexibility of quantifying casual statistical dependencies within a limited window. Since the authors mentioned a lot of causality, we hope there could be more discussion on how to tune \tau_d and \tau_e in SNN and ANN part, and what could be inferred or observed from these critical parameters.

  This paper is relatively good but there is still space to improve. If the authors are trying to look deeper in the DIB aspect, targeting tasks compatible with this framework could be pretty much expanded so that you do not have to only play with MNIST data. If focusing on the SNN, it will be beneficial to give more detailed interpretations of your model and effects in the experiment, as what the cited paper [21, Matthew Chalk et al. PNAS 2018] does.
limitations_and_societal_impact: Limitations are briefly addressed in the conclusion section.
No negative social impacts are found.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Summary.\\
This work proposes a hybrid model for the naturalization and classification of spiking signals in an unsupervised manner. The proposed hybrid model consists of an SNN-based encoder and an ANN-based decoder and is constrained by the DIB, including the encoding and decoding discrepancies. This idea is interesting.

General Overview in My Mind.\\
Despite the interest of the overall goal of the research, this paper suffers from several limitations,  which make it improper for acceptance.
main_review: Summary.

This work proposes a hybrid model for the naturalization and classification of spiking signals in an unsupervised manner. The proposed hybrid model consists of an SNN-based encoder and an ANN-based decoder and is constrained by the DIB, including the encoding and decoding discrepancies. This idea is interesting.

Readability.

The soundness and objective of the method, particularly its formalization and theoretical derivation, are hard to understand and follow.

[1] The use of some symbols is confusing, e.g., ";" in Eq. (2).

[2] As seen in Line 179, why can the unbiased estimation of gradients be achieved with only one sample?

[3] What's the binary output $h_{i,t}$?

[4] Figure 1 is hard to follow. Firstly, what's the pre-neuron learning signal $L_{i,t}$? Is it the same as $u_{i,t}$ in Eq. (15)? Secondly, what's the local eligibility trace $\boldsymbol{e}_{i,t}$? How to calculate it? Thirdly, it's better to mark the crucial symbols, such as $\boldsymbol{x}$, $\boldsymbol{r}$, and $\boldsymbol{r}$, on the roadmap of Figure 1(e). Finally, it's common sense that one can obtain the spiking inputs $\boldsymbol{x}_t$ from a static image using DVS or Poisson encoding, whereas how to reconstruct the picture, namely natural target signal, from the sequence $\boldsymbol{t}$?

Clarity and Quality.

[1] For each spiking neuron in the readout layer, the signals are generated by SRM with stochastic thresholds. So the symbols $\tau_e$ in Eq. (3) and $\tau_d$ in Eq. (4) are not fixed, right? Furthermore, how to ensure that there is an unbiased estimation of gradients in Eqs. (11) and (12)?

[2]  The experiments confused me since I lack some precision and information to conclude from the experiments. (1) How to evaluate the performance of the "naturalization" task? It's insufficient not to visualize some specific digits in Figure 3. (2) How to conclude the sentence "The latter is seen to be unable to produce representations from which class patterns can be identified." in Line 205?

[3] The comparative results in Table 1 are not promising since there lack of more contenders.

Minor Points.

[1]  There is a lot of blank space at the end of Page 6.

[2]  The format of the reference is also extremely irregular. For example, it is "Neural Computation" rather than "Neural computation" at first glance, and further, the formats of [19] and [20] are not consistent.

Anonymity.

Anonymity is not difficult to break since the authors uploaded an arxiv version of this manuscript on June 2nd, well after the deadline of NeurIPS 2021. I do not take this as an argument for rejection, but the authors should pay more attention to this.

General Overview in My Mind.

Despite the interest of the overall goal of the research, this paper suffers from several limitations,  which make it improper for acceptance.
limitations_and_societal_impact: Mentioned above.
review_title: null

Meta-review: I want to thank the authors and reviewers for engaging with this paper, as well as the reviewers for participating in an internal discussion.

We have some disagreement between the reviewers with several positive reviews and one leaning rejection.

After reading the paper, and all of the related discussion I'm going to recommend this paper for acceptance.  I feel as though work has sufficient Quality, Originality, Clarity and Significance for acceptance.

I believe that many of the remaining issues reviewers have are largely stylistic and are not enough to block acceptance.
Decision: Accept (Poster)