Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this publication, the authors present an interesting framework for approximation the mapping between sequences by feedforward spiking neural networks (SNN), providing insights on the computational capabilities of feedforward SNNs for the approximation of the mapping of input to output spike trains. And how these are influenced by hyperparameters such as the network architecture, heterogenic properties of neurons and skipping connections. The authors also provide an ansatz how these hyperparameters can be optimised in a two-steps Bayesian optimisation fashion. The performance of the approach is demonstrated by several spatiotemporal classification tasks, on datasets like DVS Gesture, N-Caltech 101 and sequential MNIST. 
main_review: Major Points

The paper is well written, the motivation of the authors is clear, and the framework presented is easy to understand. The theory part is written in a clear mathematical manner, with a theorem-proof structure, which I really like. But I have one main point of criticism that seems very important to me, while the entire theory section provides insights and interesting ideas the provided results in sec. 5 Experiments does not hold up to this. Certainly, a benchmark comparison on common data sets with other methods is very important, but it’s not adequate for the hypotheses presented in the theory part. The main argument of this paper (at least for me) is that a feedforward SNN with many memory pathways can approximate a mapping between a temporal sequence of spike trains with time-varying unknown frequencies to a pre-defined output spike trains with known frequencies, where the number of memory pathways can be increased by optimizing the skipping connections and heterogeneities in the network. This connection to the universal approximation theorem should be more supported by quantitative experiments. My suggestion to investigate the computational power of the optimised SNN would be to see how big the memory effect is as well as to exam the complexity of the mapping the SNN can approximate. One way to address this would be to train the SNN to learn a mapping from $f(x)=x $ to $g(x)=(x^n)/f(x-m)$ for different combinations of $m=1,\dots, M$ and $n=1,\dots, N$, e.g. $M=N=10$.  The resulting reconstruction errors could then be considered as a function of m and n as well as the number of network parameters. If all this should provide very good reconstructions, the mapping function could be chosen in a more complicated design.
A next point: On page 3 “[…] as for MLP-SNN and Conv-SNN network the analysis can be extended according to the specific layer dimensions.” At this point more details or explanations would be desirable I only understood this part after I had a look at the source code.

Page 7: Please provide a formal definition for the objective function, which metric is optimised?

Page 7: “[…] first optimized with fixed, manually selected neuron parameters.” How were these parameters elected or on what basis, I assume random values should not work?

Page 8 Fig 3.: I would suggest using a different symbol for the stage 1 and stage 2 optimisation. Further, I agree with the authors that the dual search converges faster, but it seems that the single search achieves a marginally better validation error. In order to better assess the advantage of faster convergence, how long does an iteration step take and what hardware was used?

Minor Points
Page 5: “Since any continuous bounded function on a compact interval […]”. For the sake of completeness, I would suggest adding the definition for a compact interval. How were these parameter selected?

Page 5 and other use $\left( … \right)$, e.g. Lemma 3

Page 7: This is really optional: Would it be possible to show some projections of the search space from the architecture search space? Maybe in the appendix or even the code-section.

Page 8: It’s a matter of taste: I would suggest that the y axis in figure 3 has the same range in both plots. then the plots can be compared directly with each other without having to pay attention to the axis section.

summary_of_the_review: The paper is well written, the motivation of the authors is clear, and the framework presented is easy to understand. The theory part is written in a clear mathematical manner, with a theorem-proof structure, which I really like. The theoretical results provide new insights and should give new impulses to other scientists in this field as well as in related fields, e.g.: for improved applications as well as building theories.

My main criticism is that the good theoretical results are not sufficiently well supported by numerical experiments, which could further consolidate the good theory results. I hope to provide a possible motivation for supplementing the results in my review. However, these would have to be provided in order to recommend a publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a theory on how to learn arbitrary input spike train - output spike train mappings, using a feedforward SNN with heterogeneous neurons and skip connections.
Then this theory is used to train a deep SNN using either BPTT or STDP. The SNN is then evaluated on IBM DVS Gesture, N-Caltech101. The accuracy that they get with BPTT is beyond the SOTA.
main_review: Pros:
* very good accuracy

Cons:
* many doubts need to be cleared about the theory
* the connection between the theory and the experiments is unclear

The theory is unclear:
* Does it only apply to sequences with constant inter-spike intervals (ISI), as Fig 1a suggests? The fact that they focus on the spike counts (gamma) also suggests that.
If true, this is a strong loss of generality, which should be properly acknowledged. Also, I don't understand how the theory can help with DVS datasets, in which the ISI are highly variable.
If false, then Fig 1 should show variable ISIs, and the authors should justify why they can summarize a spike train by its spike count.
* Lemma 1 seems wrong: what if no input spike? Then changing G has no effect on the response...
* Does the theory work in continuous or discrete-time? The equations suggest continuous time. But a timestep is mentioned on page 4. Plus BPTT works in discrete-time.

The connection between the theory and the experiments is not clear to me. Does the theory only suggests heterogeneous neurons and skip connections, but then the training is done with normal BPTT (or STDP)?
Does the theory also justify the use of learned and transferred synapses? BTW, these concepts are obscure to me. Isn't it equivalent to having a set of weights that is shared between neurons (like in a convolutional layer)?


MINOR POINTS:
* Eq 1: you should say that a is the resting potential
* Eq 2: don't use t for the integration variable
* Table 1 should specify the number of timesteps
* Table 3 should include: https://arxiv.org/abs/2007.05785

summary_of_the_review: Potentially interesting given the accuracy, but unclear
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents some theoretical understanding of sequence approximation using feedforward SNNs. The main conclusions are two folds: (1) a feedforward SNN with one neuron per layer and skip-layer connections can approximate the mapping rate-coding function; (2) a feedforward SNN constructed by heterogeneous neurons with varying dynamics and skip-connections can improve sequence approximation. Besides, the authors provide the DSBO to optimize the architecture and parameters of their proposed SNNs.


main_review: Theoretical understanding of SNNs in terms of approximation, temporal memorability, optimization, and generalization is a fundamental and challenging problem for SNNs. This paper takes a step in this direction. The motivation is interesting and significant.

Despite the interest of the overall goal of the research, this paper suffers from several limitations,  which make it improper for acceptance. There are two main reasons for rejection, (1) the writing of this paper, especially theoretical conclusions,  needs to be improved for proofreading; (2) the claimed contributions cannot be confirmed or supported by the theory and experiments.

In detail,
1. Definitions 1-5 are too redundant. It's not necessary to state these informal definitions in a formal environment. Please be clear and simple.
2. What's the difference between lemmas and theorems presented by this paper? Is it just because the conclusions of the theorems are more important or more dazzling than those of the lemmas?
3. Can the authors provide a formal description and proof for their theoretical results? The current can only be called a heuristic description.
4. What's G in Eq. (2)? Is it trainable in practice? It works just like a normalizer, and thus, can be replaced by $R_m / \tau_m e^{-1/ \tau_m}$? In fact, the legality of rate coding has already been discussed.
5. Besides, there have been great efforts on the theoretical understanding of SNNs, such as [1-4].
6. Notice that lemmas 4 and 5 only investigate the upper bound, not the least upper bound. So one cannot conclude that using heterogeneous and skip connections can improve the sequence approximation of feedforward SNNs.

The experiment is intriguing and could lead to something new. However, I am unable to follow the author's suggestions, such as how to build an experiment based on past beliefs. Furthermore, don't the optimization techniques discussed in Section 4 already exist? Both BPTT and STDP are trivial for skip connection and feedforward architecture [5]. In order to show the effectiveness of the proposed network, it is not sufficient to use only accuracy as the evaluation indicator. It's better to display the spot rasters of the memory and learner modules, just like what shows in [2] and [6].

I felt a sense of fragmentation when reading this paper. It is very much like a suture monster. I would suggest that the authors use rigorous proof to re-write clearly what the conclusions of the theory are? Where is the proof of innovation? And explain clearly what is the innovation of the experimental model or algorithm? Why is it effective?

[1] On the Algorithmic Power of Spiking Neural Networks. 2019. 
[2] Bifurcation Spiking Neural Network. 2019.
[3] Firing rate predictions in optimal balanced networks. 2013. 
[4] Spiking Analog VLSI Neuron Assemblies as Constraint Satisfaction Problem Solvers.2015.
[5] Error-Backpropagation in Temporally Encoded Networks of Spiking Neurons. 2000.
[6] Slayer: Spike layer error reassignment in time. 2018.
summary_of_the_review: In summary, I find the social impact of this work, but it still needs to be refined carefully. If there were an option for 'Weak Reject', I would select that. Since there is no such option, I recommend rejection given the many suggestions.

______________
After reading the rebuttal and other reviewers' comments, I consider arising my score to 6, although there are still many inappropriate points in the current version.
review_title: null

Meta-review: The authors provide a theory for training feed-forward spiking neural networks (SNNs) on input-to-output spike train mappings. They utilise for this heterogenous neurone and skip connections.

The resulting method is tested on DVS Gesture, N-Caltech 101 and sequential MNIST.
It achieved very good performance. The reviewers agreed that the results are interesting and significant.

In the initial reviews, the reviewers pointed out some doubts about the theory and clarity of writing.
These doubts and objections were addressed in the revision and the reviewers were quite satisfied with that.

In conclusion, the manuscript presents interesting results for SNNs with a solid theory and very good experimental results. All reviewers vote for acceptance.
Decision: Accept (Poster)