Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper makes an endeavor to enhance the memory capacity $\mathcal{C}$ with heterogeneity in neuronal dynamics, which refers to different membrane time constants. The authors also show heterogeneous STDP, which means using a probabilistic synaptic time constant, helps to suppress the firing rate while the memory capacity is maintained. The results are both analytical and empirical.
strength_and_weaknesses: **Pros**

Although using a heterogeneous time constant is not novel in this area, the elaboration upon its effect on the memory capacity of SNNs is clearly novel to me.

**Cons**

1. There are some things that need to be clarified or properly placed notations.
2. The statement in Lemma 3.2.1, that is, heterogeneous dynamics reduce the variance of membrane voltages and thus reduce the rate of spikes, seems to omit the discussion of the mean of membrane voltages.

**Questions**

1. The notations are a bit confusing even when the authors have made a table of notations. Here is a non-exhaustive lists
   -  ${\bf{w}}^{\rm{in}}$ first appears in the statement of Lemma 3.1.1, but is first defined in short proof.
   -  In Eq.2, the authors choose $[\cdot]$ to express discretized variable and distinguish it from continuous one $(\cdot)$. However, in Eq.4 and many following equations with integer $\tau$, they still use $(\cdot)$.
   - When using $\Delta$, only the meanings of $\Delta w$ and $\Delta t$ are clearly given. Notations like $\Delta\text{Var}$ in Lemma 3.2.1 are not defined and can be confusing.
2. The statement in Lemma 3.2.1 that a lower variance of membrane voltage distribution leads to a lower firing rate is not sufficient. A zero variance (constant) voltage but higher than the threshold leads to persistent firing.
clarity,_quality,_novelty_and_reproducibility: Some notations are a bit confusing. The idea is novel.
summary_of_the_review: Many contents are confusing, and need to be clarified.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The submission studies discrete-time recurrent spiking neural networks of LIF neurons with STDP both analytically and numerically.  The main analytical finding is that heterogeneity of neuron parameters increases the memory capacity of the recurrent spiking neural network. This finding is based on the fact that with increasing heterogeneity of neuron parameters (threshold and membrane time constant), the covariance is decreasing ints covariance. A lower covariance results in a higher capacity. The numerical analysis confirms that. Moreover, a SAGE-attribution analysis reveals that the main contribution to the improved performance comes from heterogeneous membrane time constants.
strength_and_weaknesses: Strength:
* It is a rare case of an analytical study of the computational capabilities of a recurrent spiking neural network.

Weaknesses:
* The main finding is that heterogeneity of membrane time constants increases the capacity, however, in biological neurons, the membrane time constant is not a free parameter but is tightly controlled by the resistance across the membrane and the capacitance of the membrane. It is thus not biologically plausible that the brain has the freedom to choose a heterogeneous distribution of membrane time constants. A more plausible candidate for that would be the heterogeneity of synaptic time constants.
* A direct comparison of analytical results and numerical simulations is missing. While figure 3 shows a similar trend as expected from theory, it would be desirable to actually directly compare them. 
* table 2: it would be better to measure the firing rates of the neurons in 1/s or 1/tau_m and not as 'spike count'.
* It would be nice to relate this work to other work with recurrent networks of LIF neurons, e.g. Brunel 2000, Jahnke/Memmesheimer/Timme 2009, Deneve/Machens 2016. 
* The theoretical part seems to rely on some kind of mean-field assumption that allows to express the covariance of the activity in terms of linearized dynamics and a stationarity assumption. As LIF networks can have a very diverse set of states (e.g. oscillations, synchronization, splay-states, asynchronous irregular states, etc. see e.g. Brunel 2000), it is not clear how this assumption is justified here. Also, no direct numerical tests of the analytical results are being presented, which makes it hard to evaluate them. 
* Details the Bayesian Optimization are missing, the numerical experiments are not fully reproducible in the current version.
clarity,_quality,_novelty_and_reproducibility: * Clarity: 
The paper is written clearly

Quality:
The quality of the theory is hard to evaluate because the analytical results are directly tested analytically.
The numerical evaluation seems to be done carefully, I liked e.g. convergence analysis on the supplement.


Novelty:
The main theoretical results seem to be a slight extension of some previous work (Aceituno 2020). Unfortunately, the main analytical result (Equations 21 and 22) seems more of a heuristic nature. An actual analytical theory of the quantitative effects of heterogeneity of membrane time constant and thresholds in the capacity is missing, so maybe one could write it as a conjecture instead. Also, a discussion of the assumptions and limitations of the analytical derivation is missing. When is it expected to fail? Are the assumptions tested numerically?

Reproducibility: 
Because no code is provided, the results are unfortunately not fully reproducible in the current form.

summary_of_the_review: The submission studies the role of the heterogeneity of membrane time constants and threshold heterogeneity on the capacity of recurrent spiking neural networks.
The two main weaknesses of the submission are a lack of biological plausibility and a lack of discussion of the scope and rigor of the "analytical" theory (which seems to be a slight extension of  (Aceituno 2020), i.e (including discussion of the implicit assumptions and when it would break down) and details (and code) that would make the findings reproducible.

Overall, I cannot recommend the manuscript in the current form for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates the role of neural and synaptic heterogeneity in improving the computational capacity and/or energy efficiency of spiking recurrent neural nets. It also provides an optimization procedure for determining the optimal degree of heterogeneity for a problem.
strength_and_weaknesses: Strengths:
+ biological motivation
+ what in hardware may be traditionally deemed as a bug becomes a computational feature of the system
+ rigorous analysis of implications
+ numerics are solid

Weakness:
- capacity measure somewhat restricting as a measure of computational power


clarity,_quality,_novelty_and_reproducibility: I found the writing very clear, precise and easy to follow

would be useful to expand slightly on the bayesian optimization procedure, in particular the  parameter distributions being optimized over
summary_of_the_review: Overall, a solid contribution with clear problem framing, theoretical and numerical support and robust numerical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed the spike-efficiency measurement for spiking neural networks and showed the relationship between spike-efficiency and the heterogeneity. They also performed some empirical experiments to demonstrate the significance of neuronal and synaptic heterogeneity in reducing spiking activity. 
strength_and_weaknesses: Strength:
1. The manuscript is clearly written with sufficient details. 
2. The idea of quantifying the capacity and heterogeneity is interesting. 

Weakness:
Although this topic appears to be interesting, there seem to lack sufficient evidence to support the claim. Specifically, I have the following concerns. 
1. The capacity can be intuitively understood as a measurement of how much variance in the past are still maintained in the current state. However, what's the relationship between this capacity and the model performance or generalization ability? As is known for the deep learning setup, overfit on the training dataset is a typical phenomenon. Thus increasing only the capacity is not enough for such cases. 

2. The definition of spike-efficiency is a bit random especially the factor 1000. Is there any reasoning on why we should define it as a ratio? Since both the memory capacity and spiking rate are unit-free variables, their relationship can be composed in multiple ways. Thus it would make more sense if this definition is derived under a certain criterion rather than simply claimed. 

3. Intuitively one may expect that if we improve the heterogeneity of parameters, it would lead to more complex models thus higher ability in memorizing certain modes. For the results in Table 2, soly increasing heterogeneity of LIF does not necessarily improve the efficiency or not. How should we understand such difference with STDP? Specially for Lemma 3.2.1, the $\Delta Var$ is not defined. Also, it seems that the proof here is a directive corralory from the heavy-tail claim, I am not 100% sure about the reliability here. 

4. The experiments are on fairly simple cases. To validate the capacity and the assumption, experiments on some more complex static/dynamic dataset may be included. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written with moderate originality in the topic. 
summary_of_the_review: Overall, this is a paper that proposes an interesting question of measuring the performance of RSNN. However, the discussion remains a bit superficial.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper focuses on the neuronal and synaptic dynamics in spike-efficient unsupervised learning, specifically showing by theory and experiments that optimizing for heterogeneity of the time constant in the STDP learning rule using Bayesian Optimization brings significant boost to accuracy. The paper describes theorems that show that the heterogeneity increases memory capacity, decreases average spiking firing rate, and overall maximizes their ration. By this account, the authors train recurrent spiking neural networks that obtain high accuracies.

There was disagreement with reviewers, mostly on whether the theorem is indeed correctly proven or not, and whether the paper relies on Hawkes processes or not to formally prove its claims. Ultimately, after reading the paper myself, I find the theoretical backing good enough at this stage for publication, and the overall contribution of the paper to be of interest, as most reviewers are positive as well.
justification_for_why_not_higher_score: There are significant changes that needed for the camera ready. Also, it is not entirely clear if the theoretical motivation is completely bullet-proof.
justification_for_why_not_lower_score: There is a good interplay between theoretical backing and empirical justification. I found the paper an interesting and useful read.
Decision: Accept: poster