Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a temporal BN method to train SNNs with high accuracy,
strengths_and_weaknesses: + The methos is simple and effective as suggested by author's results.
-The main comment I ahve is on the novelty of this work. The authors have cited many relevant works such as BNTT, TDBN etc. I think the paper's results are very similar to that of BNTT. Except for the fact that the accuracy is better in the datasets that authros have exeprimented with, I don't thinkt here is a lot of technical contribution or novelty. 
-Further, authors of BNTT and TDBN experimented on larger datasets like Tiny Imagenet. Can the authors commnet on the scalability of their approach?
-BNTT and TDBN like methods were adapted for learning in diverse scenarios, liek Federated learning [1], Segmentation [2], large scale DVS datasets, NAS optimization[3] among others. Maybe the authors can include a commentary or discussion on how their approach can be applied to more diverse learning scenarios.

[1] Venkatesha, Yeshwanth, et al. "Federated learning with spiking neural networks." IEEE Transactions on Signal Processing 69 (2021): 6183-6194.
[2] Kim, Youngeun, et al. "Beyond classification: directly training spiking neural networks for semantic segmentation." arXiv preprint arXiv:2110.07742 (2021).
[3]Kim, Youngeun, et al. "Neural architecture search for spiking neural networks." arXiv preprint arXiv:2201.10355 (2022).
questions: My main concern is teh technical novelty of this paper with respect to previous works. The authors have performed a holistic comparison. But, can they comment on how their work is technically novel? At this time, in my opinion, the paper reads to be more incremental effort.
limitations: Please see the above comments on weaknesses.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The manuscript presents a technique (TEBN) for batch normalization in SNNs that takes into account the temporal dimension of the spiking input to rescale and recenter the input distribution differently at each input timestep, even though it still estimates the mean and variance of the whole batch, throughout all timesteps. The paper's theory shows that this smoothens the loss landscape. Experiments show that TEBN improves accuracy, robustness, and latency compared to other batchnorm methods and compare to the state of the art in the SNN literature, in image recognition of common benchmark datasets, including recordings of static images through neuromorphic vision sensors.
strengths_and_weaknesses: The paper targets a true problem in the field of SNNs, and appears to provide a truly improved solution for certain important cases. In addition, it provides theoretical analysis, and experimental insights deeper than merely end accuracies, showing the impact of the method on input distributions and on robustness to hyperparameters.

On the other hand, the paper:
- does not address issues related to truly sequential data. SNNs are considered as most suitable for temporal tasks, therefore a complete evaluation of the method should involve such tasks. An example is keyword spotting on the Speech Commands dataset. Currently, all tested datasets in the paper only include static images, even if in some cases the images are recorded through a DVS.
- does not compare to layer normalization, a technique known as suitable for recurrent networks and temporal datasets, even though the authors' input includes a temporal dimension, and even though LIF SNNs are recurrent networks.
- does not explain how its results compare to the true state of the art, i.e. beyond SNNs. For example, non-spiking networks can also reduce latency and computation while maintaining their higher accuracy than SNNs (Jeffares et al., ICLR 2022 https://openreview.net/forum?id=iMH1e5k7n3L). But more generally, it would be good to put the results back in the context of the broader ML field, and discuss the differences e.g. in accuracy, latency, or computational efficiency.
- does not mention the method's strong similarity to short-term plasticity (STP), which is a long-known property of biological SNNs for temporal filtering of inputs (see e.g. Fortune & Rose, 2001 https://roselab.biology.utah.edu/publications/trends2001.pdf; Rosenbaum et al., 2012 https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002557). Temporal filtering is what the authors' TEBN also applies. Of course, STP applies a fixed, e.g. exponential kernel, whereas TEBN learns the shape of the temporal kernel. On the other hand, short-term plasticity is also learnable (Garcia-Rodriguez et al., ICML 2022 https://arxiv.org/abs/2206.14048), which then makes STP similar to what the authors here achieve, but in several aspects with more flexibility than what the authors here propose. Moreover, STP has been shown to be a powerful property of SNNs, such that SNNs with STP can even outperform LSTMs and CNNs in accuracy (Moraitis et al., 2021 https://arxiv.org/abs/2009.06808). Even though the authors do not have to perform experimental comparisons of their method against STP, STP's relevance through these works must be mentioned, and discussed as a potential alternative for future work, that is also more biologically-realistic.
questions: Could the authors please respond to the above points, and ideally address them with experiments?
Are the accuracies of other methods reported in the tables results from the authors own experiments, or from the literature? Please clarify in the paper.
limitations: The authors have not discussed any limitations of the work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors proposed a batch normalization scheme considering the temporal domain, TEBN. The proposed TEBN requires all time-step information to estimate the desired expectation and variance. The authors provided theoretical analysis and validated the effectiveness of the proposed TEBN with classification tasks. 
strengths_and_weaknesses: Strengths:
+1. The authors showed good efforts in theoretical analysis, which is desired for the NeurIPS audience. 

Weaknesses:

-1. Requiring all time steps information is not practical at all, especially for time-dependent tasks, e.g., tracking. For classification tasks, it may make sense. However, using SNN for classification tasks lacks justification as the tasks do not really need temporal information.

-2. The experimental setup is unclear, and validations are weak. For VGG, ResNet, and other architectures, which parts are based on SNNs? For the ResNet-19, did the authors only replace the ReLU with SNNs?
* The experimental results did not show the impacts of the following factor on the classification tasks.
     * mini-batch size
     * spiking threshold
     * higher time steps: The authors claimed that fewer time steps are enough for the tested classification tasks. However, how do the larger time steps impact the results?

-3. The experimental results are only based on LIF. What about other SNN neuron models, e.g., SRM, IF?



 
questions: Besides the questions, I listed above, the authors must need to describe the detailed architecture of the models used in their experiments. They cannot assume the audiences know them or let audiences play a guessing game. 
limitations: The validation is not convincing at all from my perspective. Only LIF and classification tasks cannot show the generalization of the proposed method. Without revealing the performance of the tasks that rely on temporal information, I do not think the proposed scheme is meaningful to the community. 

As such, I suggest the authors show that the proposed scheme works with time-sensitive tasks.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an efficient batch normalization (BN) method to smooth and uniform the temporal distributions of the presynaptic input in the Spiking Neural Networks (SNN). By combing the BN with temporal features, the proposed TEBN could alleviate the gradient vanishing problem to some extent. The proposed TEBN shows better classification accuracy and robustness to hyper-parameters on CIFAR-10, CIFAR-100 and DVS-CIFAR10 with fewer time-steps.
strengths_and_weaknesses: Strengths: 
1) The writing is clear and the motivation is clarified clearly. Besides, the theoretical grounding and experimental evaluation are also sufficient to show its originality and significance.  
2) Experimental results also demonstrate the superiority of the proposed TEBN method. Through Fig. 1, we can see the advantages of the proposed TEBN Intuitively compared to tdBN.
3) This paper gives a detailed illustration of how TEBN processes in a surrogate-gradient SNN training method. If the authors show the input and output simultaneously with or without TEBN, the effectiveness maybe better in a visual way. 
4) The spiking nondifferentiable leads to a non-smooth with the loss landscape, the authors analyze the smoothing optimization in sec. 5 which I think is very important for using the surrogate-gradient or ANN2SNN method training a deep SNN model.

Weaknesses: 
How does internal covariate shift appear in SNN? In my opinion, because of the different scales from each layer's output, ICS in ANN  is obvious, however, in this paper, when you used the time-series LIF neuron model, the ICS should be changed in the time window. When using rate-based coding, the spikes are more intensively compared to temporal coding, how does TCS behave in addition to sec. 4.1? 

questions: 
1) Line 40, the sentence, ‘Theoretical analysis shows that our approach can be viewed as a smoother of SNN’s optimization landscape and could help stabilize the gradient norm.’ need a revision as it is repetitive with the statement in the abstract. 
2) Line 65, please add a citation to support the statement (less than 15 layers). 
3) Page9, Figures 2 and 3. The X and Y limit should be the same for both figures since the authors could compare distributions. Besides, I expect the author to conduct some quantitative measures to compare the distributions, rather than directly state that TEBN appears more homogeneous.
4) I would recommend the author validate the efficiency of mitigating gradient explosion and vanishing (Line 215) with the proposed TEBN using a deeper model (e.g., ResNet-50).

limitations: The efficiency of mitigating gradient explosion and vanishing (Line 215) with the proposed TEBN is not validated efficiently in this work. And what is the biological meaning of the hyper-parameters in LIF model? Or if the authors explain how different hyper-parameters influence TEBN with theoretical analysis not only experimental evaluations, the results would be more sound.
review_title: null

Meta-review: metareview: The paper proposes a method of batch normalization that takes into account the temporal dimension (TEBN) and empirically shows that TEBN can significantly improve the accuracy of spiking neural networks (SNNs).  Theoretical analysis also provides new insights into how SNNs should be trained to improve accuracy (particularly in the face of temporal variation of internal covariate shift).

This paper had received conflicting evaluations from Reject to Strong Accept, and the reviewers did not reach a consensus even after fairly intense discussion.  The disagreement appears to come from what ones expect from SNNs: accuracy, robustness, latency, sparsity, biological plausibility, etc.  While it is well empirically supported (particularly with the additional experiments during rebuttal) that TEBN increases accuracy, TEBN certainly looses biological plausibility, and the operations such as variance computation needed in TEBN might not be desirable for some applications of SNNs.  Also, since much of the experiments are added during rebuttal, there is a criticism for the lack of consistency in experimental design, which also leads to mixed evaluations regarding the benefit of the proposed approach.

Overall, despite several weaknesses and uncertainties, the high accuracy certainly matters to some of the users and researchers of SNNs, and the paper clearly excels in this regard.  Hence, I recommend an acceptance.
confidence: Less certain
Decision: Accept