Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose an online learning algorithm (OTTT) that's applicable to spiking neural networks (SNNs). OTTT uses a combination of eligibility traces (trace of past activity of the neuron) and use of instantaneous loss values to achieve an online algorithm. The authors show its connection to spike representation based methods as well as three-factor learning rule and demonstrate empirically that this method works better than existing methods for training feedforward spiking neural networks.

strengths_and_weaknesses: ## Strengths

The paper derives an novel online learning rule, primarily applied to feed-forward SNNs. The connection to spike representation derived in the paper is very interesting, and connects these seemingly disparate methods. 

The empirical results are impressive, and strongly support the utility of OTTT. The fact that OTTT works with batch size of 1 is particularly impressive, and makes it of strong practical interest for neuromorphic hardware. 

The paper is also well written and the exposition is clear and easy to understand.

## Weaknesses

The results of OTTT with feedback connections and with smaller batch sizes needs more baselines, esp. compared to BPTT to understand how it stands in comparison to conventional methods. i.e. to understand if the advantage there because of OTTT or other reasons?

It would also have been interesting to see results with fully recurrent architectures.

Minor: The connection with spike representation methods is certainly interesting, but might be a bit over-emphasized and could be confined to a single section so that it doesn't interrupt the flow of OTTT. Specifically, the Spike Representation heading in Sec. 3.2 threw me off a bit, since I was looking for a connection with Sec. 4.1.
questions: ## Questions

* Does $s[\tau]$ in line 141 refer to the output spikes of each layer?
* Can OTTT also be applied to non-spiking neural network architectures?

## Suggestions

* In the abstract:
    * l.5: it's not clear why BPTT with SG leads to extremely low latency. Rephrase?
    * l.6: theoretical unclarity -> lack of theoretical clarity.
    * l.11: what does it mean for online learning to be a learning rule? Rephrase?
* l.32: The supervised training of SNNs is challenging not because of its "complex neuron model" but because of non-differentiability in the neuron model
* Sec 2.  I would suggest giving brief descriptions of individual references rather than grouping large numbers of references together with vague sentences, which doesn't convey much useful information about related work.
* Related work that uses eligibility traces should be cited: e.g. (Murray 2019; Bellec et al. 2020)
* l.142: A few details of how this can be proved would be useful.
* Table 1: It would be helpful to see comparison with equivalent non-spiking architecture trained with BPTT as an (upper) baseline to put the performance of OTTT in perspective.

Bellec, G., Scherr, F., Subramoney, A., Hajek, E., Salaj, D., Legenstein, R., Maass, W., 2020. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications 11, 3625.

Murray, J.M., 2019. Local online learning in recurrent networks with random feedback. eLife 8, e43299.
limitations: The authors do not discuss the limitations of the work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors introduce a new approach to training spiking neural networks, online training through time (OTTT). The approach only requires constant memory since it does not need to backpropagate through time. Instead, tracks presynaptic activities and leverages instantaneous loss and gradients. The proposed approach provides good results on typical benchmarks such as CIFAR-10 and 100.


strengths_and_weaknesses: Strengths:
* How to best train SNNs is still an open problem, especially if one aims for online learning on neuromorphic hardware. This article can form a substantial step in that direction.
* If I understand the mathematical proof well, the authors show that their method's gradient points in the same direction as the BPTT one, without needing to track many variables in memory.
* The results of the approach seem good.

Weaknesses:
* The work currently does not discuss other SNN  approaches that also do not backpropagate gradients through time, such as: Accurate online training of dynamical spiking neural networks through Forward Propagation Through Time, B Yin, F Corradi, SM Bohte - arXiv preprint arXiv:2112.11231, 2021https://arxiv.org/pdf/2112.11231.pdf
* Some of the main intuitions behind not needing to backpropagate through time remain unclear (see questions).
*  The tasks to which the approach is applied are quite "static", and do not immediately seem to need much memory. Moreover, the approach seems to rely on neurons moving to an equilibrium state, which would be less valid for real-world, time-varying data.
questions: 1. How does the authors' work relate to other approaches that implement forward propagation through time?

2.  What are the effects of the assumptions on the proof? For instance, in line 167 the reset moment is ignored. Line 147-150: Does the equilibrium condition depend on constant inputs? And why can only equilbria happen and not limit cycles? 

3. The "instantaneous" loss is defined as a sum over time (lie 184). How is that instantaneous and the definition higher up (line 182) not? They both sum over time, although 1/T is inside the loss function higher up.

4. Intuitively, what is remembered in the trace that allows to bypass remembering all activations over time? 

5. What are the gradients of the "spike representation"? Are these the gradients from BPTT?

6. What is meant with "time steps" in Fig 2? The time steps BPTT goes back in time? How does this apply to OTTT?

limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents online training through time (OTTT) for SNNs, which is derived from BPTT to enable learning without backward passes through time. The goal compared to other approaches is to improve memory consumption, biological plausibility, or latency. Theoretical analysis shows that the gradients of OTTT are in a direction that is suitable for optimization. Experiments show that, compared to BPTT, OTTT has an advantage in terms of memory consumption, especially as training sequence lengths increase.
strengths_and_weaknesses: The paper addresses a topic that has attracted a lot of interest by the community. It also includes significant theoretical analysis and has given effort in explaining the derivations.

On the other hand, the paper:
- does not even mention some recent methods (FPTT, OSTL) [1, 2] that have similar goals, let alone explain the differences. Therefore the extent of the advance in this paper cannot be evaluated. 
- does not provide experimental comparisons to such methods as [1-3], but only to BPTT
- if any advantage - theoretical or other - exists compared to these prior methods, the paper does not summarise clearly what element in the new theoretical derivation enables that advantage.
- the work does not aim to achieve something brand new, as successful SNN training has already been possible. Therefore its impact is expected to be limited.
- does not show results on sequential tasks, but only on image classification, even though the learning rule concerns recurrent networks
- does not compare experimentally with non-spiking networks, in terms of accuracy, memory, or computational efficiency.
- could improve its writing, by removing redundant repetitions, providing a sketch/summary of the derivation before the derivation, and by clarifying terms such as forward-in-time

[1] Yin, Bojian, Federico Corradi, and Sander M. Bohte. "Accurate online training of dynamical spiking neural networks through Forward Propagation Through Time." arXiv preprint arXiv:2112.11231 (2021).
[2] Bohnstingl, Thomas, et al. "Online spatio-temporal learning in deep neural networks." IEEE Transactions on Neural Networks and Learning Systems (2022).
[3] Bellec, Guillaume, et al. "A solution to the learning dilemma for recurrent networks of spiking neurons." Nature communications 11.1 (2020): 1-15.
questions: Could the authors explain what are their expectations from the comparisons and analyses that are missing, if they were performed?
Could these actually be peformed?
limitations: There is no discussion of limitations by the authors.
review_title: null

Meta-review: metareview: The authors propose an online training algorithm (OTTT) for spiking neural networks (SNNs) using eligibility traces and instantaneous loss values. They show empirically that this method performs better than previous ones in feed-forward spiking neural networks.

All reviewers agree that the empirical results are impressive and that the method is interesting for neuromorphic hardware. The authors also provide a mathematical analysis of the learning method.

Weaknesses:
- Networks are mostly applied to static tasks, while more temporal tasks are potentially more interesting for SNNs
- Comparison to previously proposed methods is missing.

In general, a very interesting and strong paper. I propose acceptance.
confidence: Certain
Decision: Accept