Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose a biologically-plausible temporal credit assignment rule for recurrent neural networks called ModProp. This work is motivated by recent experimental evidence on the presence of local neuromodulatory networks in the brain. Here, they propose that these synapse-type-specific local modulatory signals are received via low-pass filtering of eligibility traces at post-synaptic neurons.  
Using the framework of discrete-time rate-based RNNs, they derive the ModProp synaptic weight update rule and describe its properties. They also provide simulation results comparing the performance of ModProp vs. other biologically-plausible learning rules and Backprop through time on three temporal processing tasks. ModProp outperforms other previously proposed learning approaches, indicating that it is a promising candidate for understanding how the biological networks perform temporal credit assignment. 


strengths_and_weaknesses: This paper addresses the important question of how credit assignment might occur in biological recurrently connected neural networks. Here they explore the potential role of local neuromodulatory signaling mechanisms and propose the ModProp learning rule. The idea of synapse-type-specific local modulatory signaling complementing Hebbian learning and global neuromodulation does seem like a viable theory. Backed by the empirical results presented here, I think ModProp is a promising candidate for biologically plausible learning in recurrent networks and warrants further exploration. 

Although ModProp can technically perform temporal credit assignment over arbitrarily long durations, the influence of distant events in time is negligible. This is akin to the problem of vanishing gradients in backpropagation through time, which makes ModProp ill-suited for very long-term credit assignment. The authors do mention this drawback in the discussion section.

I think this paper is quite constrained by the page limit. Despite this constraint, I believe the authors have done a reasonably good job of describing the important concepts involved in the proposed learning rule.  
Minor suggestions: the main paper refers to certain equations (e.g. 14) and figures that are in the supplementary material. It might be helpful for the readers to point out that this content is in the methods/supplementary material. 

questions: - Have you tried the delayed XOR task with different delays between the cues? How would the performance of ModProp vary as a function of this delay?

limitations: The authors adequately address the limitations and the potential long-term societal impact of their work in the discussion section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Evaluating the parameter gradients of recurrent neural networks forward in time ([17]) is both costly and not biologically plausible. This paper introduces novel approximations to the synaptic weight gradients of recurrent neural networks, which can be evaluated forward in time, using procedures that are both less costly and more biologically plausible than full RTRL. The authors discuss possible biological implementations based on plasticity modulation and compare the performance of their method to relevant baselines in a number of experiments.
strengths_and_weaknesses: Finding biologically plausible learning rules which improve upon e-prop/RFLO is a very important research topic. The decomposition of the gradient and approximations proposed in this paper are a sensible approach forward, similar in spirit (but different from) recent algorithms such as SnAp and KeRNL.

My major concern with the current paper is the somewhat weak experimental evaluation. I leave some additional comments for the authors below, on points where I think the paper could be improved:

- The presentation of section 3.1 could be improved. In particular the proposed biological implementation (an essential point of the present work) should be clarified and discussed in greather depth. [Small additional note: there is a problem with equation references; for example, in line 140, Eq. 14 (supplemental material) is referenced instead of Eq. 2.]

- Weakness of experimental part of the paper: the paper would substantially improve if the authors demonstrated the superiority (and the cases where the different approximations fail) of modprop over e-prop in benchmarks that are not as toyish as pattern generation and delayed XOR, but perhaps not as difficult as sequential MNIST. Perhaps the copy task (with varying sequence length) is a good additional synthetic benchmark? In particular, it would be very good to assess how well the update with random modulatory weights performs in more difficult problems. Given that the focus of this work is on introducing a new approximation to the full gradient on RNNs, the somewhat weak experimental study is the major current weakness of the paper.

- Given that there is a section on efficient computer implementations, a small application (or at least a discussion) to LSTM-like models and a comparison to algorithms such as SnAp (which also reduce the bias compared to e-prop/RLFO) would also enrichen the paper and help understand how well the approximations introduced here impact performance, as well as show their potential usefulness in machine learning, beyond their biological plausibility merits.
questions: - How would the matrix power which appears in the weight update be calculated in a biological network? Even when averaging over connections of the same type, it's unclear to me how plausible this operation is.

- Can the authors discuss in more detail how to approach the problem of determining cell types? For example, could it make sense to group by time constants, when there is heterogeneity in time constants? Can the authors provide some heuristics or intuitions on how they expect such choices to affect their rule?

- Can the authors discuss KeRNL and SnAp in more detail?

- As discussed in the strengths & weaknesses section, can the authors extend their experimental evaluation to more clearly demonstrate the superiority over e-prop/RFLO, including the random modulatory version?
limitations: Apart from the relatively weak experimental evaluation of the proposed approximations, the authors have done a good job in discussing the limitations of the current work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a method of temporal credit assignment method for training recurrent neural networks, which is an approximation of real time recurrent learning. Their method can be implemented in a biologically plausible neural network with neuromodulators.
strengths_and_weaknesses: Strengths:

The authors tackle and challenging and important problem in theoretical neuroscience: biologically plausible training of RNNs (though an area I'm not very familiar with). I find their general approach quite interesting and it appears to suggest (potentially testable) predictions about the role of neuromodulators. Overall, I think it is a useful contribution to the field.

Weakness:

The derivation of their algorithm is highly technical and requires a lot of notation. I found it challenging to follow their arguments. First, the main text includes a number of forward references to the appendices, and then the equations in the main text and appendices are not aligned. Sometimes this appears due to some approximation that is made, but other times it's not clear to me. I think overall their technical arguments could be presented with more clarity. I also think that the authors could devote more space to explaining the approximations that they make, and when these approximations are valid or not.
questions: Numerics:

- Why don't you compare your algorithm with RTRL since it's an approximation of RTRL? Wouldn't that help isolate if the performance gap with BPTT is due to the approximations or because RTRL isn't as good as BPTT?

Relation to biology:

- I think it's worth further discussing the relationship of this work to [6]. Both works use cell-type specific neuromodulation, but the similarities and differences of the biological interpretations are not clearly stated.

Derivation: I think the clarify and precision of the technical arguments can be improved, especially for a novice like me who is not familiar with RTRL.

- Should Eq. (1) and Eq. (10) be the same?
- Line 129: What are the mathematical definitions of $\partial$ and $\text{d}$? One involves the chain rule and the other doesn't?
- Line 134: There are a number of forward references to equations in the appendix, which makes the paper difficult to read on its own.
- Eq. (3): Should "$\text{d}$" be "$\partial$"? 
- Is $z=\text{ReLU}(s)$? It seems like that needs to be true for Eq. 3 to hold. Why not say so in the main text?
- Line 143: Is it reasonable to assume that neural activities and weights are uncorrelated? Do you have experimental evidence or a reference? My naive impression would be that this is *not* true.
- Line 149: Based on Eq. (1), should $e_{pq,t}$ be $\frac{\text{d}s_{p,t}}{\text{d}W_{pq}}$? And wouldn't that be $\frac{\partial s_{p,t}}{\partial W_{pq}}$ in this case?
- Eq. (11): Should $\frac{\text{d} z_{j,t}}{\text{d} s_{j,t}}$ be $\frac{\partial z_{j,t}}{\partial s_{j,t}}$? Is $h_{j,t}$ simply the derivative of the ReLU function evaluated at $s_{j,t}$? 
- Line 568: A more detailed justification for approximation (a) would be greatly appreciated. I see why the approximation is exact in the linear setting. What does it mean for the $W$ and $h$ chains to be uncorrelated? Is that a reasonable assumption in the ReLU setting?
limitations: The authors make a number of approximations when deriving their algorithm and I am interested to better understand to what extend these approximations affect the performance of their algorithm. In the discussion section, the authors suggest investigating their algorithm in situations where the assumptions are violated.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: ### UPDATE 08/08/2022: Contribution score increased.


### UPDATE 08/05/2022: Score increased slightly after reading the response and updated version of the submission. 


This paper proposes a biologically plausible implementation of training recurrent neural networks as a biologically plausible alternative to back-propagation through time and already existing less plausible algorithms. This works lays out the approximation necessary for making the model believable as to what type of cell could be implementing it. The paper also proposes some experiments to valide the model. 
strengths_and_weaknesses: The paper is well-written, and results and theorems are nicely explained using informal explanations and leaving the full proof to the appendix. The paper is somewhat novel and addresses an important question, which is often less addressed than that of backprop in non-recurrent networks. 

Unfortunately, the "framework" proposed seems to be more of a patch of approximations, each solving various problems rather than a mathematically grounded approach. The paper offers a long introduction of what are the possible candidates for their algorithms to justify this work, but it is not validated, nor any predictions are made that could make us believe that it is at all plausible. 

The arguments for each approximation are mostly left in the appendix, which is the core of the paper. I would have wanted to be shown in the paper why they are possible and not have to trust the authors and move on. I observe mainly the discrepancy between the claims of the paper and what is actually shown in the paper. 


questions: What predictions can be made based on your model of the modulating candidates? 

How do you propose to present the main justification of the important assumptions that make the model plausible? 


limitations: They have addressed some of the limitations.
review_title: null

Meta-review: metareview: Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators 

The authors propose a biologically plausible method for temporal credit assignment called ModProp. They apply their framework on rate-based recurrent neural networks (RNNs), and show that it outperforms previous approaches.

All reviewers acknowledge that this work studies an interesting topic in computational neuroscience. The authors present compelling experimental results on synthetic data sets.

Weaknesses:
- Long-term dependencies cannot be tackled by this approach, which is quite common also for related approaches. 
- Somewhat weak experimental evaluation. Evaluation on more complex standard data sets would be beneficial.
- The arguments for the used approximations is left in the appendix.

In general, an interesting study with good experimental results. I propose acceptance.
confidence: Certain
Decision: Accept