Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: - This paper investigates the ability of existing bio-plausible temporal credit assignment rules to generalize. Finding that training with bio-plausible learning rules results in models that have larger and more variable train/test generalization gaps. 
- It then compares the loses’ hessian eigenspectrum of bio-plausible and SoTA learning rules finding that bio-plausible rules tend to approach high curvature regions in synaptic weight space
- It suggest an explanation for bio-plausible rules preference for high curvature regions based on worse alignment to the true gradient
strengths_and_weaknesses:  Strengths
- This paper is well written and clear
- It analyzed several learning rules and the results support the conclusion that bio-plausible methods don't generalize well
- The theoretical results offer interesting insights about the relationship between curvature and gradient alignment

Weakness
- It is well known that bio-plausible methods generally perform worse than SOTA back propagation. This work provides further empirical evidence that this is the case, but I do not find these results very surprising.

questions: Are there other advantages to bio-plausible methods or advantages in terms of generalization can only be seen on more complex tasks?
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors investigate the difference in the generalization gap in RNNs trained to minimize MSE loss with backpropagation through time and truncated BPTT. The authors claim the difference is caused by the landscape curvature of the loss, specifically, truncate BPTT approaches high-curvature regions in the synaptic weight space. The authors propose a theoretical argument to explain this phenomenon based on the first Hessian eigenvalue. The authors claim this result holds for all existing bio-inspired learning rules in RNNs and for different loss functions. 
strengths_and_weaknesses: Strengths:

* The authors address the important issue of how biological systems learn and adapt to stimuli. Specifically, the authors address that RNNs trained with truncated BPTT have a greater generalization gap than the full BPTT on three existing tasks.
* The authors focus on a well-defined metric, the 1st Hessian eigenvalue, in order to quantify their results.
* The authors provide a theoretical analysis that suggests a cause for the difference in the generalization gap between full BPTT and truncated BPTT

Weaknesses:
* The authors generalize their claims to all biologically-inspired learning rules but limit their study to three specific learning rules and three tasks. Specifically, the authors focus on truncated BPTT in RNNs on three tasks with MSE loss. These results may hold only for truncated backpropagation through time but not for all biologically-plausible learning rules. The style implies these results hold for all biologically-inspired learning rules and all loss functions. If this is true, it should be presented in the main text.  
* The authors claim their results generalize beyond MSE loss but do not demonstrate this empirically. 
* The authors make claims of potential remedies, such as using larger learning rates early in training but do not justify this claim theoretically or demonstrate it empirically. 
* It is uncertain and not addressed if the generalization gap can be reduced by fine-tuning existing truncated BPTT algorithms.
* A description of training parameters and procedure used for the biologically-plausible learning rules is not presented in the main test.
* The authors cite almost 100 articles in section 2.1. Though these articles may be relevant to biologically-plausible learning in general it is not certain they are directly relevant to networks trained with truncated BPTT which is the focus of this study. 



questions: Were potential remedies tested?

In figure 2 (a, b, c), does random initialization refer to weight initialization? Are all parameters of the learning algorithm held constant or are the distributions computed across all parameter configurations, e.g., learning rate ..etc? It appears that truncated BPTT can achieve the same generalization gap as BPTT. In the case of sMNIST it appears that it achieves zero generalization gap with a greater probability than full BPTT. 
limitations: NA
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The  paper aims to study the generalizability of biologically plausible learning rules for RNNs compared to backprop through time which is not bio-plausible. This work relies heavily on the relationship between the curvature of the loss landscape and the generalizability of the model: that wider minima are more noise tolerant and generalize better. Empirically it is shown that bio-plausible learning rules generalize worse than the common implausible ML learning rules. This is then theoretically explained as being due to the instability introduced by the truncation of the true backprop gradient being used by the bio-plausible rules. Specifically, to avoid diverging with the bio-plausible rules a smaller learning rate can be used which makes gradient descent more susceptible to converge to steeper minima.
strengths_and_weaknesses: # Strengths
## Originality
This paper addresses a topic which I have not seen in the computational neuroscience literature and does so with care for both the machine learning and computational neuroscience considerations. I think the primary novelty of this work however comes in at the end with the finding that the noise from truncation does not affect generalization in the same manner as noise from smaller batch sizes for example (truncation noise hinders generalization while other forms of noise improve generalization by driving learning out of sharp minima). This is a new consideration to me and lead me to start thinking around whether the consistency of noisy direction of learning plays a role.

## Quality
There are no points during the paper which stand out as being unreasonable, and from a scientific method point of view the work appears sound, with each step being justified either from a comp-neuro or ML perspective. Balancing the two topic was done well and I think that is a potentially understated positive for this work.

## Clarity
The paper is well written and I found it to be clear and concise. The figures are well made and helpful.

## Significance
The work appears significant from two perspectives. Firstly, work at the intersection of comp-neuro and ML is important and potentially very fruitful, however, not very common (at least not at the level of insight for both fields offered by this paper). Thus, I think this is significant to the comp-neuro community as a step towards introducing some of the more theoretical ML concepts such as loss landscape curvature. Secondly, this appears significant as it brings up the interesting concept that some noise may help generalization while other forms of noise hurt generalization. It does so in a manner which is also very useful from a learning stability perspective which also uses the Hessian matrix [1]. Thus from a theoretical ML perspective I think this work offers interesting insight and potential inspiration for future work.

# Weaknesses
## Originality and Significance
It must be noted that this work relies heavily of previous work in theoretical ML and tends to confirm what is already known (wider minima generalize better). That said, I think just because the results are not surprising does not mean they are not valuable. I am glad that this work has been done and acknowledge that it was worthy of confirming that the findings from ML carried towards bio-plausible learning rules. Indeed it seems to me that this work can still inspire future directs of ML work as I point out above. Thus I feel the strengths for originality and significance do outweigh this weakness.

That said, if the authors were to add to the discussion at the end about truncation noise reducing the stable step size I would be inclined to increase my rating to a 7 or an 8. The one citation below may be of use for this purpose. I do think that this paper is worth of acceptance and is of general interest to the machine learning and comp-neuro communities.

[1] Cohen, Jeremy M., et al. "Gradient descent on neural networks typically occurs at the edge of stability." arXiv preprint arXiv:2103.00065 (2021).
questions: Assuming I have understood the work and not misrepresented the fact above, I have no questions for clarification at present. For the rebuttal period I would find it helpful for the authors to further address the concept of different kinds of noise impacting generalization differently. 
limitations: The authors were clear that this study was not comparing bio-plausible learning rules but rather confirming that a well-established concept in ML carried to bio-plausible rules. This is the main limitation but it was clearly acknowledged. They also acknowledge that the relationship between curvature and generalization, while being the main point of study in this work, is messy and state that other factors may be at work for generalization.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investigates generalization and loss landscape curvature in RNNs trained with biologically-motivated learning rules. The paper finds empirically that generalization gaps are correlated with loss landscape curvature, with higher curvature correlating with larger generalization gaps. Next, the paper shows theoretically that approximations to gradient descent produce more curved solutions than gradient descent. This is because gradient descent approximations take smaller steps in the direction of the gradient and therefore are unable to escape highly curved local minima. The component of updates orthogonal to the gradient is less relevant to generalization. Experiments on multiple datasets support this explanation. Finally, the authors demonstrate empirically that appropriate learning rate scheduling in biologically-plausible learning rules can significantly enhance performance.
strengths_and_weaknesses: **Originality**
Although generalization in biologically-motivated learning rules has been studied previously, this paper provides surprising new insights into the nature of solutions found by biologically-plausible learning rules. Notably, to my knowledge, this paper is the first to quantitatively explain with high precision the relatively poor generalization performance of biologically-plausible learning rules (see Figure 4). 

**Quality**
The theory is sound and well justified, and the experiments are generally comprehensive. The authors may want to consider some additional experiments to sharpen their claims. 

First, the authors may want to investigate whether generalization gaps correlate with Hessian eigenvalues in networks *trained with the same rule* (rather than aggregating results across rules as done in Figure 2). This would help demonstrate the generality of the claim that generalization is tied to loss landscape curvature. 

Second, the authors may want to consider an additional variant of the three-factor rule where the update sizes are scaled such that the gradient-parallel component matches that of gradient descent. As the authors argue in equation, this can cause numerical instability. Nevertheless demonstrating this empirically would be helpful and would nicely complement the "three-factor, theory" rule.

Finally, the paper seems to suggest that one of the main limitations of biologically-plausible learning rules is that using large update sizes for them leads to numerical instability while small update sizes leads to poorly generalizing local minima. Further explaining at which point this numerical instability occurs would be valuable (e.g. is the maximum stable learning rate limited by the eigenvalues of the Hessian?).


**Clarity**
The paper is well written. The figures are particularly well illustrated and clear. The mathematical notation is generally adequately introduced and used.

Some minor issues:
The W+ and W- notation in equations 7 and 8 is not explained.
Typo- menchmark on line 140.
Typo- assymptotic on line 1054.


**Significance**
The paper may be quite significant to the field of biologically-plausible learning in RNNs. One of the important takeaways from the paper is that learning rate modulation of biologically-plausible learning rules may be a key to good performance. This may spur further research into developing better learning rate schedules for biologically-plausible learning rules. Furthermore, this paper may have implications for deep learning theory more generally.
questions: This paper considers the generalization gap of different algorithms by roughly fixing their training performance and comparing their test performance. How would the results change if the test performance were fixed and the training performance were varied (by stopping the rules at different points during training, for example)?

Do generalization gaps correlate with Hessian eigenvalues in networks trained with the same rule?

How would a three-factor rule perform empirically when scaled to match the along-gradient update sizes of gradient descent?

At what point does numerical instability occur in biologically-plausible learning rules?
limitations: The authors adequately address the limitations of the work. As the authors note, they do not consider varying architectures and leave the interaction between architecture and generalization as future work to be investigated. Moreover, the authors note that the loss landscape curvature does not fully explain generalization.
review_title: null

Meta-review: metareview: This paper applied ideas about generalization in the ML literature to biologically plausible architectures and learning rules. Especially, it explored links between curvature and generalization in biologically plausible learning.

There was active discussion about this paper, and three reviewers raised their scores during the rebuttal period. All reviewers felt this was a high quality paper, and that the results would be useful for later research. The closest thing to a criticism that came up during discussion was one reviewer describing the paper as a "high quality but incremental addition to the scientific literature."

Based upon the reviews, rebuttal, and reviewer discussion, I recommend paper acceptance. The authors should be sure to update their paper as discussed during the rebuttal period, and based upon the reviewer feedback.

PS -- Links between TBTT and loss surface curvature would also be of interest in learned optimization and meta-learning more broadly, where meta-training is often performed via truncated unrolls of the inner problem.
confidence: Certain
Decision: Accept