Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules
Authors: Yuhan Helena Liu
Arna Ghosh
Blake Aaron Richards
Eric Todd SheaBrown
Guillaume Lajoie
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neuroscience
computational neuroscience
biologically plausible learning rules
learning and plasticity
temporal credit assignment
generalization
recurrent neural networks

Abstract: To unveil how the brain learns, ongoing work seeks  biologically-plausible approximations of gradient descent algorithms for training recurrent neural networks (RNNs). Yet, beyond task accuracy, it is unclear if such learning rules converge to solutions that exhibit different levels of generalization than their non-biologically-plausible counterparts. Leveraging results from deep learning theory based on loss landscape curvature, we ask: how do biologically-plausible gradient approximations affect generalization? We first demonstrate that state-of-the-art biologically-plausible learning rules for training RNNs exhibit worse and more variable generalization performance compared to their machine learning counterparts that follow the true gradient more closely. Next, we verify that such generalization performance is correlated significantly with loss landscape curvature, and we show that biologically-plausible learning rules tend to approach high-curvature regions in synaptic weight space. Using tools from dynamical systems, we derive theoretical arguments and present a theorem explaining this phenomenon. This predicts our numerical results, and explains why biologically-plausible rules lead to worse and more variable generalization properties. Finally, we suggest potential remedies that could be used by the brain to mitigate this effect. To our knowledge, our analysis is the first to identify the reason for this generalization gap between artificial and biologically-plausible learning rules, which can help guide future investigations into how the brain learns solutions that generalize.

Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules
Yuhan Helena Liu Arna Ghosh Blake A. Richards4367, Eric Shea-Brooe Biea-Brrwn!... 1.23, and Guillaume L7,.
Department of Applied Mathematics, University of Washington, Seattle, WA, USA Allen Institute for Brain Science, 615 Westlake Ave N, Seattle WA, USA
Computationall Neuroscience Center, University of Washington, Seattle, WA, USA School of Computer Science, McGill University, Montreal, QC, Canada 'Mila- Quebec AI Institute, Montreal, QC, Canada
Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University,
Montreal, QC, Canada
"Canada CIFAR AI Chair, CIFAR, Toronto, ON, Canada Dept. de Mathématiques Statistiques, Université Montréal, Montreal, QC, Canada "Correspondence: hyliu24 uw.edu, g.lajoie @umotteel..caaa
Abstract
To unveil how the brain learns, ongoing work seeks biologically-plausbi ap- proximations of gradient descent algorithms for training recurrent neural net- works (RNNs). Yet, beyond task accuracy, it is unclear if such learning rules converge solutions that exhibit different levels of generalization than their non- moologically-plausbbe counterparts. Leveraging results from deep learning theory based on loss landscape curvature, we ask: how do biologically-plausible gradient approximations affect generalization? We first demonstrate that state-of-the- peglooically-plassibbe learning rules for training RNNs exhibit worse and more vari- able generalization performance compared to their machine learning counterparts that follow the true gradient more closely. Next, we verify that such generalization performance is correlated significantly with loss landscape curvature, and we show that baglogically-plauibbee learning rules tend to approach high-curvature regions synaptic weight space. Using tools from dynamical systems, derive theoretical arguments and present a theorem explaining this phenomenon. This predicts our numerical results, and explains why biologically-plauseee more variable generalization properties. Finally, we suggest potential remedies that could be used by the brain to mitigate this effect. To knowledge, is the first to identify the reason for this generalization gap between artificial and moologcallly-llusibbe learning rules, which can help guide future investigations into how the brain learns solutions that generalize.
rules lead to worse and
analysis
Introduction
longstanding question in neuroscience is how animals excel at learning complex behavior involving temporal dependencies across multiple timescales and thereafter generalize this learned behavior to unseen data. This requires solving the temporal credit assignment problem: how to assign the contribution of past neural states to future outcomes. To address this, neuroscientists turning to the mathematical framework provided by recurrent neural networks (RNNs) to model learning mechanisms in the brain 21 Temporal credit assignment in RNNs is typically achieved by backpropagation through time (BPTT), or other gradient descent-based optimization algorithms,
increasingly
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

none of which are petoogicalll-llassibe (or bio bio-blausioe plausible for short). Therefore, the use of RNNs as framework to understand the computational principles of learning in the brain has motivated an influx of bio-plausible learning rules that approximate gradient descent [03
The performance of such rules is typically quantified by accuracy. Although the accuracy achieved by these rules is often comparable to true gradient descent, little is known about the breadth of the emergent solutions, namely how well they generalize. Broadly speaking, generalization refers to a trained model's ability to adapt to previously unseen data, and is typically measured by the so-called generalization gap: the difference between training and testing error. This is especially important when learning complex tasks with nonlinear RNNs where the loss landscape is non- convex, and therefore, many solutions with comparable training accuracy can exist. These solutions, characterized as (local) minima in the loss landscape, can nonetheless exhibit drastically different levels of generalization (Figure [. It is not clear if gradient- methods like BPTT and the existing bio-plausible alternatives have a different tendency to converge to loss minima that provide better or worse generalization.
While the search for better predictors of the generalization performance remains an open issue in deep learning research 4 recent extensive studies identify flatness of the loss landscape at the solution point as a promising predictor for generalization [5H)]. Leveraging these empirical and theoretical findings, we ask: how do proposed biologically-motie motivated gradient approximations affect the flatness of the converged solution on loss landscape, and thereby, generalization?
Our overarching goal is to investigate generalization trends for existing bio-plausible temporal credit assignment rules, and in particular, examine how truncation- based bio-plausible gradient approximations can affect such trends. Specifically, our contributions summarized as follows:
In numerical experiments, we demonstrate across several well-known neuroscience and machine learning benchmark tasks that sfate-of-the-art (SoTA) bio-plausible learning rules for training RNNs exhibit worse and more variable generalization gaps, compared to true (stochastic) gradient descent (Figure -ACC)
Using the same experiments, show that bio- dio-plausible learning rules tend to approach high- curvature regions in synaptic weight space as measured by the loss' Hessian eigenspectrum (Figure Further, we verify that this correlates with worse generalization (Figure[p.F and 3 which is consistent with the literature.
We present a theorem to explain this phenomenon by examining the weight update equation as a discrete dynamical system, which sheds light on potential connection between gradient alignment and the preference over converging to narrower minima (Theorem [ Figure
Given the core components in designing artificial neural networks: data, objective functions, learning rules and architectures [10), we investigate different learning rules while holding the data, objective function and architecture constant. SoTA RNN learning rules investigated include three-factor rule with symmetric feedback (symmetric e-prop [T], a three-factor rule using random feedback weights (RFLO [12) and multi-factor rule using local modulatory signaling (MDGL) [3] For an in-depth explanation of how these rules are implemented and why they are bio-plausible, please refer to Appendix A.2 We also encourage the reader to visit Appendix B for Theorem proof and discussion on loss landscape geometry. In the last paragraph of the Discussion section, we discuss potential remedies implemented by the brain and provide preliminary results (Appendix Figure 5) To our knowledge, our analysis is the first to highlight and quantitatively provide a mechanistic explanation of the reason for this gap in solution quality between artificial and bio- plauplluebble learning rules for RNNs, thereby motivating further investigations into how the brain learns solutions that generalize.
2
Related works
2.1 Bio-plausible gradient approximations
Investigating bio-plausible learning rules is of interest both to identify more efficient training strategies for artificial networks and to better understand learning in the brain 2 3 14 In order for learning to reach certain goal quantified by an objective, learning algorithms often minimize a loss function 100] The error gradient, if available, tells us how each parameter should be adjusted in

---

A
RNN model
gradient graph
Example gradient graph
Error/loss
Target readout/ reward prediction
Aw
dE Flow of computation for
Learning achieved by partially following the gradient
Minima preference
?
Following gradient (BPTT)
Param
Partially follawing lapproximate rule)
Param
Min loss
(worse)
(betteri
Max
Figure 1: Setup. A) Illustration of an RNN trained to minimize error/loss function L (left). Ex- isting bio-plausible proposals for RNNs estimate the gradient by neglecting dependencies that are biologically implausible compute (right). B) Low training error/loss can be achieved by partially following a gradient (right), but the preference for converging to minima with certain generalization properties remains underexamined for these learning rules (left). C) Minima flatness matters: 1-D loss landscape illustration with two solutions that equally minimize loss L, but exhibit drastically different generalization properties: the narrower minima are more sensitive perturbation.
order to have the steepest local descent. For training RNNs, which are widely used as a model for neural circuits 63), standard algorithms that follow this gradient real time recurrent learning (RTRL) and BPTT are not bio- bio-plausible and have overwhelming memory storage demands B 64] However, learning rules that only approximate the true gradient sometimes be effective those that follow the gradient exactly [10 [65) Because of that, bio-plausible learning rules that approximate the gradient using known properties of real neurons have been proposed and led successful learning outcomes across many different tasks feedforward networks [i 66)555) with recent extensions to recurrently connected networks [TH These existing bio- plo-plausible rules for training RNNs [[I- 13 are truncation-base (which is the focus of this study), so that the untruncated gradient terms can be assigned with putative identities known biological learning ingredients: eligibility traces, which maintain preceding activity on the molecular levels [1). combined with top-down instructive signaling 766
77 82 88] as well as local cell-to-cell modulatory signaling within the network [13 89 90]. For efficient online learning in RNNs, other approximations (not necessarily bio tio-plausible) produce good performance. Given the impressive accuracy achieved by these approximate rules, several studies began to investigate their convergence properties (97], e.g. for random backpropagation weights feedforward networks [8 99). However, the trend of generalization capabilities of these rules, especially in RNNs, is underexamined.
RTRL 96 have also demonstrated
2.2 Loss landscape curvature and generalization performance
Given the central importance of understanding how neural networks perform in situations unseen during training [4, [00H106), the deep learning community has made tremendous efforts to develop tools for understanding generalization that we leverage here. That flat minima could lead to better generalization was observed more than two decades ago [107] Intuitively, under the same amount of perturbation in parameter space (e.g. loss landscape changes due the addition of new data) worse performance degradation will be seen around the narrower minima (see Figure (C). We note that perturbations in parameter space can be linked to that in input space (6] Recently, many empirical studies have consistently supported the usefulness of this predictor [1081115) In particular, the authors of (5) performed an extensive study and found that flatness- based measures have higher correlation with generalization than other alternatives. Motivated by this, several studies have

---

characterized properties of the loss functions's 's Hessian - whose eigenspectrum carries information about curvature [20] Connections between flatness and generalization performance have shed light on the reason for greater generalization gaps in large batch training [11 124, and also
have inspired optimization methods to favor flatter minima [008] [09 125 132 Despite the criticism of deaeeeeeeendence of flatness [33], where parameter rescaling can drastically change flatness but
not always generalization quality, flatness - with parameter scales taken into account [34 [35 connected to PAC-Bayesian generalization error bounds [7-9] Moreover, a recent theoretical study rigorously connects the flatness of the loss surface to generalization in classification tasks under the assumption that labels are (approximately) locally constant [6]. Leveraging the great progress from the deep learning community, we aim to study the generalization properties of bio- hio-plausible learning rules from geometric perspective.
Results
In this section, we first describe the network and learning setup we use (Figure [A]. Next, we present number of numerical experiments where we compute the generalization gap directly on three commonly used ML and neuroscience tasks (Figure -A-C), for truncation-based bio-plausible gradient approximations. For the same experiments, we also quantify loss landscape curvature along learning trajectories, and connect these quantities to generalization behavior (Figures (2D-F and Finally, we provide theoretical arguments and theorem that explains how gradient alignment in bio- plausible gradient approximations can affect curvature preference (Theorem Figure 3 and thus, generalization. Through additional experiments, we verify the predictive power of our theory. We conclude with discussions on potential remedies used by the brain (Appendix Figure 5) and future directions.
Generalization gap historgrams
Pattern generation
Sequential MNIST
Delayed match to sa-pll
Three-factor
20
RF Three-factor MDGL
35-
Count
Count
Count
BPTT
20-
00
0.05
0.0
0.15
0.20
0.25
0.30
8.00
0.05
0.10
0.15
0.20
8.0
0.1
0.2
0.3
0.4
0.5
0.6
Generalization
Generalization
Generalion cinerraizatiin
Generalization gap leading Hessian eigenvalue
Pattern generation
Sequential MNIST
Delayed match sample
Generalleation gop
P.v0lue=0.000551
P-value=0.
0.00014 0.0005114
Genera gap
0.15
0.2e-
P-value=1.96e-07
0.00
0.5
150 035 300 135 153
1st Fessian cigvals
5775
Hessian eigvals
143
00
02
1st Hessian eigvals
04
05
3.0
Figure 2: Bio-plausible temporal credit assignment rules show worse and more variable gen- eralization gap, which can be informed by loss landscape curvature. A-C) Generalization gap distributions computed at the end of training across different random weight initializations for several well-known neuroscience and machine learning tasks. The higher the generalization gap, the worse the generalization performance. BPTT (black), bio bio-plausible alternatives (magenta, yellow and green). D-F) Scatter plots showing the trend of generalization gap v.s. leading loss Hessian eigenvalue across many runs; each point corresponds to single run of the same runs in A-C.
3.1 Network and learning setup
The detailed governing equations of our setup can be found in Methods (Appendix A). We consider a RNN with Nr, input units, N hidden units and Nout readout units (Figure TA). We verified that trends hold for different network sizes and refer the reader to Appendix A.3 for more details. The

---

update formula for he E INN (the hidden state at time f) is governed by:
he+1 = (WWa(( Wa.z),,
where (6-) RR- + RN is the hidden state update function, f(-) IR - R is the activation function, Wn. E BRN (resp. W₂ E R.... xX is the recurrent (resp. input) weight matrix and R Now is the input. For o, we consider discrete- time implementation of a rate-based recurrent neural network (RNN) similar to the form in [136] (details in Appendix Aj. Readout ý E RNNout with readout weights w E R Nout X.N is defined
U = (w, f(h4))
(2)
We performed experiments three tasks: sequential MNIST 137,, pattern generation [138] and delayed match-to- saapple tasks [139]. The objective is to minimize scalar loss L E R, which is defined as
Witt y for regression tasks k=
Nout (o)))
L(W.)
2TB ii TB EE
1=
(3)
B
k= T, Tog k,t> for classification tasks
Nout (4)
given target readout y Roue, task duration T R and batch size B E R. Thet E R is the one-hot encoded target for readout unit k at time t and TR. softmax (01,t..... (NoUT,)) = exp(ix.)/ Σn' exp($w is the predicted category probability.
Different learning algorithms examined in this work are BPTT (our benchmark), which update weights by computing the exact gradient (VL(W)) NNN):
AW, VVWW,W)))
(4)
and three SoTA bio-plausible learning rules that update weights using approximate gradient:
AW, = LWWW,
(5)
where (WW),) E NNNN denotes gradient approximation and E R denotes the learning rate. These three learning rules are explained further in Appendix A (Methods) but we note that these bio-plausibe plouulauebbl learning rules are based on truncations of dependency paths on the computational graph for the exact gradient- that are biologically implausible to compute (Figure [ all figures, learning rules are labeled as "Thrre-factor" (symmetric e-prop), "RF Three-factor" (RFLO) and "MDGL", respectively. We remark that the focus here is on comparing artificial bio-plausible learn- ing rules, rather than between biological rules. Finally, we note that tasks were learned with mostly comparable training accuracies for all learning rules, and that generalization gaps reflect a testing departure from these values. We refer the reader to the Appendix for more details (Appendix A.3
3.2 Generalization gap and loss landscape curvature
To study generalization performance, our first step is compute the generalization gap empirically, Generalization gap is defined as train accuracy minus test accuracy; the larger the generalization gap, the worse the generalization performance. For various learning rules, we plot the generalization gap histogram at the end of training across runs with distinct initializations; different colors represent different learning rules (Figure Notice that these bio-plausible rules achieve worse and more variable generalization performance than their machine learning counterpart (BPTT).
We now investigate if the generalization gap behavior described above correlates well with Loss landscape curvature. We use the leading eigenvalue of the loss' Hessian (where derivatives are taken with respect to model parameters) as measure for curvature following previous practice [[2]] and note that it is practical for both empirical [118) and theoretical analyses (Theorem) I There exist
other measures for flatness and due to the scale-dependence issue of Hessian spectrum 153 we also test using parameter ccale-independes generalization gap points from Figure against the corresponding leading Hessian eigenvalue, statistically significant correlation is observed (Figure 2D-F). We also observed such correlation across runs with the learning rule fixed (Appendix Figure 12p. We note that we did not expect this relationship to be very tight since in addition to the worse generalization gap average, bio-plausible learning rules exhibit increased variance. This is an important and consistent trend we observed but might have been overlooked in previous studies.
measures (see Appendix Figures 7 and 8). When we plot
5

---

A
Sequential MNIST
B
Pattern generation
Delayed match-to-sample
1+3
1st Hessian eigval
Hessian eigual
RF Three-factor BPTT
1st Hessian eigva
Threefachor Threa
Three factor
HDGL
BPIT
0.6
0.4
0.0
0.9
0.5
1.0
15
2.0
255
Training iterations
1nd
Training iterations
104
Training iterations
l0c
Figure 3: Bio-plausible gradient approximations tend to approach high curvature regions in loss landscape. Dominant Hessian eigenvalues are plotted throughout training for bio- cloppluusbbe learning rules and BPTT. This analysis is done for A) sequential MNIST, B) pattern generation and C) delayed sample tasks. Solid lines/shaded regions: mean/standard deviation of dominant Hessian eigenvalue curves across five independent runs.
So far, we investigated the endpoints of optimization trajectories, where training performance has converged. Now, we visualize the whole training trajectory. This done is for two reasons: account for early stopping that can halt training anywhere along the trajectory due to time constraints; 2) flatness during training could give indications of avoiding or escaping high- high-curvature regions. We observe that the biologically motivated gradient approximations tend to rapidly approach high- curvature regions compared to their machine learning counterpart (Figure 3) Together, these results demonstrate a clear trend and a link between the generalization gap and loss landscape curvature: both being increased and more variable for bio-plausible rules. We also stopped BPTT early match the test accuracy of the three-factor rule, and observed similar trends as in the main text (Appendix Table D. We also note that the curvature convergence behavior seems be shared problem of temporal truncations of the gradient (Appendix Figure 9). which is what the existing bio-plausible gradient approximations for RNNs are based on. Next, we provide theoretical argument as to why truncated temporal credit assignment rules favor high curvature regions of the loss landscape.
3.3 Theoretical analysis: link between curvature and gradient approximation error
We discussed how generalization can be linked to curvature, and we now examine the link between curvature and gradient alignment during learning dynamics. We represent approximate gradients in a rule-agnostic manner, where an arbitrary approximation is represented in terms of its component along the gradient direction plus arbitrary orthogonal vector (Figure
5 = pg +
(6)
where g = VL(WW)) E N
and and R are the exact and approximate gradients, respectively (we reshaped ((WW)) into a vector here); ē E N2 is an arbitrary orthogonal vector to g. Here, scalar p IR represents the relative step length along the gradient direction that the approximate rule is making. As we will see in Theorem I p is an important quantity in our analysis. One can easily compute p from ğ and ğ by p g"gg
We express weight updates as discrete dynamical systems (with weights W,, the state variables):
Wt WT + F(W,,) W WVLWW, for BPTT
W - WT + F(W,) = W - LWW, for an approximate rule,
(8)
where 77 is the learning rate, V is an approximate gradient, and F; R + N² (resp. R N denotes the map defined by BPTT (resp. an approximate) weight update rule. Notation W+ and W denotes W at the next and current step, respectively. We note in passing that dynamical
systems view of weight updates have been used previously 140 [41
We introduce additional notations before presenting Theorem J E x (resp. J E R xN² the Jacobian of the dynamical system of BPTT (resp. an approximate rule) in Eq. (resp. Eq. 8)
X E R (resp. X e IR) is the leading eigenvalue for BPTT (resp. an approximate rule) Jacobian.

---

XL E R is the leading eigenvalue of the loss' Hessian matrix. WB E ERNN (resp. W E ERNXN is the final fixed point for BPTT (resp. an approximate rule). We now present Theorem T
Theorem 1. Consider RNN defined in Eq. D with a single scalar oulpui i and least squares loss as in Eq. 3 ppresented only at the last time step T, and weights are updated according to the difference single exumple (stochastic gradient descent)
equation for 7 resp. an approximate rule B using learning rate BiB (resp. ne). In the limit of stable fixed point convergence with zero training error, the dominant loss' Hessian eigenvalue attained by BPTT (resp. approximate rule) is bounded by |1 (WW)) < (resp. X2" (W2) < Tolre).
Proof. Full proof is in Appendix
B Here is a summary of the main steps involved:
1. The Jacobian of BPTT dynamical system (Eq. 7 is the loss" Hessian scaled by a constant;
2. Using the above relationship, we can bound | from [A/l < 1, which is the condition for a discrete-time dynamical system to converge to fixed point
3. However, the link between the Jacobian of an approximate update rule and the loss' Hessian is less obvious. Thus, we derive a link between | and [a/], and then apply the step above
The consequence of the upper bound derived in Theorem[Tis js that truncated gradient rules can converge to minima with higher dominant Hessian eigenvalue than BPTT, with the leading cigenvalue bound inversely proportional to le| (ppl < 1 usually). In practice, can vary depending on task settings and in our setup, observed it be somewhere between 0.02 and 0.3. We remark that this higher upper bound is consistent with the increased spread of curvature and generalization observed for bio-plausible rules in experiments. Theorem [] highlights scalar p (Eq. (6) relative step length along the gradient direction, as an important factor in the curvature bound. To test that, we eliminate the factor of p by reducing the learning rule of BPTT such that its step length is matched to that of the three-factor rule. This resulted in the blue curves in Figure 4 which is still trained using BPTT but with the update scaled by factor of p. By matching the step length of BPTT and a three-factor rule along the gradient direction, similar convergence behaviors were observed. Similar observations were also made when the matching step experiment was repeated at three times the learning rate for all rules (Appendix Figure[12p). This result then attributes the curvature preference behavior to relative step length along the gradient direction, and thereby indicating a link between curvature and gradient alignment under certain conditions. Consistent with earlier results, when the step length of BPTT matched to that of a three-factor rule, its generalization performance also worsened (Appendix F
We make few more remarks regarding Theorem I p is usually less than because otherwise the additional orthogonal component é would imply a larger update size for the approximate rulc compared BPTT. This would make the approximate rule more prone to numerical instabilities (Eq. [ and would require a change in solver hy hyperparameters such as overall learning rate which in turn, influence the update size. In our experiments, when the three-factor rule was scaled up to match the loong-grddiett update sizes of BPTT, we quickly ran into numerical overflow (values of NaNs in the network), which is expected for a very large learning rate. The exact point for when this numerical instability is reached depends on many factors such as the model, task, numerical precision as well as the consistency of update direction. We have also equated numerical instabilities in simulations as proxy for situations problematic for the brain in Discussion. Curiously, e did not seem to play a helpful role in finding flatter minima; this could be due to that approximation error é' is not well aligned with the sharpest directions (Appendix Figure 10 In fact, é' e'being orthogonal to the leading Hessian eigenvectors is a consequence of the assumptions behind Theorem I Despite making assumptions including scalar output, MSE loss and loss available at the last step, our empirical results indicate that our conclusions also extend to other setups: vector output, cross-entropp loss and loss that accumulates over time steps (Figure 4 In Appendix [B.3] we discuss the generality of Theorem [] by using quadratic expansion of the loss function and assuming that é is orthogonal to the leading Hessian eigenvectors.
One might assume increasing the learning rate of the approximate rules could compensate for the reduced step length due to p. However, this can raise other issues. Suppose AW =-nğ for the
7

---

A. Rule agnostic setup
B. Intuition for Theorem
Parameter 2
9
V
L
Pg
AW
dt dw
AW
x
dW
pg +
Parameter
p < 1
Narrow minima
Flat minima
Sequential MNIST
Pattern generation
Delayed match- to sa-sample
1stt Hessian eigval
theory
1st Hesslan eigval
Three-tactor, theery Thrne-lactor
Hessian eigval
1.0
Threafacter
theory
BPTT
0.8
0.6
1.0
Training iterations
Training iterations
1s4
Training iterations
lea
Figure 4: Preference for high curvature regions connected to worse gradient alignment under certain conditions. A) An arbitrary gradient approximation, 5. its component along the gradient direction, Pg. and orthogonal to the gradient é (Eq. 6). The scalar P represents the relative step length along the gradient direction. B) Illustration for Theorem [ if gradient ğ aligned with the sharpest directions but error é is not (see Appendix Figure I0), smaller step length along the gradient direction can make harder step over narrow minima. C-E) Leading loss Hessian eigenvalue v.s. training iteration for the three-factor rule, BPTT, and modified BPTT (three-factor theory). For the latter, step length along the gradient direction of BPTT was matched to three-factor rule by multiplying BPTT update with a factor of which recovers curvature trends of the three-factor rule.
exact gradient and we increase the learning rate of the approximate update until AW =-ng- ne (magnitude of i' scaled accordingly), and because 1 é:
|| = 77 I + 1| > n2ll = ||WI]
(9)
In other words, if approximate updates AW make the same amount of progress BPTT along the gradient direction, AW would have larger magnitude due to the orthogonal component and large update magnitude can be very problematic for numerical stability [142). Thus, the magnitude of é' limits the learning increasing the learning rate for approximate rules (due to é), the differences in generalization and curvature convergence between rules cannot be reduced by inereasing the learning rate for approximate rules. To balance between this numerical stability issue and the potential benefit of large learning rates, one can consider using large learning rate early training prevent premature stabilization in sharp minima followed by gradual decay to mitigate the stability issue (see Appendix Figure
that be used. Because of the numerical issues associated with
Taken together, Theorem and Eq. connect gradient alignment with the curvature of the converged solution under certain conditions. For approximation that is aligned poorly with the exact gradient, large orthogonal approximation error vector limit the step length for numerical stability reasons (Eq. (9) and small relative step length p correspond to larger curvature bound (Theorem l).
Conclusion
While developing bio-plausible learning rules of interest for both answering neuroscience questions and searching for more efficient training strategies for artificial networks, the generalization properties of solutions found by these rules are severely underexamined. Through various well-known machine learning and working memory tasks, we first demonstrate empirically that existing bio-plausible

---

temporal credit assignment rules attain worse generalization performance, which is consistent with their tendency to converge to high-curvature regions in loss landscape. Second, our theoretical analysis offers an explanation for this preference for high curvature regions based on worse alignment to the true gradient. This regime corresponds to the situation where the step length along the gradient direction is small and the approximation error vector is large. Finally, we test this theory empirically by matching the relative step length along the gradient direction resulting in similar convergence behavior (Figure
5
Discussion
Our study - a stepping stone toward understanding biological generalization using deep learning methods- raises many exciting questions for future investigations, both on the front of stronger deep learning theory and more sophisticated biological ingredients.
Deep learning theory and its implications: In this study, we investigate generalization properties using loss landscape flatness, a promising predictor with recent rigorous connection to generalization gap (6). Yet. to what extent can flatness explain generalization is still an open question in deep learning. For instance, curvature is a local measure, which means that its informativeness of robustness against global perturbation is limited. Moreover, the theoretical association between flatness and generalization is provided in the form of upper bound [6][9] and the bound may not always be tight, which is consistent with more variability in the generalization gap for bio-plausible learning rules but offers less predictive power. Despite observing a significant correlation between the generalization gap and a curvature-based measure Figure the relationship appears to be messy, suggesting other factors involved in explaining generalization. Given that developing better predictors of generalization still work in progress [4, we anticipate stronger theoretical tools to be applied for studying biological generalization in the future. Our results are also consistent with existing findings linking learning rate to loss landscape curvature in deep networks [13]]. In the case of bio-plausible gradient approximations, small step length along the gradient direction cannot be compensated by increasing the learning rate, as that would inadvertently increase the error vector, causing numerical issues (Eq.9). Curiously, the approximation error vector ě did not seem play role other than restricting the learning rate. While it is well-known that stochastic gradient noise (SGN) can help with finding flat minima due to the alignment of SGN covariance and Hessian near minima [ 143 145), that may not apply to approximation error vector é resulting primarily from temporal truncation of the gradient (see Figure ITA and Appendix Figure TO). This indicates that noise with different properties (e.g. different directions) could affect generalization differently, thereby motivating future investigations into how broad range of biological noises- which may differ from noise in ML optimization (e.g. SGN) - can impact generalization.
Moreover, our results are closely related to series of studies that examined the "catapulting" behavior in learning [1461150]. This can happen when the second order Taylor term of the loss function would dominate over the first, which would cause learning to cross the threshold for step size stability and "catapult" into a flatter region that can accommodate the step size. If the truncation noise is only aligned with the eigendirections associated with negligible eigenvalues, then it can only have limited contributions to the recond-order Taylor term. On top of that, the orthogonal noise term would require smaller learning rate be used to avoid numerical issues, as explained earlier. Overall, these would lead to a weaker second-order Taylor term relative to the first for bio-plausible temporal credit assignment rules, which would then increase the threshold for step size stability. This increased stability is closely tied to the greater dynamical stability (for the weight update difference equation) of approximation rules predicted by Theorem [ due the correspondence between loss' Hessian matrix and the Jacobian matrix of the weight update difference equation (the correspondence is explained in Theorem D proof in Appendix B
Toward more detailed biological mechanisms: On the front of more sophisticated biological ingredients that may improve generalization performance, we see two lines of approaches: 1- develop bio- plaussibee learning rules that align better with the gradient, as suggested by our rule-agnostic analysis (Theorem [ Figure and 2- instead of studying learning rules isolation, consider also other neural circuit components [151 that could interact with the learning rule. An important component would be the architecture, including connectivity structure and neuron model, found through evolution [156] (see also [159)). To address main question, this study varies learning rules

---

while holding data, objective function and architecture constant (see [TO). However, these different components can interact, and more sophisticated architecture can facilitate task learning [169-166] Given the exploding parameter space resulting from such interactions, we believe it requires careful future analysis and is outside of the scope for this one paper.
Additionally, learning rate modulation [167 168 could be one of many possible remedies employed by the brain. We conjecture that neuromodulatory mechanisms could be coupled with these learning rules to improve the convergence behavior through our scheduled learning rate experiments (Appendix Figure J. where an initial high learning rate could prevent the learning trajectory from settling in sharp minima prematurely followed by gradual decay to avoid instabilities. One possible way to realize such learning rate modulation could be through serotonin neurons via uncertainty tracking. where the learning rate is high when the reward prediction eπror is high (this can happen at the beginning of learning) [169, Since the authors of [169] showed that inhibiting serotonin led to failure in learning rate modulation, we conjecture that such inhibition might have an impact on the generalization performance of learning outcomes. On the topic of balaneing numerical instabilities and potential advantages of large learning rates, while the analog nature of biology may seem to avoid finite precision representation in digital computers that give rise to numerical instabilities, the same problems that lead to numerical instabilities in digital computers, such as big ranges between quantities added or multiplied, remains an issue for biology since quantities must be stored in noisy activity patterns of neurotransmitter release. Future investigations could investigate potential homeostatic mechanisms that regulate biological quantities to avoid such instabilities, thereby enabling larger "learning rates" to be used so to find flatter minima. Other ingredients for future investigations could include intrinsic noise with certain structures 153 170 171 (including directions and bias/variance properties) that would make them more favorable for generalization. Taken together, we hope to see follow-up investigations - riding on the rapid advancements both the front of deep learning theory and sophisticated biological mechanisms into how the brain attains solutions that generalize.
6 Acknowledgement
We thank Gauthier Gidel, Jonathan Cornford, Aristide Baratin, Thomas George and Mohammad Pezeshki for insightful discussions and helpful feedback at an early stage of this project. We also thank Henning Petzka and Michael Kamp for helpful email exchanges. This research was supported by NSERC PGS-D (Y.H.L.); NSF AccelNet IN-BIC program, Grant No. 01SE-2019976 AM02 (Y.H...): Vanier Canada Graduate scholarship (A.G.); Healthy Brains for Healthy Lives fellowship (A.G.); CIFAR Learning in Machines and Brains Program (B.A.R.). NSERC Discovery Grant RGPIN-2018 C0IIN-2018-4480 8-04821 (G.L), FRQS Research Scholar Award, Junior LAJGU0401-2533
(G.L.),
Canada Research Chair in Neural Computations and Interfacing (G.L.), Canada CIFAR Al Chair program (G.L.& & B.A.R.). We also thank the Allen Institute founder, Paul G. Allen, for his vision, encouragement, and support.
References
[1] Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current opinion in neurobiology, 55:82-89, 2019.
[2] Luke Y Prince, Ellen Boven, Roy Henha Eyono, Arna Ghosh, Joe Pemberton, Franz Scherr, Claudia Clopath, Rui Ponte Costa, Wolfgang Maass, Blake A Richards, et al. Cen gac workshop: Issues with learning in biological recurrent neural networks. arXiv preprint arXiv:2105.05382 2021.
[3] Owen Marschall, Kyunghyun Cho, and Cristina Savin. A unified framework of online learning algorithms for training recurrent neural networks. Journal of Machine Learning Research, 2 21(135):1-34, 2020.
[4] Yiding Jiang, Pierre Foret, Scott Yak, Daniel M Roy, Hossein Mobahi, Gintare Karolina Dziugaite, Samy Bengio, Suriya Gunasekar, Isabelle Guyon, and Behnam Neyshabur. Neurips 2020 competition: Predicting generalization in deep learning. arXiv preprint arXiv:0012.07976,
2020.
10

---

[5] Yiding Jiang. Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fan- tastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178. 2019.
[6] Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Relative flatness and generalization. Advances Neural Information Processing Systems, 34, 2021.
[7] Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv: 1000, 2017.
[8] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017.
[9] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bat bac-bayesian analysis. In International Conference on Machine Learning, pages 9636-9647. PMLR, 2020.
[10] Blake A Richards, Timothy P Lillicrap. Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):17111177 2019.
[11] Guillaume Bellec, Franz Scherr, Anand Subramoney. Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 11(1):1-155 dec 2020.
[12] James M Murray. Local online learning in recurrent networks with random feedback. ELife, 8:e43299, 2019.
[13] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brown and Uygar Sümbül. cell-type-speciieee neuromodulation guides synaptic credit assignment spiking neural network. Proceedings of the National Academy of Sciences, 118(51), 2021.
[14] Colin Bredenberg, Benjamin Lyo, Eero Simoncelli, and Cristina Savin. Impression learning: Online representation learning with synaptic plasticity. Advances in Neural Information Processing Systems, 34, 2021.
[15] Yanis Bahroun, Dmitri Chklovskii, and Anirvan Sengupta. A normative and biologically plausible algorithm for independent component analysis. Advances in Neural Information Processing Systems, 34, 2021.
[16] Basile Confavreux, Friedemann Zenke, Everton Agnes, Timothy Lillicrap, and Tim Vogels.
A meta-learning approach to (re) discover plasticity rules that carve a desired function into a neural network. Advances in Neural Information Processing Systems, 33:16398-16648, 2020.
[17] Gabriel Ocker and Michael Buice. Tensor decompositions of higher-order correlations by nonlinear hebbian plasticity. Advances in Neural Information Processing Systems, 34, 2021.
[18] Danil Tyulmankov, Ching Fang, Annapurna Vadaparty, and Guangyu Robert Yang. Biological key-value memory networks. Advances in Neural Information Processing Systems, 34, 2021.
[19] Alexander Meulemans, Matilde Tristany Farinha, Javier Garcia Ordonez, Pau Vil- imelis Aceituno, João Sacramento, and Benjamin F Grewe. Credit assignment in neural networks through deep feedback control. Advances in Neural Information Processing Systems, 34, 2021.
[20] Brandon McMahan, Michael Kleinman, and Jonathan Kao. Learning rule influences recurrent network representations but not attractor structure in decision-making tasks. Advances in Neural Information Processing Systems, 34, 2021.
[21] Paul Haider, Benjamin Ellenberger, Laura Kriener, Jakob Jordan, Walter Senn, and Mihai A Petrovici. Latent equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons. Advances in Neural Information Processing Systems, 34:17839- 17851, 2021.
[22] Roman Pogodin, Yash Mehta, Timothy Lillicrap, and Peter E Latham. Towards biologically plausible convolutional networks. Advances in Neural Information Processing Systems, 34:13924-13936 2021.
11

---

[23] Johannes Friedrich, Siavash Golkar, Shiva Farashahi, Alexander Genkin, Anirvan Sengupta. and Dmitri Chklovskii. Neural optimal feedback control with local learning rules. Advances in Neural Information Processing Systems, 34, 2021.
[24] Colin Bredenberg, Eero Simoncelli, and Cristina Savin. Learning efficient task-dependent representations with synaptic plasticity. Advances in Neural Information Processing Systems, 33:15714-15724, 2020.
[25] Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive predictions. Advances in Neural Information Processing Systems, 34, 2021.
[26] Elias Najarro and Sebastian Risi. Meta-learning through hebbian plasticity in random networks. Advances in Neural Information Processing Systems, 33:20719 1000079999771,,,, 2020.
[27] Pablo Tano, Peter Dayan, and Alexandre Pouget. A local temporal difference code for distributional reinforcement learning. Advances in Neural Information Processing Systems, 33: 1361--66333,,, 2020.
[28] Lea Duncker, Laura Driscoll, Krishna V Shenoy, Mancesh Sahani, and David Sussillo. Orga- nizing recurrent network dynamics by cask-compttation to enable continual learning. Advances in neural information processing systems, 33: 1384777--4,,9, 2020.
[29] Aran Nayebi, Sanjana Srivastava, Surya Ganguli, and Daniel L. Yamins. Identifying learning rules from neural network observables. Advances in Neural Information Processing Systems, 33:2639-2666 2650, 2020.
[30] Zoe Ashwood, Nicholas Roy, Ji Hyun Bak, and Jonathan W Pillow. Inferring learning rules from animal decision-making Advances in Neural Information Processing Systems, 33:3442-3453, 2020.
[31] Siavash Golkar, David Lipshutz, Yanis Bahroun, Anirvan Sengupta, and Dmitri Chklovskii. A simple normative network approximates local non ben-ieb learning in the cortex. Advances in neural information processing systems,
2020.
[32] Daniel R Kepple, Rainer Engelken, and Kanaka Rajan. Curriculum learning as tool to uncover learning principles in the brain. In International Conference on Learning Representations, 2021.
[33] Danil Tyulmankov, Guangyu Robert Yang, and LF Abbott. Meta-learning synaptic plasticity and memory addressing for continual familiarity detection. Neuron, 11053)544-557,, 2022.
[34] Eli Moore and Rishidev Chaudhuri. Using noise to probe recurrent neural network structure and prune synapses. Advances in Neural Information Processing Systems, 33:14046-14057 2020.
[35] Grace Wan Yu Ang, Clara S Tang, Y Audrey Hay, Sara Zannone, Ole Paulsen, and Claudia Clopath. The functional role of sequentially neuromodulated synaptic plasticity behavioural learning. PLOS Computational Biology, 17(6)ee 2021.
[36] Eren Sezener, Agnieszka Grabska-Barvinskka Dimitar Kostadinov, Maxime Beau, Sanjukta Krishnagopal, David Budden, Marcus Hutter, Joel Veness, Matthew Botvinick. Claudia Clopath, et al. A rapid and efficient learning rule for biological neural circuits. BioRxiv, 2021.
[37] Owen Marschall, Kyunghyun Cho, and Cristina Savin. Using local plasticity rules to train recurrent neural networks. arXiv preprint arXiv: 2019.
[38] David Clark, LF Abbott, and SueYeon Chung. Credit assignment through broadcasting global error vector. Advances in Neural Information Processing Systems, 140606 2021.
[39] Maxence Ernoult, Fabrice Normandin, Abhinav Moudgil, Sean Spinney, Eugene Belilovsky. Irina Rish, Blake Richards, and Yoshua Bengio. Towards scaling difference target propagation by learning backprop targets. arXiv preprint 2rXiv:2201.13411 2022.
[40] Jack Lindsey and Ashok Litwin-Kumnn Kumar. Learning to learn with feedback and local plasticity. Advances in Neural Information Processing Systems, 33:21213-21222 2020.
[41] Sukbin Lim, Jillian L McKee, Luke Woloszyn, Yali Amit, David J Freedman, David L. Sheinberg, and Nicolas Brunel. Inferring learning rules from distributions of firing rates in cortical neurons. Nature neuroscience, 18(12)1804-11100 2015.
12

---

[42] Artur Luczak, Bruce L. McNaughton, and Yoshimasa Kubo. Neurons learn by predicting future activity. Nature Machine Intelligence, 4(1):62-72, 72, January 2022.
[43] James C.R. Whittington and Rafal Bogacz. Theories of Error Back- BackkPrropggtion in the Brain. Trends in Cognitive Sciences, 2019.
[44] Saurabh Vyas, Matthew D Golub, David Sussillo, and Krishna V Shenoy, Computation
through neural population dynamics. Annual Review of Neuroscience, 43:249-275, 2020.
[45] Matthew G Perich, Charlotte Arlt, Sofia Soares, Megan E Young, Clayton P Mosher, Juri Minxha, Eugene Carter, Ueli Rutishauser, Peter H Rudebeck, Christopher D Harvey, al. Inferring brain-wide interactions using data-constrained recurrent neural network models. bioRxiv, pages 2020-12, 2021.
[46] Jimmy Smith, Scott Linderman, and David Sussillo. Reverse engineering recurrent neural networks with jacobian switching linear dynamical systems. Advances in Neural Information Processing Systems, 34, 2021.
[47] Michael Kleinman, Chandramouli Chandrasekaran, and Jonathan Kao, A mechanistic multi- area recurrett network model of decision-making. Advances in Neural Information Processing Systems, 34, 2021.
[48] Friedrich Schuessler, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak. The interplay between randomness and structure during learning in mns. Advances in neural information processing systems, 33:13332--1366 13362, 2020.
[49] Rylan Schaeffer, Mikail Khona, Leenoy Meshulam, Ila Rani Fiete, et al. Reverse-engineering recurrent neural network solutions a hierarchical inference task for mice. bioRxiv, 2020.
[50] Guangyu Robert Yang, Madhura R Joglekar, H Francis Song, William T Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297-306, 2019.
[51] Valerio Mante, David Sussillo, Krishna V Shenoy, and William Newsome. Context- dependent computation by recurrent dynamics in prefrontal cortex. nalure, 503(7474))77-844 2013.
[52] Joshua Glaser, Matthew Whiteway, John P Cunningham, Liam Paninski, and Scott Linderman. Recurrent switching dynamical systems models for multiple interacting neural populations. Advances in neural information processing systems, 33: 18877-1888888 2020.
[53] Jonathan Kadmon, Jonathan Timcheck, and Surya Ganguli. Predictive coding in balanced neural networks with noise, chaos and delays. Advances in neural information processing systems, 33; 33:16677-16688, 2020.
[54] Jonathan Dong, Ruben Ohana, Mushegh Rafayelyan, and Florent Krzakala. Reservoir com- puting meets recurrent kernels and structured transforms. Advances in Neural Information Processing Systems, 33:16785-16796 2020.
[55] Elia Turner, Kabir V Dabholkar, and Omri Barak. Charting and navigating the space of solutions for recurrent neural networks. Advances in Neural information Processing Systems, 34:25320-25333, 2021.
[56] Sandra Nestler, Christian Keup, David Dahmen, Matthieu Gilson, Holger Rauhut, and Moritz Helias. Unfolding recurrence by green's 's functions for optimized reservoir computing. Advances in Neural Information Processing Systems, 33:17380-17390 2020.
[57] Robert Kim and Terrence J Sejnowski. Strong inhibitory signaling underlies stable temporal dynamics and working memory in spiking neural networks. Nature Neuroscience, 24(1):129
139, 2021.
[58] Adrian Valente, Srdjan Ostojic, and Jonathan Pillow. Probing the relationship between linear dynamical systems and low-rank recurrent neural network models. arXiv preprint 100002210.00904, 2021.
[59] Tom Macpherson, Anne Churchland, Terry Sejnowski, James DiCarlo, Yukiyasu Kamitani, Hidehiko Takahashi, and Takatoshi Hikida. Natural and artificial intelligence: A brief intro- duction to the interplay between and neuroscience research. Neural Networks, 144:603-613, 1021.
13

---

[60] Ben Tsuda, Stefan C Pate, Kay M Tye, Hava T Siegelmann, and Terrence J Sejnowski. Neuromodulators generate multiple context-relevant behaviors in a recurrent neural network by shifting activity hypertubes. bioRxiv, pages 2021 2022.
[61] Nuttida Rungrataameetaweemmnt Robert Kim, John T Serences, and Terrence J Sejnowski. Probabilistic visual processing in humans and recurrent neural networks. Journal of Vision, 22(3):24-24, 2022.
[62] Brian DePasquale, Christopher Cueva, Kanaka Rajan, G Sean Escola, and LF Abbott. full- force: A target-based method for training recurrent networks. PloS one, 13(2)=e0111527, 2018.
[63] Christopher Langdon and Tatiana A Engel. Latent circuit inference from heterogeneous neural responses during cognitive tasks. bioRxiv, 2022.
[64] R. J. Williams and D. Zipser. Gradient-based learning algorithms for recurrent networks and their computational complexity. In Y. Chauvin and D. E. Rumelhart, editors, Back- propagation: Theory, Architectures and Applications, chapter 13, pages 433-486. Hillsdale, NJ: Erlbaum, 1995.
[65] Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre. Stable and expressive recurrent vision models, arXiv preprint 1:X: 2020.
[66] Timothy P Liiliccap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton.
Backpropagation and the brain. Nature Reviews Neuroscience, pages 1-12, 2020.
[67] Isabella Pozzi, Sander Bohté, and Pieter Roelfsema. A biologically plausible learning rule for deep learning the brain. arXiv preprint arXiv:1811.01120 2018.
[68] Axel Laborieux, Maxence Erouult Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling equilibrium propagation to deep convnets by drastically reducing its gradient estimator bias. Frontiers in neuroscience, 15:129, 2021.
[69] Beren Millidge, Alexander Tschantz, Anil K Seth, and Christopher L Buckley. Activation relaxation: A local dynamical approximation to backpropagation the brain. arXiv preprint arXiv:2009.0553 05359, 2020.
[70] João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. arXiv preprint arXiv.1810.11393, 2018.
[71] Yali Amit. Deep learning with asymmetric connections and hebbian updates. Frontiers in computational neuroscience, 13:18, 2019.
[72] Jonathan E Rubin, Catalina Vich, Matthew Clapp, Kendra Noneman, and Timothy Verstynen. The credit assignment problem in cortico-basal ganglia-thalamic networks: A review, problem and a possible solution. European Journal of Neuroscience, 53(7):2234-2255 2021.
[73] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, pages 1-10, 2021.
[74] Pieter R. Roelfsema and Anthony Holtmaat. Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience, 1963):166-180, feb 2018.
[75] Timothy P Lilllicrpp, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random
synaptic feedback weights support error backpropagation for deep learning. Nature communi- cations, 7(1):1-10, 2016.
[76] Jeffrey C. Magee and Christine Grienberger. Synaptic Plasticity Forms and Functions. Annual Review of Neuroscience, 43(1):95-111 jul 2020.
[77] Wulfram Gerstner, Marco Lehmann, Vasiliki Liakoni, Dane Corneil, and Johanni Brea. Fligi- bility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules. Frontiers in Neural Circuits, 12:53, jul 2018
[78] Magdalena Sanhueza and John Lisman. The CaMKII/MMIAR complex as molecular memory. Molecular Brain, 6(1):10, feb 2013.
[79] Stijn Cassenaer and Gilles Laurent. Conditional modulation of spike-timing-depeneer plastic- ity for olfactory learning, Nature, 487 feb 2012.
14

---

[80] Sho Yagishita, Akiko Hayashi- Takaaii Graham C.R. Ellis-Davies, Hidetoshi Urakubo, Shin Ishii, and Haruo Kasai. A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science, 34((6204):11116122222 sep 2014.
[81] Aparna Suvrathan. Beyond STDP - towards diverse and functionally relevant plasticity rules. Current Opinion in Neurobiology. 54:12-19, feb 2019.
[82]
Wolfram Schultz. Dopamine reward prediction-error signalling: a twoocomponent response. Nature Reviews Neuroscience, 17(3):183, 2016.
[83] Johnatan Aljadeff, James D' a'oou,, Rachel E Field, Robert C Froemke, and Claudia Clopath. Cortical credit assignment by hebbian, neuromodulatory and inhibitory plasticity. arXiv preprini arXiv: 1,000007 2019.
[84] Răzvan V Florian. Reinforcement learning through modulation of spike-timing-dependerr synaptic plasticity. Neural computation, 19(6):1488-1553 2007.
[85] Michael A. Farries and Adrienne L. Fairhall. Reinforcement Learning With Modulated Spike Timing -Dppendent Synaptic Plasticity. Journal of Neurophysiology, 98(6):3648 3665, dec
2007.
[86] Robert Legenstein, Dejan Pecevski, and Wolfgang Maass. A learning theory for reward- modulated spikeetimingg demming-dependen plasticity with application biofeedback. PLoS computa- tional biology,
2008.
[87] Zuzanna Brzosko, Susanna B Mierau, and Ole Paulsen. Neuromodulation of spike-timing- dependent plasticity: past, present, and future. Neuron, (03(4))563-584) 2019.
[88] Verena Pawlak, Jeffery R Wickens, Alfredo Kirkwood, and Jason ND Kerr. Timing is not everything: neuromodulation opens the stdp gate. Frontiers in synaptic neuroscience, 2:146, 2010.
[89] Stephen Smith, Uygar Sümbül, Lucas T Graybuck, Forrest Collman, Sharmishtaa Seshamani, Rohan Gala, Olga Gliko, Leila Elabbady, Jeremy A Miller, Trygve E Bakken, et al. Single-cell transcriptomic evidence for dense intracortical neuropeptide networks. Elife, 8:e47889, 2019.
[90] Stephen J Smith, Michael Hawrylycz, Jean Rossier, and Uygar Sümbül. New light on cortical neuropeptides and synaptic network plasticity. Current Opinion in Neurobiology, 18818. 2020.
[91] Asier Mujika, Florian Meier, and Angelika Steger. Approximating real-time recurrent learning with random kronecker factors. Advances in Neural Information Processing Systems, 31, 2018.
[92] Corentin Tallec and Yann Ollivier. Unbiased online recurrent optimization. arXiv preprint 1rXiv:/702.05044 2017.
[93] Tim Cooijmans and James Martens. On the variance of unbiased online recurrent optimization. 2019.
arXiv preprint
[94] Christopher Roth, Ingmar Kanitscheider, and Ila Fiete. Kernel mn learning (kernl). In International Conference on Learning Representations, 2018.
[95] Jacob Menick, Erich Elsen, Utku Evci, Simon Osindero, Karen Simonyan, and Alex Graves. A practical sparse approximation for real time recurrent learning. arXiv preprint DXXiv2006600232.. 2020.
[96] Friedemann Zenke and Emre O Neftei. Brain-inspired learning on neuromorphic substrates. arXiv preprint arXiv:2010. / 1931, 2020.
[97] Yinan Cao, Christopher Summerfield, and Andrew Saxe. Characterizing emergent representa- tions in a space of candidate learning rules for deep networks. Advances in Neural Information Processing Systems, 13:8660-86777 2020.
[98] Ganlin Song. Ruitu Xu, and John Lafferty. Convergence and alignment of gradient descent with random backpropagation weights. Advances in Neural Information Processing Systems,
34, 2021.
[99] Manuela Girotti, loannis Mitliagkas, and Gauthier Gidel. Convergence analysis and implicit regularization of feedback alignment for deep linear networks. arXiv preprint arXiv:2110.1083 2021.
[100] Huan Xu and Shie Mannor, Robustness and generalization. Machine learning, 86(3):391-425
2012.
15

---

[101] Zeyuan Allen-Zhu and Yuanzhi Li. Can sgd learn recurrent neural networks with provable generalization? Advances in Neural Information Processing Systems, 32, 2019.
[102] Madhu S Advani, Andrew Saxe, and Haim Sompolinsky. High High-dimensional dynamics of generalization enror in neural networks. Neural Networks, 132:428-4444 4488 2020.
[103] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[104] Valentin Thomas, Fabian Pedregosa, Bart Merriënboer, Pitrre-Antoine Manzagol, Yoshua Bengio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect on optimization and generalization. In International Conference on Artificial Intelligence and Statistics, pages 3503-3513. PMLR, 2020.
[105] Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[106] Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In International Conference on Artificial Intelligence and Statistics, pages 2269-2277, PMLR, 2021.
[107] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1): 1997.
[108] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
[109] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, (019(12)1224088 2019.
[110] Xu Sun, Zhiyuan Zhang, Xuancheng Ren, Ruixuan Luo, and Liangyou Li. Exploring the vulnerability of deep neural networks: study of parameter corruption. arXiv preprint arXiv:2006. 056220000.0560, 2020.
[111] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 1609604836 2016.
[112] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[113] Yu Feng and Yuhai Tu. The inverse variance-flatnes relation in stochastic gradient descent is critical for finding flat minima. Proceedings of the National Academy of Sciences, 118(9), 2021.
[114] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. In Interna- tional Conference on Learning Representations, 2018
[115] Huan Wang. Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Identifying general- ization properties in neural networks. arXiv preprint arXiv:1809.0777 07400,, 2018.
[116] Levent Sagun, Utku Evci, Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv: 04454, 2017.
[117] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232-2241. PMLR, 2019.
[118] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pages 581-590. IEEE, 2020.
[119] Zhenyu Liao and Michael W Mahoney. Hessian eigenspectra of more realistic nonlinear models. Advances in Neural Information Processing Systems, 34, 2021.
16

---

[120] Zeke Xie, Qian- Yuan Tang, Yunfeng Cai, Mingming Sun, and Ping Li. On the power-law spectrum in deep learning: A bridge protein science. arXiv preprint arXiv:2201.3301. 2022.
Qian-Yuan
[121] Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness adversaries. Advances Neural Information Processing Systems, 31, 2018.
[122] Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. arXiv preprint arXiv:2002.03499 03495, 2020.
[123] Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Finding flatter minima with sgd. 2018.
[124] Chiyuan Zhang, Qianli Liao, Alexander Rakhlin, Brando Miranda, Noah Golowich, and Tomaso Poggio. Theory of deep learning iib: Optimization properties of sgd. arXiv preprint arXiv:1801.02254, 2018.
[125] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural information processing systems, 30, 2017.
[126] Pavel Izmailov, Dmitrii Podoprikhin. Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:/803.05 05407, 2018.
[127] Carlo Baldassi, Fabrizio Pittorino, and Riccardo Zecehina. Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences, 117(1):161-770, 2020.
[128] Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt Kusner. Questions for flat-minima optimization of modern neural networks. arXiv preprint arXiv:2202.00661 2022.
[129] Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8 8156-8165, 2021.
[130] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958-2969 2020.
[131] Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. The normalization method for alleviating pathological sharpness in wide neural networks. Advances in neural information processing systems, 32, 2019.
[132] Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A deeper look the hessian eigenspectrum deep neural networks and its applications to regularization. arXiv preprint arXiv:2012 038001, 2020.
[133] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028. PMLR, 2017.
[134] Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd international conference on artificial intelligence and statistics, pages
PMLR, 2019.
[135] Akshay Rangamani, Nam H Nguyen, Abhishek Kumar, Dzung Phan, Sang H Chin, and Trac D Tran. A scale invariant flatness measure for deep network minima. arXiv preprint arXiv: PrXiiv1902... 02444.. 2019.
[136] Daniel B Ehrlich, Jasmine T Stone, David Brandfonbrener, Alexander Atanasov, and John D Murray. Psychrun: An accessible and flexible python package for training recurrent neural network models on cognitive tasks. Eneuro, 8(1), 2021.
[137] Yann LeCun. The mnist database of handwritten digits. http://yanm. lecun. com/exdb/mnist// 1998.
[138] Wilten Nicola and Claudia Clopath. Supervised learning in spiking neural networks with force training. Nature communications, 8(1):1-15, 2017.
17

---

[139] Travis Meyer, Xie-Lian Qi. Terrence R Stanford, and Christos Constantinidis. Stimulus selectivity in dorsal and ventral prefrontal cortex after training in working memory tasks. Journal of neuroscience, 3 (177):6666667766 2011.
[140] Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 1 166(23): 2019.
[141] Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien, Implicit regularization of discrete gradient dynamics in linear neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[142] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[143] Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. arXiv preprint 17X3v:8803001199 2018.
[144] Mingwei Wei and David J Schwab. How noise affects the hessian spectrum in overparameter- ized neural networks. arXiv preprint arXiv: wwt.vv//90.... 1910.00195, 2019.
[145] Samuel L Smith and Quoc V Le. A bayesian perspective generalization and stochastic gradient descent. arXiv preprint arXiv:1710006455, 2017.
[146] Jeremy M Cohen, Simran Kaur, Yuanzhi Li, Zico Kolter, and Ameet Talwalkar. Gra- dient descent on neural networks typically occurs at the edge of stability. arXiv preprint SKKiv:2003000065. 2021.
[147] Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint urXiv:2003.022. 02018,, 2020.
[148] Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. 'he break-even point on optimization trajectories of deep neural networks. arXiv preprint arXiv:2002.09577 09977,, 2020.
[149] Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Edward Dahl, Zachary Nado, and Orhan Firat. A loss curvature perspective on training instabilities of deep learning models. In International Conference on Learning Representations, 2021.
[150] Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on edge of stability in deep learning. arXiv preprint arXiv:2205.09745, 2022.
[151] Vladimir Ivanov and Konstantinos Michmizos. Increasing liquid state machine performance with edge-of-chaos dynamics organized by astrocyte-modullted plasticity. Advances in Neural Information Processing Systems, 34, 2021.
[152] Lukas Braun and Tim Vogels. Online learning of neural computations from sparse temporal feedback. Advances in Neural Information Processing Systems, 34, 2021.
[153] Joel Dapello, Jenelle Feather, Hang Le, Tiago Marques, David Cox, Josh McDermott, James J DiCarlo, and Yeo Chung. Neural population geometry reveals the role of stochasticity in robust perception. Advances in Neural Information Processing Systems, 34, 2021.
[154] Maurizio De Pittà, Nicolas Brunel, and Andrea Volterra. Astrocytes: Orchestrating synaptic
plasticity? Neuroscience, 323:43-61, 61, 2016
[155] James M Murray and G Sean Escola. Remembrance of things practiced with fast and slow learning in cortical and subcortical pathways. Nature Communications, 11(1):1-12, 2020.
[156] Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, (0()):1-7,0019
[157] Dhireesha Kudithipudi, Mario Aguilar-Simon Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196-21) 10, 2022.
[158] Yazan N Billeh, Binghuang Cai, Sergey I. Gratiy, Kael Dai, Ramakrishnan Iyer, Nathan W Gouwens, Reza Abbasi-Asl, Xiaoxuan Jia, Joshua H Siegle, Shawn R Olsen, et al. Systematic
18

---

integration of structural and functional data into multi-scale models of mouse primary visual cortex. Neuron, 106(3):388-40) -008, 2020.
[159] Victor Geadah, Stefan Horoi, Giancarlo Kerg, Guy Wolf, and Guillaume Lajoie. Goal-driven optimization of single-neuron properties in artificial networks reveals regularization role of neural diversity and adaptation. bioRxiv, 2022.
[160] llenna Simone Jones and Konrad Paul Kording. Can single neurons solve mnist? the computa- tional power of biological dendritic trees. arXiv preprint arXiv:2009.0/265 2020.
[161] Nicolas Perez-Nieves, Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman. Neural heterogeneity promotes robust learning. Nature communications, 12(1):1-9, 2021.
[162] Laureline Logiaco, LF Abbott, and Sean Escola. Thalamic control of cortical dynamics in
model of flexible motor sequencing. Cell reports, 35(9):109090, 2021.
[163] Guangyu Robert Yang and Manuel Molano Mazon. Next-generation of recurrent neural
network models for cogmition. 2021.
[164] Corinne Teeter, Ramakrishnan Iyer, Vilas Menon, Nathan Gouwens, David Feng. Jim Berg, Aaron Szafer, Nicholas Cain, Hongkui Zeng, Michael Hawrylyez, et al. Generalized leaky integrate-and-fire models classify multiple neuron types. Nature communications, 911):1-15, 2018.
[165] Chloc Winston, Dana Mastrovito, Eric Shea-Brown, and Stefan Mihalas. Heterogeneity in neuronal dynamics is learned by gradient descent for temporal processing tasks. bioRxiv, 2022.
[166]
Christoph Stöckl, Dominik Lang, and Wolfgang Maass. Probabilistic skeletons endow brain- like neural networks with innate computing capabilities. bioRviv, 2021.
[167] Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances Neural Information Processing Systems, 32, 2019.
[168] Niru Maheswaranathan, David Sussillo, Luke Metz, Ruoxi Sun, and Jascha Sohl-Dickstein. Reverse engineering learned optimizers reveals known and novel mechanisms. Advances in Neural Information Processing Systems, 34, 2021.
[169] Cooper D Grossman, Bilal A Bari, and Jeremiah Y Cohen. Serotonin neurons modulate learning rate through uncertainty. Current Biology, 2021.
[170] Antonio Orvieto, Hans Kersting, Frank Proske, Francis Bach, and Aurelien Lucchi. Anti- correlated noise injection for improved generalization. arXiv preprint 02218.2202.02831, 2022.
[171] Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tuo Zhao Toward understanding the importance of noise in training neural networks. In International Conference on Machine Learning. pages 7594-7602. PMLR, 2019.
[172] Stanistaw Jastrzębski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the relation between the sharpest directions of dnn loss and the sgd step length. arXiv preprint
2018.
[173] Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXhv:1611.07477 2016.
[174] Lloyd N Trefethen and David Bau III. Numerical linear algebra, volume 50. Siam, 1997.
[175] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High-dimensional geometry of population responses in visual cortex. Nature, 571(7765):361-365 2019.
[176] Arna Ghosh, Arnab Kumar Mondal, Kumar Krishna Agrawal, and Blake Richards. Investigat- ing power laws in deep representation learning. arXiv preprint arXiv:2202.0588 2022.
[177] Guozhang Chen, Franz Scherr, and Wolfgang Maass. Analysis of visual processing capabilities and neural coding strategies a detailed model for laminar cortical microcircuits in mouse vl. bioRxiv, 2021.
[178] Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-63 2019.
19

---

[179] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784)600--617 2019.
2019.
[180] Benjamin Cramer, Sebastian Billaudelle, Simeon Kanya, Aron Leibfried, Andreas Grübl, Vitali Karasenko, Christian Pehle, Korbinian Schreiber, Yannik Stradmann, Johannes Weis, et al. Surrogate gradients for analog neuromorphic computing. Proceedings of the National Academy of Sciences, 11), 2022.
[181] Dongsung Huh and Terrence Sejnowski. Gradient descent for spiking neural networks. Advances in neural information processing systems, 31, 2018.
[182] Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural compulation, 30(6):151411541, 2018.
[183] Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems, 33:12022- 12033, 2020.
[184) Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang, and Zhouchen Lin. Training feedback spiking neural networks by implicit differentiation on the equilibrium state. Advances in Neural Information Processing Systems, 34, 2021.
[185] Nicolas Perez-Nieves and Dan Goodman. Sparse spiking gradient descent. Advances in Neural Information Processing Systems, 34, 2021.
[186] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongging Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[187] Nicolas Skatchkovsky, Osvaldo Simeone, and Hyeryung Jang. Learning to time-decode in spiking neural networks through the information bottleneck. arXiv preprint arXiv:2166.01 177, 2021.
[188] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric BieaaBrrown, and Uygar Sümbül. solution to temporal credit assignment using cell-type-specific modulatory signals. bioRxiv, pages 2020-11, 2021.
[189] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. (TensorFlow) J: A system for (Large-Scale) machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265-283, 2016.
[190] Guillaume Bellec, Darjan Salaj, Anand Subramoncy, Robert Legenstein, and Wolfgang Maass. Long short-term memory and marming-to-lcanne in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
[191] Alessandro Achille, Mattco Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes]. [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
20

---

(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes] To make this easier for the readers, we have referred to the pertinent Figures and Theorem to support our summary of contributions in Introduction.
(b) Did you describe the limitations of your work? [Yes] Limitations and future work can be found in our Discussion section.
(c) Did you discuss any potential negative societal impacts of your work? [No] This work contributes to the theoretical understanding of biologically-plassbbe learning rules for recurrent neural networks. As such, we do not anticipate any direct ethical or societal impact. In the long-term, our work can have impact on related research communities such as neuroscience and deep learning, with societal impact depending on the development of these fields.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have carefully read the ethics review guidelines and confirm that this paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We state all assumptions precisely when we first presented the Theorem in Results, We further details, discussions and proofs in the appendix.
(b) Did you include complete proofs of all theoretical results? [Yes] Proofs can be found in Appendix B
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Anonymized code link is provided in Appendix A.3
(b) Did you specify all the training details (e.g.. data splits, byperparameters, how they were chosen)? [Yes] All training details are provided in Appendix [A.3]
(c) Did you report error bars (c.g., with respect to the random seed after running exper-
iments multiple times)? [Yes] We tried to provide this information all applicable figures. This is stated "Solid lines/shaded regions: mean/standard deviation obtained at the end of training across five independent runs" in the legends of several figures.
(d) Did you include the total amount of compute and the type of resources used (c.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Information pertaining to computing resources and simulation time can be found in Appendix A.3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] For dataset, we cited MNIST as well as representative neuroscience papers for the two neuroscience motivated tasks. We have also cited existing coding assets in Appendix AA.3
(b) Did you mention the license of the assets? [Yes] We mentioned that in Appendix A.3]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
21


Reviews:
[{"review_id": "nRmWsix2wj", "reviewer": "Reviewer_eejL", "summary": "The paper investigates generalization and loss landscape curvature in RNNs trained with biologically-motivated learning rules. The paper finds empirically that generalization gaps are correlated with loss landscape curvature, with higher curvature correlating with larger generalization gaps. Next, the paper shows theoretically that approximations to gradient descent produce more curved solutions than gradient descent. This is because gradient descent approximations take smaller steps in the direction of the gradient and therefore are unable to escape highly curved local minima. The component of updates orthogonal to the gradient is less relevant to generalization. Experiments on multiple datasets support this explanation. Finally, the authors demonstrate empirically that appropriate learning rate scheduling in biologically-plausible learning rules can significantly enhance performance.", "questions": "This paper considers the generalization gap of different algorithms by roughly fixing their training performance and comparing their test performance. How would the results change if the test performance were fixed and the training performance were varied (by stopping the rules at different points during training, for example)?\n\nDo generalization gaps correlate with Hessian eigenvalues in networks trained with the same rule?\n\nHow would a three-factor rule perform empirically when scaled to match the along-gradient update sizes of gradient descent?\n\nAt what point does numerical instability occur in biologically-plausible learning rules?", "limitations": "The authors adequately address the limitations of the work. As the authors note, they do not consider varying architectures and leave the interaction between architecture and generalization as future work to be investigated. Moreover, the authors note that the loss landscape curvature does not fully explain generalization.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**\nAlthough generalization in biologically-motivated learning rules has been studied previously, this paper provides surprising new insights into the nature of solutions found by biologically-plausible learning rules. Notably, to my knowledge, this paper is the first to quantitatively explain with high precision the relatively poor generalization performance of biologically-plausible learning rules (see Figure 4). \n\n**Quality**\nThe theory is sound and well justified, and the experiments are generally comprehensive. The authors may want to consider some additional experiments to sharpen their claims. \n\nFirst, the authors may want to investigate whether generalization gaps correlate with Hessian eigenvalues in networks *trained with the same rule* (rather than aggregating results across rules as done in Figure 2). This would help demonstrate the generality of the claim that generalization is tied to loss landscape curvature. \n\nSecond, the authors may want to consider an additional variant of the three-factor rule where the update sizes are scaled such that the gradient-parallel component matches that of gradient descent. As the authors argue in equation, this can cause numerical instability. Nevertheless demonstrating this empirically would be helpful and would nicely complement the \"three-factor, theory\" rule.\n\nFinally, the paper seems to suggest that one of the main limitations of biologically-plausible learning rules is that using large update sizes for them leads to numerical instability while small update sizes leads to poorly generalizing local minima. Further explaining at which point this numerical instability occurs would be valuable (e.g. is the maximum stable learning rate limited by the eigenvalues of the Hessian?).\n\n\n**Clarity**\nThe paper is well written. The figures are particularly well illustrated and clear. The mathematical notation is generally adequately introduced and used.\n\nSome minor issues:\nThe W+ and W- notation in equations 7 and 8 is not explained.\nTypo- menchmark on line 140.\nTypo- assymptotic on line 1054.\n\n\n**Significance**\nThe paper may be quite significant to the field of biologically-plausible learning in RNNs. One of the important takeaways from the paper is that learning rate modulation of biologically-plausible learning rules may be a key to good performance. This may spur further research into developing better learning rate schedules for biologically-plausible learning rules. Furthermore, this paper may have implications for deep learning theory more generally.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2yeyDZ2bdw3", "reviewer": "Reviewer_qy4u", "summary": "The authors investigate the difference in the generalization gap in RNNs trained to minimize MSE loss with backpropagation through time and truncated BPTT. The authors claim the difference is caused by the landscape curvature of the loss, specifically, truncate BPTT approaches high-curvature regions in the synaptic weight space. The authors propose a theoretical argument to explain this phenomenon based on the first Hessian eigenvalue. The authors claim this result holds for all existing bio-inspired learning rules in RNNs and for different loss functions. ", "questions": "Were potential remedies tested?\n\nIn figure 2 (a, b, c), does random initialization refer to weight initialization? Are all parameters of the learning algorithm held constant or are the distributions computed across all parameter configurations, e.g., learning rate ..etc? It appears that truncated BPTT can achieve the same generalization gap as BPTT. In the case of sMNIST it appears that it achieves zero generalization gap with a greater probability than full BPTT. ", "limitations": "NA ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The authors address the important issue of how biological systems learn and adapt to stimuli. Specifically, the authors address that RNNs trained with truncated BPTT have a greater generalization gap than the full BPTT on three existing tasks.\n* The authors focus on a well-defined metric, the 1st Hessian eigenvalue, in order to quantify their results.\n* The authors provide a theoretical analysis that suggests a cause for the difference in the generalization gap between full BPTT and truncated BPTT\n\nWeaknesses:\n* The authors generalize their claims to all biologically-inspired learning rules but limit their study to three specific learning rules and three tasks. Specifically, the authors focus on truncated BPTT in RNNs on three tasks with MSE loss. These results may hold only for truncated backpropagation through time but not for all biologically-plausible learning rules. The style implies these results hold for all biologically-inspired learning rules and all loss functions. If this is true, it should be presented in the main text.  \n* The authors claim their results generalize beyond MSE loss but do not demonstrate this empirically. \n* The authors make claims of potential remedies, such as using larger learning rates early in training but do not justify this claim theoretically or demonstrate it empirically. \n* It is uncertain and not addressed if the generalization gap can be reduced by fine-tuning existing truncated BPTT algorithms.\n* A description of training parameters and procedure used for the biologically-plausible learning rules is not presented in the main test.\n* The authors cite almost 100 articles in section 2.1. Though these articles may be relevant to biologically-plausible learning in general it is not certain they are directly relevant to networks trained with truncated BPTT which is the focus of this study. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1OK9HSJhi9", "reviewer": "Reviewer_dP4V", "summary": "The  paper aims to study the generalizability of biologically plausible learning rules for RNNs compared to backprop through time which is not bio-plausible. This work relies heavily on the relationship between the curvature of the loss landscape and the generalizability of the model: that wider minima are more noise tolerant and generalize better. Empirically it is shown that bio-plausible learning rules generalize worse than the common implausible ML learning rules. This is then theoretically explained as being due to the instability introduced by the truncation of the true backprop gradient being used by the bio-plausible rules. Specifically, to avoid diverging with the bio-plausible rules a smaller learning rate can be used which makes gradient descent more susceptible to converge to steeper minima.", "questions": "Assuming I have understood the work and not misrepresented the fact above, I have no questions for clarification at present. For the rebuttal period I would find it helpful for the authors to further address the concept of different kinds of noise impacting generalization differently. ", "limitations": "The authors were clear that this study was not comparing bio-plausible learning rules but rather confirming that a well-established concept in ML carried to bio-plausible rules. This is the main limitation but it was clearly acknowledged. They also acknowledge that the relationship between curvature and generalization, while being the main point of study in this work, is messy and state that other factors may be at work for generalization.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n## Originality\nThis paper addresses a topic which I have not seen in the computational neuroscience literature and does so with care for both the machine learning and computational neuroscience considerations. I think the primary novelty of this work however comes in at the end with the finding that the noise from truncation does not affect generalization in the same manner as noise from smaller batch sizes for example (truncation noise hinders generalization while other forms of noise improve generalization by driving learning out of sharp minima). This is a new consideration to me and lead me to start thinking around whether the consistency of noisy direction of learning plays a role.\n\n## Quality\nThere are no points during the paper which stand out as being unreasonable, and from a scientific method point of view the work appears sound, with each step being justified either from a comp-neuro or ML perspective. Balancing the two topic was done well and I think that is a potentially understated positive for this work.\n\n## Clarity\nThe paper is well written and I found it to be clear and concise. The figures are well made and helpful.\n\n## Significance\nThe work appears significant from two perspectives. Firstly, work at the intersection of comp-neuro and ML is important and potentially very fruitful, however, not very common (at least not at the level of insight for both fields offered by this paper). Thus, I think this is significant to the comp-neuro community as a step towards introducing some of the more theoretical ML concepts such as loss landscape curvature. Secondly, this appears significant as it brings up the interesting concept that some noise may help generalization while other forms of noise hurt generalization. It does so in a manner which is also very useful from a learning stability perspective which also uses the Hessian matrix [1]. Thus from a theoretical ML perspective I think this work offers interesting insight and potential inspiration for future work.\n\n# Weaknesses\n## Originality and Significance\nIt must be noted that this work relies heavily of previous work in theoretical ML and tends to confirm what is already known (wider minima generalize better). That said, I think just because the results are not surprising does not mean they are not valuable. I am glad that this work has been done and acknowledge that it was worthy of confirming that the findings from ML carried towards bio-plausible learning rules. Indeed it seems to me that this work can still inspire future directs of ML work as I point out above. Thus I feel the strengths for originality and significance do outweigh this weakness.\n\nThat said, if the authors were to add to the discussion at the end about truncation noise reducing the stable step size I would be inclined to increase my rating to a 7 or an 8. The one citation below may be of use for this purpose. I do think that this paper is worth of acceptance and is of general interest to the machine learning and comp-neuro communities.\n\n[1] Cohen, Jeremy M., et al. \"Gradient descent on neural networks typically occurs at the edge of stability.\" arXiv preprint arXiv:2103.00065 (2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0Qn0mtD-iXX", "reviewer": "Reviewer_22dn", "summary": "- This paper investigates the ability of existing bio-plausible temporal credit assignment rules to generalize. Finding that training with bio-plausible learning rules results in models that have larger and more variable train/test generalization gaps. \n- It then compares the loses’ hessian eigenspectrum of bio-plausible and SoTA learning rules finding that bio-plausible rules tend to approach high curvature regions in synaptic weight space\n- It suggest an explanation for bio-plausible rules preference for high curvature regions based on worse alignment to the true gradient", "questions": "Are there other advantages to bio-plausible methods or advantages in terms of generalization can only be seen on more complex tasks?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " Strengths\n- This paper is well written and clear\n- It analyzed several learning rules and the results support the conclusion that bio-plausible methods don't generalize well\n- The theoretical results offer interesting insights about the relationship between curvature and gradient alignment\n\nWeakness\n- It is well known that bio-plausible methods generally perform worse than SOTA back propagation. This work provides further empirical evidence that this is the case, but I do not find these results very surprising.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept