RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Bio-plausible gradient approximations
Investigating bio-plausible learning rules is of interest both to identify more efficient training strategies for artificial networks and to better understand learning in the brain [ 14 , 15 , 16 , 17 , 18 , 19 , 2 , 3 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ] . In order for learning to reach a certain goal quantified by an objective, learning algorithms often minimize a loss function [ 10 ] . The error gradient, if available, tells us how each parameter should be adjusted in order to have the steepest local descent. For training RNNs, which are widely used as a model for neural circuits [ 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 ] , standard algorithms that follow this gradient — real time recurrent learning (RTRL) and BPTT — are not bio-plausible and have overwhelming memory storage demands [ 64 , 3 ] . However, learning rules that only approximate the true gradient can sometimes be as effective as those that follow the gradient exactly [ 10 , 65 ] . Because of that, bio-plausible learning rules that approximate the gradient using known properties of real neurons have been proposed and led to successful learning outcomes across many different tasks in feedforward networks [ 66 , 1 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 ] , with recent extensions to recurrently connected networks [ 12 , 11 , 13 ] . These existing bio-plausible rules for training RNNs [ 12 , 11 , 13 ] are truncation-based (which is the focus of this study), so that the untruncated gradient terms can be assigned with putative identities to known biological learning ingredients: eligibility traces, which maintain preceding activity on the molecular levels [ 77 , 78 , 79 , 80 , 81 , 82 ] , combined with top-down instructive signaling [ 78 , 77 , 83 , 84 , 85 , 86 , 87 , 88 , 89 ] as well as local cell-to-cell modulatory signaling within the network [ 90 , 91 , 13 ] . For efficient online learning in RNNs, other approximations (not necessarily bio-plausible) to RTRL [ 92 , 93 , 94 , 95 , 96 , 97 ] have also demonstrated to produce a good performance. Given the impressive accuracy achieved by these approximate rules, several studies began to investigate their convergence properties [ 98 ] , e.g. for random backpropagation weights in feedforward networks [ 99 , 100 ] . However, the trend of generalization capabilities of these rules, especially in RNNs, is underexamined.
Loss landscape curvature and generalization performance
Given the central importance of understanding how neural networks perform in situations unseen during training [ 101 , 4 , 102 , 103 , 104 , 105 , 106 , 107 ] , the deep learning community has made tremendous efforts to develop tools for understanding generalization that we leverage here.
That flat minima could lead to better generalization was observed more than two decades ago [ 108 ] . Intuitively, under the same amount of perturbation \epsilon in parameter space (e.g. loss landscape changes due to the addition of new data) worse performance degradation will be seen around the narrower minima (see Figure 1 C). We note that perturbations in parameter space can be linked to that in the input space [ 6 ] . Recently, many empirical studies have consistently supported the usefulness of this predictor [ 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 ] . In particular, the authors of [ 5 ] performed an extensive study and found that flatness-based measures have a higher correlation with generalization than other alternatives. Motivated by this, several studies have characterized properties of the loss functions’s Hessian — whose eigenspectrum carries information about curvature [ 117 , 118 , 119 , 120 , 121 ] . Connections between flatness and generalization performance have shed light on the reason for greater generalization gaps in large batch training [ 112 , 122 , 123 , 124 , 125 ] , and also have inspired optimization methods to favor flatter minima [ 126 , 109 , 110 , 127 , 128 , 129 , 130 , 131 , 132 , 133 ] . Despite the criticism of scale-dependence of flatness [ 134 ] , where parameter rescaling can drastically change flatness but not always generalization quality, flatness — with parameter scales taken into account [ 135 , 136 ] — are connected to PAC-Bayesian generalization error bounds [ 7 , 8 , 9 ] . Moreover, a recent theoretical study rigorously connects the flatness of the loss surface to generalization in classification tasks under the assumption that labels are (approximately) locally constant [ 6 ] . Leveraging the great progress from the deep learning community, we aim to study the generalization properties of bio-plausible learning rules from a geometric perspective.
Results
In this section, we first describe the network and learning setup we use (Figure 1 A). Next, we present a number of numerical experiments where we compute the generalization gap directly on three commonly used ML and neuroscience tasks (Figure 2 A-C), for truncation-based bio-plausible gradient approximations. For the same experiments, we also quantify loss landscape curvature along learning trajectories, and connect these quantities to generalization behavior (Figures 2 D-F and 3 ). Finally, we provide theoretical arguments and a theorem that explains how gradient alignment in bio-plausible gradient approximations can affect curvature preference (Theorem 1 , Figure 3 ) and thus, generalization. Through additional experiments, we verify the predictive power of our theory. We conclude with discussions on potential remedies used by the brain (Appendix Figure 5 ) and future directions.
Generalization gap and loss landscape curvature
To study generalization performance, our first step is to compute the generalization gap empirically. Generalization gap is defined as train accuracy minus test accuracy; the larger the generalization gap, the worse the generalization performance. For various learning rules, we plot the generalization gap histogram at the end of training across runs with distinct initializations; different colors represent different learning rules (Figure 2 A-C). Notice that these bio-plausible rules achieve worse and more variable generalization performance than their machine learning counterpart (BPTT).

We now investigate if the generalization gap behavior described above correlates well with Loss landscape curvature. We use the leading eigenvalue of the loss’ Hessian (where derivatives are taken with respect to model parameters) as a measure for curvature following previous practice [ 122 ] and note that it is practical for both empirical [ 119 ] and theoretical analyses (Theorem 1 ). There exist other measures for flatness and due to the scale-dependence issue of Hessian spectrum [ 134 ] , we also test using parameter scale-independent measures (see Appendix Figures 7 and 8 ).
When we plot generalization gap points from Figure 2 A-C against the corresponding leading Hessian eigenvalue, a statistically significant correlation is observed (Figure 2 D-F). We also observed such correlation across runs with the learning rule fixed (Appendix Figure 12 ). We note that we did not expect this relationship to be very tight since in addition to the worse generalization gap on average, bio-plausible learning rules exhibit increased variance. This is an important and consistent trend we observed but might have been overlooked in previous studies.

So far, we investigated the endpoints of optimization trajectories, where training performance has converged. Now, we visualize the whole training trajectory. This done is for two reasons: 1) account for early stopping that can halt training anywhere along the trajectory due to time constraints; 2) flatness during training could give indications of avoiding or escaping high-curvature regions. We observe that the biologically motivated gradient approximations tend to rapidly approach high-curvature regions compared to their machine learning counterpart (Figure 3 ).
Together, these results demonstrate a clear trend and a link between the generalization gap and loss landscape curvature: both being increased and more variable for bio-plausible rules. We also stopped BPTT early to match the test accuracy of the three-factor rule, and observed similar trends as in the main text (Appendix Table 1 ). We also note that the curvature convergence behavior seems to be a shared problem of temporal truncations of the gradient (Appendix Figure 9 ), which is what the existing bio-plausible gradient approximations for RNNs are based on.
Next, we provide a theoretical argument as to why truncated temporal credit assignment rules favor high curvature regions of the loss landscape.
Conclusion
While developing bio-plausible learning rules is of interest for both answering neuroscience questions and searching for more efficient training strategies for artificial networks, the generalization properties of solutions found by these rules are severely underexamined.
Through various well-known machine learning and working memory tasks, we first demonstrate empirically that existing bio-plausible temporal credit assignment rules attain worse generalization performance, which is consistent with their tendency to converge to high-curvature regions in loss landscape. Second, our theoretical analysis offers an explanation for this preference for high curvature regions based on worse alignment to the true gradient. This regime corresponds to the situation where the step length along the gradient direction is small and the approximation error vector is large. Finally, we test this theory empirically by matching the relative step length along the gradient direction resulting in similar convergence behavior (Figure 4 ).
Discussion
Our study — a stepping stone toward understanding biological generalization using deep learning methods — raises many exciting questions for future investigations, both on the front of stronger deep learning theory and more sophisticated biological ingredients.
Acknowledgement
We thank Gauthier Gidel, Jonathan Cornford, Aristide Baratin, Thomas George and Mohammad Pezeshki for insightful discussions and helpful feedback at an early stage of this project. We also thank Henning Petzka and Michael Kamp for helpful email exchanges. This research was supported by NSERC PGS-D (Y.H.L.); NSF AccelNet IN-BIC program, Grant No. OISE-2019976 AM02 (Y.H.L.); Vanier Canada Graduate scholarship (A.G.); Healthy Brains for Healthy Lives fellowship (A.G.); CIFAR Learning in Machines and Brains Program (B.A.R.), NSERC Discovery Grant RGPIN-2018-04821 (G.L), FRQS Research Scholar Award, Junior 1 LAJGU0401-253188 (G.L.), Canada Research Chair in Neural Computations and Interfacing (G.L.), Canada CIFAR AI Chair program (G.L. & B.A.R.). We also thank the Allen Institute founder, Paul G. Allen, for his vision, encouragement, and support.
A.1 Hessian eigenspectrum analysis
As mentioned, we focus on the leading Hessian eigenvalue because it has been used previously [ 122 , 173 ] and is very feasible for both empirical [ 119 , 174 ] and theoretical analyses (Theorem 1 ).
The leading Hessian eigenvalue can be computed by performing power iterations on the Hessian vector product without knowing the full Hessian matrix (Algorithm 2 in [ 119 ] ). To find multiple top eigenvalues, e.g. top 200 eigenvalues, one can use the generalized power method via QR decomposition [ 175 ] . We focused on the loss’ Hessian for recurrent weights but observed a similar trend for input weights as well. We set the tolerance for stopping to 1e-6 .

Due to the scale-dependence issue of Hessian spectrum [ 134 ] , we also used scale-independent measures. For instance, we examined the power-law decay coefficient for the Hessian eigenvalues (Figure 7 in Appendix). We also looked at the recently proposed relative flatness measure [ 6 ] (Figure 8 in Appendix). We used the code in [ 176 ] to fit a power-law distribution to the top 200 eigenvalues. We found similar results had we chosen the top 50 or 100 eigenvalues instead, and 200 was chosen mainly due to computational load. We note that the link of power-law decay to generalization has also been examined in some recent studies [ 177 , 121 , 178 ] .
B.2 Discussion on tightness of the bound
Following the derivation, it is clear that the tightness of the bound will depend on how close the magnitude of leading Jacobian eigenvalue is to 1 upon convergence. That is related to the distribution of minima flatness along the loss landscape, which impacts the probability of a rule converging to a minima with flatness in a certain range. Such distribution is likely problem dependent. If the loss were convex, there would just be one minimum and the question of minima preference would become irrelevant.
Figure 1: Setup . A) Illustration of an RNN trained to minimize error/loss function L (left). Existing bio-plausible proposals for RNNs estimate the gradient by neglecting dependencies that are biologically implausible to compute (right). B) Low training error/loss can be achieved by partially following a gradient (right), but the preference for converging to minima with certain generalization properties remains underexamined for these learning rules (left). C) Minima flatness matters: 1-D loss landscape illustration with two solutions that equally minimize loss L , but exhibit drastically different generalization properties: the narrower minima are more sensitive to perturbation.; Setup . A) Illustration of an RNN trained to minimize error/loss function L (left). Existing bio-plausible proposals for RNNs estimate the gradient by neglecting dependencies that are biologically implausible to compute (right). B) Low training error/loss can be achieved by partially following a gradient (right), but the preference for converging to minima with certain generalization properties remains underexamined for these learning rules (left). C) Minima flatness matters: 1-D loss landscape illustration with two solutions that equally minimize loss L , but exhibit drastically different generalization properties: the narrower minima are more sensitive to perturbation.
Figure 2: Bio-plausible temporal credit assignment rules show worse and more variable generalization gap, which can be informed by loss landscape curvature . A-C) Generalization gap distributions computed at the end of training across different random weight initializations for several well-known neuroscience and machine learning tasks. The higher the generalization gap, the worse the generalization performance. BPTT (black), bio-plausible alternatives (magenta, yellow and green). D-F) Scatter plots showing the trend of generalization gap v.s. leading loss Hessian eigenvalue across many runs; each point corresponds to a single run of the same runs as in A-C.; Bio-plausible temporal credit assignment rules show worse and more variable generalization gap, which can be informed by loss landscape curvature . A-C) Generalization gap distributions computed at the end of training across different random weight initializations for several well-known neuroscience and machine learning tasks. The higher the generalization gap, the worse the generalization performance. BPTT (black), bio-plausible alternatives (magenta, yellow and green). D-F) Scatter plots showing the trend of generalization gap v.s. leading loss Hessian eigenvalue across many runs; each point corresponds to a single run of the same runs as in A-C.
Figure 3: Bio-plausible gradient approximations tend to approach high curvature regions in loss landscape . Dominant Hessian eigenvalues are plotted throughout training for bio-plausible learning rules and BPTT. This analysis is done for A) sequential MNIST, B) pattern generation and C) delayed match-to-sample tasks. Solid lines/shaded regions: mean/standard deviation of dominant Hessian eigenvalue curves across five independent runs.; Bio-plausible gradient approximations tend to approach high curvature regions in loss landscape . Dominant Hessian eigenvalues are plotted throughout training for bio-plausible learning rules and BPTT. This analysis is done for A) sequential MNIST, B) pattern generation and C) delayed match-to-sample tasks. Solid lines/shaded regions: mean/standard deviation of dominant Hessian eigenvalue curves across five independent runs.
Figure 4: Preference for high curvature regions connected to worse gradient alignment under certain conditions . A) An arbitrary gradient approximation, \vec{\hat{g}} , its component along the gradient direction, \rho\vec{g} , and orthogonal to the gradient \vec{e} (Eq. 6 ). The scalar \rho represents the relative step length along the gradient direction. B) Illustration for Theorem 1 : if gradient \vec{g} is aligned with the sharpest directions but error \vec{e} is not (see Appendix Figure 10 ), smaller step length along the gradient direction can make it harder to step over narrow minima. C-E) Leading loss Hessian eigenvalue v.s. training iteration for the three-factor rule, BPTT, and modified BPTT (three-factor, theory). For the latter, step length along the gradient direction of BPTT was matched to three-factor rule by multiplying BPTT update with a factor of \rho , which recovers curvature trends of the three-factor rule.; Preference for high curvature regions connected to worse gradient alignment under certain conditions . A) An arbitrary gradient approximation, \vec{\hat{g}} , its component along the gradient direction, \rho\vec{g} , and orthogonal to the gradient \vec{e} (Eq. 6 ). The scalar \rho represents the relative step length along the gradient direction. B) Illustration for Theorem 1 : if gradient \vec{g} is aligned with the sharpest directions but error \vec{e} is not (see Appendix Figure 10 ), smaller step length along the gradient direction can make it harder to step over narrow minima. C-E) Leading loss Hessian eigenvalue v.s. training iteration for the three-factor rule, BPTT, and modified BPTT (three-factor, theory). For the latter, step length along the gradient direction of BPTT was matched to three-factor rule by multiplying BPTT update with a factor of \rho , which recovers curvature trends of the three-factor rule.
Figure 5: Learning rate modulation as a possible remedy of the problem . We increased the learning rate at the beginning of training to prevent the three-factor rule from stabilizing in sharp minima prematurely, followed by a gradual decay to prevent instability. The top panels show this strategy helps to reduce the curvature of the converged solution. The bottom panels show this leads to a slight improvement in the generalization gap (vertical lines denote distribution mean). However, it is important to note that this strategy does not correct the problem; the gap still exists compared to BPTT, suggesting room for further research. Plotting conventions follow that of the previous figures.; Learning rate modulation as a possible remedy of the problem . We increased the learning rate at the beginning of training to prevent the three-factor rule from stabilizing in sharp minima prematurely, followed by a gradual decay to prevent instability. The top panels show this strategy helps to reduce the curvature of the converged solution. The bottom panels show this leads to a slight improvement in the generalization gap (vertical lines denote distribution mean). However, it is important to note that this strategy does not correct the problem; the gap still exists compared to BPTT, suggesting room for further research. Plotting conventions follow that of the previous figures.
Figure 6: Modified BPTT (three-factor, theory) resulted in worse and more variable generalization performance . Here, we follow the convention of previous generalization gap histogram plots and investigated the generalization performance of modified BPTT (three-factor, theory) in Figure 4 .; Modified BPTT (three-factor, theory) resulted in worse and more variable generalization performance . Here, we follow the convention of previous generalization gap histogram plots and investigated the generalization performance of modified BPTT (three-factor, theory) in Figure 4 .
Figure 7: Loss’ Hessian eigenspectrum for the three-factor rule exhibits significantly steeper power-law decay compared to that of BPTT . We fit a power-law function to the top 200 eigenvalues at the end of training and measure the decay parameter. Fitting to the top 50 or 100 eigenvalues resulted in similar trends. Solid lines/shaded regions: mean/standard deviation of eigenspectrum obtained at the end of training across five independent runs.; Loss’ Hessian eigenspectrum for the three-factor rule exhibits significantly steeper power-law decay compared to that of BPTT . We fit a power-law function to the top 200 eigenvalues at the end of training and measure the decay parameter. Fitting to the top 50 or 100 eigenvalues resulted in similar trends. Solid lines/shaded regions: mean/standard deviation of eigenspectrum obtained at the end of training across five independent runs.
Figure 8: Curvature preference behavior corroborated using relative flatness measure [ 6 ] . Here, the trend is consistent with that of Figure 3 . Note that the relative flatness measure can be computationally intensive for recurrent weights, so we computed it for readout weights. Plotting conventions follow that of previous figures.; Curvature preference behavior corroborated using relative flatness measure [ 6 ] . Here, the trend is consistent with that of Figure 3 . Note that the relative flatness measure can be computationally intensive for recurrent weights, so we computed it for readout weights. Plotting conventions follow that of previous figures.
Figure 9: Approaching high curvature regions seems to be a shared problem for temporal truncations of the gradient . We repeat the analysis in Figure 3 for truncated BPTT (TBPTT). Here, "Trunc X" means X time steps are truncated during the gradient calculation. We observe that TBPTT tends to converge to high curvature regions. Plotting conventions follow that of previous figures.; Approaching high curvature regions seems to be a shared problem for temporal truncations of the gradient . We repeat the analysis in Figure 3 for truncated BPTT (TBPTT). Here, "Trunc X" means X time steps are truncated during the gradient calculation. We observe that TBPTT tends to converge to high curvature regions. Plotting conventions follow that of previous figures.
Figure 10: Truncation error vector (compared to the gradient) is significantly less aligned with the top Hessian eigenvector subspace . Following [ 173 ] , we compute the cosine similarity between the error vector (of the three-factor rule) and a top Hessian eigenvector (averaged over the top 5 eigenvectors). The absolute value of the cosine similarity was taken. We observe weak alignment of the approximation error vector \vec{e} with the leading Hessian eigenvectors. Similar trends were attained had we averaged over the leading 10 or 20 eigenvectors. Plotting conventions follow that of previous figures.; Truncation error vector (compared to the gradient) is significantly less aligned with the top Hessian eigenvector subspace . Following [ 173 ] , we compute the cosine similarity between the error vector (of the three-factor rule) and a top Hessian eigenvector (averaged over the top 5 eigenvectors). The absolute value of the cosine similarity was taken. We observe weak alignment of the approximation error vector \vec{e} with the leading Hessian eigenvectors. Similar trends were attained had we averaged over the leading 10 or 20 eigenvectors. Plotting conventions follow that of previous figures.