Toward more detailed biological mechanisms:
On the front of more sophisticated biological ingredients that may improve generalization performance, we see two lines of approaches: 1- develop bio-plausible learning rules that align better with the gradient, as suggested by our rule-agnostic analysis (Theorem 1 , Figure 4 ); and 2- instead of studying learning rules in isolation, consider also other neural circuit components [ 152 , 153 , 154 , 155 , 156 , 157 , 158 , 159 ] that could interact with the learning rule. An important component would be the architecture, including connectivity structure and neuron model, found through evolution [ 157 ] (see also [ 160 ] ). To address our main question, this study varies learning rules while holding data, objective function and architecture constant (see [ 10 ] ). However, these different components can interact, and more sophisticated architecture can facilitate task learning [ 161 , 162 , 163 , 164 , 165 , 166 , 160 , 167 ] . Given the exploding parameter space resulting from such interactions, we believe it requires careful future analysis and is outside of the scope for this one paper.

Additionally, learning rate modulation [ 168 , 169 ] could be one of many possible remedies employed by the brain. We conjecture that neuromodulatory mechanisms could be coupled with these learning rules to improve the convergence behavior through our scheduled learning rate experiments (Appendix Figure 5 ), where an initial high learning rate could prevent the learning trajectory from settling in sharp minima prematurely followed by gradual decay to avoid instabilities. One possible way to realize such learning rate modulation could be through serotonin neurons via uncertainty tracking, where the learning rate is high when the reward prediction error is high (this can happen at the beginning of learning) [ 170 ] . Since the authors of [ 170 ] showed that inhibiting serotonin led to failure in learning rate modulation, we conjecture that such inhibition might have an impact on the generalization performance of learning outcomes. On the topic of balancing numerical instabilities and potential advantages of large learning rates, while the analog nature of biology may seem to avoid finite precision representation in digital computers that give rise to numerical instabilities, the same problems that lead to numerical instabilities in digital computers, such as big ranges between quantities added or multiplied, remains an issue for biology since quantities must be stored in noisy activity patterns of neurotransmitter release. Future investigations could investigate potential homeostatic mechanisms that regulate biological quantities to avoid such instabilities, thereby enabling larger "learning rates” to be used so as to find flatter minima. Other ingredients for future investigations could include intrinsic noise with certain structures [ 171 , 172 , 154 ] (including directions and bias/variance properties) that would make them more favorable for generalization. Taken together, we hope to see follow-up investigations — riding on the rapid advancements both at the front of deep learning theory and sophisticated biological mechanisms — into how the brain attains solutions that generalize.