A.3 Simulation details
We used TensorFlow [ 190 ] version 1.14 and based it on top of [ 191 ] . We modified the code for rate-based neurons (Eq. 10 and 12 ). 1 1 1 Our code is available: https://github.com/Helena-Yuhan-Liu/BiolHessRNN . We used the code in [ 176 ] for the power-law analysis (Figure 7 in Appendix). SGD optimizer was used to study the effect of gradient approximation in isolation without the complication of additional factors, as Adam optimizer with adaptive learning rate could convolute our matching step length experiments in Figure 4 . That said, we verified that the curvature convergence behavior is also observed for Adam optimizer (Figure 11 in Appendix. Learning rates were optimized by picking within \{3e-4,1e-3,3e-3,1e-2,3e-2,1e-1\} for each algorithm. For the sequential MNIST task, we explored batch sizes within \{64,256,1024\} . For the sequential MNIST task, these hyperparameters were optimized based on validation performance (the validation set loaded using tensorflow.examples.tutorials.mnist ). For the two other tasks, these hyperparameters were optimized based on the training performance, but we also tried optimizing on the test set and observed similar trends. Trainings were stopped when both the loss and leading Hessian eigenvalue stabilized. As stated, we repeated runs with different random initialization to quantify uncertainty and weights were initialized similarly as in [ 12 ] .

Simulations were completed on a computer server with x2 20-core Intel(R) Xeon(R) CPU E5-2698 v4 at 2.20GHz. The average time to complete one run of sequential MNIST, pattern generation and delayed match to sample tasks in Figure 3 were approximately 2 hours, 1 hour and 1 hour, respectively. Since the computation of second order gradient becomes prohibitively expensive as sequence length T becomes large, all tasks involved no more than 50 time steps. For instance, this was achieved for the sequential MNIST task using the row-by-row implementation. Using fewer time steps, however, should not affect the general trend as the gradient truncation effects were still significant. Because of the use of fewer steps, we dropped the leak factor \alpha in Eq. 10 (i.e. set \alpha=0 ).

For the matching step length experiments (Figure 4 ), we simply obtained \rho=\frac{\vec{\hat{g}}^{T}\vec{g}}{\vec{g}^{T}\vec{g}} for the three-factor learning rule and scaled BPTT updates by that amount. For scheduled learning rate experiments (Figure 5 ), the additional hyperparameters included initial learning rate, decay percentage and decay frequency. We used an initial learning rate that was three times the uniform rate (used in other figures) and decay the learning rate by 80\% every X iterations, where X was roughly the total number of training iterations (used in other figures) divided by 30. Since the point of that figure was to show that learning rate scheduling could lead to flatter minima than using a fixed learning rate, we did not search extensively across these additional hyperparameters as the first set of hyperparameters we tried was enough to demonstrate that point.

For the pattern generation task, our network consisted of N=30 neurons described in Eq. 10 . Input to this network was provided by a random Gaussian input ( N_{in}=1 ). The fixed target signal had a duration of 50 steps and was given by the sum of four sinusoids, with a fixed period of 10, 40, 70 and 100 steps. For learning, we used the mean squared loss function. Training for this task used full batch. For testing, we perturbed the input with additive zero-mean Gaussian noise (with \sigma picked uniformly between 0 and 0.2 across runs), to mimic the situation where the agent had to faithfully produce the desired pattern even under perturbations. Unlike the other tasks, this task measures accuracy by mean squared error, for which the lower the better. To maintain the convention of a higher generalization gap being worse, the generalization gap for this task was computed by test error minus train error.

For the delayed match to sample task, our network consisted of N=100 neurons, which include 50 neurons with (Eq. 12 ) and 50 neurons without (Eq. 10 ) threshold adaptation. The task involved the presentation of two sequential cues, each taking on a binary value, lasting 2 steps and separated by a delay of 16 steps. Input to this network was provided by N_{in}=2 neurons. The first (resp. second) input neuron sent a value of 1 when the presented cue took on a value of 1 (resp. cue 0), and 0 otherwise. The network was trained to output 1 (resp. 0) when the two cues have matching (resp. non-matching) values. For learning, we used the cross-entropy loss function and the target corresponding to the correct output was given at the end of the trial. Training for this task used full batch. For testing, we tested on increased delay, with the period picked uniformly between the training delay period and twice the training delay period, to mimic situations where the animal has to hold the memory longer than it did during the learning phase.

For the sequential MNIST task [ 138 ] , our network consisted of N=128 neurons described in Eq. 10 . Input to this network was provided by N_{in}=28 units that represented the grey-scaled value of a single row, totaling 28 steps and the network prediction was made at the last step. For learning, we used the cross-entropy loss function and the target corresponding to the correct output was given at the end of the trial. For testing, we used the existing MNIST test set [ 138 ] with additive zero-mean Gaussian input noise. As mentioned, this task did not train with a full batch but we found the trend to hold across different batch sizes.

Finally, we note that comparisons between BPTT and approximate rules were done at comparable training accuracies for the pattern generation and delayed match to sample tasks. For the sequential MNIST task, the three-factor rule achieved only around 70\% training accuracy, but the training accuracy did not explain the curvature convergence behavior. To see this, while three-factor theory (blue in Fig 4), which corresponds to BPTT with reduced step length, achieves an accuracy of >95\% but still attains similar curvature convergence to that of the three-factor rule.