A.2 Network setup and learning rule implementations
Neuron Model : We consider a discrete-time implementation of a rate-based recurrent neural network (RNN) similar to the form in [ 137 ] . The model denotes the internal hidden state as h_{t} and the observable states, i.e. firing rates, as f(h_{t}) at time t , and we use ReLU activation for f . The dynamics of those states are governed by

 h_{t+1} =\alpha h_{t}+(1-\alpha)\left(W_{h}f(h_{t})+W_{x}x_{t}\right), (10)

where \alpha=e^{-dt/\tau_{m}} denotes the leak factor for simulation time step dt and membrane time constant \tau_{m} , W_{h} denotes the weight of the recurrent synaptic connection, W_{x} denotes the strength of the input synaptic connection and x_{t} denotes the external input at time t . we use subscripts to represent indices of neurons and time steps. For instance, h_{i,t} represents the hidden activity h of neuron i at time t . W_{h,ij} represents the (ij)^{th} entry of recurrent weight matrix W_{h} . Model in Eq. 10 was used for the sequential MNIST and pattern generation tasks.

We mention in passing that the choice of ReLU activation, which has a discontinuous first derivative, means that the loss Hessian matrix is not guaranteed to be symmetric. A real matrix that is not symmetric can have complex eigenvalues come in conjugate pairs, and if they were amongst the top eigenvalues, power iterations may not converge. However, all iterations have converged in our experiments as mentioned above. Also, because of potential technical issues resulting from non-symmetric Hessian matrices, we foresee challenges in applying our methodology to spiking neural networks (SNNs), which have discontinuous activation functions. Due to the energy efficiency and biological realism of SNNs [ 179 , 180 , 181 , 182 , 183 , 184 , 185 , 186 , 187 , 188 , 97 ] , we believe extending to SNNs is an important future direction.

For the delayed match to sample task, which is a working memory task, it was found in [ 13 ] and [ 11 ] that units with an adaptive threshold as an additional hidden variable can play an important role in the computing capabilities of RNNs. Thus, we implemented adaptive threshold neuron units [ 165 ] for that task. In our rate-based implementation, this turns out to be a simple addition of a second hidden variable b_{t} that represents the dynamic threshold component:

 h_{t+1} =\alpha h_{t}+(1-\alpha)\left(W_{h}f(h_{t}-b_{t})+W_{x}x_{t}\right), (11) b_{t+1} =\beta b_{t}+(1-\beta)f(h_{t}-b_{t}), (12)

where b_{j,t} denotes the dynamic threshold that adapts based on past neuron activity. The decay factor \beta is given by e^{-dt/\tau_{b}} for simulation time step dt and adaptation time constant \tau_{b} , which is typically chosen on the behavioral task time scale [ 13 ] .

Network output and loss function :

Readout \hat{y} is defined as

\hat{y}=\langle w,f(h_{t})\rangle (13)

for readout weights w .

We quantify how well the network output \hat{y} matches the desired target y using loss function L , which is defined as

L(W_{h})=\begin{cases}\frac{1}{2TB}\sum_{i=1}^{B}\sum_{t=1}^{T}\sum_{k=1}^{N_{out}}(\hat{y}^{(i)}_{k,t}-y^{(i)}_{k,t})^{2}, for regression tasks\cr\frac{-1}{TB}\sum_{i=1}^{B}\sum_{t=1}^{T}\sum_{k=1}^{N_{out}}\pi^{(i)}_{k,t}log\hat{\pi}^{(i)}_{k,t}, for classification tasks\end{cases} (14)

for target output y , task duration T , N_{out} output neurons and batch size B . \pi_{k,t} is the one-hot encoded target and \hat{\pi}_{k,t}=\text{softmax}_{k}(\hat{y}_{1,t},\dots,\hat{y}_{N_{OUT},t})=\exp(\hat{y}_{k,t})/\sum_{k^{\prime}}\exp(\hat{y}_{k^{\prime},t}) is the predicted category probability.

Biological gradient approximations (truncation-based)

The goal of this subsection is to explain where the approximation happens for each of the bio-plausible learning rules. For full details regarding these rules, we encourage the reader to refer to the respective references. We start by writing down the gradient in terms of real-time recurrent learning (RTRL) factorization:

\frac{\partial L}{\partial W_{h,ij}} = \sum_{l,t}\frac{\partial L}{\partial h_{l,t}}\frac{\partial h_{l,t}}{\partial W_{h,ij}}, (15)

Key problems that RTRL poses to biological plausibility and computational cost reside in the second factor \frac{\partial h_{l,t}}{\partial W_{h,ij}} that arises during the factorization of the gradient (Eq. 15 ). The factor \frac{\partial h_{l,t}}{\partial W_{h,ij}} keeps track of all recursive dependencies of h_{l,t} on weight W_{h,ij} arising from recurrent connections. These recurrent dependencies can be obtained recursively as follows:

\frac{\partial h_{l,t}}{\partial W_{h,ij}} =\frac{\partial h_{j,t}}{\partial W_{h,ij}}+\sum_{m}\frac{\partial h_{l,t}}{\partial h_{m,t-1}}\frac{\partial h_{m,t-1}}{\partial W_{h,ij}} (16) =\frac{\partial h_{l,t}}{\partial W_{h,ij}}+\frac{\partial h_{l,t}}{\partial h_{l,t-1}}\frac{\partial h_{l,t-1}}{\partial W_{h,ij}}+{\underbrace{\textstyle\sum_{m eq l}W_{h,lm}f^{\prime}(h_{m,t-1})\frac{\partial h_{m,t-1}}{\partial W_{h,ij}}}_{\mathclap{ ormalsize depends on all weights $W_{h,lm$}}}}

Thus, the factor \frac{\partial h_{l,t}}{\partial W_{h,ij}} poses a serious problem for biological plausibility: it involves nonlocal terms that should be inaccessible to neural circuits, i.e. that knowledge of all other weights in the network is required in order to update the weight W_{h,ij} .

RFLO [ 12 ] (labeled as "RF Three-factor") and symmetric e-prop [ 11 ] (labeled as "Three-factor") seek to address this by truncating the expensive nonlocal terms in Eq. 17 so that the updates to weight W_{h,ij} would only depend on pre- and post-synaptic activity:

\widehat{\frac{\partial h_{l,t}}{\partial W_{h,ij}}} =\begin{cases}\frac{\partial h_{i,t}}{\partial W_{h,ij}}+\frac{\partial h_{i,t}}{\partial h_{i,t-1}}\widehat{\frac{\partial h_{i,t-1}}{\partial W_{h,ij}}},&l=i\ 0,&l eq i\end{cases} (18)

which results in a much simpler factor than the triple tensor in Eq. 17 .

After the truncation, RFLO and e-prop implement:

\widehat{\frac{\partial L}{\partial W_{h,ij}}} = \sum_{t}\frac{\partial L}{\partial h_{i,t}}\widehat{\frac{\partial h_{i,t}}{\partial W_{h,ij}}}, (19) \widehat{\frac{\partial h_{i,t}}{\partial W_{h,ij}}} = \frac{\partial h_{i,t}}{\partial W_{h,ij}}+\frac{\partial h_{i,t}}{\partial h_{i,t-1}}\widehat{\frac{\partial h_{i,t-1}}{\partial W_{h,ij}}}

The main difference between symmetric e-prop and RFLO implementation is that symmetric feedback is used for symmetric e-prop, i.e. output weight w is used as the feedback weight for the \frac{\partial E}{\partial h} , whereas RFLO uses fixed random feedback weights [ 75 ] for greater biological plausibility. We note in passing that the authors of e-prop have tested their formulation with fixed random feedback weights as well. MDGL [ 13 ] also truncates RTRL, but it restores some of the non-local dependencies – those within one connection step — that could potentially be communicated via mechanisms similar to the abundant cell-type-specific local modulatory signaling unveiled by recent transcriptomics data [ 91 , 90 ] . With that, the expensive memory trace term in Eq. 17 becomes

\frac{\partial h_{l,t}}{\partial W_{h,ij}}\approx\begin{cases}W_{h,li}f^{\prime}(h_{i,t-1})\widehat{\frac{\partial h_{i,t-1}}{\partial W_{h,ij}}},&i eq l\vspace{0.1in}\ \frac{\partial h_{i,t}}{\partial W_{h,ij}}+\frac{\partial h_{i,t}}{\partial h_{i,t-1}}\widehat{\frac{\partial h_{i,t-1}}{\partial w_{ij}}},&i=l\end{cases} (21)

MDGL involves one additional approximation: replace W_{h,li} with type-specific weights W_{ab} to mimic the cell-type-specific nature of local modulatory signaling (for cell i in group a and cell j in group b , where a,b\in C for a total of C cell groups). For simplicity, we just used W_{ab}=W_{li} , i.e. without cell-type approximation. This results in overall MDGL implementation as

\widehat{\frac{\partial L}{\partial W_{h,ij}}} = \sum_{t}\frac{\partial L}{\partial h_{i,t}}\widehat{\frac{\partial h_{i,t}}{\partial W_{h,ij}}}+\widehat{\frac{\partial h_{i,t-1}}{\partial W_{h,ij}}}\sum_{l}W_{h,li}\frac{\partial L}{\partial h_{l,t}}, (22)

Interpretation of the above update rule in terms of biological processes can be found in the MDGL paper [ 13 , 189 ] .

We note that input, recurrent and output weights were all being trained. This section illustrates the approximate gradient for updating recurrent weights W_{h} , and similar expressions were obtained for updating input weights W_{x} . The approximations, however, did not apply to output weights, as the gradient for that would not violate the aforementioned issue of nonlocality (Eq. 17 ).