SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\partial \bar {\mathbf {r}} (t , \mathbf {r})}{\partial \mathbf {W} _ {i j}} = \frac {\partial \bar {\mathbf {r}} (t , \mathbf {r})}{\partial \bar {\mathbf {r}} (t - 1 , \mathbf {r})} \frac {\partial \bar {\mathbf {r}} (t - 1 , \mathbf {r})}{\partial \mathbf {W} _ {i j}} + g (\mathbf {r} (t - 1)) \tag {E.45}
$$
$$
g (\mathbf {r} (t - 1)) _ {k} = \left\{ \begin{array}{l l} h ^ {\prime} \left(\mathbf {V} _ {i} (t)\right) \mathbf {r} _ {j} (t - 1) & \text {i f} i = k \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {E.46}
$$
this second equation provides a recursive update equation which can be stored online as a trial progresses. The  $g(\mathbf{r}(t - 1))$  term is  $p$ -local, because it appears in Eq. E.43. However,  $\frac{\partial\bar{\mathbf{r}}(t,\mathbf{r})}{\partial\bar{\mathbf{r}}(t - 1,\mathbf{r})}$ , an  $N\times N$  Jacobian matrix, is not  $p$ -local, since it depends on all neurons in the network  $\mathbf{r}(t - 1)$  as well as all parameters  $\mathbf{W}$ —neurons that do not directly synapse onto neuron  $\mathbf{r}_i$  and weights  $\mathbf{W}_{kl}$  for  $k\neq i$  are excluded from  $p$ -local updates by Property 2.1 according to the DAG defined by Eq. E.39. Furthermore, as we have seen with backpropagation, in general the credit assignment signal  $\frac{\partial\mathcal{L}(\mathbf{r}(T))}{\partial\bar{\mathbf{r}}(T)}$  is not  $p$ -local. Therefore, to characterize the  $S_p$ -locality of RTRL, we will have to proceed similarly to backpropagation, and define auxiliary variables to include in the set  $\mathbf{S}$ .
As with backpropagation, we define the auxiliary random variable  $\mathbf{e} = \frac{\partial\mathcal{L}(\mathbf{r}(T))}{\partial\bar{\mathbf{r}}(T)}$ . Because we have found the Jacobians to also violate  $p$ -locality, we will also define the set of auxiliary variables  $\mathbf{J} = \{\mathbf{J}(t) = \frac{\partial\bar{\mathbf{r}}(t,\mathbf{r})}{\partial\bar{\mathbf{r}}(t - 1,\mathbf{r})}\}$ . With these auxiliary variables, we can see that  $\frac{\partial\bar{\mathbf{r}}(t,\mathbf{r})}{\partial\mathbf{W}_{ij}}$  is  $\mathbf{J}p$ -local  $\forall t$ , and consequently, the RTRL update is  $\mathbf{e}\mathbf{J}p$ -local.
This is, of course, not biologically plausible in any way. The set  $\mathbf{J}$  allows the parameters to have access to the state of the entire network, at all time points, even from neurons that do not have any direct connections to the neuron whose synapse is being updated. Further, the entire error vector  $\mathbf{e}$  is required to compute the update. This is even less plausible than backpropagation, which only required access to  $\mathbf{e}_i$ . However, the RTRL update is an important baseline for analyzing the locality properties of other learning algorithms that are constructed as approximations of it, namely e-prop and RFLO.
E.12 e-prop
Theorem E.12. If  $p(\Theta) = \prod_{k} p(\Theta_k)$  and  $p(\mathbf{X}|\Theta)$  is defined by Eq. E.39 (with  $\mathbf{X} = \mathbf{r}$  and  $\Theta = \mathbf{W}$ ), the  $e$ -prop update for  $\mathbf{W}_{ij}$  with a loss  $\mathcal{L}(\mathbf{r}(T))$ , given by  $\mathcal{A}_{ep}(p(\mathbf{r}|\mathbf{W}),\mathcal{L}(\mathbf{r}))$  is  $\mathbf{e}_i p$ -local, where  $\mathbf{e}_i = \frac{\partial L(\mathbf{r}(T)}{\partial\bar{\mathbf{r}}_i(T)}$ .
Proof. For e-prop [37], we will also consider networks constructed according to Eq. E.39. The update is almost identical to the RTRL update, but several terms will be discarded, allowing the update to be  $\mathbf{e}_i p$ -local, as opposed to the  $\mathbf{e} \mathbf{J} p$ -local update given by RTRL. The update is as follows:
$$
\Delta \mathbf {W} _ {i j} \propto \frac {\partial \mathcal {L} (\mathbf {r} (T))}{\partial \bar {\mathbf {r}} _ {i} (T)} \frac {\partial \tilde {\mathbf {r}} _ {i} (T , \mathbf {r})}{\partial \mathbf {W} _ {i j}} \tag {E.47}
$$
$$
\frac {\partial \tilde {\mathbf {r}} _ {i} (t , \mathbf {r})}{\partial \mathbf {W} _ {i j}} = h ^ {\prime} \left(\mathbf {V} _ {i} (t)\right) \mathbf {W} _ {i i} \frac {\partial \tilde {\mathbf {r}} _ {i} (t - 1 , \mathbf {r})}{\partial \mathbf {W} _ {i j}} + h ^ {\prime} \left(\mathbf {V} _ {i} (t)\right) \mathbf {r} _ {j} (t - 1). \tag {E.48}
$$
This update combines the neuron-specific credit assignment signal  $\mathbf{e}_i$  with a local 'eligibility trace'  $\frac{\partial\tilde{\mathbf{r}}_i(t,\mathbf{r})}{\partial\mathbf{W}_{ij}}$  which performs approximate credit assignment by filtering and summing coactivity between neuron  $i$  and neuron  $j$  across timesteps. It is worth noting that the particular functional form of this eligibility trace is determined by our simplified RNN dynamics (Eq. E.39), which causes coactivity from previous timesteps to decay exponentially in proportion to the magnitude of the autapse  $\mathbf{W}_{ii}$  alternative neural network dynamics using continuous-time dynamics, or adaptive neural firing thresholds may alter the functional form of the eligibility trace [37], but do not fundamentally alter the  $p$ -locality properties of the update. Now, we only need to show that the eligibility trace is  $p$ -local.
As with RTRL, we can observe that  $h^{\prime}(\mathbf{V}_i(t))$  and  $\mathbf{r}_j(t - 1)$  both appear in the score function for our RNN (Eq. E.43) for all timesteps, as does  $\mathbf{W}_{ii} \subset \mathbf{W}_{i:}$ . Because the score function is  $p$ -local, we know that these variables are all allowed under  $p$ -locality. The eligibility trace only depends on these terms, from both the current time step and, recursively, from previous timesteps. Therefore, the eligibility trace is  $p$ -local. The e-prop update is a multiplication between  $\mathbf{e}_i$  and the eligibility trace, so by Def. 2.3 the update is  $\mathbf{e}_i p$ -local.
E.13 Random feedback local online learning (RFLO)
The RFLO update [36] is nearly identical to the e-prop update, except we replace  $\mathbf{e}_i$  with an approximate credit assignment signal  $\hat{\mathbf{e}}_i$  (which replaces symmetric feedback weights with random connections, similar to Feedback Alignment).
The update is given by:
$$
\Delta \mathbf {W} _ {i j} \propto \hat {\mathbf {e}} _ {i} \frac {\partial \tilde {\mathbf {r}} _ {i} (T , \mathbf {r})}{\partial \mathbf {W} _ {i j}} \tag {E.49}
$$
$$
\frac {\partial \tilde {\mathbf {r}} _ {i} (t , \mathbf {r})}{\partial \mathbf {W} _ {i j}} = h ^ {\prime} \left(\mathbf {V} _ {i} (t)\right) \mathbf {W} _ {i i} \frac {\partial \tilde {\mathbf {r}} _ {i} (t - 1 , \mathbf {r})}{\partial \mathbf {W} _ {i j}} + h ^ {\prime} \left(\mathbf {V} _ {i} (t)\right) \mathbf {r} _ {j} (t - 1). \tag {E.50}
$$
Following exactly the same reasoning as with e-prop, we may show that this update is  $\hat{\mathbf{e}}_i p$ -local.
Theorem E.13. If  $p(\Theta) = \prod_{k} p(\Theta_{k})$  and  $p(\mathbf{X}|\Theta)$  is defined by Eq. E.39 (with  $\mathbf{X} = \mathbf{r}$  and  $\Theta = \mathbf{W}$ ), the RFLO update for  $\mathbf{W}_{ij}$  with a loss  $\mathcal{L}(\mathbf{r}(T))$ , given by  $\mathcal{A}_{RFLO}(p(\mathbf{r}|\mathbf{W}),\mathcal{L}(\mathbf{r}))$  is  $\hat{\mathbf{e}}_i p$ -local, where  $\mathbf{e}_i = \frac{\partial L(\mathbf{r}(T))}{\partial\bar{\mathbf{r}}_i(T)}$ .
E.14 Feedback-based Online Local Learning Of Weights (FOLLOW)
The FOLLOW algorithm [50] is defined in terms of a particular continuous-time LIF circuit with postsynaptic potential kernels. For simplicity, we will focus our analysis on a linear version of the same circuit, disregarding the dynamic postsynaptic potentials and input stimuli. Disregarding these features does not affect the  $p$ -locality properties of the FOLLOW algorithm, but it would certainly degrade its performance on tasks.
The network dynamics are given by:
$$
\begin{array}{l} \mathbf {r} (t + \Delta t) = h (\mathbf {r} (t), \mathbf {e} (t)) + \sigma \boldsymbol {\eta} (E.51) \\ = \left(1 - \frac {\Delta t}{\tau}\right) \mathbf {r} (t) + \frac {1}{\tau} \left(\mathbf {W r} (t) + k \mathbf {W} ^ {f b} \mathbf {e} (t)\right) \Delta t + \sigma \boldsymbol {\eta}, (E.52) \\ \end{array}
$$
where  $\mathbf{W}^{fb}$  is an  $N\times N^o$  random feedback weight matrix,  $k$  is a positive constant, and  $\mathbf{e}(t)$  is an  $N^o$ -dimensional error feedback vector delivered at every timestep, with  $N^o$  the number of output dimensions. Because we are in continuous time, we will assume that  $\pmb{\eta}\sim \mathcal{N}(0,\Delta t)$ .
Similar to RTRL, we can write the probability distribution for the network as:
$$
p (\mathbf {r} | \mathbf {e}, \mathbf {W}) = p (\mathbf {r} (0)) \prod_ {t} \prod_ {i = 1} ^ {N} p \left(\mathbf {r} _ {i} (t + \Delta t) | \mathbf {r} (t), \mathbf {e} (t), \mathbf {W}\right) \tag {E.53}
$$
$$
p \left(\mathbf {r} _ {i} (t + \Delta t) \mid \mathbf {r} (t), \mathbf {e} (t), \mathbf {W}\right) \sim \mathcal {N} \left(h (\mathbf {r} (t), \mathbf {e} (t)), \sigma^ {2} \Delta t\right), \tag {E.54}
$$
where  $p(\mathbf{r}(0))$  is some initial distribution of firing rates. Further, we can assume that the distribution of errors at timestep  $t + \Delta t$  has any arbitrary distribution  $p(\mathbf{e}(t + \Delta t)|\mathbf{r}(t))$ .
The update for weight  $\mathbf{W}_{ij}$  is given by:
$$
\Delta \mathbf {W} _ {i j} (t) \propto \left(\mathbf {W} _ {i:} ^ {f b} \mathbf {e} (t)\right) \mathbf {r} _ {j} (t). \tag {E.55}
$$
Therefore, only the postsynaptic error current and presynaptic input are necessary to update the weights for a given synapse in this type of network. Below, we will show that this update is  $p$ -local.
Theorem E.14. If  $p(\Theta) = \prod_k p(\Theta_k)$  and  $p(\mathbf{X}|\Theta)$  is defined by Eq. E.53 (with  $\mathbf{X} = \{\mathbf{r},\mathbf{e}\}$  and  $\Theta = \mathbf{W}$ ), the FOLLOW update for  $\mathbf{W}_{ij}$ , given by  $\mathcal{A}_{FW}(p(\mathbf{r},\mathbf{e}|\mathbf{W}))$  is  $p$ -local.
Proof. To see that this is true, we need only show that the variables included in  $\Delta \mathbf{W}_{ij}$  are subsets of the variables included in the score function  $\frac{\partial\log p(\mathbf{r},\mathbf{e}|\mathbf{W})}{\partial\mathbf{W}_{ij}}$ . These variables are permissible for  $p$ -local updates by Property 2.7. The score function is given by:
$$
\begin{array}{l} \frac {\partial \log p (\mathbf {r} , \mathbf {e} | \mathbf {W})}{\partial \mathbf {W} _ {i j}} = \sum_ {t} \frac {\partial \log p (\mathbf {r} (t + \Delta t) | \mathbf {r} (t) , \mathbf {e} (t) , \mathbf {W})}{\partial \mathbf {W} _ {i j}} + \sum_ {t} \frac {\partial \log p (\mathbf {e} (t + \Delta t) | \mathbf {r} (t))}{\partial \mathbf {W} _ {i j}} (E.56) \\ = \sum_ {t} \frac {\partial \log p \left(\mathbf {r} _ {i} (t + \Delta t) \mid \mathbf {r} (t) , \mathbf {e} (t) , \mathbf {W}\right)}{\partial \mathbf {W} _ {i j}} (E.57) \\ = \sum_ {t} \frac {\left(\mathbf {r} _ {i} (t + \Delta t) - \mathbf {r} _ {i} (t) + \frac {\Delta t}{\tau} \left(- \mathbf {r} _ {i} (t) + \mathbf {W} _ {i :} \mathbf {r} (t) + k \mathbf {W} _ {i :} ^ {f b} \mathbf {e} (t)\right)\right)}{\Delta t \sigma^ {2}} \mathbf {r} _ {j} (t). (E.58) \\ \end{array}
$$
Therefore, for weight  $\Delta \mathbf{W}_{ij}$ , the permissible variables include:  $\mathbf{r}_i(t) \forall t$ , any  $\mathbf{r}_k(t)$  such that  $\mathbf{W}_{ik} \neq 0$  ( $\forall t$ ), any  $\mathbf{e}_k(t)$  such that  $\mathbf{W}_{ik}^{fb} \neq 0$  ( $\forall t$ ), and the parameters  $\mathbf{W}_{i}$  and  $\mathbf{W}_{i}^{fb}$ . The parameter update requires only  $\mathbf{r}_j(t)$  and  $\mathbf{W}_{i}^{fb}\mathbf{e}(t)$ , which is a subset of these permissible variables. Therefore, the update is  $p$ -local.
F  $p$ -locality does not guarantee biological plausibility
It is very important to clarify the exact relationship between  $p$ -locality and biological plausibility. Except for some network-wide variables that a theoretician may decide to allow through a particular choice of  $\mathbf{Sp}$ -locality, we have generally shown that  $p$ -locality is overly permissive, in that a particular choice of  $p$  may allow parameter updates to include variables that an individual synapse may not have access to. Furthermore,  $p$ -locality does not restrict the network architecture defined by  $p$  to be biologically plausible. The best way to interpret  $p$ -locality is as follows: if  $p(\mathbf{X},\Theta)$  defines a biologically plausible architecture and an algorithm  $\mathcal{A}$  is  $p(\mathbf{X},\Theta)$ -local, then the parameter update provided by  $\mathcal{A}(p(\mathbf{X},\Theta))$  will be biologically plausible. There are many network architecture and parameter update combinations that may be biologically plausible without being proven  $p$ -local (e.g. explicit approximations to backpropagation [29, 30, 31]), as there are many combinations that are  $p$ -local without being biologically plausible. Below, we will show two important instances in which  $p$ -locality does not properly diagnose a combination of network architecture and parameter update as biologically implausible.
F.1 Locality and architectural plausibility
The first example is pervasive in neural network models of the brain: networks frequently violate Dale's law, which states that neurons in a neural network are (for the most part [51]) either excitatory
(outgoing weights are positive) or inhibitory (outgoing weights are negative), but not both. In fact, in the simple network example we have provided (Section 2.4), neural firing rates are not constrained to be strictly positive, and outgoing synaptic weights are not sign-constrained. For this biologically implausible architecture,  $p$ -locality defines which variables are allowed to be included in individual parameter updates in a way that is sensible (allowing only variables involving the postsynaptic firing rate and the firing rates of all pre-synaptic neurons), but it says nothing about the aforementioned implausibilities of the network architecture. Similarly, the linear Boltzmann machine example provided in Appendix D does not constrain firing rates to be positive, and requires symmetric weights  $(\mathbf{W}_{ij} = \mathbf{W}_{ji})$ , which could not satisfy Dale's law while allowing connections between an inhibitory neuron  $i$  and an excitatory neuron  $j$  ( $\mathbf{W}_{ij} > 0$  while  $\mathbf{W}_{ji} < 0 \Rightarrow \mathbf{W}_{ij} \neq \mathbf{W}_{ji}$ ).
These examples illustrate an important fact:  $p$ -locality focuses on the plausibility of updates given an architecture that has been predetermined to be acceptable. However, it is worth noting that if we were to impose these additional constraints for the proposed networks, the accepted variables determined by  $p$ -locality would not change.
F.2 Parameterizing probabilities with neural networks
Another important caveat when working with  $p$ -locality is that the random variables  $\mathbf{Z}$  have to correspond to the relevant biophysical quantities of interest, e.g. neural firing rates  $\mathbf{X}$  and synaptic weights  $\Theta$ . If this is not the case, then  $p$ -locality can easily defy standard notions of biological plausibility. For instance, if we define a probability distribution in terms of a 3-layer neural network:
$$
p (\mathbf {X} | \boldsymbol {\Theta}) \sim \mathcal {N} (\bar {\mathbf {r}} ^ {2} (\bar {\mathbf {r}} ^ {1} (\bar {\mathbf {r}} ^ {0} (\boldsymbol {\Theta}))), \sigma^ {2}), \tag {F.1}
$$
Then the score function of this distribution is given by:
$$
\frac {\partial p (\mathbf {X} | \boldsymbol {\Theta})}{\partial \boldsymbol {\Theta}} = \frac {\mathrm {d} \log p (\mathbf {X} | \boldsymbol {\Theta})}{\mathrm {d} \bar {\mathbf {r}} ^ {2}} \frac {\mathrm {d} \bar {\mathbf {r}} ^ {(2)}}{\mathrm {d} \bar {\mathbf {r}} ^ {(1)}} \frac {\mathrm {d} \bar {\mathbf {r}} ^ {(1)}}{\mathrm {d} \bar {\mathbf {r}} ^ {(0)}} \frac {\mathrm {d} \bar {\mathbf {r}} ^ {(0)}}{\mathrm {d} \boldsymbol {\Theta}}. \tag {F.2}
$$
This equation depends on  $\mathbf{X}$ , which is the output of the network, even though  $\Theta$  parameterizes  $\bar{\mathbf{r}}^{(0)}$ . Therefore, if the random variables had been defined in as in Section 2.4, then this update would not be  $p$ -local. However, because it is the derivative of the score function, for any independent marginal  $p(\Theta)$ , it is  $p$ -local for this choice of random variables. Therefore, it is important when working with an algorithm such as Wake-Sleep or REINFORCE, that one chooses a conditional probability distribution  $p(\mathbf{X}|\Theta)$  that captures biologically plausible dependencies. When this is not done, as in [52, 53], the resulting updates have no correspondence to synaptic plasticity rules.
Note that this fact does not undermine the utility of  $p$ -locality as a concept. Our proofs for algorithms in Appendix E apply for any  $p(\mathbf{X}|\Theta)$ , as long as  $p(\Theta)$  factorizes to  $\prod_{i}p(\Theta_{i})$ . Therefore, algorithms that have universal  $p$ -local properties will respect the variable dependencies implied by  $p(\mathbf{X}|\Theta)$  whether this distribution is plausible or not, which means that the algorithms will respect variable dependencies for all plausible network architectures.