SlowGuess/NeurIPS_2023_OCR huggingface dataset:
E.6 Impression Learning (IL)
The impression learning parameter update [23] is closely related to the Wake-Sleep parameter update, and is consequently also  $\gamma p_{md}$ -local. What distinguishes IL from WS is the use of rapid alternations in the gating signal  $\gamma_t$  within a single trial with  $T$  time steps. Here,  $\mathbf{X} = [\mathbf{X}_0,\dots,\mathbf{X}_T]$ ,  $\gamma = [\gamma_0,\dots,\gamma_t]$  and  $p_{md}(\mathbf{X}|\gamma ,\boldsymbol {\Theta}) = \prod_{t = 0}^{T}p_d(\mathbf{X}_t|\mathbf{X}_{t - 1},\boldsymbol {\Theta})^{\gamma_t}p_m(\mathbf{X}_t|\mathbf{X}_{t - 1},\boldsymbol {\Theta})^{1 - \gamma}$  is a mixture distribution in which  $\gamma_t$  alternates between 0 and 1, sampling from either  $p_m$  or  $p_d$  at the time step  $t$ , respectively.
Theorem E.6. If  $p(\Theta) = \prod_k p(\Theta_k)$ , the impression learning estimator given by  $\mathcal{A}_{IL}(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}|\Theta))$  is  $\gamma p_{md}$ -local, where  $p_{md} = Mix(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}|\Theta^{(d)})) p(\Theta)$ .
Proof. Similar to WS, the update is given by:
$$
\begin{array}{l} \Delta \Theta_ {I L} \propto \int \left[ \sum_ {t = 0} ^ {T} \frac {\partial}{\partial \Theta} \left[ \left(1 - \gamma_ {t}\right) \log p _ {d} \left(\mathbf {X} _ {t} \mid \mathbf {X} _ {t - 1}, \boldsymbol {\Theta}\right) + \gamma_ {t} p _ {m} \left(\mathbf {X} _ {t} \mid \mathbf {X} _ {t - 1}, \boldsymbol {\Theta}\right) \right] \right] p _ {m d} (\mathbf {X} | \gamma , \boldsymbol {\Theta}) d \mathbf {X} (E.17) \\ \approx \sum_ {t = 0} ^ {T} (1 - \gamma_ {t}) \frac {\partial}{\partial \boldsymbol {\Theta}} \log p _ {d} (\mathbf {X} _ {t} | \mathbf {X} _ {t - 1}, \boldsymbol {\Theta}) + \gamma_ {t} \frac {\partial}{\partial \boldsymbol {\Theta}} p _ {m} (\mathbf {X} _ {t} | \mathbf {X} _ {t - 1}, \boldsymbol {\Theta}), (E.18) \\ \end{array}
$$
where in this last equality,  $\mathbf{X} \sim p_{md}(\mathbf{X})$ , and we are performing a single-sample gradient approximation. It is worth noting that unlike in the Wake-Sleep algorithm, here  $\gamma_t$  is not a constant throughout time. Instead,  $\gamma_t$  alternates between 0 and 1 with 'phase duration'  $K$ , i.e.  $\gamma_{t+1} = 1 - \gamma_t$  if mod  $(t, K) = 0$ , and  $\gamma_{t+1} = \gamma_t$  otherwise. The IL update is the score function of  $p_{md}(\mathbf{X} | 1 - \gamma, \Theta)$  which has identical dependencies to the score function of  $p_{md}(\mathbf{X} | \gamma, \Theta)$  (only a change from  $\gamma \rightarrow 1 - \gamma$  has occurred). Therefore, if  $p(\Theta) = \prod_k p(\Theta_k)$ , this parameter update is  $\gamma p_{md}$ -local by Property 2.7.
<sup>4</sup>A rigorous discussion of why this optimization process is sensible is beyond the scope of this manuscript. See [22, 47] for more detail.
E.7 Contrastive Divergence for Boltzmann machines (CD)
While the GEM learning update is provably  $p_m$ -local, it is also predicated on the assumption that the parameter marginal distribution factorizes as  $p(\Theta) = \prod_k p(\Theta_k)$ , which as we note in Appendix D can be difficult to ensure for even simple undirected graphical models. An extension of the GEM algorithm, we can show that the CD algorithm is  $\gamma p_{md}$ -local (as opposed to just  $p_m$ -local) under less restrictive assumptions. The cost of this is that CD learning usually requires costly MCMC sampling from both the posterior distribution  $p_m(\mathbf{X}_h|\mathbf{X}_o,\Theta)$  and the full joint distribution  $p_m(\mathbf{X}_h,\mathbf{X}_o|\Theta)$ .
Theorem E.7. The CD update given by  $\mathcal{A}_{CD}(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}))$  is  $\gamma p_{md}$ -local, where  $p_{md} = Mix(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X})) p(\Theta)$ .
As mentioned above, for GEM the most natural choice for  $p_d(\mathbf{X}_h|\mathbf{X})$  is given by  $p_m(\mathbf{X}_h|\mathbf{X},\Theta_0)$ . As we demonstrated, parameter updates calculated according to this rule will be  $\gamma p_{md}$ -local, but there are two important caveats. First, for GEM to produce biologically plausible updates, we still need a biological system that can sample from  $p_m(\mathbf{X}_h|\mathbf{X},\Theta_0)$ . Second, it is important to remember that we are only guaranteed that the score function is guaranteed to be  $p$ -local if the marginal parameter probability distribution factorizes as  $p(\Theta) = \prod_k p(\Theta_k)$ . For a DAG, it may be difficult to satisfy the first condition without approximation (given by the Wake-Sleep algorithm, for instance), whereas for an UG, it may be difficult to satisfy the second condition, as we saw in Section D. To make our update  $\gamma p_{md}$ -local for an undirected graphical model like the Boltzmann machine, we will require an extra step that we outline here to use the energy function Property 2.6 rather than the score function Property 2.7.
Having committed to working with an undirected graphical model, instead of sticking to the original GEM update, here we break apart the probability distribution as:
$$
\begin{array}{l} \mathcal {A} _ {C D} \left(p _ {m} (\mathbf {X} | \boldsymbol {\Theta}), p _ {d} (\mathbf {X})\right) = \int \frac {\partial}{\partial \boldsymbol {\Theta}} \log \left(p _ {m} (\mathbf {X} | \boldsymbol {\Theta})\right) p _ {d} (\mathbf {X}) d \mathbf {X} (E.19) \\ = \int \frac {\partial}{\partial \boldsymbol {\Theta}} \left[ \log \left(E (\mathbf {X}, \boldsymbol {\Theta})\right) - \log \mathcal {Z} (\boldsymbol {\Theta}) \right] p _ {d} (\mathbf {X}) d \mathbf {X} (E.20) \\ = - \int \frac {\partial}{\partial \Theta} E (\mathbf {X}, \Theta) p _ {d} (\mathbf {X}) d \mathbf {X} - \frac {\partial}{\partial \Theta} \log \mathcal {Z} (\Theta) (E.21) \\ = - \int \frac {\partial}{\partial \boldsymbol {\Theta}} E (\mathbf {X}, \boldsymbol {\Theta}) p _ {d} (\mathbf {X}) d \mathbf {X} - \frac {1}{\mathcal {Z} (\boldsymbol {\Theta})} \int \frac {\partial}{\partial \boldsymbol {\Theta}} e ^ {- \frac {E (\mathbf {X} , \boldsymbol {\Theta})}{\sigma^ {2}}} d \mathbf {X} (E.22) \\ = - \int \frac {\partial}{\partial \boldsymbol {\Theta}} E (\mathbf {X}, \boldsymbol {\Theta}) p _ {d} (\mathbf {X}) d \mathbf {X} + \frac {1}{\mathcal {Z} (\boldsymbol {\Theta})} \int \frac {\partial}{\partial \boldsymbol {\Theta}} E (\mathbf {X}, \boldsymbol {\Theta}) e ^ {- E (\mathbf {X}, \boldsymbol {\Theta})} d \mathbf {X} (E.23) \\ = - \int \frac {\partial}{\partial \boldsymbol {\Theta}} E (\mathbf {X}, \boldsymbol {\Theta}) p _ {d} (\mathbf {X}) d \mathbf {X} + \int \frac {\partial}{\partial \boldsymbol {\Theta}} E (\mathbf {X}, \boldsymbol {\Theta}) p _ {m} (\mathbf {X} | \boldsymbol {\Theta}) d \mathbf {X} (E.24) \\ \approx \frac {2}{K} \sum_ {k = 0} ^ {K} (- 1) ^ {\gamma_ {k}} \frac {\partial}{\partial \boldsymbol {\Theta}} E (\mathbf {X} _ {k}, \boldsymbol {\Theta}), (E.25) \\ \end{array}
$$
where  $\mathbf{X}_k$  is sampled from  $p_{md}$ . Now, by Property 2.6 and Property 2.8, this update is  $\gamma p_{md}$ -local. This is the Boltzmann machine learning algorithm [26], where the clamped and unclamped phases are alternated between stochastically [46]; for our previous linear Boltzmann example, in which  $\mathbf{X} = \mathbf{r}$  and  $\Theta = \mathbf{W}$ , the derivative of the energy function (Eq. D.3) with respect to a parameter  $\mathbf{W}_{ij}$  is  $\mathbf{r}_i\mathbf{r}_j$ , demonstrating that updates correspond to two phases of updates: one in which  $\mathbf{r}_o$  is clamped to a data distribution given by  $p_d(\mathbf{r}_o)$  for some  $\mathbf{r}_o \subseteq \mathbf{r}$  and Hebbian updates are positive, contrasted with an unclamped phase in which updates are negative. Note that for this model approximate sampling from a posterior distribution  $p_m(\mathbf{r}_{\neq o}|\mathbf{r}_o,\mathbf{W})p_d(\mathbf{r}_0)$  is no more difficult than sampling from the joint distribution: one simply holds  $\mathbf{r}_o$  fixed to an environmental data sample and performs Langevin sampling on all other variables.
E.8 Equilibrium Propagation (EP)
Though the derivation and setup of the equilibrium propagation algorithm [27] is very different from Contrastive Divergence, the functional form of the derived update is very similar. While equilibrium propagation typically operates on deterministic networks, here we will provide our derivation for the stochastic version with an energy function defining a joint distribution over  $\Theta$  and  $\mathbf{X}$  (as in Section D), which is somewhat more straightforward to fit into the  $p$ -locality framework.
Suppose that we have a probabilistic energy-based model whose energy function is given by:
$$
E (\mathbf {Z}, \epsilon) = E _ {0} (\mathbf {Z}) + \epsilon \mathcal {L} (\mathbf {Z}), \tag {E.26}
$$
where  $\epsilon \in \{0,\beta \}$ , where  $\beta \ll 1$ , where  $\mathcal{L}(\mathbf{Z})$  is the loss function that the parameter updates are optimizing. This can be thought of as a 'soft-clamped' system, in which nonzero  $\epsilon$  pushes the system towards slightly better performance. Intuitively, the EP parameter update attempts to change network dynamics so that the unclamped system is nudged towards the slightly better performing soft-clamped system. Then we have the following theorem:
Theorem E.8. The EP update given by  $\mathcal{A}_{EP}(p(\mathbf{X}|\Theta, \gamma))$  is  $\gamma p_{md}$ -local, where  $p_{md} = Mix(p(\mathbf{X}|\Theta, \epsilon = \beta), p(\mathbf{X}|\Theta, \epsilon = 0)) p(\Theta)$ .
The parameter update for equilibrium propagation is given by:
$$
\begin{array}{l} \Delta \Theta_ {E P} \propto - \frac {1}{\beta} \left[ \mathbb {E} _ {\epsilon = \beta} \frac {\partial E (\mathbf {Z} , \epsilon)}{\partial \boldsymbol {\Theta}} - \mathbb {E} _ {\epsilon = 0} \frac {\partial E (\mathbf {Z} , \epsilon)}{\partial \boldsymbol {\Theta}} \right] (E.27) \\ \approx \frac {2}{\beta} \sum_ {k = 0} ^ {K} (- 1) ^ {\epsilon_ {k} / \beta} \frac {\partial E \left(\mathbf {Z} _ {k} , \epsilon_ {k}\right)}{\partial \boldsymbol {\Theta}}, (E.28) \\ \end{array}
$$
where for the final equality we are using a sampling-based approximation in which we are sampling from  $\mathbf{X}_k\sim p(\mathbf{X}|\boldsymbol {\Theta},\epsilon_k)$ , and  $\epsilon_{k} / \beta \sim$  Bernoulli(0.5). This is almost identical to the Contrastive Divergence update, except that rather than clamping neural activities to a target output, they are slightly biased towards better performance. Because this is the combination of the derivative of the energy function with a mixture variable  $\gamma = \epsilon_{k} / \beta$ , by Properties 2.6 and 2.8, this update is  $\gamma p_{md}$ -local where  $p_{md} = Mix(p(\mathbf{X}|\boldsymbol {\Theta},\epsilon = \beta),p(\mathbf{X}|\boldsymbol {\Theta},\epsilon = 0))p(\boldsymbol {\Theta})$ .
E.9 Winner-take-all STDP
While Contrastive Divergence uses MCMC sampling to approximate the GEM update, Nessler et al. [48] use a particular generative model for which the posterior can be analytically calculated and resembles a simple winner-take-all neural circuit. Then, the authors derive their STDP parameter update as an approximation to the GEM algorithm. Because of this, one might imagine that the derived STDP update may, like the GEM algorithm, be  $p_m$ -local. We will see below that this is the case.
First, we define the generative model  $p_m$  used in the paper:
$$
p _ {m} (\mathbf {r}, \mathbf {s} | \mathbf {W}) = \frac {1}{\mathcal {Z}} e ^ {- E (\mathbf {r}, \mathbf {s}, \mathbf {W})} \tag {E.29}
$$
$$
E (\mathbf {r}, \mathbf {s}, \mathbf {W}) = - \left(\sum_ {i = 0} ^ {N} \mathbf {r} _ {i} \mathbf {W} _ {i 0} + \sum_ {i = 0} ^ {N} \sum_ {j = 0} ^ {N _ {s}} \mathbf {r} _ {i} \mathbf {W} _ {i j} \mathbf {s} _ {j}\right), \tag {E.30}
$$
where  $\mathcal{Z}$  is the normalizing constant and the  $\mathbf{r}$  and  $\mathbf{s}$  vectors contain binary random variables. Furthermore, in the network only one neuron  $\mathbf{r}$  is assumed to fire at any given time ( $\mathbf{r}_i \neq 0 \Leftrightarrow \mathbf{r}_k = 0 \forall k \neq i$ ). The inference distribution, conditioned on a stimulus  $\mathbf{s}$  can be calculated as follows:
$$
p _ {m} (\mathbf {r} | \mathbf {s}, \mathbf {W}) = \frac {\exp \left(\sum_ {i = 0} ^ {N} \mathbf {r} _ {i} \mathbf {W} _ {i 0} + \sum_ {i = 0} ^ {N} \sum_ {j = 0} ^ {N _ {s}} \mathbf {r} _ {i} \mathbf {W} _ {i j} \mathbf {s} _ {j}\right)}{\sum_ {i = 0} ^ {N} \exp \left(\mathbf {W} _ {i 0} + \sum_ {j = 0} ^ {N _ {s}} \mathbf {W} _ {i j} \mathbf {s} _ {j}\right)}. \tag {E.31}
$$
This probability distribution can be interpreted as a kind of winner-take-all computation, dominated by the neuron with the greatest input current [48]. Samples from this distribution are used to compute the weight updates:
$$
\Delta \mathbf {W} _ {i j} \propto \mathbf {r} _ {i} \left(c \mathbf {s} _ {i} e ^ {- \mathbf {W} _ {i j}} - 1\right) \tag {E.32}
$$
$$
\Delta \mathbf {W} _ {i 0} \propto \mathbf {r} _ {i} e ^ {- \mathbf {W} _ {i 0}} - 1, \tag {E.33}
$$
where  $c$  is a positive constant. We now prove the following theorem assessing the  $p$ -locality of this distribution:
Theorem E.9. The STDP update given by  $\mathcal{A}_{STDP}(p(\mathbf{X}|\Theta))$  is  $\gamma p_{m}$ -local.
Proof. To see this, we first note that the gradient of the energy function  $E(\mathbf{r},\mathbf{s},\mathbf{W})$  with respect to the parameters is  $p_m$ -local by Property 2.6. Therefore, any variables contained within this will also be permissible under  $p_m$ -locality.
For  $\mathbf{W}_{ij}$ , we have:
$$
\frac {\partial E (\mathbf {r} , \mathbf {s} , \mathbf {W})}{\partial \mathbf {W} _ {i j}} = - \mathbf {r} _ {i} \mathbf {s} _ {j}, \tag {E.34}
$$
so that we know  $\mathbf{r}_i$  and  $\mathbf{s}_j$  are permissible for  $\Delta \mathbf{W}_{ij}$  under  $p_m$ -locality; further, the value of a parameter itself,  $\mathbf{W}_{ij}$ , is always allowed under  $p_m$ -locality. These are the only variables on which  $\Delta \mathbf{W}_{ij}$  depends, so this update is  $p_m$ -local.
For  $\mathbf{W}_{i0}$ , we have:
$$
\frac {\partial E (\mathbf {r} , \mathbf {s} , \mathbf {W})}{\partial \mathbf {W} _ {i 0}} = - \mathbf {r} _ {i}, \tag {E.35}
$$
so that we know  $\mathbf{r}_i$  is permissible for  $\Delta \mathbf{W}_{i0}$ . By the same reasoning, this update is also  $p_m$ -local. Since all updates are therefore  $p_m$ -local, we may conclude that the full algorithm is  $p_m$ -local. However, this proof does not have the same level of generality as for the previous algorithms, because the algorithm is only defined for a single winner-take-all network model.
E.10 Backpropagation
Theorem E.10. If  $p(\Theta) = \prod_{k} p(\Theta_k)$  and  $p(\mathbf{X}|\Theta)$  is defined by Eq. 3 (with  $\mathbf{X} = \mathbf{r}$  and  $\Theta = \mathbf{W}$ ), the BP update for  $\mathbf{W}_{ij}^{(l)}$  with a loss  $\mathcal{L}(\mathbf{r})$ , given by  $\mathcal{A}_{BP}(p(\mathbf{r}|\mathbf{W}),\mathcal{L}(\mathbf{r}))$  is  $\mathbf{e}_i^{(l)}$ $p$ -local, where  $\mathbf{e}_i^{(l)} = \frac{\mathrm{d}L}{\mathrm{d}\mathbf{r}_i}$ . Similarly, the updates for feedback alignment, weight mirror, and Burstprop are  $\hat{\mathbf{e}}_i^{(l)}$ $p$ -local, where  $\hat{\mathbf{e}}_i^{(l)}$  is given by their respective gradient approximations.
Proof. As a first step, for clarity purposes we will demonstrate that backpropagation [28], is not  $p$ -local with respect to the simple feedforward neural network architecture we outlined above; we will subsequently demonstrate that it and its approximations do satisfy a particular notion of  $\mathbf{Sp}$ -locality. For a scalar loss function  $\mathcal{L}(\mathbf{r}^L)$  and a single parameter  $\mathbf{W}_{ij}^{(l)}$ , the backpropagation gradient is given by the negative gradient of the loss with respect to the parameter of choice, using the reparameterization trick [25, 24] to take for a single sample of  $\{\mathbf{r}^{(l)}\}_{l = 0:L}$  the mean (noiseless) mapping from  $\mathbf{r}^{(l - 1)}\rightarrow \mathbf{r}^{(l)}$  to be  $\bar{\mathbf{r}}^{(l)} = h\big(\mathbf{W}^{(l)}\mathbf{r}^{(l - 1)}\big)$ , so that by Eq. 5, we have  $\mathbf{r}^{(l)} = \bar{\mathbf{r}}^{(l)} + \sigma \boldsymbol{\eta}^{(l)}$ . By the chain rule, gradient descent gives:
$$
\Delta \mathbf {W} _ {i j} ^ {(l)} \propto - \frac {\mathrm {d} \mathcal {L} (\mathbf {r} ^ {(L)})}{\mathrm {d} \bar {\mathbf {r}} ^ {(L)}} \left(\prod_ {k = l} ^ {L} \frac {\mathrm {d} \bar {\mathbf {r}} ^ {(k)} \left(\mathbf {r} ^ {(k - 1)}\right)}{\mathrm {d} \bar {\mathbf {r}} ^ {(k - 1)}}\right) \frac {\mathrm {d} \bar {\mathbf {r}} ^ {(l)} \left(\mathbf {W} _ {i j} ^ {(l)}\right)}{\mathrm {d} \mathbf {W} _ {i j} ^ {(l)}}. \tag {E.36}
$$
Based on our analysis in Section 2.4, this update function is clearly not  $p$ -local, because the update depends on firing rates  $\mathbf{r}^{(k)}$  for  $k > l$ . However, while backpropagation is not in general  $p$ -local, any algorithm can be  $\mathbf{S}p$ -local: for example, if we take  $\mathbf{S} = \mathbf{Z}$ , then by Definition 2.3, any parameter update can contain any variable in the graphical model  $p(\mathbf{Z})$ . Taking  $\mathbf{S} = \mathbf{Z}$  is inherently vacuous:  $\mathbf{S}p$ -locality is only conceptually useful if we can cleanly reduce the number of variables included in  $\mathbf{S}$  for a broad set of biologically relevant neural architectures. Fortunately, for backpropagation
operating on a feedforward neural network governed by Equation 5, we do not need to include every variable in the network. By Equation E.36, we see for our example feedforward neural network that:
$$
\Delta \mathbf {W} _ {i j} ^ {(l)} \propto - \frac {\mathrm {d} \mathcal {L}}{\mathrm {d} \bar {\mathbf {r}} _ {i} ^ {(l)}} \frac {\mathrm {d} \bar {\mathbf {r}} _ {i} ^ {(l)}}{\mathrm {d} \mathbf {W} _ {i j} ^ {(l)}}, \tag {E.37}
$$
where  $\frac{\mathrm{d}\mathcal{L}}{\mathrm{d}\bar{\mathbf{r}}_i^{(l)}} = \frac{\mathrm{d}\mathcal{L}(\mathbf{r}^{(L)})}{\mathrm{d}\bar{\mathbf{r}}^{(L)}}\left(\prod_{k = l + 2}^L\frac{\mathrm{d}\bar{\mathbf{r}}^{(k)}(\mathbf{r}^{(k - 1)})}{\mathrm{d}\bar{\mathbf{r}}^{(k - 1)}}\right)\frac{\mathrm{d}\bar{\mathbf{r}}^{(l + 1)}(\mathbf{r}^{(l)})}{\mathrm{d}\bar{\mathbf{r}}_i^{(l)}}$  is the derivative of the global loss function with respect to the individual mean neuron activation  $\bar{\mathbf{r}}_i^{(l)}$ . Interestingly,  $\frac{\mathrm{d}\bar{\mathbf{r}}_i^{(l)}}{\mathrm{d}\mathbf{W}_{ij}^{(l)}}$  the derivative of the mean parameter for neuron  $\mathbf{r}_i$  is a function of only the parents of  $\mathbf{r}^{(l)}$ , which are therefore the coparents of  $\mathbf{W}_{ij}^{(l)}$ . To verify that this particular component of the weight update is  $p$ -local, we can compare its dependencies to the score function, which is in this case  $p$ -local by Property 2.7. As noted in Section 2.4, the score function is given by:
$$
\frac {\partial \log p (\mathbf {r} | \mathbf {s} , \mathbf {W})}{\partial \mathbf {W} _ {i j} ^ {(l)}} = \frac {\left(\mathbf {r} _ {i} ^ {(l)} - h \left(\mathbf {V} _ {i} ^ {(l)}\right)\right)}{\sigma^ {2}} h ^ {\prime} \left(\mathbf {V} _ {i} ^ {(l)}\right) \mathbf {r} _ {j} ^ {(l - 1)}, \tag {E.38}
$$
where  $\mathbf{V}_i^{(l)} = \mathbf{W}_i^{(l)}\mathbf{r}^{(l - 1)}$ . Because the score function is  $p$ -local, any variables that it depends on are permissible for  $p$ -local updates. The score function depends on  $\mathbf{r}_i^{(l)}$  and  $\mathbf{r}^{(l - 1)}$ , whereas  $\frac{\mathrm{d}\bar{\mathbf{r}}_i^{(l)}}{\mathrm{d}\mathbf{W}_{ij}^{(l)}}$  depends only on  $\mathbf{r}^{(l - 1)}$ . It follows that this function is also  $p$ -local.
As we already discussed,  $\frac{\mathrm{d}\mathcal{L}}{\mathrm{d}\bar{\mathbf{r}}_i^{(l)}}$  is not  $p$ -local because it depends on neurons downstream of  $\mathbf{r}_i^{(l)}$ . However, if we define an auxiliary random variable  $\mathbf{e}_i^{(l)} = \frac{\mathrm{d}\mathcal{L}}{\mathrm{d}\bar{\mathbf{r}}_i^{(l)}}$ , we see that because it multiplies  $\mathbf{e}_i^{(l)}$  with a  $p$ -local function,  $\mathbf{W}_{ij}^{(l)}$  is  $\mathbf{e}_i^{(l)}p$ -local.
Importantly, this does not mean that backpropagation is biologically plausible: this notion of locality provides no clues as to how  $\mathbf{e}_i^{(l)}$  could be calculated or approximated in the brain, and an explicit calculation of gradients could not be possible due to the weight transport problem [20]. There are many recent models that account for how  $\mathbf{e}_i^{(l)}$  could be approximated by an approximate credit assignment signal  $\hat{\mathbf{e}}_i^{(l)}$  involving either random feedback synapses that project errors backwards through the network (feedback alignment [29]) or feedback synapses that dynamically adjust through local synaptic mechanisms so that  $\hat{\mathbf{e}}_i^{(l)}$  provides an unbiased approximation (e.g. weight mirror or Kolen-Pollack alignment [30], and BurstProp [31]). Each of these algorithms decomposes into a nonlocal feedback term  $\hat{\mathbf{e}}_i^{(l)}$  and a  $p$ -local term in exactly the same way, and are consequently  $\hat{\mathbf{e}}_i^{(l)}p$ -local.
E.11 Real Time Recurrent Learning (TRL)
Consider an autonomous recurrent neural network whose directed acyclic graphical model is provided by the following equations (we will ignore stimulus-dependence for notational simplicity):
$$
p (\mathbf {r} | \mathbf {W}) = p (\mathbf {r} (0)) \prod_ {t = 1} ^ {T} \prod_ {i = 1} ^ {N} p \left(\mathbf {r} _ {i} (t) \mid \mathbf {r} (t - 1), \mathbf {W}\right) \tag {E.39}
$$
$$
p \left(\mathbf {r} _ {i} (t) \mid \mathbf {r} _ {i} (t - 1), \mathbf {W}\right) \sim \mathcal {N} \left(h \left(\mathbf {W} _ {i}: \mathbf {r} (t - 1)\right), \sigma^ {2}\right), \tag {E.40}
$$
where  $p(\mathbf{r}(0))$  corresponds to some initial distribution of activity states. This probability distribution of firing rates corresponds to the following neural sampling dynamics:
$$
\mathbf {r} (t) = h \left(\mathbf {W r} (t - 1)\right) + \sigma \boldsymbol {\eta}, \tag {E.41}
$$
where  $\pmb {\eta}\sim \mathcal{N}(0,1)$  . For this model, we have the following theorem:
Theorem E.11. If  $p(\Theta) = \prod_{k} p(\Theta_{k})$  and  $p(\mathbf{X}|\Theta)$  is defined by Eq. E.39 (with  $\mathbf{X} = \mathbf{r}$  and  $\Theta = \mathbf{W}$ ), the RTRL update for  $\mathbf{W}_{ij}$  with a loss  $\mathcal{L}(\mathbf{r}(T))$ , given by  $\mathcal{A}_{RTRL}(p(\mathbf{r}|\mathbf{W}),\mathcal{L}(\mathbf{r}))$  is  $\mathbf{eJ}$ $p$ -local, where  $\mathbf{e} = \frac{\partial L(\mathbf{r}(T)}{\partial\bar{\mathbf{r}}(T)}$ , and  $\mathbf{J} = \{\mathbf{J}(t) = \frac{\partial\bar{\mathbf{r}}(t,\mathbf{r})}{\partial\bar{\mathbf{r}}(t - 1,\mathbf{r})}\}$ .
Proof. The directed graphical model corresponding to these dynamics is depicted in Figure 1b: as with backpropagation, we will use the score function for our graphical model to identify permissible variables. For a single synapse, the score function is given by:
$$
\begin{array}{l} \frac {\partial \log p (\mathbf {r} \mid \mathbf {W})}{\partial \mathbf {W} _ {i j}} = \sum_ {t = 1} ^ {T} \frac {\partial \log p \left(\mathbf {r} _ {i} (t) \mid \mathbf {r} (t - 1) , \mathbf {W}\right)}{\partial \mathbf {W} _ {i j}} (E.42) \\ = \sum_ {t = 1} ^ {T} \frac {\left(\mathbf {r} _ {i} (t) - h \left(\mathbf {V} _ {i} (t)\right)\right)}{\sigma^ {2}} h ^ {\prime} \left(\mathbf {V} _ {i} (t)\right) \mathbf {r} _ {j} (t - 1), (E.43) \\ \end{array}
$$
where  $\mathbf{V}_i(t) = \mathbf{W}_i\mathbf{r}(t - 1)$ . Thus  $p$ -local parameter updates for  $\mathbf{W}_{ij}$  may include  $\mathbf{W}_{i}$ ,  $\mathbf{r}_i(t)$  and  $\{\mathbf{r}_k(t - 1):\mathbf{W}_{ik}\neq 0\} \forall t$ . We will now compare these allowed variables to the RTRL update. As with backpropagation, we take  $\bar{\mathbf{r}} (t,\mathbf{r}) = h(\mathbf{Wr}(t - 1))$ , so that  $\mathbf{r}(t) = \bar{\mathbf{r}} (t,\mathbf{r}) + \sigma \pmb {\eta}$ . The RTRL update minimizes a loss  $\mathcal{L}(\mathbf{r}(T))$  via the chain rule [49, 21]:
$$
\Delta \mathbf {W} _ {i j} \propto \frac {\partial \mathcal {L} (\mathbf {r} (T))}{\partial \bar {\mathbf {r}} (T)} \frac {\partial \bar {\mathbf {r}} (T , \mathbf {r})}{\partial \mathbf {W} _ {i j}} \tag {E.44}
$$