SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\partial^ {2} \log p (\boldsymbol {\Theta})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = \frac {\partial^ {2} \sum_ {k} \log p (\boldsymbol {\Theta} _ {k})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = 0, \tag {C.21}
$$
where for the last equality we have used the assumption that  $\mathbf{Z}_i \neq \Theta_k$ . These two equations collectively imply:
$$
\Rightarrow \frac {\partial^ {2} \log p (\mathbf {X} | \boldsymbol {\Theta})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = \frac {\partial^ {2} \log p (\mathbf {Z}) - \log p (\boldsymbol {\Theta})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = 0, \tag {C.22}
$$
Property 2.8 For a mixture distribution  $p_{12}(\mathbf{Z},\gamma) = p_1(\mathbf{Z})^\gamma p_2(\mathbf{Z})^{1 - \gamma}p(\gamma)$  for some binary variable  $\gamma \in \{0,1\}$  with nonzero probabilities, if  $f(\mathbf{Z})$  is  $p_1$ -local (or equivalently  $p_2$ -local), then  $f(\mathbf{Z})$  is  $p_{12}$ -local.
Proof. We again work with the contrapositive definition of  $p$ -locality, observing that:
$$
\begin{array}{l} \mathbb {E} _ {p _ {1 2} (\mathbf {Z}, \gamma)} \left[ \left(\frac {\partial \log p _ {1 2} \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = 0 (C.23) \\ \Rightarrow \sum_ {k \in \{0, 1 \}} p (\gamma = k) \mathbb {E} _ {p _ {k} (\mathbf {Z})} \left[ \left(\frac {\partial \log p _ {k} \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = 0 (C.24) \\ \Rightarrow \mathbb {E} _ {p _ {1} (\mathbf {X}; \boldsymbol {\Theta})} \left[ \left(\frac {\partial \log p _ {1} \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = 0 (C.25) \\ \Rightarrow \frac {f _ {k} (\mathbf {X})}{\partial \mathbf {Z} _ {i}} = 0, (C.26) \\ \end{array}
$$
where the third implication follows from the fact that if the sum of two nonnegative quantities is zero, then both quantities are zero, and the final implication holds from the  $p_1$ -locality of  $f(\mathbf{Z})$ .
C.1 Regularity of Joint distribution
For several of these properties we enforce mild regularity constraints on the density. This is because we want the integral of the squared score being equal to zero to imply that the score itself is equal to zero. A sufficient condition for this is that the joint density function and partial derivatives w.r.t.  $\Theta_{k}\forall k$  are, for every fixed value of  $\mathbf{Z}$  's discrete elements, continuous functions of  $\mathbf{Z}$  's continuous elements.
D Locality for a linear continuous Boltzmann machine
Consider the following example of a simplified linear recurrent neuron model with synaptic weight matrices  $\mathbf{W}$ . The joint distribution is given by:
$$
p (\mathbf {r}, \mathbf {W}) = \frac {1}{Z} e ^ {- E (\mathbf {r}, \mathbf {W})} \tag {D.1}
$$
$$
\begin{array}{l} E (\mathbf {r}, \mathbf {W}) = \left(\frac {1}{2 \tau} \| \mathbf {r} \| _ {2} ^ {2} - \frac {1}{2} \mathbf {r} ^ {T} \mathbf {W r} + \frac {1}{2} \| \mathbf {W} \| _ {2} ^ {2}\right) / \sigma^ {2} (D.2) \\ = \left(\frac {1}{2 \tau} \sum_ {i} \mathbf {r} _ {i} ^ {2} - \frac {1}{2} \sum_ {i j} \mathbf {W} _ {i j} \mathbf {r} _ {i} \mathbf {r} _ {j} + \frac {1}{2} \sum_ {i j} \mathbf {W} _ {i j} ^ {2}\right) / \sigma^ {2}, (D.3) \\ \end{array}
$$
where  $\mathbf{W}$  is assumed to be a symmetric matrix. To see why this probability distribution is relevant for neuroscience, we first note that  $E$  is a linear, continuous analog of the Hopfield energy function, which is also used for discrete-valued Boltzmann machines. There are two critical differences between this probability distribution and the linear feedforward network explored in Section 2.4: first, this distribution corresponds to the undirected graphical model shown in Figure 1c, as opposed to the DAG shown in Figure 1a; second, the marginal distribution is not a free parameter that we can choose with convenient factorization properties if we want our joint distribution to give us a version of  $p$ -locality that corresponds with our concept of biological locality. For undirected graphical models, one typically is required to define the joint distribution first, and compute conditional distributions explicitly through Bayes theorem or approximate through some form of MCMC sampling. In our case, we can see that  $p(\mathbf{r}|\mathbf{W})$  corresponds to the steady-state distribution of a stochastic differential equation using  $E$  to perform Langevin sampling:
$$
\begin{array}{l} d \mathbf {r} _ {i} = - \left[ \nabla_ {\mathbf {r} _ {i} (t)} E (\mathbf {r} _ {i} (t), \mathbf {W}) \sigma^ {2} \right] d t + \sigma d \mathbf {B} _ {i} (t) (D.4) \\ = \left[ - \frac {1}{\tau} \mathbf {r} _ {i} (t) + \sum_ {j} \mathbf {W} _ {i j} \mathbf {r} _ {j} (t) \right] d t + \sigma d \mathbf {B} _ {i} (t), (D.5) \\ \end{array}
$$
where here  $\mathbf{B}_i(t)$  corresponds to uncorrelated Brownian noise injected into the system. These stochastic sampling dynamics correspond to a noisy linear recurrent network. Therefore,  $p(\mathbf{r}|\mathbf{W})$  corresponds to the steady-state stimulus response distribution of a linear recurrent network.
Let's ask: for which neural indices  $k$  can we have  $\frac{\partial}{\partial \mathbf{r}_k} f_{\mathbf{W}_{ij}}(\mathbf{r}) \neq 0$  so that the function  $f$  is still  $p$ -local? For  $f$  to remain  $p$ -local, we would need  $\mathbb{E}_p\left[\left(\frac{\partial \log p(\mathbf{r}_k|\mathbf{r}_{\neq k},\mathbf{W})}{\partial\mathbf{W}_{ij}}\right)^2\right] \neq 0$ . For the definition of  $p$ -locality to conform to our intuitions about biological locality, we would expect the only allowable variables to be the pre- and postsynaptic neurons  $\mathbf{r}_i$  and  $\mathbf{r}_j$  — we will show that including any other variable will violate  $p$ -locality. To see why, suppose  $k \neq i, j$ . Note that we can decompose  $E$  as:
$$
E (\mathbf {r}, \mathbf {W}) = E _ {k} + E _ {\neq k} \tag {D.6}
$$
$$
E _ {k} = \left(\frac {1}{2 \tau} \mathbf {r} _ {k} ^ {2} - \frac {1}{2} \sum_ {j} \mathbf {W} _ {k j} \mathbf {r} _ {k} \mathbf {r} _ {j} - \frac {1}{2} \sum_ {j} \mathbf {W} _ {j k} \mathbf {r} _ {j} \mathbf {r} _ {k}\right) / \sigma^ {2} \tag {D.7}
$$
$$
E _ {\neq k} = \left(\frac {1}{2 \tau} \sum_ {i \neq k} \mathbf {r} _ {i} ^ {2} - \frac {1}{2} \sum_ {i j \neq k} \mathbf {W} _ {i j} \mathbf {r} _ {i} \mathbf {r} _ {j} + \frac {1}{2} \sum_ {i j} \mathbf {W} _ {i j} ^ {2}\right) / \sigma^ {2}. \tag {D.8}
$$
Under this decomposition,  $E_{k}$  has no dependency on  $\mathbf{W}_{ij}$ , and  $E_{\neq k}$  has no dependency on  $\mathbf{r}_k$ . Now we're in a position to demonstrate that for any choice of  $k$  such that  $k \neq i,j$ ,  $f_{\mathbf{W}_{ij}}(\mathbf{r})$  cannot be  $p$ -local.
$$
p \left(\mathbf {r} _ {k} \mid \mathbf {r} _ {\neq k}, \mathbf {W}\right) = \frac {p (\mathbf {r} \mid \mathbf {W})}{p \left(\mathbf {r} _ {\neq k} , \mathbf {W}\right)} \tag {D.9}
$$
$$
= \frac {p (\mathbf {r} | \mathbf {W})}{\int p (\mathbf {r} , \mathbf {W}) d \mathbf {r} _ {k}} \tag {D.10}
$$
$$
= \frac {e ^ {- E}}{\int e ^ {- E} d \mathbf {r} _ {k}} \tag {D.11}
$$
$$
= \frac {e ^ {- (E _ {k} + E _ {\neq k})}}{e ^ {- E _ {\neq k}} \int e ^ {- (E _ {k})} d \mathbf {r} _ {k}} \tag {D.12}
$$
$$
= \frac {e ^ {- (E _ {k})}}{\int e ^ {- (E _ {k})} d \mathbf {r} _ {k}} \tag {D.13}
$$
$$
\Rightarrow \frac {\partial}{\partial \mathbf {W} _ {i j}} \log p \left(\mathbf {r} _ {k} \mid \mathbf {r} _ {\neq k}, \mathbf {W}\right) = 0 \tag {D.14}
$$
$$
\Rightarrow \mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {r} _ {k} \mid \mathbf {r} _ {\neq k} , \mathbf {W}\right)}{\partial \mathbf {W} _ {i j}}\right) ^ {2} \right] = 0. \tag {D.15}
$$
Because the conditional distribution has no dependency on  $\mathbf{W}_{ij}$ , then the Fisher Information is also 0, which concludes the demonstration:  $f_{\mathbf{W}_{ij}}$  is not  $p$ -local if it is a function of  $r_k$  for  $k \neq i, j$ . Of course, this decomposition of  $E = E_k + E_{\neq k}$  would not be possible if  $k = i$  or  $j$ . To summarize, for our simple example, any parameter update for  $\mathbf{W}_{ij}$  that depends on the activity of any neuron  $\mathbf{r}_k$  that is not the pre- or postsynaptic neuron ( $k \neq i, j$ ) cannot be  $p$ -local. Alternatively, since this is an undirected graphical model, we can also inspect its corresponding graph (summarized in Figure 1c.). To verify that the graph in Figure 1c. corresponds to our network, observe that our probability
distribution factorizes according the cliques of the graph [42] as follows:
$$
p (\mathbf {r}, \mathbf {W}) = \frac {1}{\mathcal {Z}} \prod_ {i} \phi (\mathbf {r} _ {i}) \prod_ {i j} \phi \left(\mathbf {W} _ {i j}\right) \prod_ {i j} \phi \left(\mathbf {r} _ {i}, \mathbf {r} _ {j}, \mathbf {W} _ {i j}\right) \tag {D.16}
$$
$$
\phi \left(\mathbf {r} _ {i}\right) = \exp \left(\frac {1}{2 \tau \sigma^ {2}} \mathbf {r} _ {i} ^ {2}\right) \tag {D.17}
$$
$$
\phi \left(\mathbf {W} _ {i j} = \exp \left(\frac {1}{2 \sigma^ {2}} \mathbf {W} _ {i j} ^ {2}\right) \right. \tag {D.18}
$$
$$
\phi (\mathbf {r} _ {i}, \mathbf {r} _ {j}, \mathbf {W} _ {i j}) = \exp \left(- \frac {1}{2 \sigma^ {2}} \mathbf {W} _ {i j} \mathbf {r} _ {i} \mathbf {r} _ {j}\right). \tag {D.19}
$$
Looking at the graph, we can verify by inspection that the only neighbors of  $\mathbf{W}_{ij}$  are  $\mathbf{r}_i$  and  $\mathbf{r}_j$ , which confirms our detailed analysis by Property 2.2.
E Proofs of  $p$ -locality properties of normative plasticity algorithms
E.1 REINFORCE
Theorem E.1. If  $p(\Theta) = \prod_k p(\Theta_k)$ , the REINFORCE estimator given by  $\mathcal{A}_R(p(R, \mathbf{X}|\Theta))$  is Rp-local.
Proof. The REINFORCE derivation proceeds as follows: suppose that we have some probabilistic formulation of a neural network and incoming sensory stimuli  $p(\mathbf{X}|\boldsymbol{\Theta})$  and some probabilistic reward function  $p(R|\mathbf{X})$  dependent on the stimuli and neural responses. We want to maximize expected reward:
$$
\mathbb {E} [ R ] = \int R p (R | \mathbf {X}) p (\mathbf {X} | \boldsymbol {\Theta}) d \mathbf {X} d R. \tag {E.1}
$$
If we want to modify our parameters  $\Theta$  in order to improve performance, we take steps in an approximation of the direction of the gradient of the objective  $\mathbb{E}[R]$ .
$$
\begin{array}{l} \frac {\partial}{\partial \boldsymbol {\Theta}} \mathbb {E} [ R ] = \frac {\partial}{\partial \boldsymbol {\Theta}} \int R p (R | \mathbf {X}) p (\mathbf {X} | \boldsymbol {\Theta}) d \mathbf {X} d R (E.2) \\ = \int R p (R | \mathbf {X}) \frac {\partial}{\partial \boldsymbol {\Theta}} p (\mathbf {X} | \boldsymbol {\Theta}) d \mathbf {X} d R (E.3) \\ = \int R p (R | \mathbf {X}) \frac {\partial}{\partial \boldsymbol {\Theta}} e ^ {\log p (\mathbf {X} | \boldsymbol {\Theta})} d \mathbf {X} d R (E.4) \\ = \int R p (R | \mathbf {X}) \left[ \frac {\partial}{\partial \boldsymbol {\Theta}} \log p (\mathbf {X} | \boldsymbol {\Theta}) \right] p (\mathbf {X} | \boldsymbol {\Theta}) d \mathbf {X} d R (E.5) \\ \approx \frac {1}{K} \sum_ {k = 1} ^ {K} R ^ {(k)} \frac {\partial}{\partial \boldsymbol {\Theta}} \log p \left(\mathbf {X} ^ {(k)} \mid \boldsymbol {\Theta}\right), (E.6) \\ \end{array}
$$
where in this last step we have employed a Monte Carlo approximation of the expectation, where  $R^{(k)}$  and  $\mathbf{X}^{(k)} \sim p(R,\mathbf{X})$ . This update function:  $f(R,\mathbf{Z}) = R \times \frac{\partial}{\partial\Theta}\log p(\mathbf{X}|\Theta)$  is not  $p$ -local because we have  $\partial f(R,\mathbf{Z}) / \partial R \neq 0$ , while  $\frac{\partial}{\partial\Theta}p(R|\mathbf{X}) = 0$ . However, as we know, this update is the product of a score function with a marginal parameter distribution that we have assumed factorizes, which we know to be  $p$ -local by Property 2.7, with a scalar reward  $R$ . In this case, one could postulate that reward information is projected broadly to many synapses in the neural network via a neuromodulatory pathway (Figure 2a). We see that  $f(R,\mathbf{X}) = h(R,g(\mathbf{X}))$  if we take  $h(a,b) = a \times b$  and  $g(\mathbf{X}) = \frac{\partial}{\partial\Theta}\log p(\mathbf{X}|\Theta)$ ; we further see that  $g$  is  $p$ -local, and hence  $f(R,\mathbf{X})$  is by Definition 2.3  $Rp$ -local.
This might seem contrived, because any function is  $\mathbf{S}p$ -local for some sufficiently broad choice of  $\mathbf{S}$ . However, we have shown here that the REINFORCE algorithm is  $Rp$ -local for any choice of  $p$  with a marginal parameter distribution that factorizes (an easy constraint to satisfy for directed graphical model architectures). This means that we can make any of a huge variety of neural network or probabilistic model choices and still have the REINFORCE algorithm obey the same notion of locality, without having to modify our definition post-hoc.
E.2 Maximum Likelihood Estimation (MLE)
MLE is a highly popular machine learning method and the fundamental basis for several subsequent normative plasticity algorithms. This algorithm involves fitting a model,  $p_{m}(\mathbf{X}|\boldsymbol{\Theta})$ , to an empirical data distribution,  $p_{d}(\mathbf{X})$ .
Theorem E.2. If  $p(\Theta) = \prod_k p(\Theta_k)$ , the MLE update given by  $\mathcal{A}_{MLE}(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}))$  is  $p_m$ -local.
Proof. We proceed by first deriving the MLE update function. The objective function for maximum likelihood estimation is given by the KL divergence between an empirical data distribution,  $p_d(\mathbf{X})$  and a probabilistic model of the data  $p_m(\mathbf{X}|\boldsymbol{\Theta})$ . We have:
$$
\operatorname {K L} \left[ p _ {d} (\mathbf {X}) | | p _ {m} (\mathbf {X} | \boldsymbol {\Theta}) \right] = - \int \log \left(\frac {p _ {m} (\mathbf {X} | \boldsymbol {\Theta})}{p _ {d} (\mathbf {X})}\right) p _ {d} (\mathbf {X}) d \mathbf {X}. \tag {E.7}
$$
We want to minimize this objective function, which we do by gradient descent:
$$
\begin{array}{l} \mathcal {A} _ {M L E} (p (\mathbf {X}), p _ {m} (\mathbf {X})) \propto \frac {\partial}{\partial \boldsymbol {\Theta}} \int \log \left(\frac {p _ {m} (\mathbf {X} | \boldsymbol {\Theta})}{p _ {d} (\mathbf {X})}\right) p _ {d} (\mathbf {X}) d \mathbf {X} (E.8) \\ = \int \frac {\partial}{\partial \boldsymbol {\Theta}} \log \left(p _ {m} (\mathbf {X} | \boldsymbol {\Theta})\right) p _ {d} (\mathbf {X}) d \mathbf {X} (E.9) \\ \approx \frac {1}{K} \sum_ {k = 0} ^ {K} \frac {\partial}{\partial \boldsymbol {\Theta}} \log \left(p _ {m} \left(\mathbf {X} _ {k} \mid \boldsymbol {\Theta}\right)\right), (E.10) \\ \end{array}
$$
where  $\mathbf{X}_k\sim p_d(\mathbf{X})$  , and in the last approximate equality we have used a Monte Carlo sampling integral approximation. This update exclusively contains the score function of  $p_m$  , so by Property 2.7, the update is  $p_m$  -local.
E.3 Generalized EM (GEM)
MLE estimation runs into difficulties when attempting to fit latent variable models, (e.g. when  $p_{m}(\mathbf{X}_{o}) = \int p_{m}(\mathbf{X}_{o},\mathbf{X}_{h})d\mathbf{X}_{h})$ , where  $\mathbf{X}_h$  are latent variables within the model distribution that 'explain' observed data  $\mathbf{X}_o$ . Latent variable models are extraordinarily powerful, and appear in computational neuroscience in a variety of forms, including but not limited to factor analysis, hidden Markov models, and Kalman Filters [43]; for these models, we will take  $\mathbf{X} = [\mathbf{X}_o,\mathbf{X}_h]$ . Instead of performing explicit MLE, when fitting latent variable models one usually resorts to some variant of the Expectation-Maximization (EM) algorithm [44]. Here, we show that a particular variant of the EM algorithm, called Generalized EM (GEM) [45], is  $p_{m}$ -local in the same way as MLE.
GEM gains computational benefits by substituting (by any of a variety of methods) an approximate posterior distribution  $p_d(\mathbf{X}_h|\mathbf{X}_o)$  for the true, but typically intractable, model posterior  $p_m(\mathbf{X}_h|\mathbf{X}_o)$  via minimizing a variational free energy [45]. However, GEM is not just a convenient model-fitting algorithm: in subsequent sections, we will show that the  $p_m$ -locality of GEM explains why several popular normative plasticity algorithms produce biologically plausible updates.
Theorem E.3. If  $p(\Theta) = \prod_k p(\Theta_k)$ , the GEM update given by  $\mathcal{A}_{GEM}(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}))$  is  $p_m$ -local.
Proof. Rather than minimize  $\mathrm{KL}[p_d(\mathbf{X}_o)||p_m(\mathbf{X}_o|\boldsymbol {\Theta})]$ , the GEM algorithm minimizes an upper bound (the variational free energy). Taking  $\mathbf{X} = [\mathbf{X}_o,\mathbf{X}_h]$ :
$$
\begin{array}{l} \operatorname {K L} \left[ p _ {d} \left(\mathbf {X} _ {o}\right) | | p _ {m} \left(\mathbf {X} _ {o} | \boldsymbol {\Theta}\right) \right] \geq \operatorname {K L} \left[ p _ {d} \left(\mathbf {X} _ {o}\right) | | p _ {m} \left(\mathbf {X} _ {o} | \boldsymbol {\Theta}\right) \right] + \mathbb {E} _ {p _ {d} \left(\mathbf {X} _ {o}\right)} \left[ \operatorname {K L} \left[ p _ {d} \left(\mathbf {X} _ {h} | \mathbf {X} _ {o}\right) | | p _ {m} \left(\mathbf {X} _ {h} | \mathbf {X} _ {o}, \boldsymbol {\Theta}\right) \right] \right] \\ = \operatorname {K L} [ p _ {d} (\mathbf {X}) | | p _ {m} (\mathbf {X} | \boldsymbol {\Theta}) ], \tag {E.11} \\ \end{array}
$$
where the inequality follows from the positivity of the KL divergence. Here,  $p_d(\mathbf{X}_h|\mathbf{X})$  is an approximate inference distribution. Different choices of how this distribution is selected/optimized can produce very different learning algorithms, with varying degrees of biological plausibility. Obviously, the loss is minimized with respect to  $p_d(\mathbf{X}_h|\mathbf{X})$  if  $p_d(\mathbf{X}_h|\mathbf{X}) = p_m(\mathbf{X}_h|\mathbf{X},\Theta_0)$  (where  $\Theta_0 = \Theta$  prior to optimization wrt  $\Theta$ ). This choice corresponds to GEM [45]. For now, we will not concern ourselves with how  $p_d(\mathbf{X}_h|\mathbf{X})$  is selected--instead, we will focus on the locality properties of gradient updates of this loss with respect to  $\Theta_{m}$ .
Having packaged hidden and observed variables together  $(\mathbf{X} = [\mathbf{X}_o, \mathbf{X}_h])$ , our derivation proceeds exactly the same as for MLE:
$$
\begin{array}{l} \mathcal {A} _ {G E M} (p _ {m} (\mathbf {X} | \boldsymbol {\Theta}), p _ {d} (\mathbf {X})) \propto - \frac {\partial}{\partial \boldsymbol {\Theta}} \mathrm {K L} [ p _ {d} (\mathbf {X}) | | p _ {m} (\mathbf {X} | \boldsymbol {\Theta}) ] \\ = \frac {\partial}{\partial \boldsymbol {\Theta}} \int \log \left(\frac {p _ {m} (\mathbf {X} | \boldsymbol {\Theta})}{p _ {d} (\mathbf {X})}\right) p _ {d} (\mathbf {X}) d \mathbf {X} \\ = \int \frac {\partial}{\partial \boldsymbol {\Theta}} \log \left(p _ {m} (\mathbf {X} | \boldsymbol {\Theta})\right) p _ {d} (\mathbf {X}) d \mathbf {X} \\ \approx \frac {1}{K} \sum_ {k = 0} ^ {K} \frac {\partial}{\partial \boldsymbol {\Theta}} \log \left(p _ {m} \left(\mathbf {X} _ {h}, \mathbf {X} \mid \boldsymbol {\Theta}\right)\right), \tag {E.12} \\ \end{array}
$$
where as with the MLE update,  $\mathbf{X}_k\sim p_m(\mathbf{X}|\boldsymbol {\Theta})$  . This update is  $p_m$  -local for the same reason that the MLE update is.
E.4 Predictive Coding (PC)
As an additional note, if one takes the approximate posterior  $p_d(\mathbf{X}_h|\mathbf{X}_o)$  to be given by:
$$
p _ {d} \left(\mathbf {X} _ {h} \mid \mathbf {X} _ {o}\right) = \underset {p _ {d} \left(\mathbf {X} _ {h} \mid \mathbf {X} _ {o}\right)} {\operatorname {a r g m i n}} \mathrm {K L} \left[ p _ {d} (\mathbf {X}) \mid \left| p _ {m} (\mathbf {X} \mid \boldsymbol {\Theta}) \right| \right. s. t. p _ {d} \left(\mathbf {X} _ {h} \mid \mathbf {X} _ {o}\right) \sim \delta \left(\bar {\mathbf {X}} _ {h} \left(\mathbf {X} _ {o}\right)\right), \tag {E.13}
$$
where  $\delta (\cdot)$  indicates a Dirac delta distribution and  $\bar{\mathbf{X}}_h(\mathbf{X}_o)$  indicates a set of observation-dependent mean parameters, then we recover the predictive coding family of algorithms [35]. Typically, in this context for a given observed stimulus  $\mathbf{X}_o$ ,  $p_d(\mathbf{X}_h|\mathbf{X}_o)\sim \delta (\bar{\mathbf{X}}_h(\mathbf{X}_o))$  is estimated by reparameterization and gradient descent with respect to  $\bar{\mathbf{X}}_h(\mathbf{X}_o)$  (a mean parameter that is observation-dependent), which—for clever choices of  $p_m$ —loosely resembles the dynamics of a recurrent neural network relaxing to a stimulus-conditioned equilibrium state [33]. After estimating  $\bar{\mathbf{X}}_h(\mathbf{X}_o)$ , parameters  $\Theta$  are updated as in GEM. Therefore, the derivation above also applies to predictive coding algorithms, which are consequently also  $p_m$ -local.
Theorem E.4. If  $p(\Theta) = \prod_k p(\Theta_k)$ , the PC update given by  $\mathcal{A}_{PC}(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}))$  is  $p_m$ -local.
E.5 Wake-Sleep
Unlike the previous three examples, which only require sampling from the  $p_d$  distribution and only calculate parameter updates according to the  $p_m$  distribution, the Wake-Sleep algorithm parameterizes both distributions and jointly samples from a mixture of the two distributions across its 'wake' and 'sleep' phases. As we will see, this will mean that the Wake-Sleep algorithm will end up being  $\gamma p_{md}$ -local, where  $\gamma$  is the binary variable that controls whether the system is in its 'wake' or 'sleep' phase.
Theorem E.5. If  $p(\Theta, \Theta^{(d)}) = \left( \prod_k p(\Theta_k) \right) \left( \prod_k p(\Theta_k^{(d)}) \right)$ , the Wake-Sleep estimator given by  $\mathcal{A}_{WS}(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}|\Theta^{(d)}))$  is  $\gamma p_{md}$ -local, where  $p_{md} = Mix(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}|\Theta^{(d)})) p(\Theta, \Theta^{(d)})$ .
Proof. Our updates use a similar loss to the GEM algorithm, namely we take:
$$
\mathcal {A} _ {W S} \left(p _ {m} (\mathbf {X} | \boldsymbol {\Theta}), p _ {d} (\mathbf {X} | \boldsymbol {\Theta} ^ {(d)})\right) = \left[ \Delta \boldsymbol {\Theta} _ {W S}, \Delta \boldsymbol {\Theta} _ {W S} ^ {(d)} \right], \tag {E.14}
$$
where  $\Delta \Theta_{WS}$  is given by:
$$
\begin{array}{l} \Delta \Theta_ {W S} \propto - \frac {\partial}{\partial \Theta} \mathrm {K L} [ p _ {d} (\mathbf {X} | \boldsymbol {\Theta} ^ {(d)}) | | p _ {m} (\mathbf {X} | \boldsymbol {\Theta}) ] \\ \approx \frac {2}{K} \sum_ {k = 0} ^ {K} \gamma_ {k} \frac {\partial}{\partial \boldsymbol {\Theta}} \log \left(p _ {m} \left(\mathbf {X} _ {h}, \mathbf {X} \mid \boldsymbol {\Theta}\right)\right). \tag {E.15} \\ \end{array}
$$
Here,  $\gamma_{k}$ ,  $\mathbf{X} \sim p_{md}(\mathbf{X},\gamma |\boldsymbol{\Theta},\boldsymbol{\Theta}^{(d)})$ , whereas for GEM, we sampled only from  $p_d$ . Because each term of this update is 0 if  $\gamma_k \neq 1$ , this update is still an unbiased estimate of the gradient [46] and is effectively equivalent to the GEM update, except that it allows the system to alternate between sampling from  $p_m$  and  $p_d$ . This alternation is useful because it will allow also calculating parameter updates for  $\Delta \Theta^{(d)}$ . For  $\Theta^{(d)}$ , we optimize the reverse KL-divergence; by a directly analogous derivation to Eq. E.12, the update is given by:
$$
\begin{array}{l} \Delta \boldsymbol {\Theta} _ {W S} ^ {(d)} \propto - \frac {\partial}{\partial \boldsymbol {\Theta} ^ {(d)}} \mathrm {K L} [ p _ {m} (\mathbf {X} | \boldsymbol {\Theta}) | | p _ {d} (\mathbf {X} | \boldsymbol {\Theta} ^ {(d)}) ] \\ \approx \frac {2}{K} \sum_ {k = 0} ^ {K} \left(1 - \gamma_ {k}\right) \frac {\partial}{\partial \boldsymbol {\Theta} ^ {(d)}} \log \left(p _ {d} (\mathbf {X} | \boldsymbol {\Theta} ^ {(d)})\right). \tag {E.16} \\ \end{array}
$$
Now, these updates contain the score function for both  $p_m$  and  $p_d$ , as well as the scalar mixture variable  $\gamma$ . As a consequence, both updates are  $\gamma p_{md}$ -local, via Properties 2.7 and 2.8 (slightly more precisely,  $\Delta \Theta$  is  $\gamma p_m$ -local, and  $\Delta \Theta^{(d)}$  is  $\gamma p_d$ -local).