SlowGuess/NeurIPS_2023_OCR huggingface dataset:
BR was supported by NSERC (Discovery Grant: RGPIN-2020-05105; Discovery Accelerator Supplement: RGPAS-2020-00031; Arthur B. McDonald Fellowship: 566355-2022), CIFAR (Canada AI Chair; Learning in Machine and Brains Fellowship), the Canada Research Chair in Neural Computations and Interfacing. GL was supported by NSERC (Discovery Grant: RGPIN-2018-04821), CIFAR (Canada AI Chair), and a Canada Research Chair in Neural Computations and Interfacing. CB and GL acknowledge support from a Simon's Collaboration on the Global Brain Pilot Award. EW was supported by an NSERC CGS-D scholarship. CS was supported by the National Science Foundation under NSF Award No. 1922658.
[38] Panayiota Poirazzi, Terrence Brannon, and Bartlett W Mel. Pyramidal neuron as two-layer neural network. Neuron, 37(6):989-999, 2003.
[39] Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann, 1988.
[40] Steffen L Lauritzen. Elements of graphical models. Lectures from the XXXVIIth International Probability Summer School in St-Flour, France. http://www.stats.ox.ac.uk/steffen, 2011.
[41] Richard E Neapolitan et al. Learning bayesian networks, volume 38. Pearson Prentice Hall Upper Saddle River, 2004.
[42] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[43] Sam Roweis and Zoubin Ghahramani. A unifying review of linear Gaussian models. Neural computation, 11(2):305-345, 1999.
[44] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1-22, 1977.
[45] Geoffrey J McLachlan and Thriyambakam Krishnan. The EM algorithm and extensions. John Wiley & Sons, 2007.
[46] Ezekiel Williams, Colin Bredenberg, and Guillaume Lajoie. Flexible phase dynamics for bio-plausible contrastive learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 37042-37065. PMLR, 23-29 Jul 2023.
[47] Shiro Ikeda, Shun-ichi Amari, and Hiroyuki Nakahara. Convergence of the wake-sleep algorithm. Advances in neural information processing systems, 11, 1998.
[48] Bernhard Nessler, Michael Pfeiffer, Lars Buesing, and Wolfgang Maass. Bayesian computation emerges in generic cortical microcircuits through spike-timing-dependent plasticity. PLoS computational biology, 9(4):e1003037, 2013.
[49] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
[50] Aditya Gilra and Wulfram Gerstner. Predicting non-linear dynamics by stable local learning in a recurrent spiking neural network. *Elife*, 6:e28295, 2017.
[51] Thomas L O'Donohue, William R Millington, Gail E Handelmann, Patricia C Contreras, and Bibie M Chronwall. On the 50th anniversary of Dale's law: multiple neurotransmitter neurons. Trends in Pharmacological Sciences, 6:305-308, 1985.
[52] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv preprint arXiv:2006.08381, 2020.
[53] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
A Defining Markov locality and relating it to  $p$ -locality
To gain intuition for how  $p$ -locality functions, we will introduce another notion of locality, called Markov locality, which will use the language of Markov blankets. We will prove that under relatively relaxed conditions  $p$ -locality and Markov locality are equivalent. This will allow us to relate the notion of locality to various graph structures commonly used to represent probability distributions, and will be a key step in proving Properties 2.1 and 2.2.
We start by defining the Markov boundary,  $\mathcal{M}(X,S)$ , of a random variable  $X$  contained in a set of random variables  $S$ , as a minimal set such that  $p(X|S) = p(X|\mathcal{M}(X,S))$ . The Markov boundary defines a minimal set of variables such that, conditioned on these variables, conditioning on no additional random variables in  $S$  changes the probability of  $X$  [39]. Similarly, we define the Markov blanket,  $M(X,S)$  for  $X$  in  $S$  as any set of variables such that conditioning on  $M(X,S)$ , makes  $X$  conditionally independent from all other variables [39]. In this way, the Markov boundary is a Markov blanket but not all blankets are boundaries.
Definition A.1. Markov locality: Given probability distribution  $p(\mathbf{Z})$  and function  $f: \mathbb{R}^{N_{\mathbf{X}} + N_{\Theta}} \to \mathbb{R}^{N_{\Theta}}$ , the update function  $f(\mathbf{Z})$  is Markov-local with respect to the distribution  $p$  over  $\mathbf{Z}$  if and only if  $\forall k$ :
$$
\exists \mathbf {Z} \in \Omega \text {s . t .} \frac {\partial f _ {k} (\mathbf {Z})}{\partial \mathbf {Z} _ {i}} \neq 0 \Rightarrow \mathbf {Z} _ {i} \in \mathcal {M} \left(\boldsymbol {\Theta} _ {k}, \mathbf {Z}\right). \tag {A.1}
$$
Markov locality requires that the set of variables used in the parameter update  $f_{k}(\mathbf{Z})$  is a subset of the Markov boundary of the parameter itself. A Markov boundary can be thought of as the set of variables that 'locally' communicate with the parameter  $\Theta_{k}$ , thus providing a natural measure of locality.
Importantly, for Markov-locality to be of use, we would like the Markov boundaries of random variables in the model of interest to be unique. Without this requirement there will be ambiguity, for a given  $p$ , in terms of which updates are considered local and which are not. To guarantee this, we ask that the conditional independence relationships implied by  $p$  satisfy four properties, commonly referred to as graphoid properties [39, 40]. A sufficient condition for these to hold is that the distribution have a strictly positive density (see Appendix for more details B). With this, and some mild regularity assumptions, we can prove the following equivalence between Markov locality and  $p$ -locality:
Theorem A.1. Assume all quantities are as in A.1, that the conditional independence relationships implied by  $p(\mathbf{Z})$  satisfy the four graphoid properties given in Section B, and that mild regularity assumptions are satisfied by the joint distribution (see Section C.1). Then Equation 2 holds if and only if Equation A.1 also holds.
Proof. This proof relies on Lemma A.1, proved below.
We wish to prove Eq. 2  $\Longleftrightarrow$  Eq. A.1. It suffices to show the following:
$$
\mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] \neq 0 \Longleftrightarrow \mathbf {Z} _ {i} \in \mathcal {M} (\boldsymbol {\Theta} _ {k}, \mathbf {Z}) \tag {A.2}
$$
Using the contrapositive for the left and right implications separately shows that Equation A.2 is equivalent to
$$
\mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = 0 \Longleftrightarrow \mathbf {Z} _ {i} \notin \mathcal {M} (\boldsymbol {\Theta} _ {k}, \mathbf {Z}), \tag {A.3}
$$
which means that it suffices to prove Equation A.3 for the proof. Observe that
$$
E _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = 0 \Longleftrightarrow \frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \mathbf {Z} _ {k}} = 0 \quad \forall \mathbf {Z} \in \Omega , \tag {A.4}
$$
which follows from the regularity assumptions. From here, the proof follows by Lemma A.1.
Lemma A.1. Let  $\mathbf{X}$ ,  $\Theta$ ,  $\mathbf{Z}$ ,  $k$  and  $i$ , and  $p$  be defined as in Theorem A.1. Then
$$
\frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \mathbf {Z} _ {k}} = 0 \quad \forall \mathbf {Z} \in \Omega \iff \mathbf {Z} _ {i} \notin \mathcal {M} \left(\mathbf {Z} _ {k}, \mathbf {Z}\right) \tag {A.5}
$$
Proof. First, observe that
$$
\frac {\partial \log p (\mathbf {Z} _ {i} | \mathbf {Z} _ {\neq i})}{\partial \mathbf {Z} _ {k}} = 0 \quad \forall \quad \mathbf {Z} \in \Omega
$$
$$
\Longleftrightarrow \frac {\partial p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \mathbf {Z} _ {k}} = 0 \quad \forall \quad \mathbf {Z} \in \Omega \tag {A.6}
$$
by the chain rule. By applying the fundamental theorem of calculus to this derivative, which we can do by the assumption of differentiability on  $\mathbb{R}$ , we find that  $p(\mathbf{Z}_i|\mathbf{Z}_{\neq i})$  is constant w.r.t.  $\mathbf{Z}_k$  on  $\Omega$  so that
$$
p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right) = p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right) \int_ {\mathbb {R}} p \left(\mathbf {Z} _ {k} \mid \mathbf {Z} _ {\neq \{i, k \}}\right) \mathrm {d} \mathbf {Z} _ {k} \tag {A.7}
$$
$$
= \int_ {\mathbb {R}} p \left(\mathbf {Z} _ {i}, \mathbf {Z} _ {k} \mid \mathbf {Z} _ {\neq \{i, k \}}\right) \mathrm {d} \mathbf {Z} _ {k} = p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq \{i, k \}}\right) \tag {A.8}
$$
where we have also used that a probability distribution integrates to 1, and that  $p(\mathbf{Z}_k|\mathbf{Z}_{\neq \{i,k\}})$  will be equal to zero outside  $\Omega$ . Because  $\mathbf{Z} \in \Omega$  is arbitrary, from the above  $\mathbf{Z}_i$  is independent of  $\mathbf{Z}_k$  given the other random variables in  $\mathbf{Z}$ . Using Lemma B.1 (which we can do by assumption of the graphoid properties), we see that if  $\frac{\partial \log p(\mathbf{Z}_i|\mathbf{Z}_{\neq i})}{\partial \mathbf{Z}_k} = 0 \forall \mathbf{Z} \in \Omega$ , by Eq. A.6 - A.8,  $\mathbf{Z}_i \notin \mathcal{M}(\mathbf{Z}_k,\mathbf{Z})$ . Conversely, if we start with the assumption that  $\mathbf{Z}_i \notin \mathcal{M}(\mathbf{Z}_k,\mathbf{Z})$ , we immediately get  $\mathbf{Z}_k \notin \mathcal{M}(\mathbf{Z}_i,\mathbf{Z})$ , by Lemma B.1, and see that  $p(\mathbf{Z}_i|\mathbf{Z}_{\neq i})$  must not be a function of  $\mathbf{Z}_k$  for all  $\mathbf{Z}$ ; thus, the derivative w.r.t.  $\mathbf{Z}_k$  is equal to zero for  $\mathbf{Z} \in \Omega$ . Applying Equation A.6 completes the proof.
B Notes on Probabilistic Graphical Models
In this section we compile several properties, definitions, and results on Markov boundaries and Probabilistic Graphical Models (PGMs) that underlie Theorem A.1, and Properties 2.1 and 2.2. We begin by setting up notation. Let us assume that we have a joint probability distribution,  $P$ , over a set of random variables  $S$ , and that  $W, X, Y, Z \subset S$ , and  $U, V, T \in S$ . We use  $X \perp Y|Z$  to mean that the set of random variables  $X$  is independent of the set  $Y$  given set  $Z$ , and assume that the reader is familiar with the notion of a directed graph, an undirected graph, and graph separation. If a set  $X$  contains only a single random variable  $U$  then we abuse notation and write  $U$  in place of  $X$ .
The following four properties—known as the graphoid properties (or axioms—see e.g. [39, 40])—are useful for getting well-behaved Markov boundaries and in assigning graphical representation to probability distributions:
Definition B.1. Pseudo-graphoid properties:
- Symmetry:  $X \perp Y|Z \Rightarrow Y \perp X|Z$
- Decomposition:  $X \perp Y, W|Z \Rightarrow X \perp Y|Z \& X \perp W|Z$
- Weak union:  $X \perp Y, W|Z \Rightarrow X \perp Y|W, Z$
- Intersection:  $X \perp Y|Z, W$  &  $X \perp W|Z, Y \Longrightarrow X \perp Y, W|Z$
Importantly, it is known that these properties are satisfied when we have a density  $p$  that is strictly positive w.r.t. to its base product measure [40]. Here, measure is used in a measure theoretic sense; e.g. we have assumed throughout the paper that the base measure is simply a product of a multidimensional Lebesgue measure over  $\mathbb{R}^{k_1}$ , for some  $k_{1}$ , and a counting measure over  $\mathbb{N}^{k_2}$  or some subset of  $\mathbb{N}^{k_2}$ , for some  $k_{2}$ . Roughly speaking, this positivity property means that there are no purely deterministic relationships between variables.
The key result that we use to guarantee that the Markov boundaries we discuss in the paper are well-defined is given in [39]. We state a paraphrased and shortened version below for completeness:
Theorem B.1. Theorem 4, Chapter 3 in [39]: every  $P$  with conditional independence relations satisfying the four pseudo-graphoid properties has a unique Markov boundary for each  $X$ .
We now add two more simple results on Markov boundaries, used in the proof of Theorem A.1:
Lemma B.1. If  $P$  has conditional independence relations satisfying the four pseudo-graphoid properties we have:
- for every  $U, V \in S$ ,  $U \in \mathcal{M}(V, S) \iff V \in \mathcal{M}(U, S)$
- for every  $U \in S$ ,  $\mathcal{M}(U, S)$  is contained in every Markov blanket of  $U$ .
These follow simply from the graphoid properties so we omit the proof.
Lastly, we make specific what we mean when we say that a graph is an undirected or directed graphical model for a distribution.
Definition B.2. Let  $\mathcal{G}$  be an undirected graph where each node corresponds to a random variable in  $S$ . We say that  $\mathcal{G}$  is an Undirected Graph (UG) for  $\mathbf{P}$  if whenever  $X$  and  $Y$  are separated by  $Z$  in  $\mathcal{G}$ ,  $X \perp Y|Z$  is true under  $P$ . Note that this corresponds to the notion of I-map in [39].
Definition B.3. Let  $\mathcal{G}_{\mathrm{d}}$  be a directed graph with each vertex corresponding to a random variable in  $S$ . We say that  $\mathcal{G}_{\mathrm{d}}$  is a Directed Graph for  $\mathbf{P}$  if the variable under  $P$  corresponding to any node in the graph is conditionally independent of all variables corresponding to nodes that are non-descendants given the variables corresponding to parents. This is equivalent to  $\mathcal{G}_{\mathrm{d}}$  satisfying the Markov condition described in Definition 1.9 of [41].
C Proofs for  $p$ -locality properties
For the first two properties we assume the requirements of Theorem A.1 are satisfied. For all properties except 2.3, 2.4, and 2.8 we assume  $p$  satisfies mild regularity constraints (see Section C.1).
Property 2.1 Assume  $\mathcal{G}_{\mathrm{d}}$  is a Directed Acyclic Graph (DAG) for  $p$ . If  $\frac{\partial f_k(\mathbf{Z})}{\partial \mathbf{Z}_i} \neq 0$  for  $\mathbf{Z}_i$  that is not a parent, co-parent, or child of  $\Theta_k$  in  $\mathcal{G}_{\mathrm{d}}$ , then  $f$  is not  $p$ -local.
Proof. By Theorem A.1 we get that  $\frac{\partial f_k(\mathbf{Z})}{\partial \mathbf{Z}_i}$  can only be non-zero on the unique Markov boundary of  $\mathbf{Z}_i$  if it is  $p$ -local. By the definition of a DAG, the parents, co-parents, and children of  $\mathbf{Z}_i$  form a Markov blanket for it (see e.g. [41] Th. 2.13). By Lemma B.1 the boundary is included in all Markov blankets so  $\frac{\partial f_k(\mathbf{Z})}{\partial \mathbf{Z}_i}$  can only be non-zero on some subset of the parents, co-parents, and children of  $\mathbf{Z}_i$ .
Property 2.2 Assume that  $\mathcal{G}$  defines an Undirected Graph (UG) for  $p$ . If  $\frac{\partial f_k(\mathbf{Z})}{\partial \mathbf{Z}_i} \neq 0$  for  $\mathbf{Z}_i$  that is not a neighbour of  $\Theta_k$  in  $\mathcal{G}$ , then  $f$  is not  $p$ -local.
Proof. As above, by Theorem A.1 we get that  $\frac{\partial f_k(\mathbf{Z})}{\partial\mathbf{Z}_i}$  can only be non-zero on the unique Markov boundary of  $\mathbf{Z}_i$  if it is  $p$ -local. A UG for a distribution is an I-map for it, and conditioning on the neighbours in an I-map renders a node independent from the other nodes in the graph by definition—thus the neighbours form a Markov blanket. By Lemma B.1 the Markov boundary is included in every blanket so  $\frac{\partial f_k(\mathbf{Z})}{\partial\mathbf{Z}_i}$  can only be non-zero on some subset of the neighbours in the UG.
Property 2.3 For any function  $b(\mathbf{Z}) : \mathbb{R}^{N\mathbf{x} + N\Theta} \to \mathbb{R}^{N\Theta}$  defined such that  $b_{k}(\mathbf{Z}) = h_{k}(f_{k}(\mathbf{Z}), g_{k}(\mathbf{Z}))$ , where  $f$  and  $g$  are  $p$ -local and  $h_{k}$  is differentiable,  $b(\mathbf{Z})$  is also  $p$ -local.
Proof. Suppose that  $\frac{\partial b_k(\mathbf{Z})}{\partial \mathbf{X}_i} \neq 0$ . We need to show that  $\mathbb{E}_p\left[\left(\frac{\partial \log p(\mathbf{Z}_i | \mathbf{Z}_{\neq i})}{\partial \Theta_k}\right)^2\right] \neq 0$ . Knowing that  $\frac{\partial b_k(\mathbf{Z})}{\partial \mathbf{X}_i} \neq 0$ , we have:
$$
\frac {\partial h _ {k} (\mathbf {Z})}{\partial f _ {k}} \frac {\partial f _ {k} (\mathbf {Z})}{\partial \mathbf {Z} _ {i}} + \frac {\partial h _ {k} (\mathbf {Z})}{\partial g _ {k}} \frac {\partial g _ {k} (\mathbf {Z})}{\partial \mathbf {X} _ {i}} \neq 0. \tag {C.1}
$$
This implies that either  $\frac{\partial f_k(\mathbf{Z})}{\partial \mathbf{Z}_i} \neq 0$  or  $\frac{\partial g_k(\mathbf{Z})}{\partial \mathbf{Z}_i} \neq 0$  (or both). No matter which is true, by virtue of the  $p$ -locality of  $f$  and  $g$ , we have the consequence:
$$
\mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] \neq 0, \tag {C.2}
$$
which concludes our proof. This demonstrates that  $p$ -local functions can be more or less arbitrarily combined without the combination losing the  $p$ -local property.
Property 2.4 For any function  $f(\cdot): \mathbb{R}^{N_{\mathbf{X}} + N_{\Theta}} \to \mathbb{R}^{N_{\Theta}}$ , there exists a probability distribution  $p(\mathbf{Z})$  such that the random variable  $f(\mathbf{Z})$  with  $\mathbf{Z} \sim p(\mathbf{Z})$  is  $p$ -local.
Proof. We can prove this property by construction. Take  $p(\{\mathbf{X}_i : \frac{\partial f(\mathbf{X})}{\partial \mathbf{X}_i} \neq 0\} | \Theta) = \mathcal{N}(\Theta^T \Theta, \mathbf{I})$ , i.e., the distribution of every variable contained within  $f$  has mean parameter dependence on all  $\Theta$  variables. The probability distributions for all other variables  $\mathbf{Z}$  are otherwise unconstrained. Then for all  $i$  such that  $\frac{\partial f(\mathbf{X})}{\partial \mathbf{X}_i} \neq 0$ , we have:
$$
\begin{array}{l} \mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {X} _ {i} \mid \mathbf {X} _ {\neq i} , \boldsymbol {\Theta}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = \mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {X} _ {i} \mid \boldsymbol {\Theta}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] (C.3) \\ = \mathbb {E} _ {p} \left[ \left(- \frac {\partial}{\partial \boldsymbol {\Theta} _ {k}} \frac {\left(\mathbf {X} _ {i} - \boldsymbol {\Theta} ^ {T} \boldsymbol {\Theta}\right) ^ {2}}{2}\right) ^ {2} \right] (C.4) \\ = \mathbb {E} _ {p} \left[ 2 \left(\left(\mathbf {X} _ {i} - \boldsymbol {\Theta} ^ {T} \boldsymbol {\Theta}\right) \boldsymbol {\Theta} _ {k}\right) ^ {2} \right] (C.5) \\ = 4 \mathbb {E} _ {p (\boldsymbol {\Theta})} \left[ \boldsymbol {\Theta} _ {k} ^ {2} \mathbb {E} _ {p (\mathbf {X} _ {i} | \boldsymbol {\Theta})} \left[ \left(\mathbf {X} _ {i} - \boldsymbol {\Theta} ^ {T} \boldsymbol {\Theta}\right) ^ {2} \right] \right] (C.6) \\ = 4 \mathbb {E} _ {p (\boldsymbol {\Theta})} \left[ \boldsymbol {\Theta} _ {k} ^ {2} \right] \neq 0. (C.7) \\ \end{array}
$$
Property 2.5 The derivative of the log joint distribution  $\frac{\partial\log p(\mathbf{X},\Theta)}{\partial\Theta}$  is  $p$  -local.
Here, it's more useful to work with the equivalent (contrapositive) requirement for  $p$ -locality, i.e., we need to show  $\forall k, i$ :
$$
\mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = 0 \Rightarrow \frac {\partial^ {2} \log p (\mathbf {X} , \boldsymbol {\Theta})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = 0. \tag {C.8}
$$
Proof. First, we see that:
$$
\begin{array}{l} \mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = 0 (C.9) \\ \Rightarrow \frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}} = 0 (C.10) \\ \Rightarrow \frac {\partial^ {2} \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = 0, (C.11) \\ \end{array}
$$
where the first implication follows from the fact that the Fisher Information integral is effectively a weighted sum of elements, each of which is  $\geq 0$ . If the function on the right were nonzero for some  $\mathbf{Z}$ , then the Fisher Information would also be nonzero. Assuming that  $\log p$  has differentiable partial derivatives, we can interchange the order of differentiation, giving:
$$
\Rightarrow \frac {\partial^ {2} \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k} \partial \mathbf {Z} _ {i}} = 0 \tag {C.12}
$$
$$
\Rightarrow \frac {\partial}{\partial \boldsymbol {\Theta} _ {k}} \left[ \frac {\partial}{\partial \mathbf {Z} _ {i}} \left[ \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right) + \log p \left(\mathbf {Z} _ {\neq i}\right) \right] \right] = 0 \tag {C.13}
$$
$$
\Rightarrow \frac {\partial}{\partial \boldsymbol {\Theta} _ {k}} \left[ \frac {\partial}{\partial \mathbf {Z} _ {i}} [ \log p (\mathbf {Z}) ] \right] = 0 \tag {C.14}
$$
$$
\Rightarrow \frac {\partial^ {2} \log p (\mathbf {Z})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = 0, \tag {C.15}
$$
Property 2.6 For a probability distribution given by  $p(\mathbf{Z}) = \frac{1}{\mathcal{Z}} \exp(-E(\mathbf{Z}))$ , the expression  $\frac{\partial}{\partial \Theta} E(\mathbf{Z})$  is  $p$ -local.
Proof. The proof is almost identical to the proof for Property 2.5. From Property 2.5, we have that:
$$
\mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = 0 \tag {C.16}
$$
$$
\Rightarrow \frac {\partial^ {2} \log p (\mathbf {Z})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = 0. \tag {C.17}
$$
Using our definition of  $p$ , we have:
$$
\Rightarrow \frac {- \partial^ {2} (E (\mathbf {Z}) + \log \mathcal {Z})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = 0 \tag {C.18}
$$
$$
\Rightarrow \frac {\partial^ {2} E (\mathbf {Z})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = 0, \tag {C.19}
$$
Property 2.7 If the marginal parameter distribution factorizes as  $p(\Theta) = \prod_{k} p(\Theta_{k})$ , i.e. the parameters are independent from one another, then the score function  $\frac{\partial \log p(\mathbf{X}|\Theta)}{\partial \Theta}$  is  $p$ -local.
Proof. Again, we make heavy use of Property 2.5, which states:
$$
\mathbb {E} _ {p} \left[ \left(\frac {\partial \log p \left(\mathbf {Z} _ {i} \mid \mathbf {Z} _ {\neq i} , \boldsymbol {\Theta}\right)}{\partial \boldsymbol {\Theta} _ {k}}\right) ^ {2} \right] = 0 \Rightarrow \frac {\partial^ {2} \log p (\mathbf {Z})}{\partial \mathbf {Z} _ {i} \partial \boldsymbol {\Theta} _ {k}} = 0. \tag {C.20}
$$
It is important to note that the left-hand equation only holds true if  $\mathbf{Z}_i \neq \Theta_k$ : under  $p$ -locality, an update equation for parameter  $\Theta_k$  can always include its own value. So for the remainder of the proof we will assume that  $\mathbf{Z}_i \neq \Theta_k$ . Now,  $\log(p(\mathbf{X}|\boldsymbol{\Theta})) = \log p(\mathbf{Z}) - \log p(\boldsymbol{\Theta})$ . We also have: