SlowGuess/NeurIPS_2023_OCR huggingface dataset:
a
Figure 2: Different versions of  $\mathbf{Sp}$ -locality. a.  $Rp$ -locality functions as  $p$ -locality, but also allows synapses to use rewards  $R$  for parameter updates, corresponding to an additional diffuse neuromodulatory signal. b.  $\gamma p_{md}$ -locality allows synapses to use  $p_m$ -local variables and  $p_d$ -local variables, as well as the global gating variable  $\gamma$ . For the Wake-Sleep algorithm, a  $\gamma p_{md}$ -local probability distribution could correspond to a network of pyramidal neurons receiving  $p_m$ -related synapses in their apical compartments (orange) and  $p_d$ -related synapses in their basal compartments. The variable  $\gamma$  controls whether  $p_d$  or  $p_m$  synapses affect cellular dynamics. c. Backpropagation and its approximations are  $e_i p$ -local, which functions like  $p$ -locality, except it additionally permits a neuron-specific error signal to be used for parameter updates. Typically,  $e$  is constructed by sequentially propagating error signals backwards through a feedforward network.
postsynaptic neuron  $i$  and presynaptic neuron  $j$  in layer  $l$ . To make our choice of conditional probability distribution more concrete (see Figure 1a), we can see that neural firing rates for this model can be sampled by:
$$
\mathbf {r} ^ {(l)} = h \left(\mathbf {W} ^ {(l)} \mathbf {r} ^ {(l - 1)}\right) + \sigma \boldsymbol {\eta} ^ {(l)}, \tag {5}
$$
where  $\pmb{\eta}^{(l)}\sim \mathcal{N}(0,1)$ . This corresponds to an ordinary multilayer perceptron neural network with noise added at every layer. We can use this model to get intuition for how this choice of probability distribution  $p$  constrains the variables allowed for an update for a particular parameter  $\mathbf{W}_{ij}^{(l)}$ .
Now, our probability distribution corresponds to a directed graphical model with dependencies given by Eq. 3, so by Property 2.1, we know that variables that are not parents, coparents, or children of  $\mathbf{W}_{ij}^{(l)}$  violate  $p$ -locality if included in the parameter update for  $\mathbf{W}_{ij}^{(l)}$ . Figure 1 summarizes the variables allowed under  $p$ -locality for such a DAG.  $\mathbf{W}_{ij}^{(l)}$  has no parents under our graphical model, leaving only children and coparents.  $\mathbf{W}_{ij}^{(l)}$  has only one child,  $\mathbf{r}_i^{(l)}$ , whose coparents are given by  $\mathbf{W}_{ik}^{(l)}$  s.t.  $k \neq j$  (all other synapses onto  $\mathbf{r}_i^{(l)}$ ), and  $\mathbf{r}^{(l-1)}$  (all presynaptic neurons).
This shows that updates for synapses in our simple feedforward network cannot depend on neurons that are in layers  $>l$  or  $< l - 1$  while maintaining  $p$ -locality. We also found that dependencies on neurons within the postsynaptic layer  $l$  for indices  $k \neq i$  were also not permissible. As a concrete example, by Property 2.7, we found that the score function of  $p(\mathbf{X}|\Theta)$  is  $p$ -local if the parameters are assumed to be independently distributed. Our example satisfies this criterion. Its score function is given by:  $\frac{\partial \log p(\mathbf{r}|\mathbf{s},\mathbf{W})}{\partial \mathbf{W}_{ij}^{(l)}} = \frac{\left(\mathbf{r}_i^{(l)} - h(\mathbf{V}_i^{(l)})\right)}{\sigma^2} h'(\mathbf{V}_i^{(l)})\mathbf{r}_j^{(l - 1)}$ , where  $\mathbf{V}_i^{(l)} = \mathbf{W}_i^{(l)}\mathbf{r}^{(l - 1)}$ . By inspection, this function only requires information about the post- and presynaptic neurons, as well as the summed input to the postsynaptic neuron,  $\mathbf{V}_i$ .
This simple example clearly outlines why  $p$ -locality aligns with expected notions of locality. More importantly, the use of  $p$ -locality also has the benefit of generalizing cleanly to alternative network architectures. For example, if we were to instead inspect a multilayer recurrent neural network (Figure 1b), we would see that for  $\mathbf{W}_{ij}^{(l)}$  the firing rates  $\mathbf{r}_i^{(l)}(t)$  and all presynaptic neural firing rates would be permissible for all time steps, because the activities at each timestep are children or coparents of  $\mathbf{W}_{ij}^{(l)}$ . For another classical example of a biologically plausible probability distribution corresponding to an UG (as in Figure 1c.), see Appendix D for a discussion of a linear continuous Boltzmann machine.
2.5 Sp-locality
In almost all situations, a given class of plasticity algorithms will produce updates that are almost  $p$ -local, in that synapses are required to have access to some small number of global state variables that a biological system could in principle project diffusely throughout a network, but which would not be allowed under generic  $p$ -locality (e.g. reward, which is usually produced far downstream of any given synapse in a neural network). For this eventuality, we introduce a notion called  $\mathbf{S}p$ -locality, which is best considered a hybrid of  $\mathbf{S}$ -locality and  $p$ -locality.
Definition 2.3. Given parameters  $\Theta \in \mathbb{R}^{N_{\Theta}}$  and random variables  $\mathbf{X} \in \mathbb{R}^{N_{\mathbf{X}}}$ , concatenated as  $\mathbf{Z} = [\mathbf{X}, \Theta]^\top$ , and a function  $f(\mathbf{Z}): \mathbb{R}^{N_{\mathbf{X}} + N_{\Theta}} \to \mathbb{R}^{N_{\Theta}}$ , the update function  $f(\mathbf{Z})$  is  $\mathbf{Sp}$ -local with respect to probability distribution  $p(\mathbf{Z})$  and some set  $\mathbf{S} = \{\mathbf{S}_k : \mathbf{S}_k \subseteq \mathbf{Z}\}$  if and only if  $\forall k f_k = h_k(\mathbf{S}_k, g_k(\mathbf{Z}))$ , where  $h_k$  is an arbitrary function and  $g_k(\mathbf{Z})$  is  $p$ -local.
The value of this definition is that it negotiates a compromise between the architecture-generality of  $p$ -locality, and the flexibility of S-locality. Obviously if we take  $\mathbf{S}_k = \mathbf{Z} \forall k$ , all functions are S-local, and likewise, if we define  $p$  sufficiently generally, all functions are  $p$ -local (Property 2.4). However, if some class of optimization algorithms operating on a predefined probability distribution  $p$  are provably guaranteed to produce parameter updates that are  $\mathbf{S}p$ -local for some small set  $\mathbf{S}$ , then we will have obtained a very concise description of the types of information required by that algorithm.
3 Applying  $\mathbf{S}p$ -locality to normative plasticity models
Given a probability distribution over a neural network states and parameters,  $p(\mathbf{X}, \Theta)$ , which could be experimentally motivated or even observed, it is natural to ask which learning algorithms respect locality principles in the way parameter updates are made during learning. Alternatively, if we denote an algorithm which operates on a probability distribution  $p(\mathbf{X}, \Theta)$  to output an update  $f(\mathbf{Z})$  by  $\mathcal{A}(p(\mathbf{X}, \Theta)) = f(\mathbf{Z})$ , where  $f(\mathbf{Z}): \mathbb{R}^{N_{\mathbf{X}} + N_{\Theta}} \to \mathbb{R}^{N_{\Theta}}$ , we can ask which probability distributions  $p(\mathbf{X}, \Theta)$  and variable collections  $\mathbf{S}$  leave  $f(\mathbf{Z})$ $\mathbf{S}p$ -local. This is one way  $\mathbf{S}p$ -locality could potentially facilitate experimentally testing biologically plausible learning models.
To demonstrate the utility of our approach, in this section we characterize the  $\mathbf{S}p$ -locality properties of a wide variety of algorithms that have historically been used to produce normative models of synaptic plasticity, as summarized in Table 1. Though our framework covers many algorithms, here we expand on three prototypical algorithms in detail, focusing on features that are relevant for our discussion of the algorithms' locality properties. More algorithmic details as well as theorem proofs are provided in Appendix E. Schematics depicting biological interpretations for the types of  $\mathbf{S}p$ -locality discussed below are provided in Figure 2.
3.1 REINFORCE
REINFORCE [13], also known as policy gradient learning, produces reward-modulated Hebbian parameter updates for neural networks [1] similar to the one discussed in Section 2.4. Here we will show that REINFORCE is  $Rp$ -local, meaning that it is  $\mathbf{S}p$ -local where  $\mathbf{S} = \{\mathbf{S}_k = R \forall k\}$ , which assumes each synapse has access to a global scalar reward signal  $R$  that needs to be delivered diffusely to all synapses in a network.
Theorem 3.1. If  $p(\Theta) = \prod_k p(\Theta_k)$ , the REINFORCE estimator given by  $\mathcal{A}_R(p(R, \mathbf{X}|\Theta))$  is Rp-local.
As an example, consider the network defined by Eq. 3. The REINFORCE update for this network, for a single sample of the network state  $\mathbf{r}$  and reward  $R$ , is given by:
$$
\Delta \mathbf {W} _ {i j} ^ {(l)} = R \frac {\partial \log p (\mathbf {r} | \mathbf {s} , \mathbf {W})}{\partial \mathbf {W} _ {i j} ^ {(l)}} = R \frac {\left(\mathbf {r} _ {i} ^ {(l)} - h \left(\mathbf {V} _ {i} ^ {(l)}\right)\right)}{\sigma^ {2}} h ^ {\prime} \left(\mathbf {V} _ {i} ^ {(l)}\right) \mathbf {r} _ {j} ^ {(l - 1)}, \tag {6}
$$
where again  $\mathbf{V}_i^{(l)} = \mathbf{W}_{i}^{(l)}\mathbf{r}^{(l - 1)}$ . This is just the scalar multiplication of the reward signal  $R$  with the score function; because the score function is  $p$ -local, the full update is  $Rp$ -local (see Appendix E for more detail).
3.2 Wake-Sleep (WS)
Whereas REINFORCE has been used to model reinforcement learning in neural networks, the Wake-Sleep algorithm has been used to model unsupervised learning [22]. It assumes that a neural network has two modes of operation, controlled by a scalar variable  $\gamma$  that determines whether the network is in inference mode ('wake',  $\gamma = 1$ ) or in generative mode ('sleep,'  $\gamma = 0$ ). In all, neural activity samples are drawn from the mixture distribution given by  $p_{md}(\mathbf{r},\gamma |\mathbf{W},\mathbf{M}) = p_d(\mathbf{r}|\mathbf{W})^\gamma p_m(\mathbf{r}|\mathbf{M})^{(1 - \gamma)}p(\gamma)$ , where  $p(\gamma)$  defines the probability of sampling from the wake phase ( $p_d$ ;  $d$  corresponds to 'data') or the sleep phase ( $p_m$ ;  $m$  corresponds to 'model'), and  $\mathbf{W}$  corresponds to the feedforward weights while  $\mathbf{M}$  correspond to feedback weights. For conciseness, in all subsequent sections we will denote the mixture distribution  $p_1(\mathbf{Z})^\gamma p_2(\mathbf{Z})^{1 - \gamma}p(\gamma)$  by  $Mix(p_1(\mathbf{Z}),p_2(\mathbf{Z}))$ . As an example, building on the feedforward network defined by Eq. 3 by adding a generative feedback pathway, we have neural dynamics given by:
$$
\mathbf {r} ^ {(l)} = \gamma h _ {w} \left(\mathbf {W} ^ {(l)} \mathbf {r} ^ {(l - 1)}\right) + (1 - \gamma) h _ {s} \left(\mathbf {M} ^ {(l)} \mathbf {r} ^ {(l + 1)}\right) + \sigma \boldsymbol {\eta} ^ {(l)}, \tag {7}
$$
where  $h_w(\cdot)$  corresponds to the 'wake' nonlinearity, and  $h_s(\cdot)$  corresponds to the 'sleep' nonlinearity. Under this formulation,  $\gamma$  gates, for all neurons, whether activity is driven by the feedforward or feedback pathways; there are several hypotheses for how this could be implemented at a neuronal level, including global neuromodulatory or inhibitory gating of the apical and basal dendrites of pyramidal neurons in the cortex [23] (Fig 2b).
The Wake-Sleep algorithm updates  $\mathbf{M}$  to fit  $p_m$  as a generative model of the network's input data, and updates  $\mathbf{W}$  to fit  $p_d$  to perform approximate inference with respect to that generative model, in a manner similar to variational autoencoders (VAE) [24, 25]; unlike VAE training with backpropagation, however, the Wake-Sleep algorithm is  $\gamma p_{md}$ -local, meaning  $\mathbf{S} = \{S_k = \gamma \forall k\}$  (see Appendix E for the proof).
Theorem 3.2. If  $p(\Theta, \Theta^{(d)}) = \left( \prod_{k} p(\Theta_{k}) \right) \left( \prod_{k} p(\Theta_{k}^{(d)}) \right)$ , the Wake-Sleep estimator given by  $\mathcal{A}_{WS}(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}|\Theta^{(d)}))$  is  $\gamma p_{md}$ -local, where  $p_{md} = Mix(p_m(\mathbf{X}|\Theta), p_d(\mathbf{X}|\Theta^{(d)})) p(\Theta, \Theta^{(d)})$ .
As a concrete example, the Wake-Sleep parameter update for  $\mathbf{W}_{ij}^{(l)}$  for a single sample of the state  $\mathbf{r}$  from the network above is given by:
$$
\Delta \mathbf {W} _ {i j} ^ {(l)} = (1 - \gamma) \frac {\partial \log p _ {d} (\mathbf {r} | \mathbf {s} , \mathbf {W})}{\partial \mathbf {W} _ {i j} ^ {(l)}} = (1 - \gamma) \frac {\left(\mathbf {r} _ {i} ^ {(l)} - h _ {w} (\mathbf {V} _ {i} ^ {(l)})\right)}{\sigma^ {2}} h _ {w} ^ {\prime} (\mathbf {V} _ {i} ^ {(l)}) \mathbf {r} _ {j} ^ {(l - 1)}, \tag {8}
$$
where  $\mathbf{V}_i^{(l)} = \mathbf{W}_{i:}^{(l)}\mathbf{r}^{(l - 1)}$ ; a similar update holds for M. Similar to REINFORCE, this parameter update is only the combination of a scalar variable  $\gamma$  and the score function, and is consequently  $\gamma p_{md}$ -local (see Appendix E for the proof). Interestingly, Wake-Sleep is not the only algorithm to obey this form of locality: several other normative plasticity models, including Boltzmann machine learning [26], equilibrium propagation [27], and impression learning [23] have essentially the same  $p$ -locality properties. However, some of these have additional requirements on the probability distribution that make them less biologically plausible. For example, Boltzmann machine learning is  $\gamma p_{md}$ -local only for distributions that can be captured with energy-based models (which typically require weight symmetry).
3.3 Backpropagation (BP) and its approximations
Here we provide a characterization of the locality of the backpropagation algorithm [28], which as we will show using our definitions, is only a local algorithm with biologically implausible assumptions [20]. This demonstration is primarily important because the  $\mathbf{Sp}$ -locality properties of several biologically plausible backpropagation approximations, including feedback alignment [29], weight mirror [30], and Burstprop [31] satisfy a similar notion of  $\mathbf{Sp}$ -locality, but using more biologically realistic assumptions about  $\mathbf{S}$  (see Table 1 and Appendix E). Derivations for the backpropagation algorithm and its approximations require more stringent network assumptions (here we assume a feedforward multilayer perceptron network as in Section 2.4), because the algorithm itself is not well-defined for arbitrary probabilistic network architectures.
Optimization Alg.	Locality supported	Architectural restrictions
REINFORCE	Rp-local	DAG
Maximum Likelihood Estimation	pm-local	DAG
Generalized EM	pm-local	DAG
Predictive Coding	pm-local	MAP (weight symmetry)
Wake-Sleep	γpmd-local	DAG
Impression learning	γpmd-local	DAG
Contrastive Divergence	γpmd-local	K-Step EBM (weight symmetry)
Equilibrium Propagation	γpmd-local	EBM (weight symmetry)
WTA-STDP	pm-local	WTA
Backpropagation	ei(l)p-local	MLP (weight transport)
Feedback alignment	ˆi(l)p-local	MLP
Weight mirror	ˆi(l)p-local	MLP
Burstprop	ˆi(l)p-local	MLP
RTRL	eJp-local	RNN
e-prop	ei,p-local	RNN
RFLO	ˆi,p-local	RNN
FOLLOW	p-local	error-RNN
Table 1: Summarizing  $\mathbf{S}p$ -locality for classical learning algorithms. First column: the optimization algorithm of interest. Second column: the variant of  $\mathbf{S}p$ -locality that we have proven in Appendix E. Third column: the network architectures required for both the algorithm and our proofs to work. DAG = Directed Acyclic graphical model, MAP = Maximum a posteriori gradient descent dynamics, WTA = winner-take-all circuit, EBM = Energy-based model, MLP = Multilayer perceptron, RNN = recurrent neural network, error-RNN = error-driven recurrent neural network. 'K-Step' refers to the small, finite number of sampling steps required, as opposed to sampling from a steady-state distribution or calculating an equilibrium, which are more time-intensive. Weight symmetry indicates that symmetric recurrent connectivity is required for the original proposed algorithm. Weight transport indicates that symmetric weights are required exclusively for error propagation.
Theorem 3.3. If  $p(\Theta) = \prod_{k} p(\Theta_{k})$  and  $p(\mathbf{X}|\Theta)$  is defined by Eq. 3, the BP update for  $\mathbf{W}_{ij}^{(l)}$  with a loss  $\mathcal{L}(\mathbf{X})$ , given by  $\mathcal{A}_{BP}(p(\mathbf{X}|\Theta),\mathcal{L}(\mathbf{X}))$  is  $\mathbf{e}_i^{(l)}$ $p$ -local, where  $\mathbf{e}_i^{(l)} = \frac{\mathrm{d}L}{\mathrm{d}\bar{\mathbf{r}}_i}$ . Similarly, the updates for feedback alignment, weight mirror, and Burstprop are  $\hat{\mathbf{e}}_i^{(l)}$ $p$ -local, where  $\hat{\mathbf{e}}_i^{(l)}$  is given by their respective gradient approximations.
This notion of  $\hat{\mathbf{e}}_i^{(l)}p$ -locality, where  $\mathbf{S} = \{\mathbf{S}_{ij}^{(l)} = \hat{\mathbf{e}}_i^{(l)}\forall i,j,l\}$ , includes more restrictive biological assumptions than the one provided by  $Rp$ -locality and  $\gamma p_{md}$ -locality, because it relies heavily on the neuron-specific error  $\hat{\mathbf{e}}_i^{(l)}$ , which must be then accounted for using physiological mechanisms. For our network architecture given by Eq. 3, the functional form of all backpropagation approximations listed in Table 1 is given by:
$$
\Delta \mathbf {W} _ {i j} ^ {(l)} = - \hat {\mathbf {e}} _ {i} h ^ {\prime} \left(\mathbf {V} _ {i} ^ {(l)}\right) \mathbf {r} _ {j} ^ {(l - 1)}, \tag {9}
$$
where  $\hat{\mathbf{e}}_i$  is an algorithm-specific approximation of  $\frac{\mathrm{d}L}{\mathrm{d}\bar{\mathbf{r}}_i}$ . Similar to  $\gamma p_{md}$ -local algorithms, recent work has suggested that this error signal could be propagated through networks by a top-down error signal relayed to the apical dendrites of pyramidal neurons [31] (see Fig. 2c). Thus, these algorithms make specific biological predictions, driven by the need for  $\hat{\mathbf{e}}_i^{(l)}$  to be accessible to each synapse.
We have constructed a formal definition of locality,  $\mathbf{S}p$ -locality, that combines set-based and probabilistic graphical model-based formalisms (S-locality and  $p$ -locality, respectively). Our goal was standardizing and operationalizing notions of locality, and we also demonstrated the usefulness of these definitions in characterizing normative plasticity models. Notably, this approach enabled the identification of distinct classes of  $\mathbf{S}p$ -locality (Table 1) which subsume learning algorithms spanning
more than thirty years of research, from Boltzmann machines [26] to modern incarnations like equilibrium propagation [27]. Importantly, our framework can be applied to any neural network architecture, incorporating networks that are recurrent [16, 32, 23, 27, 26, 33], spiking [34, 12], and multilayer [29, 22, 30] into a single framework. This is thanks, in large part, to the architecture-generality of  $p$ -locality, which abstracts away specific details of network models, defining locality instead in terms of statistical dependencies—this makes our framework much more powerful than simpler linguistic descriptions of locality. As we discuss in Appendix F, this abstraction comes at the cost of requiring researchers to specify a distribution,  $p$ , that is grounded in biological constraints. We note that for non-biological probability distributions the variables allowed to be "local" under  $p$ -locality can violate standard intuitions; however, for standard neural networks like those discussed in Section 2.4 and Appendix D,  $p$ -locality behaves as one would expect, with "local" learning rules being those that use pre- and postsynaptic activity.
Table 1 organizes existing normative plasticity models into distinct classes. The first locality class,  $Rp$ -locality, requires individual synapses to have access to reward information; this class encompasses many reward-modulated Hebbian plasticity models [1]. The second class,  $p_{(m)}$ -locality, encompasses predictive coding and its variations [33, 35], and aligns with more traditionally Hebbian plasticity updates (often at the cost of unrealistic network architectures; see Table 1). The third class,  $\gamma p_{md}$ -locality, requires neural networks to have two distinct modes of operation (e.g. the 'wake' and 'sleep' modes in the Wake-Sleep algorithm) gated by a scalar global variable  $\gamma$ . Thus,  $\gamma p_{md}$ -locality requires synapses to have access to the value of  $\gamma$  to compute updates, and subsumes many algorithms that require generative feedback pathways for learning [22, 23]. The final class of algorithms are  $\hat{\mathbf{e}}_i^{(l)}p$ -local for feedforward neural networks, where  $\hat{\mathbf{e}}_i^{(l)}$  is a neuron-specific approximate error signal, and includes feedback alignment [29], weight mirror [30], and Burstprop [31], as well as temporal variants of these algorithms [36, 37]. These four categories— $Rp$ -locality,  $p_{(m)}$ -locality,  $\gamma p_{md}$ -locality, and  $\hat{\mathbf{e}}_i^{(l)}$ -locality—subsume, to our knowledge, almost all existing normative plasticity models.
Because these plasticity models are sometimes only constructed for specific network architectures (which may violate known facts about the brain), we also specify the broadest class of network architectures for which we have successfully derived a model's  $\mathbf{S}p$ -locality properties (proven in Appendix E). As noted in Appendix F, 'biological plausibility' requires both network architectures and parameter updates to be in line with existing experimental evidence, while  $\mathbf{S}p$ -locality provides a formal framework for assessing only the latter of these two factors. We therefore stress that  $\mathbf{S}p$ -locality properties are only relevant for neuroscience if they can be determined for biologically plausible network architectures. As a consequence, classes of network architectures that encompass a broad range of biologically plausible network models (such as DAGs) are desirable, whereas algorithms that are only viable for implausible architectures (e.g. those that imply weight symmetry or weight transport) are undesirable.
Interestingly, the distinct classes of locality also delineate distinct sets of experimentally testable predictions for plasticity. Critically,  $\mathbf{S}p$ -locality abstracts away details that are not important for testing predictions, and helps identify important features of learning algorithms. For example, the REINFORCE (Eq. 6) and Wake-Sleep (Eq. 8) algorithms are equivalent with respect to the  $p$  in  $\mathbf{S}p$ -locality, which tells us that the set of allowed local variables under  $p$ -locality itself (e.g. pre- and post-synaptic information) are not the best targets for experimental testing between these algorithms, and instead the focus must be on the variables included in 'S' (a reward signal  $R$  versus a clamping variable  $\gamma$  that switches the network between different modes of synaptic plasticity and neural activity). Thus, by identifying whether plasticity is modulated by reward, neuron-specific error teaching signals, transitions in network dynamics, or none of the above, one may narrow the space of possible candidate plasticity models down to  $Rp-$ ,  $\hat{\mathbf{e}}_i^{(l)}$ ,  $\gamma p_{md}$ , or  $p_{(m)}$ -locality, respectively. These distinct phenomena are necessary features of the learning algorithms in question for any neural network architecture used and are consequently good targets for experimental testing. In contrast, other features of parameter updates are more about specific architectural choices, e.g. switching from a rate-based network [16] to spiking [15], from feedforward [22] to recurrent [32], or single-to multi-compartment [38]. Thus, beyond clarifying and categorizing the locality properties of normative plasticity models, our hope is that our framework will also support experimental efforts to differentiate between such models, helping the field to focus in on the most important predictions to test experimentally.