SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Julian Rossbroich<sup>1,2</sup>
Friedemann Zenke<sup>1,2</sup>
{firstname这个名字}  $\text{三}$  fmi.ch
<sup>1</sup>Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland
$^{2}$  Faculty of Science, University of Basel, Basel, Switzerland
How neuronal circuits achieve credit assignment remains a central unsolved question in systems neuroscience. Various studies have suggested plausible solutions for back-propagating error signals through multi-layer networks. These purely functionally motivated models assume distinct neuronal compartments to represent local error signals that determine the sign of synaptic plasticity. However, this explicit error modulation is inconsistent with phenomenological plasticity models in which the sign depends primarily on postsynaptic activity. Here we show how a plausible microcircuit model and Hebbian learning rule derived within an adaptive control theory framework can resolve this discrepancy. Assuming errors are encoded in top-down dis-inhibitory synaptic afferents, we show that error-modulated learning emerges naturally at the circuit level when recurrent inhibition explicitly influences Hebbian plasticity. The same learning rule accounts for experimentally observed plasticity in the absence of inhibition and performs comparably to back-propagation of error (BP) on several non-linearly separable benchmarks. Our findings bridge the gap between functional and experimentally observed plasticity rules and make concrete predictions on inhibitory modulation of excitatory plasticity.
How do neurons far away from the sensory periphery and motor output system update their connections to contribute to network computation meaningfully? This question, formally known as the "credit assignment problem," is one of the outstanding questions in systems neuroscience. Classic learning theories assume synaptic plasticity in the brain is mainly Hebbian, i.e., changes in a synapse's efficacy depend merely on the pre- and postsynaptic activity of the neurons it connects [1]. A plethora of experiments in different brain areas support the notion of Hebbian plasticity [2, 3] and diverse phenomenological plasticity models quantitatively capture observed synaptic plasticity dynamics [4-8]. Theories of reinforcement learning suggested further extensions of Hebbian learning to three-factor rules that account for reward modulation, whereby a global modulatory factor, e.g., dopamine, explicitly influences the sign of plasticity [9, 10]. Again, there is ample experimental evidence for the role of neuromodulators in gating plasticity in various brain areas [11, 12]. Yet, recent work argues that global neuromodulation is insufficient to learn complex function mappings in large networks [13, 14] and that more fine-grained control over the sign of plasticity is required, as in the BP algorithm used in deep learning [15-18]. This realization motivated several modeling studies on how biological networks could approximate BP [19-25]. One central assumption in virtually all of these models is that the sign of plasticity is explicitly modulated by a neuron-specific local error signal akin to the gradient-based update used in BP [26]. However, learning rules with explicit error
Figure 1: Explicit error modulation of the sign of plasticity is inconsistent with phenomenological plasticity models. (a) In neuronal circuits, top-down feedback connections target excitatory neurons, as well as inhibitory and dis-inhibitory circuits that have been implicated in gating of plasticity. In phenomenological plasticity models, the sign of plasticity is typically determined by postsynaptic quantities such as the membrane voltage [7], firing rate [6], or calcium concentration [8] without explicit error modulation. (b) In normative models, the sign of plasticity is typically subject to a hypothetical, explicit error modulation with little experimental evidence. Explicit error modulation makes specific predictions of the shape of the learning rule, (Supplementary Fig. S1; see Appendix A), at odds with experimentally observed plasticity (see Panel (a)). (b) Theories of bio-plausible gradient-based learning typically focus on approximating direct error-modulation as in BP, but differ in how errors are computed and relayed. Existing models suggest separate temporal phases, e.g. equilibrium propagation (EP) [29], putative compartments to represent error signals locally, e.g., dendritic error coding [22, 30], or burst-multiplexing [23, 31]. Still, there is little evidence for an explicit error signal that alters the sign of plasticity [26] and is consistent with phenomenological plasticity models.
modulation are inconsistent with phenomenological models of Hebbian plasticity (Fig. 1), raising the question of how theories of bio-plausible credit assignment tie into the phenomenology of Hebbian learning.
Here, we address this question using a normative control theory approach. We set out from a plausible dis-inhibitory circuit motif ubiquitously found in the brain [27, 28] and derive a Hebbian learning rule with an explicit inhibitory current dependence. We demonstrate that this learning rule accounts for key experimental observations and allows for control over the sign of plasticity through top-down synaptic input to specific interneurons. Our work suggests that error signals could naturally be encoded in top-down inputs to inhibitory neurons and bridges the gap between normative theories of gradient-based learning and phenomenological models of Hebbian plasticity. Our main contributions are:
- We extend Deep Feedback Control (DFC), a recent adaptive control theory framework for error-based learning [24], to a biologically plausible dis-inhibitory microcircuit motif.
- We show how a Hebbian plasticity rule with an explicit inhibition dependence can naturally decode credit signals from this microcircuit.
- We demonstrate that this rule enables error-based learning in hierarchical networks, naturally stabilizes runaway Hebbian plasticity, and resembles phenomenological plasticity rules under simulated experimental conditions.
- Finally, we demonstrate that our learning rule performs comparable to BP in deep neural networks trained on computer vision benchmarks.
2 Background and previous work
In this article we strive to reconcile phenomenological plasticity models constrained by experiments and models of biologically plausible credit assignment based on normative theories of gradient-based learning. In the following we review essential literature of both approaches.
2.1 Phenomenological synaptic plasticity models
There is widespread experimental support for the notion of Hebbian synaptic plasticity in the brain, captured in the form of classical long-term potentiation (LTP) and long-term depression (LTD) [2] or spike-timing-dependent plasticity (STDP) [3]. An extensive mathematical model catalog captures the phenomenology of these findings [4-8]. Common to most of the above models is that postsynaptic quantities define a plasticity threshold which separates LTD from LTP induction, which effectively determines the sign of the synaptic weight change [32-34] (Fig. 1a). A plethora of phenomenological plasticity models exist that capture such dependence on firing rate [6], voltage [7], and postsynaptic calcium concentration [5, 8]. For isolated neurons, classic work has demonstrated that Hebbian plasticity can extract principal components from structured data [35] or capture receptive field formation observed experimentally [36]. However, these models do not extend to deep hierarchical networks, nor can they account for the modulation of plasticity through local error signals required for solving the credit assignment problem. Thus, the mechanisms by which synaptic plasticity observed under experimental conditions could give rise to coordinated learning at the circuit- and network level remain elusive.
2.2 Models of biologically plausible credit assignment
The above models are contrasted by normative rules derived from gradient-based learning principles, which often aim at approximating BP [15] (Fig. 1b-c). A crucial aspect of BP is the separation of forward- and backward signaling, which algorithmically separates credit signaling from neuronal activity [37]. This separation presents a significant challenge for biologically plausible implementations, as it presumes a distinct separation through either learning phases or separate pathways.
Equilibrium propagation (EP) offers one possible, if only partial, solution to this dilemma. EP posits that local errors are derived as variations in neuronal activity at a network equilibrium state [29]. Yet, classic EP still requires separate processing phases for each input, inconsistent with neurobiology. However, recent work suggests possible ways of alleviating the requirement for distinct phases through neural oscillations [38].
An alternative to separate phases for forward and backward passes is to separate them spatially. Predictive coding models [19, 25, 39] exemplify this idea, whereby errors are computed in dedicated neuron-specific error units by comparing each neuron's activity to a top-down prediction. Recent work has suggested that the electrotonically segregated apical dendrites of cortical pyramidal neurons [37, 40] could represent local errors in learning [21, 22].
However, to approximate BP, a common theme across these models is their dependence on explicit error-modulation of plasticity [30] (Fig. 1b-c). While error-modulated learning rules prove functionally useful and, in some cases, can match the performance of BP, experimental evidence for their existence is inconclusive. In particular, they fall short of capturing established properties of experimentally observed plasticity, such as a threshold between LTD and LTP that is governed by postsynaptic activity [32-34, 41].
Finally, Payeur et al. [23] proposed a unique multiplexing approach by encoding forward and feedback signals in the event and burst rate of output spike trains. Such burst-dependent plasticity rules capture essential facets of phenomenological plasticity [31] (Supplementary Fig. S1; see Appendix A). However, the model primarily applies to cortical Layer 5 pyramidal cells with electrically segregated dendritic trees. In contrast, it may not work for layer 2/3 neurons or other brain areas without segregated dendrites.
Since most of the above models focused on approximating BP, they face another potential issue: they usually require weak feedback, which causes only a slight perturbation, or nudge, of the input-driven equilibrium. This requirement contrasts significantly with a wealth of experimental literature suggesting that feedback connections in the brain substantially influence neuronal activity [42-44]. Rather than striving to find biologically plausible separations between forward and backward signaling, recent modeling studies used an approach grounded in adaptive control theory [25, 45-48]. These models leverage strong feedback signals to steer neuronal activity to align with a given target output. While strong feedback aligns more closely with neurobiological observations, these models are also dependent on explicitly error-modulated learning rules, wherein the error is computed from either the difference between the controlled and uncontrolled states of neuronal activity similar to EP or the difference between activity in segregated neuronal compartments, as in dendritic error coding.
Still, it remains to be seen how such learning could be implemented at the circuit level with plasticity rules that capture experimental findings. In this article, we propose a putative circuit-level solution to this issue by mapping the notion of feedback control onto a known dis-inhibitory circuit motif and combining it with an inhibition-modulated Hebbian plasticity rule.
3 Model
To study whether biological microcircuits could naturally interpret dis-inhibitory feedback signals as local errors to control the sign of synaptic plasticity, we consider a continuous time dynamic multi-layer network comprised of excitatory and inhibitory neurons in each layer with dis-inhibitory feedback connections.
3.1 Neuronal dynamics
The membrane potential dynamics of the excitatory and inhibitory neurons in layer  $i$  are described by the following ordinary differential equations (ODEs):
$$
\tau_ {\mathrm {E}} \frac {d}{d t} \mathbf {u} _ {i} ^ {\mathrm {E}} = - \mathbf {u} _ {i} ^ {\mathrm {E}} (t) + \mathbf {W} _ {i} \mathbf {r} _ {i - 1} ^ {\mathrm {E}} (t) - \mathbf {r} _ {i} ^ {\mathrm {I}} (t) \tag {1}
$$
$$
\tau_ {\mathrm {I}} \frac {d}{d t} \mathbf {u} _ {i} ^ {\mathrm {I}} = - \mathbf {u} _ {i} ^ {\mathrm {I}} (t) + \mathbf {r} _ {i} ^ {\mathrm {E}} (t) - \mathbf {Q} _ {i} \mathbf {c} (t) \tag {2}
$$
with  $\mathbf{W}_i$  the afferent synaptic weights from the previous layer and  $\mathbf{r}_i = \phi (\mathbf{u}_i)$  a smooth, monotonically increasing nonlinear activation function. Note that here we made the simplifying assumption that each excitatory neuron has an associated inhibitory neuron and that both are connected locally within a microcircuit. For all simulations, we use the soft rectifying nonlinearity  $\phi (u) = \beta \log (1 + \exp (u - \gamma))$ , in which  $\beta$  and  $\gamma$  are parameters controlling the scale and shift of the activation function, respectively. Dis-inhibitory feedback is mediated through top-down control signals  $\mathbf{c}(t)$  relayed to each layer  $i$  through an associated feedback weight matrix  $\mathbf{Q}_i$  (Fig. 2). The input layer  $\mathbf{r}_0(t)$  is data-dependent and not influenced by top-down feedback. For a constant input current  $\mathbf{r}_0$ , the network dynamics settle to the equilibrium state
$$
\mathbf {\ddot {u}} _ {i} ^ {* \mathrm {E}} = \mathbf {W} _ {i} \mathbf {\dot {r}} _ {i - 1} ^ {* \mathrm {E}} - \mathbf {\dot {r}} _ {i} ^ {* \mathrm {I}}, \quad \mathbf {\dot {u}} _ {i} ^ {* \mathrm {I}} = \mathbf {\dot {r}} _ {i} ^ {* \mathrm {E}} - \mathbf {Q} _ {i} \mathbf {\dot {c}}. \tag {3}
$$
Figure 2: Illustration of a multi-layer network with dis-inhibitory control microcircuits (left). Each network unit consists of an excitatory and inhibitory neuron that are recurrently connected (right). The top-down control signal to each layer  $\mathbf{Q}_i\mathbf{c}(t)$  is relayed through dis-inhibitory afferents (orange).
3.2 Feedback control
As in previous work on DFC [24, 47], our model uses feedback control to drive the output activity of the network towards the target activity  $\mathbf{r}_L^{\mathrm{tar}}$  by minimizing the magnitude of the output error:
$$
\mathbf {e} (t) = - \left. \frac {\partial \mathcal {L} \left(\mathbf {r} _ {L} , \mathbf {r} _ {L} ^ {\text {t a r}}\right)}{\partial \mathbf {r} _ {L}} \right| _ {\mathbf {r} _ {L} = \mathbf {r} _ {L} (t)} ^ {T} \tag {4}
$$
where  $\mathcal{L}(\mathbf{r}_L,\mathbf{r}_L^{\mathrm{tar}})$  is a label-dependent supervised loss function defined on the network's output activity. For a simple mean squared error (MSE) loss,  $\mathcal{L} = 1 / n\sum_{n}1 / 2\| \mathbf{r}_{L}^{\mathrm{tar}}(n) - \mathbf{r}_{L}(n)\|_{2}^{2}$ , the error for each datapoint  $n$  is directly related to the difference between the network's output and the target, specifically  $\mathbf{e}(t) = \mathbf{r}_L^{\mathrm{tar}} - \mathbf{r}_L(t)$ .
Leaky proportional-integral controller. In our model, we use a leaky proportional-integral controller to compute the feedback control signals  $\mathbf{c}(t)$ :
$$
\mathbf {c} (t) = k _ {\mathrm {p}} \mathbf {e} (t) + k _ {\mathrm {i}} \mathbf {c} ^ {\text {i n t}}, \quad \tau_ {c} \frac {d}{d t} \mathbf {c} ^ {\text {i n t}} = \mathbf {e} (t) - \mathbf {c} ^ {\text {i n t}} (t) \tag {5}
$$
where  $k_{\mathrm{p}}$  and  $k_{\mathrm{i}}$  are the proportional and integral control constants, respectively.
Dis-inhibitory feedback connectivity. Next, we have to connect the controller to the controlled quantities. In the context of our model, we assume that control signals are relayed via inhibitory interneurons. We thus have to specify the feedback weights connecting the control signals to the local microcircuits defined in the previous section. While we make no claims about how suitable control feedback is generated in neurobiology, in this article we merely assume that suitable control signals exist and that they are mediated via inhibitory interneurons. To that end, the feedback weights  $\mathbf{Q}_i$  need to be chosen such that the output loss  $\mathcal{L}(\mathbf{r}_L,\mathbf{r}_L^{\mathrm{tar}})$  is minimized at the controlled equilibrium state. To fulfill this requirement, the column space of the concatenated feedback weights of the network,  $Q\triangleq \left[\mathbf{Q}_1^T,\dots ,\mathbf{Q}_L^T\right]^T$ , must be equal to the row space of the network Jacobian  $J$  at steady-state [24]. This Jacobian characterizes how infinitesimal perturbations of each controlled quantity, e.g., a neuronal activation, relates to changes in the network's output  $\mathbf{r}_L$ . In contrast to previous models relying on top-down control, our network consists of recurrently connected excitatory and inhibitory units whilst top-down input is targeting the inhibitory population exclusively. Since we want to model control through dis-inhibitory circuits, we assume that the Jacobian for the controller is defined with respect to the inhibitory membrane potential at each layer:
$$
J \triangleq [ \mathbf {J} _ {1}, \dots , \mathbf {J} _ {L} ] = \left[ \frac {\partial \mathbf {r} _ {L}}{\partial \mathbf {u} _ {1} ^ {\mathrm {I}}}, \dots , \frac {\partial \mathbf {r} _ {L}}{\partial \mathbf {u} _ {L} ^ {\mathrm {I}}} \right] \tag {6}
$$
where we use  $\frac{\partial\mathbf{r}_L}{\partial\mathbf{u}_i^{\mathrm{I}}} = \nabla_{\mathbf{u}_i^{\mathrm{I}}}\mathbf{r}_L$  to denote the Jacobian matrix of partial derivatives of the vector  $\mathbf{r}_L$  with respect to the vector  $\mathbf{u}_i^{\mathrm{I}}$ . It has been shown that a wide range of possible feedback weights  $Q$  can match the row space of  $J$  in the DFC framework. One simple way of ensuring this condition is met is to set the feedback weights at each layer proportional to the transposed Jacobian, i.e.,  $-\mathbf{Q}_i = \mathbf{J}_i^T$ . However, the Jacobian depends on the input and the neuronal activity. It is thus changing over time until an equilibrium state is reached. To avoid changing feedback weights over time for a given input, we compute them based on the network Jacobian at the uncontrolled equilibrium state with  $\mathbf{c} = 0$
$$
- \mathbf {Q} _ {i} = \tilde {\mathbf {J}} _ {i} ^ {T} = \left. \left(\frac {\partial \mathbf {r} _ {L}}{\partial \mathbf {u} _ {i} ^ {\mathrm {I}}}\right) ^ {T} \right| _ {\mathbf {u} _ {i} ^ {\mathrm {I}} = \tilde {\mathbf {u}} _ {i} ^ {\mathrm {I}}} \tag {7}
$$
with  $\tilde{\mathbf{u}}_i^{\mathrm{I}}$  corresponding to the inhibitory membrane potentials in Layer  $i$  at the uncontrolled equilibrium state.
Learning as minimization of control in dis-inhibitory neuronal circuits. Given suitable feedback weights and a strong influence on the network activity by the controller, neuronal activity can change considerably compared to the uncontrolled steady-state. Tracing the steps of [47], learning rules can be derived from a minimization of control objective
$$
\mathcal {H} = \frac {1}{2} \sum_ {n} \| Q \hat {\xi} (n) \| _ {2} ^ {2} \tag {8}
$$
where  $\hat{\mathbf{c}}(n)$  is the steady-state feedback control signal for datapoint  $n$ . Formally, it can be shown that minimizing the above surrogate loss  $\mathcal{H}$  also minimizes the output loss  $\mathcal{L}$  (see Appendix B.1). We start with the following learning rule which minimizes  $\mathcal{H}$ :
$$
\tau_ {w} \frac {\mathrm {d}}{\mathrm {d} t} \mathbf {W} i = \left[ \underbrace {\left(\overline {{\mathbf {r}}} _ {i} ^ {* \mathrm {E}} - \overline {{\mathbf {u}}} _ {i} ^ {* \mathrm {I}}\right)} _ {\text {e r r o r}} \odot \underbrace {\phi^ {\prime} \left(\overline {{\mathbf {u}}} _ {i} ^ {* \mathrm {E}}\right)} _ {\text {p o s t s y n a p t i c}} \right] \underbrace {\left(\overline {{\mathbf {r}}} _ {i - 1} ^ {* \mathrm {E}}\right) ^ {T}} _ {\text {p r e s y n a p t i c}} \tag {9}
$$
where  $\phi^{\prime}(\mathbf{u})$  is the derivative of the activation function and  $\odot$  denotes element-wise multiplication. The sign of the weight change is determined by the error projected onto each neuron by the feedback controller at the equilibrium, which following Eq. (3) is encoded as  $\mathbf{Q}_i\mathbf{\bar{c}} = \mathbf{\bar{r}}_i^{\mathrm{E}} - \mathbf{\bar{u}}_i^{\mathrm{I}}$ . While Eq. (9) minimizes  $\mathcal{H}$  when provided with sensible feedback signals, it does not constitute a local learning rule because it explicitly depends on the inhibitory membrane potentials  $\mathbf{\bar{u}}_i^{\mathrm{I}}$ , which excitatory neurons cannot access directly (cf. Fig. 1).
3.3 A Hebbian learning rule for error-modulated learning through dis-inhibitory control
We were wondering whether excitatory neurons could compute an effective local approximation of Eq. (9) by estimating the inhibitory membrane potentials from locally available quantities. To that
end, we first re-write Eq. (9) as
$$
\tau_ {W} \frac {\mathrm {d}}{\mathrm {d} t} \mathbf {W} _ {i} = \left[ \left(\mathbf {\hat {r}} _ {i} ^ {* \mathrm {E}} - \phi^ {- 1} \left(\mathbf {\hat {r}} _ {i} ^ {* \mathrm {I}}\right)\right) \odot \phi^ {\prime} \left(\mathbf {\hat {u}} _ {i} ^ {* \mathrm {E}}\right) \right] \left(\mathbf {\hat {r}} _ {i - 1} ^ {* \mathrm {E}}\right) ^ {T}, \tag {10}
$$
where we substituted the inhibitory membrane potentials  ${\bf \ddot{u}}_i^{\mathrm{I}}$  with the inverse activation function  $\phi^{-1}\left(\mathbf{\dot{r}}_i^{\mathrm{I}}\right)$ . Mathematically, Eqs. (9) and (10) are equivalent, but conceptually, it re-frames the problem of non-locality since it would require an excitatory neuron to compute the inverse activation function from the recurrent inhibitory current, which is locally available. While it is hard to imagine how neurons would invert the activation function of other neurons exactly, we assume that they could conceivably compute a linear approximation, leading to a learning rule of the following general form:
$$
\tau_ {W} \frac {\mathrm {d}}{\mathrm {d} t} \mathbf {W} _ {i} = \left[ \left(\overset {*} {\mathbf {f}} _ {i} ^ {\mathrm {E}} - \theta_ {i} - \delta_ {i} \overset {*} {\mathbf {f}} _ {i} ^ {\mathrm {I}}\right) \odot \phi^ {\prime} \left(\overset {*} {\mathbf {u}} _ {i} ^ {\mathrm {E}}\right) \right] \left(\overset {*} {\mathbf {f}} _ {i - 1} ^ {\mathrm {E}}\right) ^ {T}. \tag {11}
$$
Here the parameter  $\theta_{i}$  takes the role of a postsynaptic plasticity threshold, common to many phenomenological plasticity models [6-8], while  $\delta_{i}$  adds an inhibitory current dependence to this threshold. In practice, we obtain  $\theta_{i}$  and  $\delta_{i}$  through a first-order Taylor expansion of the inverse activation function around a given linearization point  $\tilde{r}$  (see Appendix B). In the next sections, we will see that, depending on the inhibitory activation function and the linearization parameters, the model reconciles aspects of phenomenological plasticity with an effective error-modulation mechanism as demanded by normative theories of gradient-based learning.
4 Learning with dis-inhibitory control accounts for key plasticity experiments