SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Most experiments on synaptic plasticity are performed in vitro under highly controlled conditions, in which pairs of connected neurons are isolated. This enables researchers to investigate the plasticity at single synapses in the absence of interfering activity from the local microcircuit or long-range synaptic afferent connections. Such experimental conditions would likely interfere with any putative error-modulation of synaptic plasticity since long-range synaptic afferents are either severed during sample preparation or do not transmit plausible activity levels. To account for such possible experimental confounding factors and to compare experimentally observed Hebbian plasticity, we probe our error-modulated learning rule in three different settings: full microcircuit with "closed-loop feedback", intact "microcircuit without feedback", and isolated neurons resembling in-vivo experimental conditions with "direct control of inhibition" (Fig. 3). In each setting, we vary the excitatory input to the excitatory neuron and calculate the resulting weight change as a function of the postsynaptic firing rate using Eq. (11). For a qualitative comparison to previously suggested error-driven plasticity rules, we evaluated learning rules using dendritic error coding [22, 24, 30] or burst-dependent plasticity [23, 31] in comparable settings (Supplementary Fig. S1; see Appendix A).
Dis-inhibition controls the sign of plasticity in the intact microcircuit. We first considered a simplified version of the intact microcircuit with top-down dis-inhibitory feedback, for which our learning rule was derived. In this circuit, top-down control drives neuronal activity towards a target value  $r^{\mathrm{tar}}$ . We computed the weight updates dictated by our learning rule for two different targets as a function of the neuronal firing rate. In this setting, the learning rule exhibits two stable fixed points separated by an unstable one (Fig. 3a). Importantly, a stable fixed point exists at the target firing rate, i.e.  $r^{\mathrm{E}} = r^{\mathrm{tar}}$  (Fig. 3a). At this fixed point, the sign of plasticity is determined by the top-down controller through the error decoded by the learning rule (11). This behavior is in line with the behavior of learning rules with explicit error-modulation in this simplified setting (cf. Fig. 1b, Supplementary Fig. S1).
Moreover, in contrast to purely error-modulated plasticity rules, our rule exhibits an LTD region flanked by a stable fixed point in neuronal firing rates at zero and an unstable fixed point at intermediate firing rates. The existence of this LTD regime is a direct consequence of the local approximation used in its derivation (cf. Eq. (11)) and depends on the chosen parameters of the linearization (Supplementary Fig. S2; see Appendix B). For proper error-modulated learning in our framework, we have to ensure that each neuron's activity does not exclusively stay in this region over time and across different inputs. In neurobiology, this activity regime could, for instance, be attained through homeostatic plasticity [51]. Thus our learning rule exhibits error-modulated plasticity at the upper fixed point when embedded in an intact microcircuit with top-down feedback.
Figure 3: A Hebbian learning rule for error-modulated learning through dis-inhibitory control resembles plasticity observed in single-neuron electrophysiology experiments. (a) Weight change  $\Delta W$  of a single synapse as a function of postsynaptic firing rate  $r_{\mathrm{E}}$ . Different colored lines indicate two different postsynaptic firing rate targets  $r^{\mathrm{tar}}$  indicated by colored arrows. The top-down feedback onto the interneuron is proportional to the error  $r^{\mathrm{tar}} - r^{\mathrm{E}}$ . In the intact microcircuit with closed-loop feedback, our learning rule naturally leads to error-modulated learning. (b) Same as (a), but with top-down connections ablated. The two shades of green represent two different linear approximations of the inverse inhibitory activation function (see inset; inverse activation function in black). The plasticity rule resembles a multi-stable Hebbian plasticity rule [49]. (c) Same as before, but for an isolated neuron without microcircuit. Different shades of blue correspond to different amounts of inhibitory current. Different levels of injected inhibitory current lead to different values for the plasticity threshold, but the stable fixed point disappears in the absence of recurrent inhibition. (d) Experimentally observed plasticity with activity-dependent LTD and LTP redrawn from [50]. The data qualitatively resembles our learning rule in the open-loop setting (cf. panel (c)).
Plasticity in an isolated microcircuit is self-stabilizing. To examine how our learning rule behaves in the absence of top-down control signals, we investigated plasticity dynamics in an isolated microcircuit without control feedback, while local circuit connectivity between excitatory and inhibitory neurons was left intact. In this setting, the inhibitory membrane potential does not encode an error signal that can be decoded by the learning rule. While explicitly error-modulated learning rules would not exhibit any synaptic weight change in this setting, our Hebbian learning rule predicts weight changes due to the imperfect approximation of the inverse function. Fig. 3b depicts the resulting plasticity dynamics for two different linear approximations. Embedded in a local microcircuit, the plasticity rule still exhibits both an LTD and LTP regime and a stable fixed point that depends on the learning rule parameters. Notably, the plasticity rule embedded in an isolated, but intact microcircuit is self-stabilizing through recurrent inhibition. Interestingly, the necessity of such a stable fixed point at higher activity levels for stable learning has been postulated previously in theoretical work [49, 52]. As before, the presence and location of the stable fixed point depend on the choice of plasticity parameters (Supplementary Fig. S2).
Plasticity induction changes under direct control of inhibition. Experiments on excitatory plasticity in vitro are commonly performed under conditions designed to minimize the interference of inhibitory activity, for example by applying GABA antagonists [53]. To study plasticity induction in our model under such simulated experimental conditions, we blocked recurrent connections from excitatory to inhibitory neurons, so that inhibitory activity is independent of excitatory activity. Additionally, we controlled inhibitory activity, as could be achieved, for instance, through current injection or optogenetic manipulations in experiments. In the absence of any inhibitory activity, i.e.  $r^{\mathrm{I}} = 0$ , our learning rule reduces to the form  $\Delta w \propto r_{\mathrm{pre}}(r_{\mathrm{post}} - \theta) \phi'(r_{\mathrm{post}})$  and loses its stable fixed point (Fig. 3c). Thus, in the absence of inhibition, the weight update prescribed by our plasticity model resembles Hebbian plasticity rules commonly observed under experimental conditions (Fig. 3d). However, our model predicts that direct control over inhibitory inputs to the excitatory neuron should shift the postsynaptic plasticity threshold to larger values.
In summary, dis-inhibitory control accounts for key plasticity experiments while also supporting error-driven learning in top-down controlled microcircuits. Additionally, its self-stabilizing capabilities provide a possible explanation as to why in vitro experiments have failed to uncover a
Figure 4: Online learning using dis-inhibitory control of Hebbian plasticity in a student-teacher task. (a) A teacher network (left) of size 30-20-2 implements a nonlinear mapping from an array of input sine waves  $\mathbf{x}(t)$  (bottom) to an output target  $f(\mathbf{x}(t))$ . A student network (right) of the same size learns to approximate the teacher function by minimizing the control signal  $\mathbf{Q}_i\mathbf{c}(t)$  of a dis-inhibitory feedback controller at each layer. (b) One example teacher output neuron (black) and the corresponding student neuron (teal) before and after learning. The yellow bar indicates when feedback control is active. (c) MSE loss  $\mathcal{L}$  and least control loss  $\mathcal{H}$  over time for student networks trained with the exact update rule derived in Eq. (10) and the linear threshold rule (Eq. (11)).
stabilizing mechanism for excitatory synaptic plasticity. Next, we test whether our learning rule allows hierarchical networks to solve nonlinear function approximation problems.
5 Dis-inhibitory control orchestrates learning in multi-layer networks
To explore our model's ability to train multi-layer networks, we first designed a simple continuous-time low-dimensional student-teacher learning task (Fig. 4a; see Appendix C for details). In this task, a randomly initialized teacher network with fixed parameters  $f(\mathbf{x}(t))$  is given a set of 20 sine wave inputs with randomly chosen amplitudes, frequencies, and phases  $\mathbf{x}(t)$ . An architecturally identical student network but with different initial weights receives the same input  $\mathbf{x}(t)$  and is tasked to reproduce the teacher's output, i.e.  $r_L^{\mathrm{tar}}(t) = f(\mathbf{x}(t))$ , evaluated through an MSE loss function. A dis-inhibitory feedback controller as described in the previous section is continually driving the student network's activity towards lower loss in real time. We trained the student network with either the exact non-linear inhibitory threshold rule Eq. (10) or the Hebbian learning rule with a linear inhibitory threshold Eq. (11). Neuronal and weight dynamics were simulated in continuous time using an explicit  $5^{\mathrm{th}}$  order Runge-Kutta method. To make sure that the feedback controller is aligned with the changing Jacobian during learning, the feedback weights were plastic and continuously evolving towards the average network Jacobian (see Appendix C).
Before learning, the student network did not follow the target closely in the open-loop setting, i.e., when the control signal was turned off. However, as soon as disinhibitory control was activated, the output activity closely followed the target (Fig. 4b). We then trained the student network for a total of 300 seconds of continuous sine wave inputs using either Eq. (10) or Eq. (11). After learning, the student network output closely followed the target in the open-loop setting, accompanied by a substantial reduction of the open-loop MSE loss  $(c(t) = 0)$  and the surrogate loss  $\mathcal{H}$  over the course of training (Fig. 4c). Finally, we observed that the linear threshold learning rule resulted in comparable performance to the exact inverse learning rule. Thus, an inhibitory modulated Hebbian learning rule is capable of solving a nonlinear learning task when credit is relayed through local dis-inhibitory microcircuits acting as feedback controller.
5.1 Training multi-layer networks on classification tasks through dis-inhibitory control
Having confirmed that our learning rule is capable of error-driven learning through minimization of control on a simple continuous-time learning task, we wondered whether we could train a multi-layer perceptron (MLP) on standard image classification tasks. To that end, we implemented a MLP with excitatory-inhibitory microcircuit units. Networks were comprised of either one or three hidden layers
and used a parameterized soft rectifier activation function for all units. Numerical integration was performed using a fifth-order Runge-Kutta method with an adaptive step size to allow the network to reach an equilibrium state for each input (see Appendix C for details). Networks were trained either with the exact inverse rule in Eq. (10), or the linear threshold rule (Eq. (11)). To make training more robust and less dependent on initialization, the linear approximation of the inverse function was obtained through first-order Taylor expansion around a neuron-specific parameter  $\tilde{\mathbf{r}}_i$  which tracked the average inhibitory firing rate across each batch (see Appendix C). With these settings we trained the networks on MNIST [54] and Fashion-MNIST [55]. For comparison, we also trained conventional MLPs with the same neuron numbers but strictly feed-forward connectivity using BP.
Table 1: Test accuracy in % for networks trained with BP or dis-inhibitory feedback control. Reported values are mean ±stdev ( $n = 10$ ). For validation accuracy see Appendix C.
Number of hidden layers	MNIST	Fashion-MNIST
1	3	1	3
Backprop	98.1 ± 0.2	98.3 ± 0.1	89.3 ± 0.3	89.4 ± 0.2
Dis-inhibitory control	Exact inverse	97.7 ± 0.2	98.0 ± 0.2	89.1 ± 0.2	89.1 ± 0.2
Linear threshold	97.1 ± 0.1	96.5 ± 0.1	87.6 ± 0.4	86.8 ± 0.2
Exact inverse (Avg J)	97.8 ± 0.1	97.0 ± 1.5	88.7 ± 1.8	88.2 ± 0.5
Linear threshold (Avg J)	96.5 ± 0.1	96.0 ± 0.3	86.4 ± 0.3	84.6 ± 0.3
We observed that networks trained using dis-inhibitory control with the exact inverse learning rule performed almost on par with standard BP on both datasets (Table 1). Using the linear threshold rule led to a slight drop in accuracy in all cases. It is possible that this gap could be narrowed further by choosing different linearization parameters  $\tilde{\mathbf{r}}_i$  since the performance of the exact inverse rule suggests that improving the approximation of the error translates to higher accuracy.
In above simulations, the Jacobian used for calculating the feedback signals is input-dependent and feedback weights are thus different for each stimulus. In neurobiology, feedback would presumably be relayed via synaptic connections that do not change this rapidly. To test whether successful learning is possible with more slowly varying feedback weights we repeated the above simulations with feedback weights that slowly tracked the average Jacobian (see Appendix C). This change did not compromise learning, although it resulted in a small but noticeable drop in accuracy compared to the ideal data-dependent Jacobian (Table 1). In summary, the combination of disinhibitory control with a Hebbian learning rule with an inhibition-dependent threshold allows training MLPs on vision datasets such as MNIST and Fashion-MNIST to accuracy values close to networks trained with BP.
In this article, we introduced a Hebbian learning rule with an inhibition-dependent threshold, which, through dis-inhibitory microcircuit dynamics, allows top-down feedback to control the sign of plasticity. Notably, the learning rule captures essential aspects of classic phenomenological Hebbian plasticity models under simulated experimental conditions disrupting recurrent inhibition in the local microcircuit. In contrast to standard Hebbian plasticity models, our model is stable when recurrent inhibition is intact without requiring additional homeostatic or compensatory mechanisms. Finally, we show how dis-inhibitory control is sufficient to train MLPs on vision tasks with performance levels close to classical BP.
Dis-inhibitory control reconciles error-based learning with classic Hebbian plasticity models. The learning rule we put forward in this article has a postsynaptic plasticity threshold  $\theta$  (cf. Eq. (11)), a neuron-specific parameter potentially subject to its own temporal dynamics. This threshold makes it reminiscent of the BCM rule [36]. Moreover, our model extends classic plasticity models by an explicit dependence on inhibition, which adds a dynamic, rapidly evolving component to the plasticity threshold. We derived this dual-threshold mechanism within a normative minimization-of-control objective. Its functional role is to decode a credit signal, relayed through dis-inhibitory afferents, from changes in the inhibitory current. Notably, the inhibitory threshold in our model confers an additional helpful property in that it induces a rapid compensatory plasticity mechanism that prevents
pathological runaway LTP, usually associated with Hebbian plasticity, through recurrent inhibition and in the absence of top-down control signals or any additional homeostatic mechanisms. Theoretical studies have argued that such a stabilizing mechanism for high firing rates should exist to counteract the runaway potentiation of Hebbian plasticity [49, 52].
Experimentally testable predictions. Our model puts forth several testable predictions. First, we anticipate a direct modulation of plasticity induction through inhibitory currents in conventional excitatory long-term plasticity induction protocols. We predict that postsynaptic plasticity thresholds [32, 33] should be influenced by inhibitory current injection, even when the postsynaptic activity is kept constant in experiments. Most plasticity experiments proceed in the presence of GABA antagonists or sodium channel blockers, which may obscure the direct impact of inhibition on the plasticity threshold. Nevertheless, abundant experimental evidence supports the idea that inhibition influences classic induction protocols at excitatory synapses [56-59] and that GABAergic afferents can switch the sign of plasticity [60, 61]. Second, our model suggests that blocking dis-inhibitory circuits during an error-based learning paradigm should block or influence learning. Consistent with this hypothesis, dis-inhibitory microcircuits have been implicated with the gating of plasticity and behaviorally relevant learning in the Amygdala [62, 63], Hippocampus [64], and sensory cortices [65-69] (for a review, see [70]). Conversely, activating the same circuitry should affect or trigger learning during specific tasks.
Limitations. Several limitations should be considered when interpreting this study's results. First, while our learning rule minimizes the loss, it does not necessarily follow the negative gradient. This difference could lead to sub-optimal learning dynamics and we will explore its impact in future work.
Moreover, our circuit model requires specific one-to-one connectivity between excitatory and inhibitory interneurons that is inconsistent with circuit motifs observed in the brain. In biological microcircuits, excitatory neurons usually exceed the number of inhibitory neurons [71], and inhibitory interneurons typically provide inputs to many local excitatory cells and vice versa. Nevertheless, it may be possible to consider our model's inhibitory threshold (cf. Eq. (11)) as a local inhibitory current derived from multiple presynaptic targets. In this scenario, the challenge for excitatory neurons is to estimate their respective contribution to the inhibitory current they receive. In future work, we will explore the possibility of learning an estimate of the expected inhibitory current by implementing inhibitory plasticity to achieve an excitatory-inhibitory balanced state [72] and its co-dependence with excitatory plasticity [73]. Inhibitory plasticity and its interactions with excitatory plasticity is supported by experiments [74-76] and has been the focus of recent computational models [77, 78]. However, neither of these studies explored the functional relevance of co-dependent plasticity for credit assignment and circuit-level learning.
Finally, the present model does not adhere to Dale's law in synaptic connections between hidden layers or the feedback pathway. Instead, we integrated the local population of dis-inhibitory interneurons into the dynamics of a top-down controller that modifies inhibitory activity bidirectionally. Biologically, this could be achieved through high baseline firing rates of dis-inhibitory interneurons or top-down feedback connections that target inhibitory interneurons directly [42]. In future work, we will explore detailed cortical architectures for dis-inhibitory feedback controllers that allow bi-directional modulation.
In summary, we made the first step to reconcile realistic circuit models and phenomenological plasticity rules with normative theories relying on error-modulated plasticity to solve the credit assignment problem. Specifically, we showed how top-down feedback signals targeting specific interneurons could efficiently modulate neuronal activity and plasticity. Our results highlight the potential of learning algorithms beyond BP, showcase their ability to incorporate diverse plasticity phenomena observed in neurobiology, and open the door for exciting future research.
We thank all members of the Zenke Group for valuable comments and discussions. This project was supported by the Swiss National Science Foundation [grant number PCEFP3_202981] and the Novartis Research Foundation.
[1] D O Hebb. The organization of behavior; a neuropsychological theory. The organization of behavior; a neuropsychological theory. Wiley, Oxford, England, 1949.
[2] Robert C Malenka and Mark F Bear. LTP and LTD: an embarrassment of riches. Neuron, 44(1): 5-21, 2004. doi: 10.1016/j.neuron.2004.09.012.
[3] Daniel E Feldman. The spike-timing dependence of plasticity. Neuron, 75(4):556-571, 2012. doi: 10.1016/j.neuron.2012.08.001.
[4] Wulfram Gerstner and Werner M Kistler. Mathematical formulations of hebbian learning. Biological cybernetics, 87(5-6):404-415, 2002. doi: 10.1007/s00422-002-0353-y.
[5] Harel Z. Shouval, Mark F. Bear, and Leon N. Cooper. A Unified Model of NMDA Receptor-Dependent Bidirectional Synaptic Plasticity. Proc Natl Acad Sci U S A, 99(16):10831-10836, 2002. doi: 10.1073/pnas.152343099.
[6] Jean-Pascal Pfister and Wulfram Gerstner. Triplets of Spikes in a Model of Spike Timing-Dependent Plasticity. J Neurosci, 26(38):9673–9682, 2006. doi: 10.1523/JNEUROSCI.1425-06.2006.
[7] Claudia Clopath, Lars Busing, Eleni Vasilaki, and Wulfram Gerstner. Connectivity reflects coding: a model of voltage-based STDP with homeostasis. Nat Neurosci, 13(3):344-352, 2010. doi: 10.1038/nn.2479.
[8] Michael Graupner and Nicolas Brunel. Calcium-based plasticity model explains sensitivity of synaptic changes to spike pattern, rate, and dendritic location. Proc Natl Acad Sci U S A, 109 (10):3991-3996, 2012. doi: 10.1073/pnas.1109359109.
[9] Nicolas Frémaux and Wulfram Gerstner. Neuromodulated Spike-Timing-Dependent plasticity, and theory of Three-Factor learning rules. Frontiers in neural circuits, 9:85–85, 2016. doi: 10.3389/fncir.2015.00085.
[10] Lukasz Kuśmierz, Takuya Isomura, and Taro Toyoizumi. Learning with three factors: modulating hebbian plasticity with errors. Current opinion in neurobiology, 46:170-177, 2017. doi: 10.1016/j.conb.2017.08.020.
[11] Verena Pawlak, Jeffery R Wickens, Alfredo Kirkwood, and Jason N D Kerr. Timing is not Everything: Neuromodulation Opens the STDP Gate. Front Synaptic Neurosci, 2:146, 2010. doi: 10.3389/fnsyn.2010.00146.
[12] Zuzanna Brzosko, Susanna B Mierau, and Ole Paulsen. Neuromodulation of Spike-Timing-Dependent plasticity: Past, present, and future. *Neuron*, 103(4):563–581, 2019. doi: 10.1016/j.neuron.2019.05.041.
[13] Justin Werfel, Xiaohui Xie, and H Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. Neural computation, 17(12):2699-2718, 2005. doi: 10.1162/089976605774320539.
[14] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature reviews. Neuroscience, 2020. doi: 10.1038/s41583-020-0277-3.
[15] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 323(6088):533-536, 1986. doi: 10.1038/323533a0.
[16] F Crick. The recent excitement about neural networks. Nature, 337(6203):129-132, 1989. doi: 10.1038/337129a0.
[17] Pieter R Roelfsema and Anthony Holtmaat. Control of synaptic plasticity in deep cortical networks. Nature reviews. Neuroscience, 19(3):166-180, 2018. doi: 10.1038/nrn.2018.6.
Appendix A Comparison with explicitly error-modulated plasticity rules
Normative theories of functional learning in neuronal circuits of the brain commonly derive biologically plausible solutions for the credit assignment problem that approximate BP. Like BP, these models result in explicitly error-modulated update rules for synaptic weights of the general form
$$
\Delta w _ {i j} \propto \operatorname {p r e} _ {j} \times g (\operatorname {p o s t}) _ {i} \times e _ {i} \tag {12}
$$