SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks

1 INTRODUCTION . State-of-the-art ( SOTA ) artificial neural networks ( ANNs ) achieve impressive results in a variety of machine intelligence tasks ( Sejnowski , 2020 ) . However , they largely rely on mechanisms that diverge from the original inspiration from biological neural networks ( Bengio et al. , 2015 ; Illing et al. , 2019 ) . As a result , only a small part of this prolific field also contributes to computational neuroscience . In fact , this biological implausibility is also an important issue for machine intelligence . For their impressive performance , ANNs trade off other desired properties , which are present in biological systems . For example , ANN training often demands very large and labelled datasets . When labels are unavailable , self-supervised learning schemes exist , where supervisory error signals generated by the network itself are exploited and backpropagated from the output towards the input to update the network ’ s parameters ( Goodfellow et al. , 2014 ; Devlin et al. , 2018 ; Chen et al. , 2020 ) . However , this global propagation of signals in deep networks introduces another limitation . Namely , it prevents the implementation of efficient distributed computing hardware that would be based on only local signals from neighbouring physical nodes in the network , and is in contrast to local synaptic plasticity rules that partly govern biological learning . Several pieces of work have been addressing parts of the biological implausibility and hardware-inefficiency of backpropagation in ANNs ( Bengio et al. , 2015 ; Lillicrap et al. , 2016 ; Guerguiev et al. , 2017 ; Pfeiffer & Pfeil , 2018 ; Illing et al. , 2019 ; Pogodin & Latham , 2020 ; Millidge et al. , 2020 ; Pogodin et al. , 2021 ) . such as the need for exactly symmetric forward and backward weights or the waiting time caused by the network ’ s forward-backward pass between two training updates in a layer ( weight transport and update-locking problems ) . Recently , an approximation to backpropagation that is mostly Hebbian , i.e . relies on mostly pre- and post- synaptic activity of each synapse , has been achieved by reducing the global error requirements to 1-bit information ( Pogodin & Latham , 2020 ) . Two schemes that further localize the signal that is required for a weight update are Equilibrium Propagation ( Scellier & Bengio , 2017 ) and Predictive Coding ( Millidge et al. , 2020 ) . Both methods approximate backpropagation through Hebbian-like learning , by delegating the global aspect of the computation , from a global error signal , to a global convergence of the network state to an equilibrium . This equilibrium is reached through several iterative steps of feed-forward and feed-back communication throughout the network , before the ultimate weight update by one training example . The biological plausibility and hardware-efficiency of this added iterative process of signal propagation are open questions that begin to be addressed ( Ernoult et al. , 2020 ) . Moreover , learning through backpropagation , and presumably also its approximations , has another indication of biological implausibility , which also significantly limits ANN applicability . Namely , it produces networks that are confused by small adversarial perturbations of the input , which are imperceptible by humans . It has recently been proposed that a defence strategy of `` deflection '' of adversarial attacks may be the ultimate solution to that problem ( Qin et al. , 2020 ) . Through this strategy , to cause confusion in the network ’ s inferred class , the adversary is forced to generate such a changed input that really belongs to the distribution of a different input class . Intuitively , but also strictly by definition , this deflection is achieved if a human assigns to the perturbed input the same label that the network does . Deflection of adversarial attacks in ANNs has been demonstrated by an elaborate scheme that is based on detecting the attacks ( Qin et al. , 2020 ) . However , the human ability to deflect adversarial perturbations likely does not rely on detecting them , but rather on effectively ignoring them , making the deflecting type of robustness an emergent property of biological computation rather than a defence mechanism . The biological principles that underlie this property of robustness are unclear , but it might emerge from the distinct algorithms that govern learning in the brain . Therefore , what is missing is a biologically plausible model that can learn from fewer data-points , without labels , through local plasticity , and without feedback from distant layers . This model could then be tested for emergent adversarial robustness . A good candidate category of biological networks and learning algorithms is that of competitive learning . Neurons that compete for their activation through lateral inhibition are a common connectivity pattern in the superficial layers of the cerebral cortex ( Douglas & Martin , 2004 ; Binzegger et al. , 2004 ) . This pattern is described as winner-take-all ( WTA ) , because competition suppresses activity of weakly activated neurons , and emphasizes strong ones . Combined with Hebbian-like plasticity rules , WTA connectivity gives rise to competitivelearning algorithms . These networks and learning schemes have been long studied ( Von der Malsburg , 1973 ) and a large literature based on simulations and analyses describes their functional properties . A WTA neuronal layer , depending on its specifics , can restore missing input signals ( Rutishauser et al. , 2011 ; Diehl & Cook , 2016 ) , perform decision making i.e . winner selection ( Hahnloser et al. , 1999 ; Maass , 2000 ; Rutishauser et al. , 2011 ) , and generate oscillations such as those that underlie brain rhythms ( Cannon et al. , 2014 ) . Perhaps more importantly , its neurons can learn to become selective to different input patterns , such as orientation of visual bars in models of the primary visual cortex ( Von der Malsburg , 1973 ) , MNIST handwritten digits ( Nessler et al. , 2013 ; Diehl & Cook , 2015 ; Krotov & Hopfield , 2019 ) , CIFAR10 objects ( Krotov & Hopfield , 2019 ) , spatiotemporal spiking patterns ( Nessler et al. , 2013 ) , and can adapt dynamically to model changing objects ( Moraitis et al. , 2020 ) . The WTA model is indeed biologically plausible , Hebbian plasticity is local , and learning is input-driven , relying on only feed-forward communication of neurons – properties that seem to address several of the limitations of ANNs . However , the model ’ s applicability is limited to simple tasks . That is partly because the related theoretical literature remains surprisingly unsettled , despite its long history , and the strong and productive community interest ( Sanger , 1989 ; Földiák & Fdilr , 1989 ; Földiak , 1990 ; Linsker , 1992 ; Olshausen & Field , 1996 ; Bell & Sejnowski , 1995 ; Olshausen & Field , 1997 ; Lee et al. , 1999 ; Nessler et al. , 2013 ; Pehlevan & Chklovskii , 2014 ; Hu et al. , 2014 ; Pehlevan & Chklovskii , 2015 ; Pehlevan et al. , 2017 ; Isomura & Toyoizumi , 2018 ) . Nessler et al . ( 2009 ; 2013 ) described a very related theory but for a model that is largely incompatible with ANNs and thus less practical . It uses spiking and stochastic neurons , input has to be discretized , and each input feature must be encoded through multiple binary neurons . Moreover , it was only proven for neurons with an exponential activation function . It remains therefore unclear which specific plasticity rule and structure could optimize an ANN WTA for Bayesian inference . It is also unclear how to minimize a common loss function such as cross-entropy despite unsupervised learning , and how a WTA could represent varying families of probability distributions . In summary , on the theoretical side , an algorithm that is simultaneously normative , based on WTA networks and Hebbian unsupervised plasticity , performs Bayesian inference , and , importantly , is composed of conventional , i.e . non-spiking , ANN elements and is rigorously linked to modern ANN tools such as cross-entropy loss , would be an important advance but has been missing . On the practical side , evidence that Hebbian WTA networks could be useful for presently pertinent issues of modern ANNs such as adversarial robustness , generation of synthetic images , or faster learning , has remained limited . Here we aim to fill these gaps . Recently , when WTA networks were studied in a theoretical framework compatible with conventional machine learning ( ML ) , but in the context of short-term as opposed to long-term Hebbian plasticity , it resulted in surprising practical advantages over supervised ANNs ( Moraitis et al. , 2020 ) . A similar theoretical approach could also reveal unknown advantages of long-term Hebbian plasticity in WTA networks . In addition , it could provide insights into how a WTA microcircuit could participate in larger-scale computation by deeper cortical or artificial networks . Here we construct `` SoftHebb '' , a biologically plausible WTA model that is based on standard ratebased neurons as in ANNs , can accommodate various activation functions , and learns without labels , using local plasticity and only feed-forward communication , i.e . the properties we seek in an ANN . Importantly , it is equipped with a simple normalization of the layer ’ s activations , and an optional temperature-scaling mechanism ( Hinton et al. , 2015 ) , producing a soft WTA instead of selecting a single `` hard '' winner neuron . This allows us to prove formally that a SoftHebb layer is a generative mixture model that objectively minimizes its Kullback-Leibler ( KL ) divergence from the input distribution through Bayesian inference , thus providing a new formal ML-theoretic perspective of these networks . We complement our main results , which are theoretical , with experiments that are small-scale but produce intriguing results . As a generative model , SoftHebb has a broader scope than classification , but we test it on image classification tasks . Surprisingly , in addition to overcoming several inefficiencies of backpropagation , the unsupervised WTA model also outperforms a supervised two-layer perceptron in several aspects : learning speed and accuracy in the first presentation of the training dataset , robustness to noisy data and to one of the strongest white-box adversarial attacks , i.e . projected gradient descent ( PGD ) ( Madry et al. , 2017 ) , and without any explicit defence . Interestingly , the SoftHebb model also exhibits inherent properties of deflection ( Qin et al. , 2020 ) of the adversarial attacks , and generates object interpolations . 2 THEORY . A supporting diagram summarising the theoretical and neural model , and a succinct description of the learning algorithm are provided in the beginning of Appendix A . Definition 2.1 ( The input assumptions ) . Each observation jx ∈ Rn is generated by a hidden `` cause '' jC from a finite set of K possible such causes : jC ∈ { Ck , ∀k ≤ K ∈ N } . Therefore , the data is generated by a mixture of the probability distributions attributed to each of the K classes Ck : p ( x ) = K∑ k=1 p ( x|Ck ) P ( Ck ) . ( 1 ) In addition , the dimensions of x , xi are conditionally independent from each other , i.e . p ( x ) = ∏n i=1 p ( xi ) . The number K of the true causes or classes of the data is assumed to be known . The term `` cause '' is used here in the sense of causal inference . It is important to emphasize that the true cause of each input is hidden , i.e . not known . In the case of a labelled dataset , labels may correspond to causes , and the labels are deleted before presenting the training data to the model . We choose a mixture model that corresponds to the data assumptions but is also interpretable in neural terms ( Paragraph 2.4 ) : Definition 2.2 ( The generative probabilistic mixture model ) . We consider a mixture model distribution q : q ( x ) = ∑K k=1 q ( x|Ck ) Q ( Ck ) , approximating the data distribution p. We choose specifically a mixture of exponentials and we parametrize Q ( Ck ; w0k ) also as an exponential , specifically : q ( xi|Ck ; wik ) = ewik· xi ||x|| , ∀k ( 2 ) Q ( Ck ; w0k ) = e w0k , ∀k . ( 3 ) In addition , the parameter vectors are subject to the normalization constraints : ||wk|| = 1 , ∀k , and∑K k=1 e w0k = 1 . The model we have chosen is a reasonable choice because it factorizes similarly to the data of Definition 2.1 : qk : = q ( x|Ck ; wk ) = n∏ i=1 q ( xi|Ck ; wik ) = e ∑n i=1 wik xi ||x|| = euk , ( 4 ) where uk = wk·x||wk||·||x|| , i.e . the cosine similarity of the two vectors . A similar probabilistic model was used in related previous theoretical work Nessler et al . ( 2009 ; 2013 ) ; Moraitis et al . ( 2020 ) , but for different data assumptions , and with certain further constraints to the model . Namely , ( Nessler et al. , 2009 ; 2013 ) considered data that was binary , and created by a population code , while the model was stochastic . These works provide the foundation of our derivation , but here we consider the more generic scenario where data are continuous-valued and input directly into the model , which is deterministic and , as we will show , more compatible with standard ANNs . In Moraitis et al . ( 2020 ) , data had particular short-term temporal dependencies , whereas here we consider the distinct case of independent and identically distributed ( i.i.d . ) input samples . The Bayes-optimal parameters of a model mixture of exponentials can be found analytically as functions of the input distribution ’ s parameters , and the model is equivalent to a soft winner-take-all neural network ( Moraitis et al. , 2020 ) . After describing this , we will prove here that Hebbian plasticity of synapses combined with local plasticity of the neuronal biases sets the parameters to their optimal values . Theorem 2.3 ( The optimal parameters of the model ) . The parameters that minimize the KL divergence of such a mixture model from the data are , for every k , optw0k = lnP ( Ck ) ( 5 ) and optw ∗ k = optwk || optwk|| = µpk ( x ) ||µpk ( x ) || , ( 6 ) where optwk = c · µpk ( x ) , c ∈ R+ , µpk ( x ) is the mean of the distribution pk , and pk : = p ( x|Ck ) . In other words , the optimal parameter vector of each component k in this mixture is proportional to the mean of the corresponding component of the input distribution , i.e . it is a centroid of the component . In addition , the optimal parameter of the model ’ s prior Q ( Ck ) is the logarithm of the corresponding component ’ s prior probability . This Theorem ’ s proof was provided in the supplementary material of Moraitis et al . ( 2020 ) , but for completeness we also provide it in our Appendix . These centroids and priors of the input ’ s component distributions , as well as the method of their estimation , however , are different for different input assumptions , and we will derive a learning rule that provably sets the parameters to their Maximum Likelihood Estimate for the inputs addressed here . The learning rule is a Hebbian type of synaptic plasticity combined with a plasticity for neuronal biases . Before providing the rule and the related proof , we will describe how our mixture model is equivalent to a WTA neural network .

Summary: The work presents an interesting equivalency between a generative probabilistic mixture model and a winner-take-all Hebbian local learning rule-based learning. The approach seems promising, especially the adversarial robustness results compared to a multi-layer perceptron.   **UPDATED SCORE** I would like to thank the authors for their revisions and modification based on my feedback. I am changing my score from 5 --> 6 to reflect that. I would also like to mention that there have been works based on Hebbian/Non-Hebbian Learning that outperform backdrop-based  approaches in challenging scenarios such as continual supervised/reinforcement learning. There are other strategies for backprop networks such as implicit formulation (treating NN as a dynamical system) that improves adversarial robustness without adversarial training. Moreover, from a practical usability standpoint, even though SoftHebb is faster, the accuracy still trails the backprop-based networks.