SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks

1 INTRODUCTION . State-of-the-art ( SOTA ) artificial neural networks ( ANNs ) achieve impressive results in a variety of machine intelligence tasks ( Sejnowski , 2020 ) . However , they largely rely on mechanisms that diverge from the original inspiration from biological neural networks ( Bengio et al. , 2015 ; Illing et al. , 2019 ) . As a result , only a small part of this prolific field also contributes to computational neuroscience . In fact , this biological implausibility is also an important issue for machine intelligence . For their impressive performance , ANNs trade off other desired properties , which are present in biological systems . For example , ANN training often demands very large and labelled datasets . When labels are unavailable , self-supervised learning schemes exist , where supervisory error signals generated by the network itself are exploited and backpropagated from the output towards the input to update the network ’ s parameters ( Goodfellow et al. , 2014 ; Devlin et al. , 2018 ; Chen et al. , 2020 ) . However , this global propagation of signals in deep networks introduces another limitation . Namely , it prevents the implementation of efficient distributed computing hardware that would be based on only local signals from neighbouring physical nodes in the network , and is in contrast to local synaptic plasticity rules that partly govern biological learning . Several pieces of work have been addressing parts of the biological implausibility and hardware-inefficiency of backpropagation in ANNs ( Bengio et al. , 2015 ; Lillicrap et al. , 2016 ; Guerguiev et al. , 2017 ; Pfeiffer & Pfeil , 2018 ; Illing et al. , 2019 ; Pogodin & Latham , 2020 ; Millidge et al. , 2020 ; Pogodin et al. , 2021 ) . such as the need for exactly symmetric forward and backward weights or the waiting time caused by the network ’ s forward-backward pass between two training updates in a layer ( weight transport and update-locking problems ) . Recently , an approximation to backpropagation that is mostly Hebbian , i.e . relies on mostly pre- and post- synaptic activity of each synapse , has been achieved by reducing the global error requirements to 1-bit information ( Pogodin & Latham , 2020 ) . Two schemes that further localize the signal that is required for a weight update are Equilibrium Propagation ( Scellier & Bengio , 2017 ) and Predictive Coding ( Millidge et al. , 2020 ) . Both methods approximate backpropagation through Hebbian-like learning , by delegating the global aspect of the computation , from a global error signal , to a global convergence of the network state to an equilibrium . This equilibrium is reached through several iterative steps of feed-forward and feed-back communication throughout the network , before the ultimate weight update by one training example . The biological plausibility and hardware-efficiency of this added iterative process of signal propagation are open questions that begin to be addressed ( Ernoult et al. , 2020 ) . Moreover , learning through backpropagation , and presumably also its approximations , has another indication of biological implausibility , which also significantly limits ANN applicability . Namely , it produces networks that are confused by small adversarial perturbations of the input , which are imperceptible by humans . It has recently been proposed that a defence strategy of `` deflection '' of adversarial attacks may be the ultimate solution to that problem ( Qin et al. , 2020 ) . Through this strategy , to cause confusion in the network ’ s inferred class , the adversary is forced to generate such a changed input that really belongs to the distribution of a different input class . Intuitively , but also strictly by definition , this deflection is achieved if a human assigns to the perturbed input the same label that the network does . Deflection of adversarial attacks in ANNs has been demonstrated by an elaborate scheme that is based on detecting the attacks ( Qin et al. , 2020 ) . However , the human ability to deflect adversarial perturbations likely does not rely on detecting them , but rather on effectively ignoring them , making the deflecting type of robustness an emergent property of biological computation rather than a defence mechanism . The biological principles that underlie this property of robustness are unclear , but it might emerge from the distinct algorithms that govern learning in the brain . Therefore , what is missing is a biologically plausible model that can learn from fewer data-points , without labels , through local plasticity , and without feedback from distant layers . This model could then be tested for emergent adversarial robustness . A good candidate category of biological networks and learning algorithms is that of competitive learning . Neurons that compete for their activation through lateral inhibition are a common connectivity pattern in the superficial layers of the cerebral cortex ( Douglas & Martin , 2004 ; Binzegger et al. , 2004 ) . This pattern is described as winner-take-all ( WTA ) , because competition suppresses activity of weakly activated neurons , and emphasizes strong ones . Combined with Hebbian-like plasticity rules , WTA connectivity gives rise to competitivelearning algorithms . These networks and learning schemes have been long studied ( Von der Malsburg , 1973 ) and a large literature based on simulations and analyses describes their functional properties . A WTA neuronal layer , depending on its specifics , can restore missing input signals ( Rutishauser et al. , 2011 ; Diehl & Cook , 2016 ) , perform decision making i.e . winner selection ( Hahnloser et al. , 1999 ; Maass , 2000 ; Rutishauser et al. , 2011 ) , and generate oscillations such as those that underlie brain rhythms ( Cannon et al. , 2014 ) . Perhaps more importantly , its neurons can learn to become selective to different input patterns , such as orientation of visual bars in models of the primary visual cortex ( Von der Malsburg , 1973 ) , MNIST handwritten digits ( Nessler et al. , 2013 ; Diehl & Cook , 2015 ; Krotov & Hopfield , 2019 ) , CIFAR10 objects ( Krotov & Hopfield , 2019 ) , spatiotemporal spiking patterns ( Nessler et al. , 2013 ) , and can adapt dynamically to model changing objects ( Moraitis et al. , 2020 ) . The WTA model is indeed biologically plausible , Hebbian plasticity is local , and learning is input-driven , relying on only feed-forward communication of neurons – properties that seem to address several of the limitations of ANNs . However , the model ’ s applicability is limited to simple tasks . That is partly because the related theoretical literature remains surprisingly unsettled , despite its long history , and the strong and productive community interest ( Sanger , 1989 ; Földiák & Fdilr , 1989 ; Földiak , 1990 ; Linsker , 1992 ; Olshausen & Field , 1996 ; Bell & Sejnowski , 1995 ; Olshausen & Field , 1997 ; Lee et al. , 1999 ; Nessler et al. , 2013 ; Pehlevan & Chklovskii , 2014 ; Hu et al. , 2014 ; Pehlevan & Chklovskii , 2015 ; Pehlevan et al. , 2017 ; Isomura & Toyoizumi , 2018 ) . Nessler et al . ( 2009 ; 2013 ) described a very related theory but for a model that is largely incompatible with ANNs and thus less practical . It uses spiking and stochastic neurons , input has to be discretized , and each input feature must be encoded through multiple binary neurons . Moreover , it was only proven for neurons with an exponential activation function . It remains therefore unclear which specific plasticity rule and structure could optimize an ANN WTA for Bayesian inference . It is also unclear how to minimize a common loss function such as cross-entropy despite unsupervised learning , and how a WTA could represent varying families of probability distributions . In summary , on the theoretical side , an algorithm that is simultaneously normative , based on WTA networks and Hebbian unsupervised plasticity , performs Bayesian inference , and , importantly , is composed of conventional , i.e . non-spiking , ANN elements and is rigorously linked to modern ANN tools such as cross-entropy loss , would be an important advance but has been missing . On the practical side , evidence that Hebbian WTA networks could be useful for presently pertinent issues of modern ANNs such as adversarial robustness , generation of synthetic images , or faster learning , has remained limited . Here we aim to fill these gaps . Recently , when WTA networks were studied in a theoretical framework compatible with conventional machine learning ( ML ) , but in the context of short-term as opposed to long-term Hebbian plasticity , it resulted in surprising practical advantages over supervised ANNs ( Moraitis et al. , 2020 ) . A similar theoretical approach could also reveal unknown advantages of long-term Hebbian plasticity in WTA networks . In addition , it could provide insights into how a WTA microcircuit could participate in larger-scale computation by deeper cortical or artificial networks . Here we construct `` SoftHebb '' , a biologically plausible WTA model that is based on standard ratebased neurons as in ANNs , can accommodate various activation functions , and learns without labels , using local plasticity and only feed-forward communication , i.e . the properties we seek in an ANN . Importantly , it is equipped with a simple normalization of the layer ’ s activations , and an optional temperature-scaling mechanism ( Hinton et al. , 2015 ) , producing a soft WTA instead of selecting a single `` hard '' winner neuron . This allows us to prove formally that a SoftHebb layer is a generative mixture model that objectively minimizes its Kullback-Leibler ( KL ) divergence from the input distribution through Bayesian inference , thus providing a new formal ML-theoretic perspective of these networks . We complement our main results , which are theoretical , with experiments that are small-scale but produce intriguing results . As a generative model , SoftHebb has a broader scope than classification , but we test it on image classification tasks . Surprisingly , in addition to overcoming several inefficiencies of backpropagation , the unsupervised WTA model also outperforms a supervised two-layer perceptron in several aspects : learning speed and accuracy in the first presentation of the training dataset , robustness to noisy data and to one of the strongest white-box adversarial attacks , i.e . projected gradient descent ( PGD ) ( Madry et al. , 2017 ) , and without any explicit defence . Interestingly , the SoftHebb model also exhibits inherent properties of deflection ( Qin et al. , 2020 ) of the adversarial attacks , and generates object interpolations . 2 THEORY . A supporting diagram summarising the theoretical and neural model , and a succinct description of the learning algorithm are provided in the beginning of Appendix A . Definition 2.1 ( The input assumptions ) . Each observation jx ∈ Rn is generated by a hidden `` cause '' jC from a finite set of K possible such causes : jC ∈ { Ck , ∀k ≤ K ∈ N } . Therefore , the data is generated by a mixture of the probability distributions attributed to each of the K classes Ck : p ( x ) = K∑ k=1 p ( x|Ck ) P ( Ck ) . ( 1 ) In addition , the dimensions of x , xi are conditionally independent from each other , i.e . p ( x ) = ∏n i=1 p ( xi ) . The number K of the true causes or classes of the data is assumed to be known . The term `` cause '' is used here in the sense of causal inference . It is important to emphasize that the true cause of each input is hidden , i.e . not known . In the case of a labelled dataset , labels may correspond to causes , and the labels are deleted before presenting the training data to the model . We choose a mixture model that corresponds to the data assumptions but is also interpretable in neural terms ( Paragraph 2.4 ) : Definition 2.2 ( The generative probabilistic mixture model ) . We consider a mixture model distribution q : q ( x ) = ∑K k=1 q ( x|Ck ) Q ( Ck ) , approximating the data distribution p. We choose specifically a mixture of exponentials and we parametrize Q ( Ck ; w0k ) also as an exponential , specifically : q ( xi|Ck ; wik ) = ewik· xi ||x|| , ∀k ( 2 ) Q ( Ck ; w0k ) = e w0k , ∀k . ( 3 ) In addition , the parameter vectors are subject to the normalization constraints : ||wk|| = 1 , ∀k , and∑K k=1 e w0k = 1 . The model we have chosen is a reasonable choice because it factorizes similarly to the data of Definition 2.1 : qk : = q ( x|Ck ; wk ) = n∏ i=1 q ( xi|Ck ; wik ) = e ∑n i=1 wik xi ||x|| = euk , ( 4 ) where uk = wk·x||wk||·||x|| , i.e . the cosine similarity of the two vectors . A similar probabilistic model was used in related previous theoretical work Nessler et al . ( 2009 ; 2013 ) ; Moraitis et al . ( 2020 ) , but for different data assumptions , and with certain further constraints to the model . Namely , ( Nessler et al. , 2009 ; 2013 ) considered data that was binary , and created by a population code , while the model was stochastic . These works provide the foundation of our derivation , but here we consider the more generic scenario where data are continuous-valued and input directly into the model , which is deterministic and , as we will show , more compatible with standard ANNs . In Moraitis et al . ( 2020 ) , data had particular short-term temporal dependencies , whereas here we consider the distinct case of independent and identically distributed ( i.i.d . ) input samples . The Bayes-optimal parameters of a model mixture of exponentials can be found analytically as functions of the input distribution ’ s parameters , and the model is equivalent to a soft winner-take-all neural network ( Moraitis et al. , 2020 ) . After describing this , we will prove here that Hebbian plasticity of synapses combined with local plasticity of the neuronal biases sets the parameters to their optimal values . Theorem 2.3 ( The optimal parameters of the model ) . The parameters that minimize the KL divergence of such a mixture model from the data are , for every k , optw0k = lnP ( Ck ) ( 5 ) and optw ∗ k = optwk || optwk|| = µpk ( x ) ||µpk ( x ) || , ( 6 ) where optwk = c · µpk ( x ) , c ∈ R+ , µpk ( x ) is the mean of the distribution pk , and pk : = p ( x|Ck ) . In other words , the optimal parameter vector of each component k in this mixture is proportional to the mean of the corresponding component of the input distribution , i.e . it is a centroid of the component . In addition , the optimal parameter of the model ’ s prior Q ( Ck ) is the logarithm of the corresponding component ’ s prior probability . This Theorem ’ s proof was provided in the supplementary material of Moraitis et al . ( 2020 ) , but for completeness we also provide it in our Appendix . These centroids and priors of the input ’ s component distributions , as well as the method of their estimation , however , are different for different input assumptions , and we will derive a learning rule that provably sets the parameters to their Maximum Likelihood Estimate for the inputs addressed here . The learning rule is a Hebbian type of synaptic plasticity combined with a plasticity for neuronal biases . Before providing the rule and the related proof , we will describe how our mixture model is equivalent to a WTA neural network .

Summary: The paper wants to provide an optimization theory for WTA networks that it claims to be missing. More precisely they consider soft WTA networks that are implemented as ANNs.They apply their model to MNIST and Fashion-MNIST. In addition some robustness to adversarial attacks is demonstrated.