Learning to live with Dale's principle: ANNs with separate excitatory and inhibitory units

1 INTRODUCTION . In recent years , artificial neural networks ( ANNs ) have been increasingly used in neuroscience research for modelling the brain at the algorithmic and computational level ( Richards et al. , 2019 ; Kietzmann et al. , 2018 ; Yamins & DiCarlo , 2016 ) . They have been used for exploring the structure of representations in the brain , the learning algorithms of the brain , and the behavioral patterns of humans and non-human animals ( Bartunov et al. , 2018 ; Donhauser & Baillet , 2020 ; Michaels et al. , 2019 ; Schrimpf et al. , 2018 ; Yamins et al. , 2014 ; Kell et al. , 2018 ) . Evidence shows that the ability of ANNs to match real neural data depends critically on two factors . First , there is a consistent correlation between the ability of an ANN to learn well on a task ( e.g . image recognition , audio perception , or motor control ) and the extent to which its behavior and learned representations match real data ( Donhauser & Baillet , 2020 ; Michaels et al. , 2019 ; Schrimpf et al. , 2018 ; Yamins et al. , 2014 ; Kell et al. , 2018 ) . Second , the architecture of an ANN also helps to determine how well it can match real brain data , and generally , the more realistic the architecture the better the match ( Schrimpf et al. , 2018 ; Kubilius et al. , 2019 ; Nayebi et al. , 2018 ) . Given these two factors , it is important for neuroscientific applications to use ANNs that have as realistic an architecture as possible , but which also learn well ( Richards et al. , 2019 ; Kietzmann et al. , 2018 ; Yamins & DiCarlo , 2016 ) . Although there are numerous disconnects between ANNs and the architecture of biological neural circuits , one of the most notable is the lack of adherence to Dale ’ s principle , which states that a neuron releases the same fast neurotransmitter at all of its presynaptic terminals ( Eccles , 1976 ) . Though there are some interesting exceptions ( Tritsch et al. , 2016 ) , for the vast majority of neurons in †Corresponding author : blake.richards @ mcgill.ca adult vertebrate brains , Dale ’ s principle means that presynaptic neurons can only have an exclusively excitatory or inhibitory impact on their postsynaptic partners . For ANNs , this would mean that units can not have a mixture of positive and negative output weights , and furthermore , that weights can not change their sign after initialisation . In other words , a unit can only be excitatory or inhibitory . However , most ANNs do not incorporate Dale ’ s principle . Why is Dale ’ s principle rarely incorporated into ANNs ? The reason is that this architectural constraint impairs the ability to learn—a fact that is known to many researchers who have tried to train such ANNs , but one that is rarely discussed in the literature . However , when we seek to compare ANNs to real brains , or use them to explore biologically inspired learning rules ( Bartunov et al. , 2018 ; Whittington & Bogacz , 2019 ; Lillicrap et al. , 2020 ) , ideally we would use a biologically plausible architecture with distinct populations of excitatory and inhibitory neurons , and at the same time , we would still be able to match the learning performance of standard ANNs without such constraints . Some previous computational neuroscience studies have used ANNs with separate excitatory and inhibitory units ( Song et al. , 2016 ; Ingrosso & Abbott , 2019 ; Miconi , 2017 ; Minni et al. , 2019 ; Behnke , 2003 ) , but these studies addressed questions other than matching the learning performance of standard ANNs , e.g . they focused on typical neuroscience tasks ( Song et al. , 2016 ) , dynamic balance ( Ingrosso & Abbott , 2019 ) , biologically plausible learning algorithms ( Miconi , 2017 ) , or the learned structure of networks ( Minni et al. , 2019 ) . Importantly , what these papers did not do is develop means by which networks that obey Dale ’ s principle can match the performance of standard ANNs on machine learning benchmarks , which has become an important feature of many computational neuroscience studies using ANNs ( Bartunov et al. , 2018 ; Donhauser & Baillet , 2020 ; Michaels et al. , 2019 ; Schrimpf et al. , 2018 ; Yamins et al. , 2014 ; Kell et al. , 2018 ) . Here , we develop ANN models with separate excitatory and inhibitory units that are able to learn as well as standard ANNs . Specifically , we develop a novel form of ANN , which we call a “ Dale ’ s ANN ” ( DANN ) , based on feed-forward inhibition in the brain ( Pouille et al. , 2009 ) . Our novel approach is different from the standard solution , which is to create ANNs with separate excitatory and inhibitory units by constraining whole columns of the weight matrix to be all positive or negative ( Song et al. , 2016 ) . Throughout this manuscript , we refer to this standard approach as “ ColumnEi ” models . We have departed from the ColumnEI approach in our work because it has three undesirable attributes . First , constrained weight matrix columns impair learning because they limit the potential solution space ( Amit et al. , 1989 ; Parisien et al. , 2008 ) . Second , modelling excitatory and inhibitory units with the same connectivity patterns is biologically misleading , because inhibitory neurons in the brain tend to have very distinct connectivity patterns from excitatory neurons ( Tremblay et al. , 2016 ) . Third , real inhibition can act in both a subtractive and a divisive manner ( Atallah et al. , 2012 ; Wilson et al. , 2012 ; Seybold et al. , 2015 ; Pouille et al. , 2013 ) , which may provide important functionality . Given these considerations , in DANNs , we utilize a separate pool of inhibitory neurons with a distinct , more biologically realistic connectivity pattern , and a mixture of subtractive and divisive inhibition ( Fig . 1 ) . This loosely mimics the fast feedforward subtractive and divisive inhibition provided by fast-spiking interneurons in the cortical regions of the brain ( Atallah et al. , 2012 ; Hu et al. , 2014 ; Lourenço et al. , 2020 ) . In order to get DANNs to learn as well as standard ANNs we also employ two key insights : 1 . It is possible to view this architecture as being akin to normalisation schemes applied to the excitatory input of a layer ( Ba et al. , 2016 ; Ioffe & Szegedy , 2015 ; Wu & He , 2018 ) , and we use this perspective to motivate DANN parameter initialisation . 2 . It is important to scale the inhibitory parameter updates based on the Fisher information matrix , in order to balance the impact of excitatory and inhibitory parameter updates , similar in spirit to natural gradient approaches ( Martens , 2014 ) . Altogether , our principle contribution is a novel architecture that obey ’ s Dale ’ s principle , and that we show can learn as well as standard ANNs on machine learning benchmark tasks . This provides the research community with a new modelling tool that will allow for more direct comparisons with real neural data than traditional ANNs allow , but which does not suffer from learning impairments . Moreover , our results have interesting implications for inhibitory plasticity , and provide a means for future research into how excitatory and inhibitory neurons in the brain interact at the algorithmic level . 2 BIOLOGICALLY INSPIRED NETWORKS THAT OBEY DALE ’ S PRINCIPLE . 2.1 MODEL DEFINITION . Our design for DANNs takes inspiration from the physiology of feedforward inhibitory microcircuits in the neocortex and hippocampus . Based on these circuits , and an interpretation of layers in ANNs as corresponding to brain regions , we construct DANNs with the following architectural constraints : 1 . Each layer of the network contains two distinct populations of units , an excitatory and an inhibitory population . 2 . There are far fewer inhibitory units than excitatory units in each layer , just as there are far more excitatory neurons than inhibitory neurons ( ∼ 5-10 times ) in cortical regions of the brain ( Tremblay et al. , 2016 ; Hu et al. , 2014 ) . 3 . As in real neural circuits where only the excitatory populations project between regions , here only excitatory neurons project between layers , and both the excitatory and inhibitory populations of a layer receive excitatory projections from the layer below . 4 . All of the synaptic weights are strictly non-negative , and inhibition is enforced via the activation rules for the units ( eq . 1 ) . 5 . The inhibitory population inhibits the excitatory population through a mixture of subtractive and divisive inhibition . This constrained architecture is illustrated in Figure 1 . Formally , we define the network as follows . Input to the network is received as a vector of positive scalar values x ∈ Rd+ , which we consider to be the first excitatory population . Each hidden layer , ` , is comprised of a vector of excitatory units h ` ∈ Rne+ and inhibitory units hI ` ∈ R ni + , in-line with constraint ( 1 ) above . ( We will drop the layer index when it is unnecessary for clarity . ) Note , for the first layer ( ` = 1 ) , we have h ` = x and ne = d. Next , based on constraint ( 2 ) we set ne > > ni , and use 10 % inhibitory units as default . Following constraint ( 3 ) , both the excitatory and inhibitory units receive inputs from the excitatory units in the layer below ( h ` −1 ) , but the inhibitory units do not project between layers . Instead , excitatory units receive inputs from the inhibitory units of the same layer . In-line with constraint ( 4 ) , we have three sets of strictly non-negative synaptic weights , one for the excitatory connections between layers , WEE ` ∈ R ne×ne + , one for the excitatory projection to the inhibitory units WIE ` ∈ R ni×ne + , and one for the inhibitory projections within layer W EI ` ∈ R ne×ni + . Finally , per constraint ( 5 ) , we define the impact of the inhibitory units on the excitatory units as comprising both a subtractive and a divisive component : h ` = f ( z ` ) z ` = g ` γ ` ( zE ` −WEI ` hI ` ) + β ` ( 1 ) where zE ` = W EE ` h ` −1 h I ` = f I ( zI ` ) = f I ( WIE ` h ` −1 ) γ ` = W EI ` ( e α ` hI ` ) where for each layer ` , β ` ∈ Rne is a bias , g ` ∈ Rne+ controls the gain , γ ` is the divisive inhibitory term , and α ` ∈ Rni is a parameter that controls the strength of this divisive inhibition . Here denotes elementwise multiplication ( Hadamard product ) and the exponential function and division are applied elementwise . In the rest of this manuscript we set f to be the rectified linear function ( ReLU ) . Though a ReLU function is not a perfect match to the input-output properties of real neurons , it captures the essential rectification operation performed by neurons in physiologically realistic low activity regimes ( Salinas & Sejnowski , 2000 ) . In this paper , we model the inhibitory units as linear ( i.e . f I ( zI ) = zI ) since they receive only positive inputs and have no bias , and therefore their activation would always be in the linear part of the ReLU function . Although we make make this modelling choice mainly for mathematical simplicity , there is some biological justification , as the resting membrane potential of the class of fast-spiking interneurons most related to our model is relatively depolarised and their spike outputs can follow single inputs one-to-one ( Hu et al. , 2014 ; Galarreta & Hestrin , 2001 ) . In future work , for example in which inhibitory connections are included between inhibitory units , we expect that the use of nonlinear functions for inhibitory units will be important .

Summary: Inspired by the observations of feedforward inhibition in the brain, the authors propose a novel ANN architecture that respects Dale’s rule (DANN). They provide two improvements for training DANNs: better initialization and update scaling for synaptic weights. As a result, they empirically demonstrate that DANNs perform no worse than the ANNs that do not respect Dale’s rule.