Learning to live with Dale's principle: ANNs with separate excitatory and inhibitory units
Authors: Jonathan Cornford
Damjan Kalajdzievski
Marco Leite
Amélie Lamarquette
Dimitri Michael Kullmann
Blake Aaron Richards
Venue: ICLR 2021 Poster (2021)

Abstract:  The units in artificial neural networks (ANNs) can be thought of as abstractions of biological neurons, and ANNs are increasingly used in neuroscience research. However, there are many important differences between ANN units and real neurons. One of the most notable is the absence of Dale's principle, which ensures that biological neurons are either exclusively excitatory or inhibitory. Dale's principle is typically left out of ANNs because its inclusion impairs learning. This is problematic, because one of the great advantages of ANNs for neuroscience research is their ability to learn complicated, realistic tasks. Here, by taking inspiration from feedforward inhibitory interneurons in the brain we show that we can develop ANNs with separate populations of excitatory and inhibitory units that learn just as well as standard ANNs. We call these networks Dale's ANNs (DANNs). We present two insights that enable DANNs to learn well: (1) DANNs are related to normalization schemes, and can be initialized such that the inhibition centres and standardizes the excitatory activity, (2) updates to inhibitory neuron parameters should be scaled using corrections based on the Fisher Information matrix. These results demonstrate how ANNs that respect Dale's principle can be built without sacrificing learning performance, which is important for future work using ANNs as models of the brain. The results may also have interesting implications for how inhibitory plasticity in the real brain operates.

Published as a conference paper at ICLR 2021
LEARNING TO LIVE WITH DALE'S PRINCIPLE: ANNs WITH SEPARATE EXCITATORY AND INHIBITORY UNITS
Jonathan Cornford Damjan Kalajdzievskii". Marco Leite*, Dimitri M. Kullmann', Blake Richards McGill University, "MILA, Université de Montréal, 'UCL, University of Cambridge
Amélie Lamarquette
ABSTRACT
The units in artificial neural networks (ANNs) can be thought of as abstractions of biological neurons, and ANNs are increasingly used in neuroscience research. However, there are many important differences between ANN units and real neu- rons. One of the most notable is the absence of Dale's s principle, which ensures that biological neurons are either exclusively excitatory or inhibitory. Dale's principle is typically left out of ANNs because its inclusion impairs learning. This is prob- lematic, because one of the great advantages of ANNs for neuroscience research is their ability to learn complicated, realistic tasks. Here, by taking inspiration from feedforward inhibitory interneurons in the brain we show that we can develop ANNs with separate populations of excitatory and inhibitory units that learn just as well as standard ANNs. We call these networks Dale's s ANNs (DANNs). We present two insights that enable DANNS to learn well: (1) DANNS are related normalization schemes, and can be initialized such that the inhibition centres and standardizes the excitatory activity, (2) updates to inhibitory neuron parameters should be scaled using corrections based on the Fisher Information matrix. These results demonstrate how ANNs that respect Dale's principle can be built without sacrificing learning performance, which is important for future work using ANNs as models of the brain. The results may also have interesting implications for how inhibitory plasticity in the real brain operates.
INTRODUCTION
In recent years, artificial neural networks (ANNs) have been increasingly used in neuroscience research for modelling the brain at the algorithmic and computational level (Richards et al., 2019; Kietzmann et al., 2018; Yamins & DiCarlo, 2016). They have been used for cxploring the structure of representations in the brain, the learning algorithms of the brain, and the behavioral patterns of humans and non-human animals (Bartunov et al., 2018 8; Donhauser & Baillet, 2020; Michaels et al., 2019; Schrimpf et al., 2018; Yamins et al., 2014; Kell et al., 2018). Evidence shows that the ability of ANNs to match real neural data depends critically on two factors. First, there consistent correlation between the ability of an ANN to learn well on a task (e.g. image recognition, audio perception, or motor control) and the extent which its behavior and learned representations match real data (Donhauser & Baillet, 2020; Michaels et al., 2019; Schrimpf et al., 2018; Yamins et al., 2014; Kell et al., 2018). Second, the architecture of an ANN also helps to determine how well can match real brain data, and generally, the more realistic the architecture the better the match (Schrimpf et al., 2018; Kubilius et al., 2019; Nayebi et al., 2018). Given these two factors, it is important for neuroscientific applications to use ANNs that have realistic an architecture possible, but which also learn well (Richards et al., 2019; Kietzmann et al., 2018; Yamins & DiCarlo, 2016).
Although there are numerous disconnects between ANNs and the architecture of biological neural circuits, one of the most notable the lack of adherence to Dale's principle, which states that a neuron releases the same fast neurotransmitter at all of its presynaptic terminals (Eccles, 1976). Though there are some interesting exceptions (Tritsch et al., 2016), for the vast majority of neurons in
"Corresponding author: Mlake.richards megill.ca

---

Published as a conference paper at ICLR 2021
adult vertebrate brains, Dale's principle means that presynaptic neurons can only have an exclusively excitatory or inhibitory impact on their postsynaptic partners. For ANNs, this would mean that units cannot have mixture of positive and negative output weights, and furthermore, that weights cannot change their sign after initialisation. In other words, a unit can only be excitatory or inhibitory. However, most ANNS do not incorporate Dale's principle.
Why is Dale's ' principle rarely incorporated into ANNs? The reason is that this architectural constraint impairs the ability learn-a fact that is known to many researchers who have tried train such ANNs, but one that is rarely discussed in the literature. However, when we seek to compare ANNs to real brains. or use them to explore biologically inspired learning rules (Bartunov et al., 2018; Whittington & Bogacz, 2019; Lillicrap et al., 2020), idcally we would use biologically plausible architecture with distinct populations of excitatory and inhibitory neurons, and at the same time, we would still be able to match the learning performance of standard ANNs without such constraints.
Some previous computational neuroscience studies have used ANNs with separate excitatory and inhibitory units (Song et al., 2016; Ingrosso & Abbott, 2019; Miconi, 2017; Minni et al., 2019; Behnke, 2003), but these studies addressed questions other than matching the learning performance of standard ANNs, e.g. they focused on typical neuroscience tasks (Song et al., 2016), dynamic balance (Ingrosso& & Abbott, 2019), biologically plausible learning algorithms (Miconi. 2017), or the Iearned structure of networks (Minni et al., 2019). Importantly, what these papers did not do is develop means by which networks that obey Dale's ' principle can match the performance of standard ANNs on machine learning benchmarks, which has become an important feature of many computational neuroscience studies using ANNs (Bartunov et al.. 2018; Donhauser & Baillet, 2020; Michaels et al., 2019; Schrimpf al., 2018; Yamins et al., 2014; Kell al., 2018).
Here, we develop ANN models with separate excitatory and inhibitory units that are able to learn as well as standard ANNs. Specifically, we develop a novel form of ANN, which we call a "Dale's 's ANN" (DANN), based on feed-forward inhibition in the brain (Pouille et al., 2009). Our novel approach is different from the standard solution, which is to create ANNs with separate excitatory and inhibitory units by constraining whole columns of the weight matrix to be all positive or negative (Song et al., 2016). Throughout this manuscript, we refer to this standard approach as "ColumnEi" models. We have departed from the ColumnEI approach in our work because it has three undesirable attributes. First, constrained weight matrix columns impair learning because they limit the potential solution space (Amit al., 1989; Parisien et al., 2008). Second, modelling excitatory and inhibitory units with the same connectivity patterns is biologically misleading. because inhibitory neurons in the brain tend to have very distinct connectivity patterns from excitatory neurons (Tremblay et al., 2016). Third, real inhibition can act in both a subtractive and a divisive manner (Atallah et al., 2012; Wilson et al., 2012; Seybold et al., 2015; Pouille et al., 2013), which may provide important functionality.
Given these considerations, in DANNS, we utilize a separate pool of inhibitory neurons with a distinct, more biologically realistic connectivity pattern, and a mixture of subtractive and divisive inhibition (Fig. 1). This loosely mimics the fast feedforward subtractive and divisive inhibition provided by fast-spiking interneurons in the cortical regions of the brain (Atallah et al., 2012; Hu et al., 2014; Lourenço et al.. 2020). In order to get DANNs to learn as well as standard ANNs we also employ two key insights:
1. It is possible to view this architecture as being akin to normalisation schemes applied to the excitatory input of a layer (Ba et al., 2016; loffe Szegedy, 2015; Wu & He, 2018), and we use this perspective to motivate DANN parameter initialisation.
2. It is important to scale the inhibitory parameter updates based on the Fisher information matrix. in order to balance the impact of excitatory and inhibitory parameter updates, similar in spirit to natural gradient approaches (Martens, 2014).
Altogether, our principle contribution is novel architecture that obey's Dale's principle, and that we show can learn as well as standard ANNs on machine learning benchmark tasks. This provides the research community with new modelling tool that will allow for more direct comparisons with real neural data than traditional ANNs allow, but which does not suffer from learning impairments. Moreover, results have interesting implications for inhibitory plasticity, and provide means for future research into how excitatory and inhibitory neurons in the brain interact at the algorithmic level.
2

---

Published as a conference paper at ICLR 2021
2 BIOLOGICALLY INSPIRED NETWORKS THAT OBEY DALE'S PRINCIPLE
2.1 MODEL DEFINITION
Our design for DANNs takes inspiration from the physiology of feedforward inhibitory microcircuits in the neocortex and hippocampus. Based on these circuits, and an interpretation of layers in ANNs as corresponding to brain regions, we construct DANNs with the following architectural constraints:
1. Each layer of the network contains two distinct populations of units, excitatory and an inhibitory population.
2. There are far fewer inhibitory units than excitatory units in each layer, just as there are far more excitatory neurons than inhibitory neurons (~ 5-10 times) in cortical regions of the brain (Tremblay et al., 2016; Hu et al., 2014).
3. As in real neural circuits where only the excitatory populations project between regions, here only excitatory neurons project between layers, and both the excitatory and inhibitory populations of a layer receive excitatory projections from the layer below.
4. All of the synaptic weights are strictly non-negative, and inhibition enforced via the activation rules for the units (eq. 1).
5. The inhibitory population inhibits the excitatory population through a mixture of subtractive and divisive inhibition.
This constrained architecture is illustrated in Figure 1.
h
h
W
8-1
h
WIE
WEL 2-1
WIE
WEI
h
h/
Figure 1: Illustration of DANN architecture. Lines with arrow ends indicate excitatory projections. Lines with bar ends indicate inhibitory projections, which can be both subtractive and divisive.
Formally, we define the network follows. Input to the network is received vector of positive scalar values x E IRq which we consider to be the first excitatory population. Each hidden layer, is comprised of a vector of excitatory units h, E IR and inhibitory units hJ E IR in-line with constraint (1) above. (We will drop the layer index when it is unnecessary for clarity.) Note, for the first layer (e = 1), we have h, = x and n4e d. Next, based on constraint (2) we set ne >> n, and use 10% inhibitory units default. Following constraint (3), both the excitatory and inhibitory units receive inputs from the excitatory units in the layer below (ha-1), but the inhibitory units do not project between layers. Instead, excitatory units receive inputs from the inhibitory units of the same layer. In-line with constraint (4), we have three sets of strictly non-negative synaptic weights, one for the excitatory connections between layers, WEE E RR Xn. one one for the excitatory projection to the inhibitory units W!E E R xn, and one for the inhibitory projections within layer WEI e R*.x*. xn, Finally, per constraint (5), we define the impact of the inhibitory units the excitatory units as comprising both a subtractive and divisive component:
h f(zz)
Z:
g
E
Wo
h!) + B
7l
where zf
WWE
h
h = fl(2t) f'(!BBhe-1)
7 = o h!)
3

---

Published as a conference paper at ICLR 2021
where for each layer f, Be E R is a bias, It R. controls the gain, "Ee is the divisive inhibitory term, and ag E R is a parameter that controls the strength of this divisive inhibition. Here @ denotes elementwise multiplication (Hadamard product) and the exponential function and division are applied elementwise. In the rest of this manuseript we set f to be the rectified linear function (ReI.U). Though a ReL.U function is not perfect match to the input-output properties of real neurons, it captures the essential rectification operation performed by neurons in physiologically realistic low activity regimes (Salinas & Sejnowski, 2000). In this paper, we model the inhibitory units as linear (i.e. fz(z) = zz) since they receive only positive imputs and have no bias, and therefore their activation would always be in the linear part of the ReLU function. Although we make make this modelling choice mainly for mathematical simplicity, there is some biological justification, as the resting membrane potential of the class of fast-s interneurons most related to our model is relatively depolarised and their spike outputs can follow single inputs one-to-one (Hu et al., 2014; Galarreta & Hestrin, 2001). In future work, for example in which inhibitory connections are included between inhibitory units, we expect that the use of nonlinear functions for inhibitory units will be important.
3 PARAMETER INITIALISATION FOR DALE'S ANNS
In biology. excitation and inhibition are balanced (Isaacson & Scanziani, 2011). and we use this biological property to derive appropriate weight initialisation for DANNs. First we initialise excitatory parameters from an exponential distribution with rate parameter XE, WEE Hd Exp(X*), and then inhibitory parameters are initialised such that excitation and subtractive inhibition are balanced, i.c. B[=E] E[( (( )x}, Ak. This can be achieved in number of ways (see appendix C.2). In line with biology, we choose to treat excitatory weights onto inhibitory and excitatory units the same, and sample WT sid Exp(A") and set W <- 1/ng. We note that for DANN layer with a single inhibitory neuron, e.g. at an output layer with 10 excitatory neurons, the noise inherent sampling a single weight vector may result in a poor match between the excitatory and inhibitory inputs, so in this case we initialise WIB as 1 Σi EE explicitly (where E-EE the jth row of EE ).
J.
Next, we consider the relationship between this initialisation approach and normalisation schemes (Ba et al., 2016; loffe & Szegedy, 2015). Normalisation acts to both center and scale the unit activities in a network such that they have mean zero and variance one. The weight initialisation given above will produce centered activities at the start of training. We can also draw a connection between the divisive inhibition and standardisation if we assume that the elements of x are sampled from a rectified normal distribution, x fid max (0, (O, ())) )). Under this assumption, the mean and standard deviation of the excitatory input are proportional (see Appendix D). For example, if we consider the relationship e- [R[=]] Var(=B)1/2 for each unit k we get the scalar proportionality constant as:
/2n 1/Vd.
B[=E] = d - E
EE [E [c:]
Var(z E d Var w EE
(B[x²)) Var(z))
= d E(w
EB
02-
= d Var(w w J0?
EE
2T
(2)
2xx
2T
with expectation over the data and the parameters, and where EE x refer to any element of WEE x. Therefore, since E|W- Var BB)(WBB) for weights drawn from an exponential distribution, we have
Var(= B[=B]
V2T Vd
c
This proportionality means that you can perform centering and standardisation operations using the same neurons. For DANNs, exx will dictate the expected standard deviation of the layer's activation z, as it controls the proportionality between subtractive and divisive inhibition for each inhibitory unit. If e is set to c, then the divisive inhibition approximates dividing F by its standard deviation, as E[=E] c E(wB! BI
R. (e 3 =+] = E[7n]- We note that due to the proportionality between the mean and standard deviation of EF other values of exx will also control the layer's variance with depth. However, given these considerations, standardisation at initialisation. We find that these initialisation schemes enable DANNs to learn well. We next turn to the question of how perform parameter updates in DANNS in order learn well.
initialise e +- v2m I/vd, thereby achieving
4

---

Published as a conference paper at ICLR 2021
4 PARAMETER UPDATES FOR DALE'S ANNS
Unlike a layer in a column constrained network, whose affine function is restricted by sign constrained columns, layer in a DANN is not restricted in its potential function space. This is because excitatory inputs to a layer can still have an inhibitory impact via feedforward inhibition. However, the inhibitory interneuron architecture of DANN layers introduces disparities in the degree to which updates to different parameters affect the layer's ouiput distribution. This can be seen intuitively, for example if a single element of WTF is updated, this has an effect on each element of Similarly, an update El will change depending on the alignment of x and all of the jth inhibitory unit's weights. Therefore, instead of using the euclidean metric measure distance between parameter settings, we employ alternative approach. Similar to natural gradient methods, we use approximation of the Kullback-Leible divergence (KL divergence) of the layer's 's output distribution for our metric. In order to help ensure that both excitatory and inhibitory parameter updates have similar impacts the KL divergence, we scale the updates using correction terms derived below. We provide an extended derivation of these scaling factors in the Appendix E.
to way
Given a probability distribution parameterized by some vector 0, a second order approximation to the KI. divergence for change to the parameters 0 is
DKL [P(y)x;)0) P[y]x; 0 + 6)] ~ F(O)5
(4)
2
Oog P(y|x; x; 0) θ log P(y x; 0)
F(O)= =
E xxx.
(5)
d0
00
Where F(0) is the Fisher Information matrix (or just the Fisher). In order to calculate the Fisher for the parameters of a neural network, we must interpret the network's outputs in a probabilistic manner. One approach is to view a layer's activation
parameterising a conditional distribution from the natural exponential family P(y x; 0) = P(y (z), independent in each coordinate of y i (similar to a GLM, and as done in Ba et al. (2016)). The log likelihood of such a distribution can be written as?
y y.2. z - 17())
log Pyy|x: 0)
+ c(y,, x)
(6)
E y 0 = fzz = (z) Cov(ylx: 0) = diag(of(z))
(7)
where f(z) is the activation function of the layer, and db, 7, define the particular distribution in the exponential family. Note that here we are taking (z) and fz( to denote
dn and of d%
Uh respectively.
In our networks, we have used softmax activation functions at the output and ReLU activation functions in the hidden layers. In this setting, the log likelihood of the output softmax probability layer would only be defined for one-hot vector y and would correspond to ó 1, c(y, Φ) o, and 77(z) = log(E, For the ReLU activation functions, the probabilistic model corre- sponds to tobit regression model, in which y is censored observation of latent variable y ~ (2,, (rlg((ff(z))) In this case, one could consider either the censored or pre-censored latent random variable, depending on modelling preference. As fits well with the above framework we analyze the pre-censored random variable ý, i.e. f(z) = z in equation 6. Returning to the general case, where we consider layer's activation as parameterising a conditional distribution from the natural exponential family, the fisher of layer is:
Oz (y (z)) (y - r2())) T Oz T
F(0) x
E
(8)
00
00
Oz diag( (f(()) Oz T
E Pxx)
(9)
00
00
Note the general form of the exponential family is log P(y x) xT(())-(()) + c(y, 6). but here we only
consider distributions from the natural exponential family, where Tyy) y. as this includes distributions of interest for us, such as Normal and Categorical, and also common distributions including Exponential, Poisson, Gamma, etc.
5

---

Published as a conference paper at ICLR 2021
To estimate the approximate KL divergence resulting from the simple case of perturbing individual parameter 0 0 of single-layer DANN, we only need to consider the diagonal entries of the Fisher:
DKL Po+6,
8
E P(x)
F(z))(
Ozk 00
(10)
2Φ
where 63 represents 1-hot vector corresponding 9 multiplied by scalar d. We now consider the approximate KL divergence after updates single element of W W W and o::
8² E
DKL. DKI_[Po Po-
&
(zx
(11)
2Φ
DKL [Po
Po+ów!R!
~
8²
Σ E
f'(2z)
(
9k
(w ags)
(12)
2Φ
78
R
82 d
DKL Po+a
~
Σ
E
(2))
9i
xn) Pwu ais) )2))
(13)
2Φ
%
7
8²
+
E
((zx))
-i Ix Fm W 3h Wi (ats)² 7/
IE
2
DKL [Po Po+s..
82
E
f' (z())) 7/k
Ik: x2,²
El LE w Wij (ati - 1)²
(14)
26
h:
+
8
The
Σ
d
((x))
9h 1² xm w (w B1y2 ( (axi k 1)²
26
nym
8. (= (WBL21) + and expectations over the data, Pxx)
Where 0kj 7k
Therefore, as a result of the feedforward inhibitory architecture of DANNs, for parameter update 6, the effect on the model's distribution will be different depending on the updated parameter-type. While the exact effect depends on the degree of co-variance between terms, the most prevalent differences between and within the excitatory and inhibitory parameter-typer are the sums input and output dimensions. For example, an inhibitory weight update of to wiz change the model distribution approximately n. times EE In order to balance the impact of updating different parameter-types, we update DANN to Wi
layer wil is expected to than excitatory weight update of 5
parameters after correcting for these terms: updates WE were scaled by vne E by d-1 and by (dy/ne)-1 As result, inhibitory unit parameters updates are scaled down relative to excitatory parameter updates. This leads to an interesting connection to biology, because while inhibitory neuron plasticity is well established, the rules and mechanisms governing synaptic updates are different from excitatory cells (Kullmann & Lamsa, 2007; Kullmann et al., 2012), and historically interneuron synapses were thought be resistant to long-term weight changes (MeBain et al., 1999).
Next, we empirically verified that our heuristic correction factors captured the key differences between parameter-types in their impact on the KI. divergence. To do this we compared parameter gradients before and after correction, to parameter gradients multiplied by approximation of the diagonal of the Fisher inverse for each layer (which we refer to as Fisher corrected gradients), see Appendix F.3. The model was trained for 50 epochs on MNIST, and updated using the Fisher corrected gradients. Throughout training. we observed that the heuristic corrected gradients were more aligned to the derived correction
Fisher corrected gradients than the uncorrected gradients were (Fig. 2). Thus, factors help to balance the impact of excitatory and inhibitory updates on the network's behaviour. Below, we demonstrate that these corrections are key getting DANNs to learn well.

---

Published as a conference paper at ICLR 2021
Cosine similarity
0.8
Uncorrected
Corrected
20
30
Epoch #
Figure 2: Empirical verification of update correction terms: Cosine of the angle between gradients multiplied by an approximation of the diagonal of the Fisher inverse for each layer, and either uncorrected gradients (black) or corrected gradients (orange) over 50 epochs. Plot displays a moving average over 500 updates
5 EXPERIMENTAL RESULTS
Having derived appropriate parameter initialisation and updates for DANNs, we now explore how they compare to traditional ANNs and ColumnEi models on simple benchmark datasets. In brief, we find that column constrained models perform poorly, failing even to achieve zero training-set error, whereas DANNs perform equivalently to traditional ANNs.
5.1 IMPLEMENTATION DETAILS
All models were composed of layers: in general hidden layers of dimension 500 with ReLU activation function followed by a softmax output with 10 units, and experiments were run 50 epochs with batch size 32. Unless stated, for DANNs and ColumnEi models, 50 inhibitory units were included per hidden layer. For DANN models, the softmax output layer was constructed with one inhibitory unit. For ColumnEi models, each hidden layer's activation is Wx where 500 columns of W were constrained to be positive and 50 negative (therefore for ColumnEi models h, was of dimension 550). ColumnEi layer weights were initialised that variance did scale with depth and that activations were centered (see Appendix C.I for further details). All benchmark datasets (MNIST, Kuzushiji MNIST and Fashion MNIST) were pre-processed so that pixel values were in [0, 1]. Learning rates were selected according to validation eтrоr averaged over 3 random seeds, after random search (Orion; Bouthillier et (2019), log uniform [10, 1e-5], 100 trials, 10k validation split). Selected models were then trained on test data with 6 random seeds. Plots show mean training error per epoch, and mean test set error every 200 updates over random seeds. Tables show final error mean + standard deviation. For further implementation details and link to the accompanying code see Appendix F.
Note that because our goal in this paper is not to achieve state-of-the-art performance, did not apply regularisation techniques, such as dropout and weight decay, or common modifications to stochastic gradient descent (SGD). Instead the goal of the experiments presented here simply to determine whether, in the simplest test case scenario, DANNs can learn better than ColumnEi models and as well as traditional ANNs.
5.2 COMPARISON DANNs TO COLUMN-EI MODELS AND MLPS
Training Error
Test Error
2
0
10
20
30
40
50
0
10
20
30
40
50
Epoch
Epoch
MLP
DANN
DANN nc
DANN 1i
ColumnEi
Figure 3: Model comparison on MNIST dataset. ne no update corrections, one inhibitory unit

---

Published as a conference paper at ICLR 2021
We first compared model performance the MNIST dataset (Fig 3). We observed that ColumnEi models generalised poorly, and failed to achieve training error within the 50 epochs. This confirms the fact that such models cannot learn well as traditional ANNs. In contrast, we observed that DANNs performed equivalently to multi-layer perceptrons (MLPs), and even generalised marginally better. This was also the case for ColumnEi and DANN models constructed with more inhibitory units (Supp. Fig. 6, 100 inhibitory units per layer). In addition, performance was only slightly worse for DANNS with one inhibitory unit per layer. These results show that DANN performance generalizes to different ratios of excitatory-te cecitatory-toinnibiiie in-iiiiiioooy units. We also found that not correcting parameter updates using the corrections derived from the Fisher significantly impaired optimization, further verifying the correction factors (Fig 3).
Next, we compared DANN performance to ML.Ps trained with batch and layer normalization on more challenging benchmark datasets (Fig 4). Again we found that DANNs performed equivalently to these standard architectures, whereas ColumnEi models struggled to achieve acceptable performance.
Training
Test
5.0
K-MNIST
10
Error
0.0
20
F-MNIST
10
0
10
20
30
40
50
10
20
30
40
Epoch #
Epoch #
MLP
LayerNorm
BatchNorm
DANN
ColumnEi
Figure 4: Model comparison on Fashion MNIST and Kuzushiji MNIST datasets.
We also explored methods for improving DANN performance (Appendix F.4). First, in order to maintain the positive DANN weight constraint, if after a parameter update a weight was negative, we reset it to zero, i.e. 0 <- max(0, 0), and as result the actual update is no longer that suggested by SGD. therefore experimented with temporarily reducing the learning rate whenever this parameter clipping would reduce the cosine of the angle made between the gradient and actual updates below certain constraint (see Appendix F.4). Second, we note that the divisive inhibition term, 7, appears in the denominator of the weight gradients (Appendix E.2) and, therefore, if becomes small, the gradients will become large, potentially resulting in inappropriate parameter updates. We therefore would be particularly effective for DANNs. We tested both of these modifications to DANNs trained on Fashion MNIST (Supp. Fig. 5). However, we found that they provided no observable improvement, indicating that the loss landscape and gradients were well behaved over optimization.
wondered if constraining the gradient
Finally, we provide an analysis and preliminary experiments detailing how the DANN architecture described above may be extended to recurrent and convolutional neural networks in future work (Appendix B). brief, we unroll recurrent networks over time and place inhibition between both network layers and timesteps, corresponding to fast feedforward and local recurrent inhibition, respectively. For convolutional architectures, we can directly apply the DANN formulation to activation maps if inhibitory and excitatory filters are of the same size and stride. Supporting this, found that a DANN version of VGG16 (Simonyan Zisserman, 2014) converged equivalently to standard VGG16 architecture (Supp.Fig.7) F)..
Altogether, our results demonstrate that: (1) the obvious approach to creating ANNs that obey Dale's principle (ColumnEi models) do not learn well as traditional ANNs, (2) DANNs learn better than ColumnEi models and as well as traditional ANNs, (3) DANN learning is significantly improved by taking appropriate steps scale updates in excitatory and inhibitory units appropriately.
8

---

Published as a conference paper at ICLR 2021
6 DISCUSSION
Here we presented DANNs, a novel ANN architecture with separate inhibitory and excitatory units. We derived appropriate parameter initialisation and update rules and showed experimentally that, unlike ANNs where some columns are simply constrained to be positive or negative, DANNs perform equivalently to traditional ANNs on benchmark datasets. These results are important as they are, as far as we know, the first example of an ANN architecture that fully adheres to Dale's i law without sacrificing learning performance. However, our results also raise an interesting question: why does nature employ Dale's s principle? After all, we did not see any improvement over normal ANNs in our experiments. There are two possible hypotheses. First, it is possible that Dale's principle represents an evolutionary local minima, whereby early phylogenetic choices led to constraints on the system that were difficult to escape via natural selection. A Alternatively, Dale's principle may provide some computational benefit that we were unable to uncover given the specific tasks and architectures used here. For example, it has been hypothesized that inhibition may help to prevent catastrophic forgetting (Barron al., 2017). We consider exploring these questions an important avenue for future research.
There are number of additional avenues for future work building upon DANNs, the most obvious of which are to further extend and generalize DANNs to recurrent and convolution neural networks (see Appendix B). It would also be interesting to explore the relative roles of subtractive and divisive inhibition. While subtractive inhibition is required for the unconstrained functional space of DANN layers, divisive inhíbition may confer some of the same optimisation benefits as normalisation schemes. A related issue would be to explore the continued balance of excitation and inhibition during optimization, because while DANNS are initialised such that these are balanced, and inhibition updated during training, and the
approximates normalisation schemes, the inhibitory parameters model is free to diverge from this initialisation. As a result, the distribution of layer activations may be unstable over successive parameter updates, potentially harming optimization. In the brain, variety of homeostatic plasticity mechanisms stabilize neuronal activity. For example, reducing excitatory input naturally results in reduction in inhibition in real neural circuits (Tien & Kerschensteiner, 2018). It would therefore be interesting to test the inclusion of a homeostatic loss to encourage inhibition to track excitation throughout training. Finally, we note that while fast feedforward inhibition in the mammalian cortex was the main source of inspiration for this work, future investigations may benefit from drawing on a broader range of neurobiology, for example by incorporating principles of invertebrate neural circuits, such as the mushroom bodies of insects (Serrano et al., 2013).
In summary, DANNs sit at the intersection of a number of programs of research. First, they are a new architecture that obeys Dale's 's principle, but which can still learn well, allowing researchers to more directly compare trained ANNs to real neural data (Schrimpf et al., 2018; Yamins al., 2014). Second, DANNs contribute towards computational neuroscience and machine learning work on inhibitory interneurons in ANNs, and in general towards the role of inhibitory circuits and plasticity in neural computation (Song et al., 2016; Sacramento et al., 2018; Costa et al., 2017: Payeur et al., 2020; Atallah et al., 2012; Barron et al., 2017). Finally, the inhibition in DANNS also has an interesting connection normalisation methods used to improving learning in deep networks (loffe & Szegedy, 2015; Wu & He, 2018; Ba et al., 2016). As DANNs tie these distinct programs of research together into a single model, we hope they can serve as a basis for future research at the intersection of deep learning and neuroscience.
ACKNOWLEDGEMENTS
We would like to thank Shahab Bakhtiari, Luke Prince, and Arna Ghosh for their helpful comments on this work. This work was supported by grants to BAR, including a NSERC Discovery Grant (RGPIN-2020-05I1 105)0 an Ontario Early Career Researcher Award (ER17-13-242), Healthy Brains, Healthy Lives New Investigator Start-up (2b-NISU-8), and funding from CIFAR (Learning in Machines and Brains Program, Canada CIFAR AI Chair), the Wellcome Trust and by the Medical Research Council (UK). Additionally, DK was supported by the FRQNT Strategic Clusters Program 2020-RR4-265502-UNIQUPP

---

Published as a conference paper at ICLR 2021
REFERENCES
Daniel J Amit, C Campbell, and KYM Wong. The interaction space of neural networks with sign-constrained synapses. Journal of Physics A: Mathematical and General, 22(21):4687, 1989.
Bassam V Atallah, William Bruns, Matteo Carandini, and Massimo Scanziani. Parvalbumin- expressing interneurons linearly transform cortical responses visual stimuli. Neuron, 73(1): 159-170, 2012.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450, 2016.
arXiv preprint
Helen C Barron, Tim P Vogels, Timothy E Behrens, and Mani Ramaswami. Inhibitory engrams in perception and memory. Proceedings of the National Academy of Sciences, 114(2)):666666677 2017.
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lilli- crap. Assessing the scalability of biologically-motivan deep learning algorithms and architectures. In Advances in Neural Information Processing Systems, pp. 9368-9378, 2018.
Sven Behnke. Hierarchical neural networks for image interpretation, volume 2766. Springer, 2003.
Xavier Bouthillier, Christos Tsirigotis, François Corneau-Tremblay Pierre Delaunay, Reyhane Askari, Dendi Suhubdy, Michael Noukhovitch, Dmitriy Serdyuk, Arnaud Bergeron, Peter Henderson, Pascal Lamblin, Mirko Bronzi, and Christopher Beckham. Orfon asynchronous distributed hyperparameter optimization, October 2019. URL https: /ttp:///oiii.// /ttp:///ooo.oog/1.22222/////oe 3478592.
Rui Costa, Ioannis Alexandros Assael, Brendan Shillingford, Nando de Freitas, and Tim Vogels. Cortical microcircuits as gated-recurrent neural networks. In Advances in neural information processing systems, pp. 272-283, 2017.
Peter W Donhauser and Sylvain Baillet. Two distinct neural timescales for predictive speech processing. Neuron, 105(2):385-393, 2020.
John Carew Eccles. From electrical chemical transmission the central nervous system: the closing address of the sir henry dale centennial symposium cambridge, 19 september 1975. Notes and records of the Royal Society of London, 30(2):219-230, 1976.
Mario Galarreta and Shaul Hestrin. Spike transmission and synchrony detection in networks of gabaergic interneurons. Science, 292(5525):2255-292) 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Hua Hu, Jian Gan, and Peter Jonas. Fast-spiking, parvalbumint gabaergic interneurons: From cellular design to microcircuit function. Science, 345(6196): 2014.
Alessandro Ingrosso and LF Abbott. Training dynamically balanced excitatory-inhibitory networks. PloS one, 14(8):c0220547, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03166; 2015.
Jeffry S Isaacson and Massimo Scanziani. How inhibition shapes cortical activity. Neuron, 72(2): 231-243, 2011.
Alexander JE Kell, Daniel LK Yamins, Frica N Shook, Sam V Norman- Haigner, and Josh H McDermott, A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630-64 644, 2018.
Tim Christian Kietzmann, Patrick McClure, and Nikolaus Kriegeskorte. Deep neural networks in computational neuroscience. BioRxiv, pp. 133504, 2018.
10

---

Published as a conference paper at ICLR 2021
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj. Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. In Advances in Neural Information Processing Systems, pp. 12805-12816, 2019.
Dimitri M Kullmann and Karri P Lamsa. Long-term synaptic plasticity in hippocampal interneurons. Nature Reviews Neuroscience, 899):687- 2007.
Dimitri M Kullmann, Alexandre W Moreau, Yamina Bakiri, and Elizabeth Nicholson. Plasticity of inhibition. Neuron, 75(6):951-92,, 2012.
Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint 1rXiv:660.0366400 2016.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, pp. 1-12, 2020.
Joana Lourenço, Angela Michela De Stasi, Charlotte Deleuze, Mathilde Bigot, Antonio Pazienti,
Andrea Aguirre, Michele Giugliano, Srdjan Ostojic, and Alberto Bacci. Modulation of coordinated activity across cortical layers by plasticity of inhibitory synapses. Cell reports, 30(3):630-644 2020.
James Martens. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1193, 2014.
James Martens and Roger Grosse. Optimizing neural networks with frotecker-factored approximate curvature. In International conference machine learning, pp. 2408-2417, 2015.
Chris J McBain, Tamas F Freund, and Istvan Mody. Glutamatergic synapses onto hippocampal interneurons: precision timing without lasting plasticity. Trends neurosciences, 22(5):228- 235, 1999.
Jonathan A Michaels, Stefan Schaffelhofer, Andres Agudelo-Toro, and Hansjörg Scherberger. A neural network model of flexible grasp movement generation. bioRxiv, pp. 742189, 2019.
Thomas Miconi. Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks. Elife, 6:e20899, 2017.
Sun Minni, Li Ji-An, Theodore Moskovitz, Grace Lindsay, Kenneth Miller, Mario Dipoppa, and Guangyu Robert Yang. Understanding the functional and structural differences across excitatory and inhibitory neurons. 2019.
Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J DiCarlo, and Daniel Yamins. Task-driven convolutional recurrent models of the visual system. In Advances in Neural Information Processing Systems, pp. 5290-529000 2018.
Christopher Parisien, Charles H Anderson, and Chris Eliasmith. Solving the problem of negative synaptic weights in cortical models. Neural computation, 20(6):: 2008.
Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake Richards, and Richard Naud. Burst- dependent synaptic plasticity can coordinate learning in hierarchical circuits. bioRxiv, 2020.
Frédéric Pouille, Antonia Marin-Burgin, Hillel Adesník, Bassam V Atallah, and Massimo Scanziani. Input normalization by global feedforward inhibition expands cortical dynamic range. Nature neuroscience, 12(12):1577, 2009.
Frederic Pouille, Oliver Watkinson, Massimo Scanziani. and Andrew J Trevelyan. The contribution of synaptic location to inhibitory gain control in pyramidal cells. Physiological reports, 1(5), 2013.
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22 1777777771111770, 2019.

---

Published as a conference paper at ICLR 2021
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, pp. 8721 2018.
Emilio Salinas and Terrence Sejnowski. Impact of correlated synaptic input on output firing rate and variability in simple neuronal models. Journal of neuroscience, 20(16);6 193-66699,,,, 2000.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy Kailyn Schmidt, al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007, 2018.
Eduardo Serrano, Thomas Nowotny, Rafael Levi, Brian H Smith, and Ramón Huerta. Gain control network conditions in early sensory coding. PLoS Comput Biol, 907)xe003333, 2013.
Bryan A Seybold, Elizabeth AK Phillips, Christoph E Schreiner, and Andrea R Hasenstaub. Inhibitory actions unified by network integration. Neuron, 87(6): 181-1192. 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
H Francis Song, Guangyu R Yang, and Xiao-Jing Wang. Training neural networks for cognitive tasks: a simple and flexible framework. PLoS computational biology, 12(2)=e1044792 2016.
recurrent
Nai-Wen Tien and Daniel Kerschensteiner. Homeostatic plasticity in neural development. Neural development, 13(1):1-7, 2018.
Robin Tremblay, Soohyun Lee, and Bernardo Rudy. Gabaergic interneurons in the neocortex: from cellular properties to circuits. Neuron, 91(2):260-292, 2016.
Nicolas Tritsch, Adam Granger, and Bernardo L Sabatini. Mechanisms and functions of gaba co-release. Nature Reviews Neuroscience, 17(3):139-145, 2016.
James CR Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in cognitive sciences, 2019.
Nathan R Wilson, Caroline A Runyan, Forea L Wang, and Mriganka Sur. Division and subtraction by distinct cortical inhibitory networks in vivo. Nature, 483(7411) 2012.
Yuxin Wu and Kaiming He. Group normalization. Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models understand sensory cortex. Nature neuroscience, 19(3):356, 2016.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performaace-optiiizeed hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 11 (11(2))86619 011-86611-6622 8624, 2014.
12

---

Published as a conference paper at ICLR 2021
SUPPLEMENTARY MATERIAL
A
SUPPLEMENTARY RESULTS
10
15
Training Error
Test Error
14
13
10
Epoch
Epoch #
DANIN grad
DANN scaling
Figure 5: DANNs trained on Fashion MNIST with gradient normalisation and learning rate scaling.
Training Enro
Test Error
3
2
0
10
20
30
400
50
10
20
30
40
50
Epoch
Epoch
#
MLP
Column 500
ColumnEI
Figure 6: Model comparison on MNIST dataset as in Fig. 3 but including models with 100 inhibitory units, ne no update corrections, #i no. inhibitory units, ColumnEi #e ti.
Table 1: MNIST results
Model
# inhib
Learning rate
Train error
Test enror
MLP
0
0.2976
0.0 + 0.0
1.44 + 0.034
DANN
0.05327
0.0 0.0
1.56 + 0.041
DANN
50
0.1107
0.0 0.0
1.325 + 0.066
DANN
100
0.3576
0.0 = 0.0
1.244 + 0.067
DANN ne
50
0,003981
0.293 + 0.039
2.167 + 0.113
ColumnEi
50
0.05273
0.662 + 0.071
3.035 + 0.231
ColumnEi
100
0.06533
0.574 + 0.092
2.857 + 0.08
13

---

Published as a conference paper at ICLR 2021
Table 2: K-MNIST results
Model
# inhib
Learning rate
Train error
Test enror
MLP
0
0.213
0.0 0.0
6.842 + 0.119
LayerNorm
0
0.1166
0.0 + 0.0
7.171 + 0.134
BatchNorm
0
0.7235
0.123 0.017
6.747 = 0.129
DANN
50
0.156
0.0 + 0.0
6.728 + 0.23
ColumnEi
50
0.05426
1.886 + 0.191
12.763 + 0.44
Table 3: Fashion MNIST results
Model
inhib
Learning rate
Train error
Test error
MLP
0.08403
1.793 = 0.119
10.626 + 0.466
LayerNorm
0.04743
2.136 + 0.076
10.445 + 0.455
BatchNorm
0.1098
1.104 +: 0.044
9.992 + 0.218
DANN
50
0.01973
3.832 + 0.031
10.962 0.365
ColumnEi
50
0.02265
12.365 + 0.217
14.986 = 0.674
B EXTENSION OF DANNS TO OTHER ARCHITECTURES
Here we discuss how our results and analysis of fully-connected feedforward Dale's ANNs may be applied to convolutional and recurrent neural networks.
B. EXTENSION TO CONVOLUTIONAL NEURAL NETWORKS
Consider the response of a standard convolutional layer of output channels with filters of size k x k at a single position j over m input channels:
2j Wx, + b
(15)
Here, W a x k2m matrix whose rows correspond the kernel weights of each output channel, and the vector of length k2mm contains the values over the n input channels for the spatial location i Concatenating each input location x, as the columns of matrix X, the full output of the convolutional layer over all input locations can be expressed as Z = WX + b. where b is broadcast over the columns of Z. We can readily make an equivalent DANN formulation for convolution layer by assuming the same kernel size and stride for excitatory and inhibitory filter-sets EEE and WEI,
(W Xj= WEI x,) + B,
Zj
(16)
7 = BE WlE
Here the inhibitory channels are mapped to each excitatory output channel by WE for subtractive inhibition, and are first scaled by ex for divisive inhibition. For parameter initialisation, by following the approach of He et al. (2015) and considering the response of the layer at a single location, we use the same initialisations as those derived in section but where the input dimension d is the product of kernel size and input channels, k.2m. Next, the correction factors to parameters updates apply in section the KL divergence is summed over each valid input location j, which results in approximately the same multiplicative factor for each parameter, but does not change the approximate relative differences between parameter types:
DKL Po Po+55]
~
Σ
E P(x)
(((na))
Pin )²)
(17)
20
20
where consider the full response Z of the layer over all valid kernel locations.
14

---

Published as a conference paper at ICLR 2021
In order to confirm extension to convolutional neural networks conducted preliminary experiments with DANN versions of convolutional neural networks as described above. Below, we show results of training a standard VGG16 architecture, and DANN version of the VGG16 architecture (Supp. Fig. 7) on CIFAR-10. As can be seen, the DANN network trains approximately well the standard VGG16 model.
80
80
Training Error
Test Error
100
100
Epoch #
Epoch #
DANN VGG18
Control VGG16
Figure 7: Convolutional network results on CIFAR-10 with control and DANN VGGI6 models. Plots show mean training and test set error 6 random seeds.
Both control and DANN VGG16 architectures were trained on CIFAR- 10 with stochastic gradient descent with batch size 128, without dataset augmentation, dropout, or batch normalisation layers. Best model learning rates (control 0.089, DANN 0.03458) were selected after random search according to average final validation error over random seeds, and conditional on all seeds beginning to converge within 5 epochs (convergence defined as validation eπrоr 90%). The random search was performed with learning rates sampled from a log-uniform [lc-4,1] distribution, seeds per trial, 60 trials, 1550 epochs, and with 10k validation split. Final epoch test error over 6 random seeds was 21.00 .08 + 0.811 for the control VGG16 model, and 21.178 + 0.348 for the DANN-VGG1 model with constrained weights. VGG16 models were adapted from code here? and for the DANN VGG16 model we used 10 inhibitory filters per 64 excitatory filters, and 10% inhibitory units in the fully connected layers.
B.2 EXTENSION TO RECURRENT NEURAL NETWORKS
We can readily make a connection between the fully-connected Dales ANNs described in Section 2.1 and recurrent neural networks (RNNs) by considering the similarities between depth and time. As has been previously noted, shallow RNN unrolled over time can be expressed as a deep neural network with weight sharing (Liao & Poggio, 2016).
g: 3 Wh-1t+ +B
h f(z))
(18)
7
where
W
WEE El BI
7: = BC W
where in this simple case, recurrent processing steps over time are applied to the input x = ho- In this view, layer depth corresponds time, and inhibition between layers corresponds to fast feedback inhibition.
We note that if there are sequence of inputs coming at each time-step, Tt, then this formulation can still hold, but with a simple modification to incorporate the time-varying inputs. Specifically, we need to add additional input weights, U:
h f(z))
2:
g 7
Wh
gx O UX +3 7x
(19)
15

---

Published as a conference paper at ICLR 2021
where
W W W W
7/ y= W (e"1 hi-1
= UEE - U
7x = (c @ x.)
All of the existing DANN approaches developed above can be applied to this case.
C PARAMETER INITIALISATIONS
In this section we provide further details regarding parameter initialisations.
Throughout we assume that the elements of the imput, x, to a layer are iid and also the output of layer 1 whose pre-activations were distributed (00, o2-1). ). Therefore r will follow a rectified normal distribution:
00
x² /20
dr 0t-1
E[a]
Tt-1V 12
2T
00
x² /2n
0
B[r²1
x
dx
0t-1
2m
(20)
π - 1
Var(x)
E[r²²]
2 E[x]²
d;-1
2TT
2x -
Var(x) + E[x]² = 02-1
2x
where here, and throughout the text, non-indexed non-bold to refers any element of vector or matrix, e.g [[ refers to the expectation of any element of x, Var(z) refers to the variance of any element of x, etc. In addition, for all models we draw positively constrained weights iid from exponential distributions, and make use of the following properties for w ~ Exp(A)
1
E [w]
Ā
Var(w)=
X = E[w]]
(21)
E E[w?]
2Vrr(w) = 2E[w]²
A
C.I COLUMN CONSTRAINED EI MODELS AND WEIGHT INITIALISATION
Here we provide detail on the parameter initialisation of column constrained models. Layer activations are z = Wx where columns of W are constrained to be positive or negative. Therefore, for convenience, let denote W [W+ W-], and Vi,j. Note for this model, The + nι = d, the input dimensionality. As for DANN models, throughout training we preserve the sign constraints of the weights by resetting weights using rectification around zero, i.e. W+ <- max(0, W+), W- <- min(0, W-).
xx),'), and assume x," x! are iid
At initialisation for the column constrained model for each layer we require B[2x] 0,
Var(=)) = 02_1-
E E[2a]
1
[[x]
n.
(w
[[
n1, .. w
[[x]]
n.
[[r]
(22)
E ww
Elwrt
n;
16

---

Published as a conference paper at ICLR 2021
Where w w0 refer to any element of W W
ne
Var(za) = Σ Var(waa + Σ Var(w
Var(w 2)) + Var(w x))
(23)
(B/w ( + Var(w BB(a] + Var(w (ar(a)) + (n,(Blu 13 (ar + Var(w [x]]] + Var(w Var(z))
As weights are drawn from an exponential distribution, Var(w = Bl we have
Var(z*) = n. t (2Var((1)) [[x]*) + n (2Var(() P + E[x}²)
= n. + Var (z) + n,,[u ( Var(z))
= (B[n²]+ + Var x n E w + n
(24)
2x
n2 + n
= =/-11
El
2x
Therefore E[w=]] 1/( )
2n
Note that as the input to the network is all positive, the first weight matrix has no negative columns. We therefore use the bias vector to center the activations of the first layer (in other layers it is initialised to zeros).
[[ n. B..uut+EE + B
(25)
Therefore we initialise all elements of 3 to -w [B[r]]
C.2 INITIALISATION OF DANN INHIBITORY WEIGHTS FOR BALANCED EXCITATION AND SUBTRACTIVE INHIRITION
Here provide details of inhibitory parameter initialisation such that E[2 EE sid
([ Bl
22)x), for
Ex)
1
E E
E
W
EE ks
d
A B[[]]
(26)
d
E (( 21
)b]
E
Wkk
EI
10 J2
n.
El
dd [w
[[(r]
These expectaions are equal when both sets excitatory weights are drawn from the same distribution, IE sid Exp(>F) and FI 1 1/ng- Or alternativlly. inhibitory weights can both drawn from
the same distribution, WIB WE iid Exp(v/Enn Note, that although the above always holds in expectation, in the case of multiple inhibitory units we can apply the law of large numbers to conclude that the subtractive inhibition and excitatory input will be approximately equal.
Note that while this initialisation is general to different settings of E we initialise /(TT - 12 (see section D.
17

---

Published as a conference paper at ICLR 2021
D PROPORTIONAL RELATIONSHIP BETWEEN EXCITATORY INPUT MEAN AND STANDARD DEVIATION
Here we provide further details regarding the proportionality between z mean and standard deviation. This proportionality constant depends on which statistic or distribution that of interest for activations (c.g. layer-statistics or unit batch-statistics in layer and batch normalisation).
D.1 UNIT STATISTICS OVER DATA AND PARAMETER DISTRIBUTIONS
As discussed main text, if we consider e. over the data and parameters, c 2 1/vd:
B[=E] Var(= for a unit k with expectation
B d Efwe F.F.
E[21
d Elw BB
T6-1
2t
Var(= Var
xx))
EE
Wki
(27)
d Var (w BB
= Var w BE
E[xx] + d E/o
= d Var w BE
(E[x²] + Var(z))
2
d Var (w EE J
2m
Where E2 Var(w for weights drawn from an exponential distribution. Therefore
RR=E) c
Var(=
(28)
2T
c
Vd
Additionally, that for Var(2 are drawn from should be
0 the variance of the distribution that elements of EE
2T
Var(w EE)=
(29)
d. 12T 1) and so can set XE +- 1(2 for Var(=
of-
D.2 UNIT STATISTICS OVER THE DATA DISTRIBUTION
instead we consider unit k, with excitatory weights WEE and expectation and variance taken only over the data we have the approximation:
E[2]
W
RJ d. TE [R. EE
(30)
d 0i-1 E(w
2
Likewise the variance over the data can be approximated
Var(= = Var(x)
(wER) EB52
(31)
Rd. d. Var(x) El(w R
= d o/-1 2
2 E(u 2
18

---

Published as a conference paper at ICLR 2021
Therefore
E Ex-p(a)
-C= Va
V2T 2 c
(32)
Vd
D.3. LAYER STATISTICS OVER THE DATA AND PARAMETER DISTRIBUTIONS
Alternatively we can consider the mean and standard deviation of the layer statistics H2R, Ox as calculated if one was to apply layer normalisation z Here again, these statistics are proportionally related, but with the constant v
If we were to apply layer normalisation to = the layer statistics would be as follows:
g 2x
=2x)++++
UsZ
Σ Ne
1
No
E 5
EE J:
1
E
M2
)2
11g
11
(33)
We now derive the relationship that the expectation layer statistics are proportionally related by Eluga] vTTVvd = - 1/2 The expectation of E Elpa=] is straightforward:
Eluz=] = d [[[ [[x]]
(34)
Turning to the derivation 1E(02E):
E
E
122 )
(35)
T1e
1
BE w x ne
Σ w
EE 2
(36)
The
1
(22))
(37)
where we have defined & = Wi:: Var(2,). As
F.
En
The
w
E. x.
We can obtain [[({)) by deriving B[2] and
Ti
Wij
w.j
FE
Wkj
FE
(w
wkj
F.F.
(38)
The
The
k=
we see that [[ 0, and therefore B[2]] 0. For the variance of Var(2:) we start with Var(1;;)
The
Var(w;;))
Var(
EE (w
wkj
EEE)
(39)
n
k=1,oti
nhe
nC
Var(w
w
EE
Cov(w
Wkj W
EE
w
k2)
(40)
1,
Rm 1, k. 44,, Aigh
((n. 1)2 L) + (n. - 1)(re - 2) (ar(
(41)
(ne 1) Var
(42)
Th
19

---

Published as a conference paper at ICLR 2021
For S ne calculate Var(2,, keeping in mind that for i I ne, S d, 2, iid, and equation (38) shows that úhj are iid the j 'tth coordinate, so we see that
Var(2.) Var Σ wiززر)) = d(ar((ux))
(43)
Remembering the values of E(ib), Var(w)), that E = Var(X) + E[X]?, and for independent X,Y, Var(XY)= Var(X)) Var(X) Var(Y))B[X] we have
Var(2,) = d(Var Var((>) + Var E[x] + (ar())
(44)
d( ( + Var xz) B [w]3) = d(Var(0)BBx>
(45)
Var u BE B[x²"] d(n. 1)
(46)
The
Now putting these terms together we can derive Elo
1 ne
[o
2
EE [(()))
(47)
The
Var (2))
(48)
The
d Var EB). -[x²"
(49)
Therefore returning to E[uaa]-c- = (B ) 1/2 and keeping in mind that the variance of an exponential random variable is it's mean squared,
(d Var w BE E 1///2
/ x
(50)
d. B EE
E[r)
Vd
E(ar
We have assumed that follows rectified normal distribution. Therefore, E[x] of
Ø--1 <2n
E[x?]
Resulting in:
C= V Vd
(51)
We note that for DANN layer with single inhibitory unit, MgB = = as W -
Σ EE
and WEI 1. Therefore DANN divisive inhibition, 7, can be made equivalent to layer standard deviation at initialisation in expectation if ex <- However, these calculations apply for the case of multiple interneuron one makes the approximation AgE & (l)EE) for any i.
20

---

Published as a conference paper at ICLR 2021
E PARAMETER UPDATES AND FISHER INFORMATION MATRIX
E. LAYER FISHER INFORMATION MATRIX
We view a layer's activation parameterising conditional distribution from the exponential family P(y|x;0) = P(y (2), independent in each coordinate y z.
log P(y[x: 0) y 7z)) + cly の)
(52)
E y 0 = = = (z) Cov(y|x: 0) = diag (
(53)
where fz) is the activation function of the layer, and Φ, 1) c define the particular distribution the exponential family. Note we take vy [" to denote the 0y af Un oz
F(0) is defined as:
log P(y|x; 0) log P(y x; 0) 00
F(0)
E P(y
(54)
00
As
θz à
y n(2)
log P(y|x; 0)
cy.,o)
00
00
(55)
1 Oz
On
(y
3 00
Oz
we have
Oz (y (z)))) (y - (z))
Oz T
F(0)
E
(56)
P(x) Plyl)y-ppPPyyyy xx
00
00
Oz
(y - (2)) (y (2))
Oz 7
E P(x)
E
x; 0
(57)
00 y (xy(x:0)
00
Oz Cov
[y x; 0
Oz T
E Pxx)())
(58)
00
d²
00
Ōz diag( f (z)) Oz" T
E Px>((x)
(59)
00
00
where we recognise the covariance matrix is diagonal:
Cov [y|x; 0] = diag(Vart (ar(y1|x; 0)..., Var [x: 0))
To analyse the approximate KL divergence resulting from the simple case of perturbing individual parameters of single-layer DANN, only need to consider the diagonal entries of the Fisher.
Oz diag(I
Oz 7
DKL. [Po Po+aø]
R
2
E x- ((x)
00
Φ
00
5
(60)
82
On
Oz T
E
diag(f (z))
(61)
26 PxxxP())
00
00
82 E
dz Oz T
(f(
(62)
20 2xxxP(xx)
2 00
82
Ozh 2
E
f'(zz)
(63)
20 Pxx(
30
21

---

Published as a conference paper at ICLR 2021
where 63 represents 1-hot vector corresponding to 6, multiplied by a scalar d.
E.2 DERIVATIVES
Here we provide derivatives for DANN layer activations with respect to the different parameter groups. The equations for the layer activation can be written
g 3 W z
-z +
7
(64)
where
z = WEE z z'=WlEx y=
W
BB (z)Oz
azk Ou
0zk
Note
0 for l: + i.
Oz, ow EE ()
Di (=)
E
(WEI2!)) + Bi)
(65)
EW EE
7
1j
9i
0
(66)
%i
BE
9i
(67)
V
02:
9i (z.
(WF12 ):) + 3.)
(68)
ET
Tw EI
9i
07: (2 (WWll)),
9i 2]
(69)
73 Ow
El
7
g: a
(2 F (WE22)))
21 2]
(70)
?
7
Di
I
(2)
E
(WEIZ!), + 1)
(71)
7i
Di
TE
2
c05
(2F
FW + 1)
(72)
%
7s
Ozy ET Ou
Ozy
In contrast
+ for 1: #
daz
azk Fm!!
gk
(z
E - EWEL 2 )a)) + +Bx
(73)
Tw
7kk
9k d7
(2k
E (WEIZ!) )n)
9k
(74)
2 iw
7k
Ih c
w
(2 - ( Aw (122)) )b0
gk gk Ela x
(75)
9k
El
a.
(2b (W 1)
(76)
7kk
7/6
22

---

Published as a conference paper at ICLR 2021
0zi
9i (24 -(W"lz) ).) + B.)
(77)
Dag
daj
Di
0% E (WBlZ))
(78)
7 daz
9i El
wij
2 (WElz))
(79)
7
d
Wij
El
jk
IE
_ (2F (WElz)))
(80)
7i
k
02.
(2)
E
(W El z ){)
(81)
0g.
7
02.
(82)
ab.
(83)
E.3 APPROXIMATE KI. DIVERGENCE FOR WEIGHT UPDATES
we consider an update to an element ij of EE the approximate KL divergence is
82 Th
Ozk 2 Suuf
DKL [o Po+ Pa+swmmn
E 2Φ Pxx)
Σ
('(x)(
(84)
82
((x))(
E
20 x~ P(x)
73
as Du.
0 for + i.
Oza In contrast, for update to element ij of WIE we n. terms, Du!!
+ 0 for 1: +
82 n
Ozk 8'(zx)
DkI. [Pol || Potswag]
2Φ
pxx)
8² ne
f((x)
gk
w x xki)) 3
EI
(85)
2Φ xxx
7/
8² The
Σ
Ih f'(x)( (w ani)?
26 xx)(
7x
Ea
where 0kj
(2f (WE12)))) +
7ik
23

---

Published as a conference paper at ICLR 2021
dzg (
For an update owe, while a squared sum over d terms.
for A the derivative contains a 2j term, so there instead
8² Re
Ozk ET awl
DKL [Po
Po+owai
E
(zx)
2Φ
P(x)
8²
E
(z))
Ozi ET Duly
)
2Ø x~ P(x)
82
9i f((2)
2jss)) 2
E
P(xx
7
8²
E
(x)))))) )2 (((as)) 1
20x~
8²
E
f(()) Σ
IE v )2 ( ( (
20 P(x)
71
8²
E
f(())
18
(x )² + Σ
jjn jm In x
IE IE
(
26 P(x)
T
82
gi (a:z)² (w (sm
E
2Φ
xx~
82
E
((2)) 2 2n
9i (ass)) )
26 mom
xxx)
7/
(86)
Finally, for alpha
82
f'(z)) Ozk 2
)zk)2
DKL Pol Potd...
~
Σ
E x
2Φ
Das
82
E
f'(zw) -
d
9k
Wks Wi x; - (2t - EI 2()) 2
ea:
2
P(x)
7ik
8
E
f((z)( Σ
gk w w ij
x (aka 1))²
2Φ
P(x)
%
8
Re
d
E
S'(2x) Σ
9k
Wkk W x. (axs - 1))
2Φ
X P(x)
7kk
Σ
((
gk El 2 w xn im am (aki 1)2)
Wki
7
nytmm
82 Re
d
f(xx)(
gk
w w x, (aps - 1))²]
2Φ
xxP())
8 12
Ok w f((x)(
) ? xn ti
u I' (aks 1)²
2Φ
xxx(
7k
(87)
24

---

Published as a conference paper at ICLR 2021
F ALGORITHMM
Here we provide pseudo-code for implementaioon details. Please see the following link for code: http:://gihnbb.cm//inncddd////
F.1 PARAMETER INITIALIZATION
Algorithm Parameter initialization for DANNs
for layer L do require The n4ps WEE ~ exp(( A) n4
WIE
ΣE 2== W
EE
WEI, t
else:
WI ~ exp(1²)
W + 1/n2,
end
1 log
225I
Vd
g. 1 end for
Where number of excitatory output units is ne, number of inhibitory units n,, and input dimensionality and N dd - T)/v 2n.
F.2 PARAMETER UPDATES
For DANN parameter updates we used the algorithms detailed below. Note that gradients were corrected detailed in Section 4 and Algorithm 3.
All below algorithms are computed using the loss gradients V0 of parameter 0, in given model computed minibatch sample.
Algorithm 2 Parameter updates
Require learning rate updates AO
for each layer do
WF EE + W 7A WEF
WE IE 1/ IE
WEI +- WEI 7 WEI
<- AAa
g 1dg
- 3 - 1AB
WFE mar FF 0)
WIE + mar WIF 0)
WI mar W 0)
f- mar(g.
end for
F.3 DANN GRADIENT CORRECTION ALGORITIIMS
For the majority of experiments we scaled gradients using the heuristic correction terms derived in Section (and see Appendix E). In this case applied the following algorithm before Algorithm 2.
25

---

Published as a conference paper at ICLR 2021
Algorithm 3 DANN gradient correction
for each layer / do
require ne. Rii d
AWEE + VWEE
AWIE - The VWIE
A AWEI + -V
Aa + n.
Va
Ag Vg
AB + VB
end for
We also tested that our heuristic correction factors approximated gradient multiplication by the diagonal of F,- for each layer (see Figure 2).
Algorithm 4 Gradient correction by approximation of diag(F
Require learning rate 7). fisher momentum k, fisher learning rate
for each batch (za, 3/4 do
Compute p = softmax
Compute cross entropy loss L(p, yt)
for each layer do 0L
V
a0t
end for
Sample I Categorical(p)
Compute cross entropy loss [p,, 0) for each layer do
F' b-btccc
Σ
|batchl AL 12
00
F. = kF + (1 1)) Ft-1
F. 1 1/FF;
Fr F. F.WEE where F FWBE
the elements of F corresponding to EFF
AO' + F VO
end for
end for
Here we note this update can considered very rough diagonal approximation to natural gradient descent. In addition, various efficient approximations to natural gradient descent that have been utilized such as KFAC Martens & Grosse (2015) could not be applied due to the structure of DANNs, the mathematical assumptions of KFAC, which were made for feedforward networks with activations as matrix multiplications, do not apply.
F.4 LEARNING RATE SCALING AND GRADIENT NORMALISATION
We also tested whether constraining the gradient norm and scaling the learning rate based on parameter clipping improved DANN performance. For these experiments we applied the following algorithms.
Algorithm 5 Gradient normalisation
for each layer & do Require ~ M
1 la > M:
a + M 4
else:
A +- V
end if
end for
26

---

Published as a conference paper at ICLR 2021
Algorithm 6 Learning rate scaling
for each layer é do
Require V0 M, E
i+1
c 0
while M: x &
c t CosineSimilartyymaxx(,. -1/00),) 0 v)
i + i+1
end for
The learning rate scaling method temporarily reduces the learning rate whenever parameter clipping would reduce the cosine of the angle, made between the gradient and actual updates, below certain constraint. For any optimization problems caused by actual clipped updates not following the gradient, learning rate scaling is a principled way of following the direction of the gradient.
We also note, this technique can be generally applied to any other model which is constrained so that it cannot have updates freely follow gradient descent. If the constrained parameter space is open subset of euclidean space, and we allow the learning rate to be arbitrarily small (Algorithm 6 with lim; & 0), updates will always follow the direction of the gradient.
27


Reviews:
[{"review_id": "hVaJNmU49No", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Inspired by the observations of feedforward inhibition in the brain, the authors propose a novel ANN architecture that respects Dale’s rule (DANN). They provide two improvements for training DANNs: better initialization and update scaling for synaptic weights. As a result, they empirically demonstrate that DANNs perform no worse than the ANNs that do not respect Dale’s rule.\n\nAlthough, I find the contribution interesting, my enthusiasm is tempered by the following two issues:\n\n1.\tAlthough feedforward inhibition has its place in the brain, most connections of inhibitory interneurons with excitatory neurons are reciprocal, resulting in feedback inhibition. Therefore, feedforward inhibition seems like a secondary factor here.\n\n2.\tThe DANNs are shown to be just no worse than ANNs that do not respect Dale’s rule. If biology “invested the effort” to evolve inhibitory interneurons respecting Dale’s rule, this is probably because they confer a computational advantage, not just lack of disadavantage. \n\nThe formulation of Dale’s rule on page 1 is not consistent with the current biological knowledge. A better version would be: “A neuron releases the same fast neurotransmitter at each of its pre-synaptic terminals”. Note that this does not mean that the action of a neuron is always excitatory or always inhibitory on all of its post-synaptic partners. It is possible, as often the case in invertebrates, that different post-synaptic partners have different receptors resulting in de- or hyper-polarization in different post-synaptic neurons. \n  \nAlthough the paper is generally well written, the authors could make it clearer. In particular, it would help if they defined symbols such as the circled dot or variables such as y when they are first used.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UOaNyAsFi3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: It is shown that Dale’s principle can be observed in feedfoward ANNs if one uses inhibitory neurons in the form of feedforward inhibition, while the other neurons are purely excitatory.\n\nPros: This is a nice and new insight. It appears to be useful for understanding the design of biological neural networks, and at least one type of uses of inhibitory neurons in them.\n\nCons:  Apparently this insight provides no benefit for designing ANN. Furthermore the biological insight is rather limited because biological neural networks are not feedforward networks. Also, the chosen tasks (3 variations of MNIST) are relatively simple, and are solved with relatively shallow networks, with just 4 hidden layers. In my view this evaluation does not support the much more general claim in the Abstract that „ANN’s that respect Dale’s principle can be built without sacrificing learning performance“.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PXCX5Vt_W_B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Most neurons in the brains are either excitatory (E) or inhibitory (I) - sometimes referred to as Dale’s law.  Practically Dale’s principle is often left out of Artificial Neural Networks (ANNs) because having the E and I separation often impairs learning, although this has not been well documented in the literature (probably due to that this is also interpreted as a negative result). In this paper, the authors propose a new scheme to construct and train the feedforward E/I network by incorporating several ingredients, including feedforward inhibition and E/I balance among others. It is shown that this particular kind of E/I networks (DANNs) trained on MNIST and variations of MNIST could achieve a level of performance that is comparable to those without E/I separation.\n\nQuality: I think this is an interesting submission of good quality, with some novel ideas and promising preliminary results. \nClarity: The writing is generally clear.\nOriginality: As far as I can tell, the results are original.\nSignificance: Although the results are promising, I have reservations about the significance of these results as the performance of the models are still worst than the standard ANNs.\n\nPros:  \n1.To my knowledge, this is the first E/I network that could achieve comparable performance with the standard ANN model on MNIST task (although at the same time, I have to say that not too many papers have studied and reported this issue).\n2. The ingredients in the proposed model is well motivated in neuroscience, such as the feedforward inhibition, and E/I balance, as well as no connections between I neurons across the different layers.\n3.   The results on the MNIST and its variations look promising. \n4.  The paper is fairly well written and the basic ideas are clear. \n\nCons: \n1.The role of the subtractive and divisive components need to be better explained. Are both of them necessary for getting the results shown later?\n2. The authors assume the number of E neurons is far larger than that of the I neurons. This is not quite true in physiology. The E/I ratio reported is often around 4:1. The authors assumed 10% of neurons are I neurons- this is on the smaller end. Another related concern is that, in cortex, despite of a smaller number, I neurons are often responsible for controlling the dynamics/computation due to the dense connectivity from I to E neurons. I am a little bit worried that the paper is studying a quite different regime, in which the E neurons are dominating. Also, would adding more I neurons decrease the performance of the network? If that is the case, that would be concerning.\n3. The initialization of E/I network has been carefully studied previously in the context of training balanced E/I recurrent neural networks (e.g., Ingrosso & Abbott, 2019, which the authors cited). How does the authors scheme different from the previous work?\n4. The method assumes inhibitory units are linear units. Several questions arise. First, is this a mathematical issue or a numerical issues? Second, does this imply the firing rate of inhibitory neuron can be both positive and negative?\n5. In fig4, DANN performs significantly worse than LayerNorm and BathNorm.\n6.The algorithms is not tested on slightly more challenging benchmark datasets such as CIFAR10 or ImageNet. Relatedly, would DANN scale up to larger networks?\n\nQuestions to be clarified:\n*Are their connections between the I neurons within the same layers?\n*page 4, “Unlike a column constrained network, a layer in a DANN is not restricted in its potential function space. “ - It is unclear what this sentence means…\n*Between Eq 4 and Eq 5, the authors mentioned the exponential family. What particular distribution was used? Gaussian or any exponential family distribution would produce similar results?\n*The authors wrote: “As a result, inhibitory unit parameters updates are scaled down relative to excitatory parameter updates. This is intriguing given the differences between inhibitory and excitatory neuron plasticity…including the relative extent of weight changes in excitatory and inhibitory neurons (McBain et al., 1999). ” I think these comparisons to the neuroscience literature are too vague and potentially mis-leading. To make this useful, it would helpful to make the comparison more specific and clear. \n*I am worried that the experiments for the ColumnEi model was not treated fairly. In section 5.1, it is mentioned that 50 columns are negative. Did the authors try to make increase this number to see if the performance would be improved for the ColumnEi model?\n\n\n*********updated after rebuttal period\nI still consider this as an interesting contribution, and stand with my original rating. \nIt would be useful if the discrepancies and similarity between the connectivity structures in the model and the anatomy could be more carefully discussed in the paper.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "10hN6uh2oaE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a great investigation on how to scale the gain of the inhibitory weights to balance the impact that the changes that the excitatory and inhibitory connections have on the layer’s output. I think using the KL distance that naturally connects with the Fisher Information is neat. I appreciate the effort that the authors make to connect the manner neural circuits are designed and connect it with ANN. You never know when the breakthrough can arise.\n\nI love the experiments that the authors present illustrating with clarity the impact that having the proper gain modulation of the inhibitory changes have in the speed of convergence.\n\nMy single constructive criticism is that the inspiration in cortical circuits do not prevent the authors to get inspiration from smaller neural circuits like in insects for example. The Mushroom Bodies of the insects are the equivalent of the cortex and present feedforward inhibition. The number of layers is much smaller but the neural principles that operate are fairly consistent across multiple animal species. Drawing from that experience, the mutual inhibition within layer may provide a natural mechanism to keep balance in the output distribution as shown for example in mean field models that investigate the regulation of activity in a dynamical neural layer (see for example https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003133). \n\nOther that this comment I learn and enjoy from reading this paper. I think it should be accepted.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was unanimously rated above the acceptance threshold by the
reviewers.  While all reviewers agree it is worth accepting, they
differed in their enthusiasm.  Most reviewers agree that  major
limitations of the paper include that the paper provides no insight into why
Dale's principle exists and the actual results are not truly
state-of-the-art.  Nevertheless there is agreement that the paper
presents results worth publicizing to the ICLR audience.  The comparison
of the inhibitory network to normalization schemes is interesting.
Also, please reference the Neural Abstraction Pyramid work.


Decision: Accept (Poster)