Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Inspired by the observations of feedforward inhibition in the brain, the authors propose a novel ANN architecture that respects Dale’s rule (DANN). They provide two improvements for training DANNs: better initialization and update scaling for synaptic weights. As a result, they empirically demonstrate that DANNs perform no worse than the ANNs that do not respect Dale’s rule.

Although, I find the contribution interesting, my enthusiasm is tempered by the following two issues:

1.	Although feedforward inhibition has its place in the brain, most connections of inhibitory interneurons with excitatory neurons are reciprocal, resulting in feedback inhibition. Therefore, feedforward inhibition seems like a secondary factor here.

2.	The DANNs are shown to be just no worse than ANNs that do not respect Dale’s rule. If biology “invested the effort” to evolve inhibitory interneurons respecting Dale’s rule, this is probably because they confer a computational advantage, not just lack of disadavantage. 

The formulation of Dale’s rule on page 1 is not consistent with the current biological knowledge. A better version would be: “A neuron releases the same fast neurotransmitter at each of its pre-synaptic terminals”. Note that this does not mean that the action of a neuron is always excitatory or always inhibitory on all of its post-synaptic partners. It is possible, as often the case in invertebrates, that different post-synaptic partners have different receptors resulting in de- or hyper-polarization in different post-synaptic neurons. 
  
Although the paper is generally well written, the authors could make it clearer. In particular, it would help if they defined symbols such as the circled dot or variables such as y when they are first used.


review_title: Showing that Dale's principle does not hurt the performance of feedforward ANNs does not illuminate its computational purpose   

final_score: rating: 6: Marginally above acceptance threshold
review_content: Most neurons in the brains are either excitatory (E) or inhibitory (I) - sometimes referred to as Dale’s law.  Practically Dale’s principle is often left out of Artificial Neural Networks (ANNs) because having the E and I separation often impairs learning, although this has not been well documented in the literature (probably due to that this is also interpreted as a negative result). In this paper, the authors propose a new scheme to construct and train the feedforward E/I network by incorporating several ingredients, including feedforward inhibition and E/I balance among others. It is shown that this particular kind of E/I networks (DANNs) trained on MNIST and variations of MNIST could achieve a level of performance that is comparable to those without E/I separation.

Quality: I think this is an interesting submission of good quality, with some novel ideas and promising preliminary results. 
Clarity: The writing is generally clear.
Originality: As far as I can tell, the results are original.
Significance: Although the results are promising, I have reservations about the significance of these results as the performance of the models are still worst than the standard ANNs.

Pros:  
1.To my knowledge, this is the first E/I network that could achieve comparable performance with the standard ANN model on MNIST task (although at the same time, I have to say that not too many papers have studied and reported this issue).
2. The ingredients in the proposed model is well motivated in neuroscience, such as the feedforward inhibition, and E/I balance, as well as no connections between I neurons across the different layers.
3.   The results on the MNIST and its variations look promising. 
4.  The paper is fairly well written and the basic ideas are clear. 

Cons: 
1.The role of the subtractive and divisive components need to be better explained. Are both of them necessary for getting the results shown later?
2. The authors assume the number of E neurons is far larger than that of the I neurons. This is not quite true in physiology. The E/I ratio reported is often around 4:1. The authors assumed 10% of neurons are I neurons- this is on the smaller end. Another related concern is that, in cortex, despite of a smaller number, I neurons are often responsible for controlling the dynamics/computation due to the dense connectivity from I to E neurons. I am a little bit worried that the paper is studying a quite different regime, in which the E neurons are dominating. Also, would adding more I neurons decrease the performance of the network? If that is the case, that would be concerning.
3. The initialization of E/I network has been carefully studied previously in the context of training balanced E/I recurrent neural networks (e.g., Ingrosso & Abbott, 2019, which the authors cited). How does the authors scheme different from the previous work?
4. The method assumes inhibitory units are linear units. Several questions arise. First, is this a mathematical issue or a numerical issues? Second, does this imply the firing rate of inhibitory neuron can be both positive and negative?
5. In fig4, DANN performs significantly worse than LayerNorm and BathNorm.
6.The algorithms is not tested on slightly more challenging benchmark datasets such as CIFAR10 or ImageNet. Relatedly, would DANN scale up to larger networks?

Questions to be clarified:
*Are their connections between the I neurons within the same layers?
*page 4, “Unlike a column constrained network, a layer in a DANN is not restricted in its potential function space. “ - It is unclear what this sentence means…
*Between Eq 4 and Eq 5, the authors mentioned the exponential family. What particular distribution was used? Gaussian or any exponential family distribution would produce similar results?
*The authors wrote: “As a result, inhibitory unit parameters updates are scaled down relative to excitatory parameter updates. This is intriguing given the differences between inhibitory and excitatory neuron plasticity…including the relative extent of weight changes in excitatory and inhibitory neurons (McBain et al., 1999). ” I think these comparisons to the neuroscience literature are too vague and potentially mis-leading. To make this useful, it would helpful to make the comparison more specific and clear. 
*I am worried that the experiments for the ColumnEi model was not treated fairly. In section 5.1, it is mentioned that 50 columns are negative. Did the authors try to make increase this number to see if the performance would be improved for the ColumnEi model?


*********updated after rebuttal period
I still consider this as an interesting contribution, and stand with my original rating. 
It would be useful if the discrepancies and similarity between the connectivity structures in the model and the anatomy could be more carefully discussed in the paper.



review_title: An interesting submission on training ANNs with E/I neuronal division

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: It is shown that Dale’s principle can be observed in feedfoward ANNs if one uses inhibitory neurons in the form of feedforward inhibition, while the other neurons are purely excitatory.

Pros: This is a nice and new insight. It appears to be useful for understanding the design of biological neural networks, and at least one type of uses of inhibitory neurons in them.

Cons:  Apparently this insight provides no benefit for designing ANN. Furthermore the biological insight is rather limited because biological neural networks are not feedforward networks. Also, the chosen tasks (3 variations of MNIST) are relatively simple, and are solved with relatively shallow networks, with just 4 hidden layers. In my view this evaluation does not support the much more general claim in the Abstract that „ANN’s that respect Dale’s principle can be built without sacrificing learning performance“.


review_title: Dale's principle may not reduce the performance of feedforward ANNs if one uses negative weights only for feedforward inhibition.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This is a great investigation on how to scale the gain of the inhibitory weights to balance the impact that the changes that the excitatory and inhibitory connections have on the layer’s output. I think using the KL distance that naturally connects with the Fisher Information is neat. I appreciate the effort that the authors make to connect the manner neural circuits are designed and connect it with ANN. You never know when the breakthrough can arise.

I love the experiments that the authors present illustrating with clarity the impact that having the proper gain modulation of the inhibitory changes have in the speed of convergence.

My single constructive criticism is that the inspiration in cortical circuits do not prevent the authors to get inspiration from smaller neural circuits like in insects for example. The Mushroom Bodies of the insects are the equivalent of the cortex and present feedforward inhibition. The number of layers is much smaller but the neural principles that operate are fairly consistent across multiple animal species. Drawing from that experience, the mutual inhibition within layer may provide a natural mechanism to keep balance in the output distribution as shown for example in mean field models that investigate the regulation of activity in a dynamical neural layer (see for example https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003133). 

Other that this comment I learn and enjoy from reading this paper. I think it should be accepted.

review_title: Gain modulation of inhibitory feedforward inhibition

Meta-review: This paper was unanimously rated above the acceptance threshold by the
reviewers.  While all reviewers agree it is worth accepting, they
differed in their enthusiasm.  Most reviewers agree that  major
limitations of the paper include that the paper provides no insight into why
Dale's principle exists and the actual results are not truly
state-of-the-art.  Nevertheless there is agreement that the paper
presents results worth publicizing to the ICLR audience.  The comparison
of the inhibitory network to normalization schemes is interesting.
Also, please reference the Neural Abstraction Pyramid work.


Decision: Accept (Poster)