Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The idea behind the paper is quite interesting and enticing. The authors attempt to emulate real-life human brain neural network structure with artificial neural networks similar in architecture to the visual cortex.
The authors claim that they observe similar patterns emerge in the Neural Networks when it comes to the interaction and topological properties of excitatory and inhibitory neurons.

Unfortunately, the paper's claims seems to be undermined by several factors. First, the properties of the artificial neural networks that the authors build into the model (excitatory/inhibitory neurons relative abundance, inter-layer projection) are likely to be sufficient to explain the properties authors claim are informative in their model.

In addition to that, some architectural choices by the authors are highly disputable and counter-intuitive. For instance ReLU activation for Neurons would be contradicting any biological reality.

Besides, to be biologically innovative, the authors' approach seems to miss a number of crucial factors - notably interaction between neurotransmitters, timing of discharges and synapses offsets, allowing single neurons to perform complex logical computations requiring artificial neural networks with simple scalar non-linear activation functions multi-layer sub-networks to learn.

The case the authors cite - similarity of architectures of convolutional neural networks and the pinwheel architecture of visual cortex V1/V2 areas, is one of the few examples of convergent architectures of real-world and artificial neural networks specializing in a task.

Finally, the role of Inhibory neurons in the visual cortex seems to be
well understood, both biologically and mathematically (see for instance https://www.sciencedirect.com/science/article/pii/S0896627303003325 or https://www.sciencedirect.com/science/article/pii/S0896627303003325 or https://www.sciencedirect.com/science/article/pii/S092842570300072X)

Overall, the quality of the article at least in its current state, does not seem to be ready for acceptance to ICLR, but I'm willing to adjust my opinion after reading the opinion of more qualified reviewers' in this area and the authors response.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: [Update after rebuttal period]

I would like to thank the authors for the detailed response and for addressing the concerns on such a tight schedule.

Overall my two concerns appear to have been right: 1. The object classification task is not really relevant to elicit the observed behavior and 2. Inhibitory neurons are not essential (at least when training with batch norm). 

Do the conclusions of the paper about understanding of E/I cells in the brain still hold? I am not really sure. I would like to urge the authors to carefully assess their conclusions again in light of the new evidence and perhaps rephrase abstract, intro and discussion where necessary.

Having said that, I still like the paper and its approach. I think it provides valuable insights to the field (e.g. neuro-inspired architecture design, concerns regarding training objectives, etc.) and important steps in the right direction and could be a valuable contribution even if the original claims are not entirely substantiated.


[Original review]

This paper seeks to understand why excitatory neurons in cortex are more stimulus selective than inhibitory cells and why excitatory cells are more sparsely connected to each other. The authors train a recurrent neural network model with a number of biological constraints on CIFAR-10. These constraints include Dale's law, an unequal number of E vs. I cells and only excitatory connections between layers (areas) of the network. I would summarize the main findings as follows:

- Excitatory principal neurons are more selective and more sparsely connected to each other than local inhibitory neurons, consistent with biology

- Stimulus selectivity and performance depend only on the number of excitatory cells, but neither on the E/I ratio nor the number of I cells

- High selectivity appears to be a general property of principal cells, but does not depend on the sign of local connections, while sparse connectivity is mainly linked to the excitatory local connections


Strengths:
+ Architectural decisions are well motivated from a neuroscientific perspective
+ Provides a hypothesis why certain connectivity and selectivity patterns emerge in the brain by incorporating biological knowledge into neural network models trained on a specific task
+ Well written and clear, logic development of the arguments

Weaknesses:
- Unclear whether classification task is necessary to elicit the authors' observations
- Interneurons seem unnecessary, raising concerns about relevance of results
- Also trains on ImageNet, but only some results are shown; most from CIFAR


Overall I like the paper from the perspective of a neuroscientist, as it provides a kind of normative account of why things in the brain might look the way they do. My enthusiasm is somewhat limited, however, because I am not convinced the classification task is actually what drives the authors' observations. I am willing to increase my score and argue more strongly for the paper if the authors can address this concern, detailed in the following:

A few observations lead me to believe the classification task the networks are trained may not be important to elicit the observed behavior. 

1.) The emergent properties in stimulus selectivity show up almost immediately after training starts according to Fig. 5A+B. Performance, on the other hand, takes a few more epochs to pick up according to Fig. 10. 

2.) The connectivity preferences that emerge in Figure 5C appear at already 50 epochs where CIFAR-10 performance is at 50%. To put that into perspective, a linear SVM already achieves 40% on this task, and a one-layer multilayer perceptron with only 100 hidden neurons reaches that performance. 

3.) The results of Fig. 11 suggest that the number of inhibitory channels is not important. Did the authors try a stripped down version with only excitatory cells?

I suggest the authors train on CIFAR-10 with shuffled labels on the training set [Zhang et al. 2016; https://arxiv.org/abs/1611.03530]. Although performance on the test set would be at chance (no object recognition capabilities), it would be interesting to see whether the selectivity and connectivity properties still emerge.

Finally, (related to 3.) I wonder whether it makes sense to draw conclusions about differences between E and I cells from a model trained on a task where the number of inhibitory cells seems irrelevant. Wouldn't one need to have at least both types of neurons to be required for the task in order to draw such conclusions?


Minor Comments:

- I sometimes found the nomenclature of the multiple models you tried a bit confusing and hard to follow — especially in parts of the Appendix

- The type of operations (convolution, element-wise) in equation 1, together with the meaning of nonlinearities like \sigma_c are only defined one page after, making that section a bit hard to follow. 
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper explores why the brain might have separate excitatory (E) and
inhibitory (I) cells and how their different properties affect their
network connectivity.  The paper is novel and addresses an interesting
question.

The paper follows Dale's law and has separate populations of E and I
cells.  The cells differ in proportion (4 times more E cells) and in
projection patterns (E cells project to the next layer).  From these
constraints they found that some of the other observed differences
between E and I cells developed, specifically greater selectivity of E
cells and greater connectivity of I cells.

Both subtractive and divisive inhibition are modeled in the Standard model.

Through careful controls, they show that increased selectivity of E
neurons depends on having a lot of E neurons (but there was not a
corresponding effect for I neurons).  Likewise the sparse connectivity
of E neurons was due to a large number of E neurons.  They also showed
that being projection neurons was more important than being E cells
for strong selectivity.  Finally networks without Dale's law imposed
perform slightly better.

These are all interesting findings that will benefit the field of
neuroscience but may also lead to insights for the next architecture
boost to improve deep learning.  I am in favor of acceptance.




review_title: Official Blind Review #3

Meta-review: This paper explores the role of excitatory and inhibitory neurons, and how their properties might differ based on simulations.  A few issues were raised during the review period, and I commend the authors for stepping up to address these comments and run additional experiments.  It seems, though, that the reviewer's worries were born out in the results of the additional experiments: "1. The object classification task is not really relevant to elicit the observed behavior and 2. Inhibitory neurons are not essential (at least when training with batch norm)."  I hope the authors can make improvements in light of these observations, and discuss their implications in a future version of this paper. 
Decision: Reject