Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Dale's law is the idea that (most) neurons express only a single neurotransmitter, making them excitatory or inhibitory but not both. This paper studies a few models of neural networks constrained to follow Dale's law and contrasts them with networks where the neurons are free to form excitatory and inhibitory connections. The main results are that "Daleian" networks can generally produce nearly the same output as "non-Daleian" ones. They also present better robustness to noise in the weights and can more easily learn distributions of responses with biologically-plausible rules.
strengths_and_weaknesses: Strengths: The paper is very well-written. A couple of places in the margin I wrote "interesting" or "good". This is an important neuro-AI area of study and a significant difference between how networks are modeled by computer scientists and the biological reality. The figures are clear and look nice. Code is available for reproducibility.

Weaknesses: 

The study is limited to only certain kinds of network models. Most notably, this work does *not* study artificial neural network models and tasks prevalent in machine learning, so some of the results might be misleading or confusing to the NeurIPS community. In particular, the only models used are a max-entropy style statistical model of recurrent dynamics and a simple dynamical RNN, both linear (in the max-ent model the sufficient statistics are linear) in the previous system state (in contrast to nonlinearities used for machine learning and present in real brains). Furthermore, the stimuli are assumed to be static vectors (eqn 1). The networks don't really have to solve any kind of task, as far as I can tell, except to reproduce certain distributions. These model choices are minimally motivated in the text, despite standing out to me as not standard compared to the work I know.

My confidence in the results is lower because it feels like a lot of details were relegated to the supplement, which I did not have time to read.

Small points by line

1: "implies" seems like wrong word

2: "number of possible architectures" this sentence struck me as imprecise, is this a new (minor) result? If so present as such. 

89: "whose entries are all larger than zero" is not obvious, explain

100: $sign(i)$ is incorrect notation, all indices $i > 0$, what you mean is there is a label $s_i \in \pm 1$ for each neuron $i$

106: "100 ms" is arbitrary, you make no connection to a timescale in the text, the only mention of a step size I see is in Fig 1b, which was hard to find

181: Introducing this new model seems to require a new section. I had a hard time distinguishing between models 1 & 2 throughout the results, as the paper switches back and forth. Be explicit throughout.

183: add comma after "Here" in "Here neural activity"

207: "In learning and optimizing their function" is awkward

questions: Why did you choose the models that you did?

If you account for weight values (i.e. as real numbers), what can you say about the measure of Daleian networks vs non-Daleian?

What were the distributions of stimuli and responses that you tested and how do those choices influence the performance of Daleian/not networks?

Sec 2.3: Is studying robustness via the Hessian valid if the signs of W are constrained? Some perturbations could break Dale's law.

Sec 2.4: Please explain the task more clearly. How is it similar/different from a machine learning task? Where does the gradient come from and what 

Sec 3: Dale's law certainly *has* been studied in the machine learning community before, as recognized in the introduction. However, the discussion reads like this past work didn't occur. Discuss in more detail how your results fit into previous work. 
limitations: The models seem narrow to me, can the authors better motivate them and better discuss their limitations?

The last paragraph of section 3 discusses limitations much too briefly, expand.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper empirically studies the computational benefits of Dale's law of spiking NNs and rate-based NNs. tivity
strengths_and_weaknesses: Strength:
1) The paper studies an important puzzle in computational neuroscience, why the biological neural nets contain the signs of output weights and what could be the computational implications. The paper is well written and all central claims are well supported by abundant experimental evidence.
2) The functional comparison of random and learned Daleian and non-Daleian NNs is a novel analytical tool to probe the computational consequence of computational constraints.
3) Sensitivity analysis of synaptic weights shows a surprising robustness benefit of Daleian NNs.

Weakness:
Lack of theoretical justifications to better understand
- when does a non-Daleian NN hard to be approximated by a Daleian NN
- why is a Daleian NN less sensitive to weight perturbations
- why does non-Daleian neuronal learning perform worse than Daleian neuronal learning
questions: 1) line 141, gradient descent doesn't keep the sign constraints of Daleian networks. Did you use the gradient projection?
2) Figure 4b, hard to compare the D and nD distributions. It looks that nD (syn) has a lower accuracy than D (syn), which is inconsistent with Figure 4c? 
3) Random Daleian NNs are more "informative", doesn't that depend on stimuli distribution?
limitations: The authors mentioned limitations in network size and the assumption of weight distribution. It's also worth noticing that there are more excitatory neurons than inhibitory neurons in the cortex, but in this work they are balanced.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers networks that abide the Dale's law: a neuron can be either excitatory or inhibitory, but not both simultaneously. In experiments, they show that Daleian networks can approximate non-Daleian ones very well, are more robust to noise and are easier to train using fewer parameters. 
strengths_and_weaknesses: ### Strengths
1. The experimental results are important for theoretical neuroscience.
2. The paper is well-written and easy to follow. 

### Weaknesses
1. No theoretical analysis of the Daleian approximation and increased noise robustness of Daleian networks.

### Summary
I give the score 8 (strong accept) as the paper addresses an important questions in theoretical neuroscience, showing previously unknown benefits (at least to my knowledge) of Dale's law.
questions: ### Main questions
I guess the biggest take away for me: we can use models that violate Dale’s law, because there’s always a similar model that doesn’t. Which brings out two questions: 
1) Wouldn’t the Dale law abiding learning algorithms be different? The answer is probably yes, see Clark et al. 2021 (https://proceedings.neurips.cc/paper/2021/hash/532b81fa223a1b1ec74139a5b8151d12-Abstract.html). I think it's worth discussing in more detail. 

2) If you take a trained deep network and converted it into a Daleian one (e.g. using a subset of training data, making sure the accuracy didn’t change much), would it produce more brain-like responses (see http://www.brain-score.org/ and their papers)? I’m fine with this being in the discussion, but it’d be great to have it in the results.

The authors should mention why a theoretical analysis of the Daleian approximation is hard (because it’s non-convex, I assume).

Sec. 2.4 is a really interesting result. Maybe it would be possible to run the same experiment for at least CIFAR10 and a small conv net? (Not for the rebuttal deadline, but a comment on that would be appreciated).

### Small corrections
Line 32: “directed” is usually used to describe graphs, rather than networks. Maybe say asymmetric or remove it altogether?

Maybe Eq. 2 can include the definition of D_JS

Lines 167-168: unfinished sentence. …how capable Daleian networks are?

Line 192: move to an equation (not inline)

Line 306: citation needed on log-normal distribution (although I agree it’s widely known)
limitations: I think the paper adequately addressed the limitations and potential negative impacts.
review_title: null

Meta-review: metareview: This paper examines the impact of Dale's principal from neuroscience on neural network computation. Dale's principle says that neurons only release a single neurotransmitter type from their axons, which in principle, means neurons are either excitatory or inhibitory, but not both. Using both spiking and rate-based recurrent networks, the authors show that networks that respect Dale's principle can recapitulate the same computations as those that do not while exhibiting greater robustness to noise. This provides some account for why Dale's principle may provide actual benefits to neural computation.

The reviewers agreed that this paper is well-written and addresses an important question. The decision to accept was unanimous.
confidence: Certain
Decision: Accept