The computational and learning benefits of Daleian neural networks
Authors: Adam Haber
Elad Schneidman
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural coding
dale’s principle
spiking neural networks
recurrent neural networks
neural architecture
learning
information coding

Abstract: Dale’s principle implies that biological neural networks are composed of neurons that are either excitatory or inhibitory. While the number of possible architectures of such Daleian networks is exponentially smaller than the number of non-Daleian ones, the computational and functional implications of using Daleian networks by the brain are mostly unknown. Here, we use models of recurrent spiking neural networks and rate-based ones to show, surprisingly, that despite the structural limitations on Daleian networks, they can approximate the computation performed by non-Daleian networks to a very high degree of accuracy. Moreover, we find that Daleian networks are more functionally robust to synaptic noise. We then show that unlike non-Daleian networks, Daleian ones can learn efficiently by tuning of single neuron features, nearly as well as learning by tuning individual synaptic weights. Importantly, this suggests a simpler and more biologically plausible learning mechanisms. We therefore suggest that in addition to architectural simplicity, Dale's principle confers computational and learning benefits for biological networks, and offer new directions for constructing and training biologically-inspired artificial neural networks.

The computational and learning benefits of Daleian neural networks
Adam Haber
Elad Schneidman Department of Brain Sciences
Department of Brain Sciences Weizmann Institute of Science
Weizmann Institute Science
Rehovot, Israel adam. www.hhbbereeezzann.... ac
Rehovot, Israel
eladdschheeldnann@eeiiimmaa
Abstract
Dale's principle implies that biological neural networks composed of neurons that are either excitatory or inhibitory. While the number of possible architectures of such Daleian networks is exponentially smaller than non non-Daleian ones, the computational and functional implications of using Daleian networks by the brain models of recurrent spiking neural networks
mostly unknown. Here, and rate-based based networks to show, surprisingly, that despite the structural limita- tions on Daleian networks, they can approximate the computation performed by non-Daleian networks to a very high degree of accuracy. Moreover, we find that Daleian networks are more functionally robust synaptic noise. We then show that unlike non non- -aneiaa networks, Daleian ones can learn efficiently by tuning single neuron features, nearly well as learning by tuning individual synaptic weights suggesting a simpler and more biologically plausible learning mechanism. We thus suggest that in addition architectural simplicity, Dale's 's principle confers compu- tational and learning benefits for biological networks, and offers new directions for constructing and training biologically-inppin artificial neural networks.
Introduction
The synaptic connectivity of neurons in the brain is shaped by variety of physical, chemical, and biological factors [1, 2}, even before learning comes into play. For example, the location of neurons and their morphology induce dissadee-dependent connectivity [3], whereas molecular lock-and-ker mechanisms induce cell-type specific connectivity patterns [4]. The functional and computational
implications of such constraints from the architecture of connectomes cell types are fundamental to our understanding how real neural circuits are built and function.
particularly stringent constraint the design of biological neural networks imposed by Dale's principle, which asserts that the axonal branches of a neuron release the same set of neurotransmitter molecules [5, 6). While some exceptions to Dale's principle have been reported [7-10], the implied identity of neurons as either excitatory or inhibitory has been central to our characterization and understanding neural cireuits' structure and function [11-14]. Moreover, the balance between excitatory and inhibitory populations has been at the core of many computational models of the design of neural circuits and their function [15].
Dale's 's principle has far-reaching implications for the architecture of neural circuits: For directed networks of N neurons, there N (N - 1) possible synapses, and so for the general case in which each synapse can be either excitatory, inhibitory, non-existent there are 3 NNN-N11) possible signed networks connecting the neurons. However, only (2 2 - 1) N of these networks abide by Dale's principle (see Supplementary Information, SI). Thus, already for 15 neurons, non-Daleian networks outnumber the Daleian ones by a factor of ~ 30 The staggering difference in richness of synaptic
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

connectivity maps that non-Daleian networks offer compared to Daleian ones, and their potentially higher plasticity (the ability to change synaptic signs learning) suggest that non- mon-Daleian networks may be able to carry computations that Daleian networks cannot. However, Daleian networks are clearly simpler terms of biological design and development, since the set of neurotransmitters and receptors expressed by each neuron is smaller, compared to the expected requirements of regulation of connections in non- non-Daleian ones. Moreover, since synapses cannot change their sign during learning in Daleian networks, they are limited to significantly smaller "search space", which might limit learning but could make it easier for Daleian architectures to find networks that perform a desired function. The obvious question is then: what is the brain "missing" by imposing Dale's principle on biological neural networks - if at all?
Analysis of the statistical properties of Daleian connectivity matrices in terms of their eigenvalue spectra suggested that the variance of synaptic weights in this case shapes network dynamics [16], and described the covariance structure of neural activity in such networks [17, 18). One functional analysis of networks violating Dale's s principle [19] suggested they may allow for robust balanced network states using fewer neurons than Daleian networks. Studies of training neural networks that obey Dale's principle suggested that feed-forward artificial neural networks (ANNs) with separate inhibitory and multi-layered perceptrons that are not sign-specific [20]. For spiking neural network models, a scheme for constructing networks that compatible with biophysical constraints connectivity and perform tasks that are solved by biological neural circuits, has been suggested in [21].
excitatory units can reach similar performance as
Here, we study the implications of Dale's principle on the function of recurrent neural networks, and their ability to learn. We simulate the responses of large ensembles of Daleian and non-Daleian networks to rich sets of stimuli, and measure the functional similarity between them in terms of the overlap of the distributions of their spiking responses. Surprisingly, we find that almost all non-Daleian networks have Daleian "neighbor" that is close in functional space, namely it responds similar way to the same stimulus. Thus, the computations implemented by non-Daleian networks can be accurately approximated by Daleian ones. We analyze the sensitivity of networks to synaptic perturbations, and find that computations performed by Dalcian networks are more robust to random synaptic noise. We further show that Daleian networks arbitrary non-Dalcian networks perform using a simpler and more collogically-plasiiie learning mechanism. Overall, our results suggest that Daleian networks are beneficial both from both design perspective and a computational one, and offer potential venues for exploring biologicalll-inspired design of artificial neural networks.
learn to approximate the computation that
2 Results
To characterize the functional differences between Daleian and non-Daleian networks, we studied the encoding of stimuli by these two kinds of networks, using two classes of models of recurrent neural networks: networks of spiking neurons and networks of firing rate-based neurons. We compared the responses of Daleian networks and non-Daleian ones for the same stimulus or class of stimuli, and explored the nature and implications of their differences.
2.1 Dalcian and non-Dain con-Daleian networks tile the space of neural response distributions in a similar manner
We started by considering the case of networks of N spiking neurons, where the synaptic connectivity of network is given by an N x N real matrix, W. where Wyy is the synapse from neuron i to neuron and W., 0 for all i (i.c., no self-synapses). The stimulus to the network is given by vector s E RN, such that s; is a time-independent stimulus to neuron i. The biases of neurons towards spiking are represented
0 E RNN such that 0; sets the baseline activity level of neuron i. The activity of the network is discretized into time bins, such that a binary vector x E {0, 1) denotes the spiking and silence of the neurons in time bin f, where I, if neuron i spiked in that time bin, and x; = 0 otherwise; time bins were set here to 20 ms. The dynamics of the network is given by
1
Pw. (Xe(xx-1,5)
exp (x-_1 W. X, + 0 x, + s.),)
(1)

---

so the probability of neuron j to spike in time t is given by sigmoidal function of the sum of its synaptic inputs Σ; Wi, 2 24-1, 1. its bias towards spiking is t,. the stimulus it receives is $j, and Z is the partition function. Equation (1) defines a transition matrix between the 2 binary states of the network, M, whose entries are all strictly positive (due to the definition of eq. 1), and so this Markov chain converges to unique stationary distribution over the states of the network for a given stimulus Tw.e(x(s), This unique stationary distribution satisfies Tw ( -M Tw (w.e(x(s), (xls). or (x)s( (M - I) = O, where I is the identity matrix. Since Tw (x/s () is a vector from the left nullspace of (M I). and we know that this nullspace has dimension TWe (x(s) can be uniquely found by singular value decomposition (SVD) of (M- - 1) (see Methods).
To compare Daleian and non-Daleian networks, we computed the stationary distributions Tw, o (x)s) of 5000 Daleian networks and 5000 non-Daleian networks of N = 10 neurons, each presented with 100 randomly selected stimuli (Fig. la-c). For the non-Daleian (nD) networks, synapses were sampled ij ~ (O, vxy,,, where i,j E 1...N .N. For the Daleian (D) networks, half of the neurons were selected to be excitatory and half inhibitory, and the outgoing synaptic weights of each neuron were sampled from the positive or negative parts of a normal distribution. namely, W ~ [O J )l (i), where (i) = 1 for excitatory neurons and -1 for inhibitory ones.
from a Gaussian distribution WaD
The stimuli were sampled from normal distribution, s (O, 1) (i E 1....N .N), as in this range the network responses were not dominated by the stimulus or the recurrent activity in the network. but by the combination of both (see SI). Importantly, while the stationary distributions are an asymptotic property of the model, the rate of convergence of these Markov chains for our networks is very fast, approaching T (x]s) at an exponential rate (the rate is governed by the spectral gap of M, which here corresponds to about 5 time steps, or 100 mss see SI). We note that the dynamics defined by eq. (1) converge to a (x|s( irrespective of the initial conditions of the network, which justifies the comparison of networks by the stationary distributions of their responses to the same stimulus.
We quantified the funetional similarity between all pairs of networks within and between the two ensembles, using the dissimilarity of the stationary response distributions:
Diune (Wa, Wils) = Dss on (x (x|ss)
where Dys is the Jensen-Shannon divergence bounded and symmetric measure of the overlap of the distributions [22] (see Methods). Figure 1d shows a sub-sample of the resulting 10, 000 x 10, 000 dissimilarity matrix between all networks in the Daleian and non-Daleian ensembles for one such stimulus. The dissimilarity matrices for different stimuli were highly correlated (see SI). The structure of the dissimilarity matrix suggests that Daleian networks do not inhabit specific part of functional space, but rather that their response distributions tile the space of stationary distributions in a similar way to that of the non-Daleian networks. This further demonstrated by the 2-dimensional embedding of the dissimilarity matrix (Fig. Ie), which does not show any obvious separation between Daleian and non-Dalcian networks. Furthermore, the profiles of distances from randomly chosen Daleian or non-Daleian network to the other networks are consistent over different choices of networks, as shown Fig. If.g- reflecting that for typical non-Daleian network, there is a Daleian network which is functionally close to it. We further computed for each stimulus the percent of non-Daleian networks whose nearest network over the two ensembles was a Daleian network, and found that, across all stimuli, the probability that the nearest network to a non-Daleian network would be Daleian is approximately 50%, reflecting, again, that Daleian networks cover the space of response distributions in a similar manner to that of non-Daleian ones. Importantly, repeating the analysis for the relations between networks, averaged over the set of stimuli, namely the matrix given by (D) fune (W,, Wil s))s (where ()a denotes averaging over all stimuli) gave similar structure in terms of the "coverage" of the same space (see SI).
Thus, the results for our sampled networks and stimuli suggest that for given non-Daleian network, there is Daleian network that is nearby in terms of its function. To verify that this not the result of the details of the spiking neural network models that we used, our sampling of the space of non-Daleian and Daleian networks, or the size of the networks - we studied different distributions of synaptic weights (see SI), as well as another class of network models and larger networks' size (see below). Moreover, for both classes of network models, we asked, rather than relying on sampling of the space of networks how well may Dalcian networks learn to approximate non-Daleian ones?
3

---

e
Functional dissimilrity Drme
embedding of Dirne
Dalein
0.4
Embeddingg dim [anu)
20
0.3
Stimulus
Network W
Spkkng resporse (x)e))))))
non-Daleon
0.2
0.1
20-
0.0
Tere
+
-20
0
20
Embeekding dim [aau.
Ensemble of non-Daleian
networks
networks
Distance from Daleian
9
Distance from non-Daleian
Deleian
15
Dalcbn Dalekn
Density
10-
Dalekan
Density
10
Funtional comparison (W.. Wg)
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Drune [bits]
Drane [bits]
Figure I: Daleian and non-Daleian recurrent neural networks cover the same functional space. (a) In non-Daleian networks, the same neuron can have both excitatory and inhibitory outgoing synapses. In Daleian networks, all the synapses going out of neuron either exclusively excitatory (red neurons) or inhibitory (blue neurons). (b) We simulate the responses of networks of spiking neurons whose connectivity given by W to a large set of random stimuli, {s and compute the stationary response distribution to each stimulus Tw:e (x|s)) (c) We sample 10,000 Dalcian and non-Daleian networks, and compute the response distribution of all networks the same stimuli. We then measure the functional dissimilarity between networks by the overlap of these distributions. (d) example of the pairwise functional dissimilarity matrix between 300 Daleian and 300 non-Daleian networks presented with the same stimulus (only 600 networks are shown for visualization purpose). Rows and columns within the block of Daleian vs. Daleian networks (and non-Daleian vs. non- Daleian ones), were each ordered by hierarchical clustering (see Methods). The black horizontal and vertical black lines mark the transition from Dale to non-Dale networks. (e) A 2-dimensional tSNE embedding of the dissimilarity matrix from (d). Each dot marks network, with Dalcian networks shown in green, and non-Dalcian shown in black. (f) A representative example of the distributions of functional dissimilarity values between one randomly chosen Daleian (marked by green arrow in (d)) network and the rest of the networks in the ensemble.
(g) Similar to (f) but for a randomly
selected non- on-Dala network (black arrow in (d)). Distance profiles shown in (f) and (g) are typical for all networks in the ensemble (SI).
2.2 Daleian networks learn approximate non-Daleian ones with high accuracy
We asked how accurately we can approximate the response of randomly selected non-Daleian network with synaptic connectivity WnD to stimulus s, using Daleian network W by op- timizing synaptic weights. We started from a randomly chosen Daleian connectivity ma- trix W ini with continuous synaptic weights, where half of the neurons were randomly chosen to have only positive outgoing synapses, the other half had only negative outgoing synapses, and the diagonal terms were all zero (Fig, 2a). We used gradient descent on WD minimize C(WP) P) = DJs TwD aeppeeee [I | beg (x|s) To keep the excitatory/inhibitoyy identity neurons, we optimized the magnitude of their synaptic weights, maintaining their sign throughout learning.
We performed such learning for 2000 different non-Daleian networks W> each with different stimulus s, yielding 2000 Daleian approximations (see Methods for optimization details). We found that for a random non-Daleian network there exists, with high probability, Daleian network B that is orders-of-magniuue
closer than typical non-Daleian network C. Over the whole set, optimized Daleian networks were two ondrrs-ofmmaanitudee closer to target networks compared the typical functional distance between networks in our sampled ensembles above, and one order-of -f-agniuud

---

closer than the nearest network in the ensemble (Fig. 2b). The average dissimilarity of an optimized Daleian network random non-Daleian responding to arbitrary stimuli reached fune ~ 5 10 bits, which means that 2000 time bins are needed to distinguish between the responses of the target non-Daleian network and the optimized Daleian one. (For reference, optimizing non-Daleian network to approximate non-Daleian network gives Duun 3 10 bits).
We then asked how well can an optimized Daleian network approximate a non-Daleian one on set of stimuli, i.c. how similar they are in terms of the function they compute. We therefore repeated the learning described above, but this time minimizing (C(W>))) Again, the optimized Daleian networks were highly accurate in approximating the non-Daleian ones, now reaching an average D June value of 10 bits, when training only W 2 when training both W and (Fig. 2c). These differences mean it would take about 200 time bins to tell apart the responses of the Dalcian network from the non-Daleiad one; for time bins of 20 ms, this means it would take more than 4000 ms of neural activity to the networks apart.
a
Optimized
Optimized non Dale
Nearest
Dale
10:1
moise
10a1
Nearest
Dalelan
Target
10
3%
10%
100%
Synaptic noise level
b
Single stimulus
Multiple stimuli
Dale (leemeed)
Dale: (leamed)
non Dale (ncarest)
Denaty
non-Calle (earmed)
(leamed)
Dale Onearest)
Dendty
non-Dale (laame))
non Dale (kamed)
17 Tune [bits]
Dtune (bits)
Figure 2: Daleian networks can learn to accurately approximate the function of non-Daleian networks. (a) For a given oon-Dalcian network WaD and stimulus s, we compute the distance to the nearest Daleian network in the sampled ensemble (blue) and the nearest non-Daleian network in the sampled ensemble (gray). We also optimize a random Daleian network to approximate the target non-Daleian network (these learned networks will be later denoted in green) and learn non- nal-aan network that is optimized to approximate the non- non-Daleian network (later shown in orange). In the last two cases, learning is done by gradient descent all synapses, see text. (b) The histograms of distances the sampled and learned networks from 2000 different nol-Daleian networks and 2000 different stimuli. Different colors show the distances to the target non-Daleian networks from the different classes of networks; color as described in (a). For reference, we also show a histogram of the mean distance from each non-Daleian network to all other networks in the ensemble for the corresponding stimulus (black). (c) The histograms of distances of the learned networks from 2000 different non-Daleian, each responding 30 different stimuli. Different colors show the distances to the target non-Daile non-Daleian networks from the different classes of networks (Daleian/non-Dene ) such that learning done either by optimizing synaptic weights alone or both the synaptic weights and the neuronal thresholds 0. (d) Functional dissimilarity between the stationary distribution of a network Wn responding to stimulus and noisy variations of W/nD for different levels of synaptic noise (noise is independent for each synapse). Error bars correspond to standard deviations over 100 different noise realizations. Results shown for 30 different networks and 30 different stimuli.
We emphasize that we did not optimize the types of neurons of Daleian networks that learn to approximate a non-Daleian one, i.e. we did not pick which neurons would be excitatory and which inhibitory in each network. This means that our optimized Daleian networks are an upper bound on the ability of Daleian networks to approximate non-Daleian ones, and thus reflect how capable Dalcian networks are. Interestingly, the learned networks had significantly broader distributions
5

---

of synaptic weights, and Daleian optimized networks were significantly sparser than non-Daleian optimized networks, similar the properties of real neuronal circuits (see SI).
To give a more biologically relevant interpretation to these differences, we asked what level of synaptic noise would result in similar functional dissimilarity values. Thus, for randomly sampled network WaD and stimulus s, we computed the Aansen-Shannon divergence between the stationary distribution the original WnD and noisy versions of it, W W + e, for different levels of multiplicative synaptic noise (see SI). Averaging over noise realizations, networks, and stimuli we found that functional dissimilarity value of [Js[ Twr o (x [x|s]] Tw. ". (x[s]] 5 10 bits, typical value for the optimized Daleian approximation, corresponds to synaptic noise level of ~ 25% (Fig. 2d). Given the variability of activation strength of synaptic connections in real neural networks, and their dynamic nature, these results suggest that Daleian networks can approximate non-Daleian ones to degree that is very close to the limit set by synaptic and neuronal noise [23].
connectivity of non-Daleian target WrD
(WaDy?)-1
Structural comparison
Prediction accuracy
Matrix values 1910
200
Dale (learned)
Presynaptic
Density
100
-2-
-0.5
0.5
0.95
0.96 0.97 0.98 0.99
Matrix values Wap
connectivity of Daleian approximation W.P
(I- - -WW
Functional comparison
Matrix values ir (wolp""
200
non-Dale (learned)
Presynaptic
Density
100
-1
0
0.95 0.96 0.97 0.98 0.99 T. TD)
Postsynaptic
Postsynaptic
Matrix values p027pp
Figure 3: Daleian rate-based networks can learn to accurately approximate rate-based non- Daleian networks (a) An example of the connectivity matrix of non-Daleian target the continuous- time recurrent firing-rate network 50 neurons, WnD (top) and the connectivity of Daleian approximation WDD (b) Corresponding (T (W*DyT) matrix (top) and (1 DW matrix (bottom). (c) Comparison of individual entries in the non-Daleian network and Daleian approxima- tion: structural similarity is shown for connectivity matrices (top); functional comparison (bottom). (d) Accuracy of the Daleian approximation (green) and a non-Daleian approximation (orange) for firing-rate-basee model networks with N 100 neurons, measured by the correlation of their steady state activity vectors, averaged over 1000 different stimuli.
Next, we studied how well Daleian networks can approximate non-Daleian ones using another commonly studied class of network models: Continuous time recurrent firing-rate networks [24], which offer mathematical tractability, and also allowed us to extend our analysis to much larger networks. Here, neural activity is described by firing rates of the N neurons, which are denoted by r E R Correspondingly, the dynamics given by differential equation with integration time constant 7, and so the response of the network to stimulus s, is given by
dr dt
W r
(3)
and the steady state solution is given by r.s (1 (1- WT) -1-W" s. We asked how well we can approxi- mate the computation performed by an arbitrary rate-based network WnD of N 100 neurons using a Daleian network D We repeated the analysis we performed for spiking networks, but here, for each randomly chosen non- -aleial network we used gradient descent to minimize the loss function:
C(WD)= = (1- (WnD)7) - -(1 (1-(WP))
-1-(WDD)) (WD)))

---

We found that the optimized Daleian matrices WD are very different from WnD of the non- network they approximate (Fig. 3a,c), while (1- ( are similar to (1- (W -1 (Fig.3ba 3b,c). Thus, non-Daleian networks and their Daleian approximations have very similar stimulus- to io-steady-state mapping: The mean Pearson correlation, for 1000 stimuli, between the steady states of non-Daleian targets and those of their Daleian approximations was ~ 99% (Fig. 3d). These corre- lation values were very similar to the accuracy of learning to approximate a non non-Daleian networks by another non-Daleian network.
We conclude that non-Daleian networks can be approximated with high accuracy by Daleian ones. Moreover, since our learning of the optimized Daleian approximations is based on simple gradient descent and randomly chosen initial Daleian network structure, is clear that our results are an upper- bound on how close one can get to a non-Dalcian network with Daleian one. Having established that there is no clear functional loss in using Daleian networks compared to the supposedly more powerful non- Daleeaa ones, we turned to explore the sensitivity and learning dynamics of Daleian and non- non-Daleian networks, and sought functional benefits of using Daleian networks.
2.3 Robustness and adaptability of Dalcian networks are superior to non-Daleian ones
In learning and optimizing their function, neural networks need to balance changes that improve their performance while retaining their previously learned function. This suggests that certain network changes would result in altered function, whereas others should not. To explore the "design" of learnability of non-Daleian and Daleian networks and their robustness, we next asked how sensitive they are to changes in their synaptic connectivity maps and to the properties of individual neurons.
We first explored the effeet of changes in synaptic weights on the function of a given network by computing fw (AW) = D [Tw. e (x [s) (ww.///wwwx/// and then the Hessian of f. given by the 2ع
matrix [Myleg.tt aw., We note that in this case, the Hessian of H, (evaluated at W) is the Fisher Information Matrix of (x|s) which provides principled interpretation of the analysis and the results that follow [25]. The Hessian measures the curvature of f in all directions of parameter space, where each direction corresponds to changing the weight of a single synapse. The trace of the H then measures the overall "flatness" of f around W. From a biological perspective, the flatness corresponds to the robustness of the network to synaptic fluctuations. We thus computed
Tr [M,] - E...EI...N aW?
a²,, for each of the networks in the Daleian and non-Daleian ensembles,
and for each sampled stimulus. To compare the sensitivities of Daleian and non-Daleian networks, we computed the ratio between the average trace of Daleian networks and that of non- non-Daleian networks, separately for each stimulus. This ratio quantifies whether Daleian networks are more sensitive (ratio > 1) or less sensitive (ratio < 1) to random synaptic changes compared to non- non-Dalcian networks. We found that for the 98% of the stimuli, Daleian networks were less sensitive than non non-Dalcian ones to perturbations of synaptic weights (Fig. 4a). Given the inherent noise in biological synapses, such robustness carries clear functional advantage.
a
Sensitivity to W
b
Sensitivity to 0
Daleian
Daleian more sensitive
Daleian
Daleian
30
less sensitive
less sensitive
Density
Density
40
20
20
10
0.90
0.95
1.00
1.05
1.10
0.90
0.95
1.00
1.05
1.10
Sensitivity ratio (Dalelan / nan-Dalelan)
Sensitivity ratio (Daleian/ non-Daleian)
Figure 4: Daleian networks are less sensitive to synaptic fluctuations and more sensitive to changes of single neuron parameters. (a) Histogram of the ratio of the average sensitivity to synaptic perturbation of Daleian and the average sensitivity of non-Daleian networks, shown for 1000 different stimuli. Average sensitivity is measured using the trace of each of the Hessian matrices of 1000 networks for a given stimulus. (b) Similar to (a) but sensitivity is computed with respect to neuronal biases vector 0.
7

---

Increased robustness to synaptic fluctuations may hinder the ability of network to adjust its reactions to changes the environment. We therefore repeated the sensitivity analysis above, but this time f
computed the Hessian with respect to changes in the bias of individual neurons 8;, [Hy 20,00,
and computed Tr [ EieI... 8² for each of the networks and each stimulus. We then a computed the ratio between the average sensitivity of Daleian networks to threshold perturbations, and that of non-Daleian networks, separately for each stimulus. We found that for 84% of the stimuli, Daleian networks were, on average, more sensitive to changes in the threshold parameters compared to non non-Daleian ones (Fig. 4b). Thus, the differences sensitivities of the two classes of networks mean that Daleian networks are more resilient to random synaptic fluctuations, and may learn effectively by tuning the baseline firing rates of individual neurons.
2.4 Daleian networks learn efficiently by tuning only single neuron properties
We asked how learning by changing the properties of individual neurons, rather than individual synapses, would work for Daleian and non networks. We evaluated learning performance using the loss function:
C(W) Das [Trarger (x) (x]ss
where Trarget is the distribution that we wish to approximate with the optimized network, W. Given the homeostatic scaling of changes in synaptic weights [26] and the identification of "architectural" features that shape the function of neural networks [27, 28]. it is clear that learning by adapting neuronal features could have computational benefits and be more plausible or efficient than learning by changing individual synapses. We thus compared the results of learning by gradient descent over (Wij}: ). to learning by "macroscopic" neuron-level features. Specifically, we used scaling factor for all outgoing synapses of neuron Xqur > O, scaling factor for all incoming synapses of a neuron A > 0, and the firing bias A5: during learning, each synapse re-scaled according to the pre-synaptic and post-synaptic scaling factors W., - Xa Xan WWe (Aput and X are initialized to I before learning), and each neuron's 's bias is adapted individually 0, 42-00 0, + AO; (A0, is initialized to 0 before learning). Thus, instead of learning by changing the N(N - 1) terms of the full synaptic connectivity map, we learned by changing 3N parameters, corresponding to known biological features at the level of individual neurons (Fig. 5a). Clearly, synaptic learning is more potent than the neuron-level learning (any learning trajectory in weights-space due to neuronal learning is also achievable by synaptic learning, but not vice versa). But, it turns out that they act very differently in terms of the accuracy of synaptic and neuron-level learning on Daleian vs. non- -anlaal networks.
We then asked how well the different classes of networks would approximate a distribution Tunget corresponding to natural response distribution of biological neural networks. We used real spiking patterns of groups of 10 neurons recorded from the prefrontal cortex of macaque monkeys performing a visual classification task [29]. to compare the accuracy of synaptic learning and neuron-level learning in mapping random stimulus to the empirical response distributions of real neurons. We found that while both Dalcian and non- non-Daaeian networks achieved higher accuracy using synaptic learning (Fig. 5b), learning by changing single neurons' properties achieved higher performance for Dalcian networks compared to non- -aleiaa networks (Fig. Sc,d). This suggests that the constraints set by Dale's 's principle change the relative effectiivness of these learning mechanisms.
2.5 Information coding by Daleian networks
As we have shown that Daleian networks are able to accurately approximate the responses of non-Daleian networks for a given stimulus, we asked how the two classes may differ in their encoding of set of stimuli, which would reflect on the ability of downstream neurons to decode stimuli from network responses. We therefore computed the information that the responses of a network whose synaptic connectivity is given by W carries about set of l: 500 different stimuli. We first computed the entropy of the stationary distribution for each stimulus, H (Tw.o = Σx Tw.o(x(ss)) logg Tw.a (x):)
The total entropy of the network's activity is given by H [=w.o] = =- Tw. (x) logg Twe (x). where Tw.o (x) = - Two (x) s:), where we assumed uniform distribution over stimuli (using non-uniform stimulus distributions gave similar results; see SI). We then computed the mutual
8

---

a
Synapse-level learning
Neuron-level learning
b
Daleian
non-Daleian
19"
10"
Fuze (synaptic)
10"1
10"3
19
Wg - W2 + AWW
Wi + A
10
10-3
10
10.2
10:2
16-1
10
10-3
10-
20-2
10-2
16-4
Dlunc (neuronal)
Dtune (neuronal)
C
Accuracy of different learning mechanisms
d
Ratio of synaptic and neuronal learning
2.0
2.0
Dale (synaptic)
Dale (neuronal)
1.5
Dale
non-Dale
Density
non-Dale
non-Dale
Density
(synaptic)
(neuronal)
1.0
1.0
0.5-
0.5
0.0
10-1
10-3
10
0.0
10-
D Diune [bits]
Ratio [a.u.]
Figure 5: Daleian networks learn efficiently by tuning single-neuron properties. (a) In synaptic learning each synapse is separately tuned (left), while in neuronal learning, changes to neuronal properties induce correlated changes across synapses of the changed neuron (right; see text). (b) Accuracy of synaptic learning (measured by functional similarity to Targeti lower D Funo means higher accuracy) is compared neuron-level learning. Points show the results for 2000 different (s, Turget) pairs; each point shows the mean Duune when learning the mapping from random stimulus s to random target distribution Target. over 30 Daleian (left) or non- non-Daleian (right) initial networks. (c) Comparison of the learning accuracy of Dalcian and non- -aleian networks using synaptic and neuron-level learning mechanisms. Histograms show the distributions of the average fune
fnne over 30 Daleian/non Daneinn networks trained to map a random stimulus to a target distribution Larget- (d) The ratio between the accuracy of synaptic learning and neuron-level learning for Daleian and non-Daleian networks, as measured by the ratio of mean Dfune values from (b).
information between the stimuli and the stationary responses,
Iw (s: x) = H [[mw,o(x)) (H Tw (x(s)))
(4)
where ()s denotes averaging over stimuli. For 500 Dalcian networks and 500 non-Daleian networks of N = 10 neurons, we found that for randomly sampled networks, the responses of Daleian networks were more informative about the stimulus. Then, using both synaptic and neuronal learning, we trained the networks to maximize the mutual information in eq. (4) (Fig. 6a). We found that Daleian networks reach similar performance using synaptic learning, but are more informative (on average) when using neuron-level learning (Fig. 6b).
3 Discussion
We found that despite the clear constraint that Dale's principle imposes on the architecture of neural networks, it does not seem to incur a clear cost in terms of the computational capacity of Daleian networks or their ability learn. We were able approximate the responses of arbitrary non-Daleian networks to a range of stimuli, with Daleian networks, to very high degree of accuracy. In fact, this accuracy is so high that is on par with the level of intrinsic noise level of real biological networks. Moreover, we found that Daleian networks are more robust to synaptic noise, and that they can learn by tuning of neuron-level parameters, nearly as well as can be achieved by general synaptic learning rules. Our results suggest that in addition to the clear structural and developmental benefits of using Daleian neural networks, the brain may actually gain from using them, as they offer similar computational power, higher robustness to noise, and simpler learning mechanisms.

---

MI [bits]
2-
30
1.75
Dale (init)
Dale (neuronal)
Synaptic
non-Dale (init)
1.25
Density
20
non-Dale (synaptic)
Dale
MI [bits]
2-
non-Dale (neuronal)
(synaptic)
1.5
Neuronal
1.25
0
1000
2000
3000
1.2
1.4
1.6
1.8 MI [bits]
2
2.2
2.4
Training step
Figure 6: Random Daleian networks are more informative than random non-Daleian ones, whereas trained Daleian networks carry similar information trained non-Daleian. (a) The mean of the information values carried by 500 Daleian (blue) and 500 non-Daleian (gray) networks, is shown as function of training steps. Each network was trained optimize its information on a given set of 500 stimuli using synaptic learning (top) neuronal learning (bottom). Error bars represent 1 standard deviation. (b) Distribution of information values carried by networks on a of 500 random stimuli before learning: Daleian networks shown in yellow, non-Daleian in orange. Distribution of information values network trained to maximize the carried information by neuronal learning (Daleian in blue, non-Dale in gray), and using synaptic learning (Dale in purple, non-Dale in green).
Our analysis is naturally limited by the network models have explored, our of dense and relatively small networks, and the choices of connectivity patterns and stimulus statistics. Analysis of a wider range of connectivity parameters, log-normally distributed synaptic weights [30], and richer stimuli have given similar results, yet the exploration of larger and sparser networks, more naturalistic stimuli, and different connectivity patterns is warranted.
Other differences between Daleian and non- con-Daleian networks suggest future directions study: Randomly sampled Daleian networks, well
optimized to approximate non- non-Daleian ones show pairwise correlations profile that is skewed towards positive correlations, similar to real neural networks, whereas randomly sampled non-Daleian ones show balanced distribution of correlations (see SI). These differences suggest organization features Daleian networks that remain to be uncovered. Somewhat diametrically, been shaped by evolution, development, and learning to implement specific computations. It is likely then that within the vast space of non-Daleian networks there would be ones that have interesting and distinct architectures both in terms of structure and the computations they carry.
that real networks, which mostly Daleian, have
The potential implications of results for ANNs also intriguing. Theoretical studies of Deep Neural Networks and their applications have usually ignored Dale's principle, as its inclusion typically impairs learning. Recent results potential remedies that would allow ANNs to rely distinct classes of neurons [31] and in particular Excitatory and Inhibitory ones [20] suggest how the integration of design features and properties of real neural networks into artificial ones might be beneficial [32, 33]. In particular, the idea that structural constraints on networks can affect learning trajectories and define the loss surface has a long history in constructing and training artificial neural networks [34-36]. We hypothesize that some of the benefits that Dale's principles confers on recurrent biology-inspired neural networks may extend artificial ones. It will be especially interesting to explore the parallels of our results in terms of shaping the search space of sign- nign-constrained ANNs, the possibility of efficient learning by neuronal parameters rather than synaptic ones, the robustness and sensitivity they may offer, and their potential for preventing catastrophic forgetting.
Acknowledgments and Disclosure of Funding
This work was supported by Simons Collaboration on the Global Brain grant 542997, Israel Science Foundation grant 137628, Israeli Council for Higher Education/Weizmann Data Science Research Center, Martin Kushner Schnur, and Mr. Mrs. Lawrence Feis. ES is the incumbent of Joseph and Bessie Feinberg Chair.
10

---

References
[1] Kosuke Takagi. Energy constraints on brain network formation. Scientific Reports, 11(1):1-8, 2021.
[2] Michael W Reimann, Anna-Lena Horlemann, Srikanth Ramaswamy, Eilif B Muller, and Henry Markram. Morphological diversity strongly constrains synaptic connectivity and plasticity. Cerebral Cortex, 27(9):457-44885 2017.
[3] Bernhard Hellwig. A quantitative analysis of the local connectivity between pyramidal neurons in layers 2/3 the rat visual cortex. Biological cybernetics, 1112):111-121, 2000.
[4] Joshua R Sanes and S Lawrence Zipursky. Synaptic specificity, recognition molecules, and assembly neural circuits. Cell, 181(3):536556, 2020.
[5] John Carew Eccles. From electrical to chemical transmission in the central nervous system: the closing address of the sir henry dale centennial symposium cambridge, 19 september 1975. Notes and records of the Royal Society of London, 30(2):219-230 1976.
[6] Piergiorgio Strata, Robin Harvey, et Dale's principle. Brain research bulletin, 50(5):349-350, 1999.
[7] DW Golding. Patterns of peptide discharge iiphaargemmmlllattionn for dale's principle. The Peptidergic Neuron, pages 7-12. Springer, 1996.
[8] Wayne S Sossin, Alejandro Sweet-Cordent and Richard Scheller. Dale's hypothesis revisited: different neuropeptides derived from common prohormone are targeted to different processes. Proceedings of the National Academy of Sciences, 87(12):8845- 4848888844448 1990.
[9] D Sulzer and S Rayport. Dale's principle and glutamate corelease from ventral midbrain dopamine neurons. Amino acids, 19(1):45 2000.
[10] Erik Svensson, John Apergis-Schoute, Geoffrey Burnstock, Michael P Nusbaum, David Parker, and Helgi B Schiöth. General principles of neuronal co-transmission: insights from multiple model systems. Frontiers in neural circuits, page 117, 2019.
[11] Carl Van Vreeswijk and Haim Sompolinsky. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science, 274(52933: 1996.
[12] Tim P Vogels, Kanaka Rajan, and Larry Abbott. Neural network dynamics. Annu. Rev. Neurosci., 28:357-376, 2005.
[13] Nicolas Brunel. Dynamics of sparsely conntected networks of excitatory and inhibitory spiking neurons. Journal of Computational Neuroscience, 8:183-208, 2000.
[14] Michael Okun and Ilan Lampl. Instantaneous correlation of excitation and inhibition during ongoing and sensory-evoked activities. Nature neuroscience, 11555533--337, 2008
[15] David C Somers, Sacha B Nelson, and Mriganka Sur. An emergent model of orientation selectivity cat visual cortical simple cells. Journal of Neuroscience, 15(8):544855453 1995.
[16] Kanaka Rajan and Larry F Abbott. Eigenvalue spectra of random matrices for neural networks. Physical review letters, 97(11))888104, 2006.
[17] Yu Hu and Haim Sompolinsky. The spectrum of covariance matrices of randomly connected recurrent ncuronal networks. bioRxiv, 2020.
[18] Jonathan Kadmon and Haim Sompolinsky. Transition to chaos in random neuronal networks. Physical Review X, 5(4)0041030, 2015.
[19] Victor J Barranca, Asha Bhuiyan, Max Sundgren, and Fangzhou Xing. Functional implications of dale's law balanced neuronal network dynamics and decision making. Frontiers in Neuroscience, 16, 2022.

---

[20] Jonathan Cornford, Damjan Kalajdzievski, Marco Leite, Amélie Lamarquette, Dimitri Michael Kullmann, and Blake Aaron Richards. Learning live with dale's principle: Anns with separate excitatory and inhibitory units. In International Conference on Learning Representations, 2020.
[21] Brian DePasquale, Mark M Churchland, and LF Abbott. Using firing-rate dynamics to train recurrent networks of spiking model neurons. arXiv preprint 1eXv::600.007200 2016.
[22] Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Trans. Information Theory, 37(1):14--15, 1991.
[23] Noam E. Ziv and Naama Brenner. Synaptic tenacity or lack thereof: spontaneous remodeling of synapses. Trends in neurosciences, 41(2):89 99.89999999999 2018.
[24] Sadra Sadeh and Claudia Clopath. Theory of neuronal perturbome in cortical networks. Pro- ceedings of the National Academy of Sciences,
2020.
[25] Benjamin B Machta, Ricky Chachra, Mark K Transtrum, and James P Sethna. Parameter space compression underlies emergent theories and predictive models. Science, 342(61558))004-6677, 2013.
[26] Gina G. Turrigiano and Sacha B. Nelson. Homeostatic plasticity the developing nervous system. Nature Reviews Neuroscience, 5(9):97-077, 2004.
[27] Adam Haber and Elad Schneidman. Learning the architectural features that predict functional similarity of neural networks. Phys. Rev. X, 12:021051, 2022.
[28] R Milo, S Shen-Orr, S Itzkovitz, N Kashtan, D Chklovskii, and U Alon. Network motifs: simple building blocks of complex networks. Science, 298(5594):824-827, 2002.
[29] Roozbeh Kiani, Christopher J Cueva, John B Reppas, and William T Newsome, Dynamics of neural population responses in prefrontal cortex indicate changes of mind on single trials. Current Biology, 24(13):1542-15 2014.
[30] György Buzsáki and Kenji Mizuseki. The log-dynamic brain: how skewed distributions affect network operations. Nature Reviews Neuroscience, 15(4))264-78, 2014.
[31] Briar Doty, Stefan Mihalas, Anton Arkhipov, and Alex Piet. Heterogeneous 'cell types' can improve performance of deep neural networks. bioRxiv, 2021.
[32] Michael Pfeiffer and Thomas Pfeil. Deep learning with spiking neurons: opportunities and challenges. Frontiers in neuroscience, page 774, 2018.
[33] Jordan Guerguiev, Timothy P Lillicrap. and Blake A Richards. Towards deep learning with segregated dendrites. ELife, 6:e22901, 2017.
[34] Devon Graham, Junhao Wang, and Siamak Ravanbakhsh. Equivariant entity-relationship networks. arXiv preprint (eXXvv:903.000000 2019.
[35] Steven Nowlan and Geoffrey E Hinton. Simplifying neural networks by soft Neural computation, 4(4):473-493, 1992.
[36] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[37] Charles Miller Grinstead and James Laurie Snell. Introduction probability. American Mathematical Soc., 1997.
[38] Ziv Bar-Joseph, David K Gifford, and Tommi S Jaakkola. Fast optimal leaf ordering for hierarchical clustering. Bioinformatics, 17(suppL_1::222-S2)) 2001.
[39] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv.1412.6980, 2014.
12

---

[40] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne,, and Qiao Zhang. JAX: composable transformations of Python+Numbl programs, 2018. URL http:// /ttt:://ithub com/ wtt google/jax.
[41] David C. MacKay. Information Theory, Inference Learning Algorithms. Cambridge University Press, New York, NY, USA, 2002.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Limitations of our work are
described in the final paragraph of the discussion.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes] We specify the parameters of the models from which the network ensembles were sampled, as well as details about the optimization objective functions and hyperparameters.
(b) Did you include complete proofs of all theoretical results? [Yes] The only theoretical claim in the paper the ratio between Dalcian andn non- non-Daleian networks of given size, whose derivation is in the Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as URL)? [Yes] Link to repository containing the code will be attached as supplemental material.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g.. with respect the random seed after running experi- ments multiple times)? [Yes] We quantify and present the variability of our results with respect different networks, stimuli and noise realizations throughout the work.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All analysis was conducted on an internal cluster.
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the paper describing the neural recordings we have used in Section 2.4.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ZSu6No9-1sg", "reviewer": "Reviewer_p2FF", "summary": "The paper empirically studies the computational benefits of Dale's law of spiking NNs and rate-based NNs. tivity", "questions": "1) line 141, gradient descent doesn't keep the sign constraints of Daleian networks. Did you use the gradient projection?\n2) Figure 4b, hard to compare the D and nD distributions. It looks that nD (syn) has a lower accuracy than D (syn), which is inconsistent with Figure 4c? \n3) Random Daleian NNs are more \"informative\", doesn't that depend on stimuli distribution?", "limitations": "The authors mentioned limitations in network size and the assumption of weight distribution. It's also worth noticing that there are more excitatory neurons than inhibitory neurons in the cortex, but in this work they are balanced.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1) The paper studies an important puzzle in computational neuroscience, why the biological neural nets contain the signs of output weights and what could be the computational implications. The paper is well written and all central claims are well supported by abundant experimental evidence.\n2) The functional comparison of random and learned Daleian and non-Daleian NNs is a novel analytical tool to probe the computational consequence of computational constraints.\n3) Sensitivity analysis of synaptic weights shows a surprising robustness benefit of Daleian NNs.\n\nWeakness:\nLack of theoretical justifications to better understand\n- when does a non-Daleian NN hard to be approximated by a Daleian NN\n- why is a Daleian NN less sensitive to weight perturbations\n- why does non-Daleian neuronal learning perform worse than Daleian neuronal learning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TSWVTA_PRTS", "reviewer": "Reviewer_yFoy", "summary": "Dale's law is the idea that (most) neurons express only a single neurotransmitter, making them excitatory or inhibitory but not both. This paper studies a few models of neural networks constrained to follow Dale's law and contrasts them with networks where the neurons are free to form excitatory and inhibitory connections. The main results are that \"Daleian\" networks can generally produce nearly the same output as \"non-Daleian\" ones. They also present better robustness to noise in the weights and can more easily learn distributions of responses with biologically-plausible rules.", "questions": "Why did you choose the models that you did?\n\nIf you account for weight values (i.e. as real numbers), what can you say about the measure of Daleian networks vs non-Daleian?\n\nWhat were the distributions of stimuli and responses that you tested and how do those choices influence the performance of Daleian/not networks?\n\nSec 2.3: Is studying robustness via the Hessian valid if the signs of W are constrained? Some perturbations could break Dale's law.\n\nSec 2.4: Please explain the task more clearly. How is it similar/different from a machine learning task? Where does the gradient come from and what \n\nSec 3: Dale's law certainly *has* been studied in the machine learning community before, as recognized in the introduction. However, the discussion reads like this past work didn't occur. Discuss in more detail how your results fit into previous work. ", "limitations": "The models seem narrow to me, can the authors better motivate them and better discuss their limitations?\n\nThe last paragraph of section 3 discusses limitations much too briefly, expand. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is very well-written. A couple of places in the margin I wrote \"interesting\" or \"good\". This is an important neuro-AI area of study and a significant difference between how networks are modeled by computer scientists and the biological reality. The figures are clear and look nice. Code is available for reproducibility.\n\nWeaknesses: \n\nThe study is limited to only certain kinds of network models. Most notably, this work does *not* study artificial neural network models and tasks prevalent in machine learning, so some of the results might be misleading or confusing to the NeurIPS community. In particular, the only models used are a max-entropy style statistical model of recurrent dynamics and a simple dynamical RNN, both linear (in the max-ent model the sufficient statistics are linear) in the previous system state (in contrast to nonlinearities used for machine learning and present in real brains). Furthermore, the stimuli are assumed to be static vectors (eqn 1). The networks don't really have to solve any kind of task, as far as I can tell, except to reproduce certain distributions. These model choices are minimally motivated in the text, despite standing out to me as not standard compared to the work I know.\n\nMy confidence in the results is lower because it feels like a lot of details were relegated to the supplement, which I did not have time to read.\n\nSmall points by line\n\n1: \"implies\" seems like wrong word\n\n2: \"number of possible architectures\" this sentence struck me as imprecise, is this a new (minor) result? If so present as such. \n\n89: \"whose entries are all larger than zero\" is not obvious, explain\n\n100: $sign(i)$ is incorrect notation, all indices $i > 0$, what you mean is there is a label $s_i \\in \\pm 1$ for each neuron $i$\n\n106: \"100 ms\" is arbitrary, you make no connection to a timescale in the text, the only mention of a step size I see is in Fig 1b, which was hard to find\n\n181: Introducing this new model seems to require a new section. I had a hard time distinguishing between models 1 & 2 throughout the results, as the paper switches back and forth. Be explicit throughout.\n\n183: add comma after \"Here\" in \"Here neural activity\"\n\n207: \"In learning and optimizing their function\" is awkward\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "08oEv-O5sV", "reviewer": "Reviewer_eksJ", "summary": "The paper considers networks that abide the Dale's law: a neuron can be either excitatory or inhibitory, but not both simultaneously. In experiments, they show that Daleian networks can approximate non-Daleian ones very well, are more robust to noise and are easier to train using fewer parameters. ", "questions": "### Main questions\nI guess the biggest take away for me: we can use models that violate Dale’s law, because there’s always a similar model that doesn’t. Which brings out two questions: \n1) Wouldn’t the Dale law abiding learning algorithms be different? The answer is probably yes, see Clark et al. 2021 (https://proceedings.neurips.cc/paper/2021/hash/532b81fa223a1b1ec74139a5b8151d12-Abstract.html). I think it's worth discussing in more detail. \n\n2) If you take a trained deep network and converted it into a Daleian one (e.g. using a subset of training data, making sure the accuracy didn’t change much), would it produce more brain-like responses (see http://www.brain-score.org/ and their papers)? I’m fine with this being in the discussion, but it’d be great to have it in the results.\n\nThe authors should mention why a theoretical analysis of the Daleian approximation is hard (because it’s non-convex, I assume).\n\nSec. 2.4 is a really interesting result. Maybe it would be possible to run the same experiment for at least CIFAR10 and a small conv net? (Not for the rebuttal deadline, but a comment on that would be appreciated).\n\n### Small corrections\nLine 32: “directed” is usually used to describe graphs, rather than networks. Maybe say asymmetric or remove it altogether?\n\nMaybe Eq. 2 can include the definition of D_JS\n\nLines 167-168: unfinished sentence. …how capable Daleian networks are?\n\nLine 192: move to an equation (not inline)\n\nLine 306: citation needed on log-normal distribution (although I agree it’s widely known)", "limitations": "I think the paper adequately addressed the limitations and potential negative impacts.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The experimental results are important for theoretical neuroscience.\n2. The paper is well-written and easy to follow. \n\n### Weaknesses\n1. No theoretical analysis of the Daleian approximation and increased noise robustness of Daleian networks.\n\n### Summary\nI give the score 8 (strong accept) as the paper addresses an important questions in theoretical neuroscience, showing previously unknown benefits (at least to my knowledge) of Dale's law.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept