A theory of weight distribution-constrained learning
Authors: Weishun Zhong
Ben Sorscher
Daniel Lee
Haim Sompolinsky
Venue: NeurIPS 2022 Accept (2022)
Keywords: feedforward neural networks
weight distribution
optimal transport
connectomics

Abstract: A central question in computational neuroscience is how structure determines function in neural networks. Recent large-scale connectomic studies have started to provide a wealth of structural information such as the distribution of excitatory/inhibitory cell and synapse types as well as the distribution of synaptic weights in the brains of different species. The emerging high-quality large structural datasets raise the question of what general functional principles can be gleaned from them. Motivated by this question, we developed a statistical mechanical theory of learning in neural networks that incorporates structural information as constraints. We derived an analytical solution for the memory capacity of the perceptron, a basic feedforward model of supervised learning, with constraint on the distribution of its weights. Interestingly, the theory predicts that the reduction in capacity due to the constrained weight-distribution is related to the Wasserstein distance between the cumulative distribution function of the constrained weights and that of the standard normal distribution. To test the theoretical predictions, we use optimal transport theory and information geometry to develop an SGD-based algorithm to find weights that simultaneously learn the input-output task and satisfy the distribution constraint. We show that training in our algorithm can be interpreted as geodesic flows in the Wasserstein space of probability distributions. Given a parameterized family of weight distributions, our theory predicts the shape of the distribution with optimal parameters. We apply our theory to map out the experimental parameter landscape for the estimated distribution of synaptic weights in mammalian cortex and show that our theory’s prediction for optimal distribution is close to the experimentally measured value. We further developed a statistical mechanical theory for teacher-student perceptron rule learning and ask for the best way for the student to incorporate prior knowledge of the rule (i.e., the teacher). Our theory shows that it is beneficial for the learner to adopt different prior weight distributions during learning, and shows that distribution-constrained learning outperforms unconstrained and sign-constrained learning. Our theory and algorithm provide novel strategies for incorporating prior knowledge about weights into learning, and reveal a powerful connection between structure and function in neural networks. 

A theory of learning with constrained weight-distribution
Weishun Zhong Harvard and MIT wszhong@mit.co
Ben Sorscher
Stanford University
bsorsch@stanford.co
Daniel Lee Coreell Tech
Haim Sompolinsky
Harvard and Hebrew University hsompo I ins inskymmmb.......mmm...mm.mmm.kkr harvard. edu
dd1468cornel1.cem
Abstract
A central question in computational neuroscience is how structure determines function in neural networks. Recent large-scale connectomic studies have started to provide a wealth of structural information such as the distribution of excita- tory/inhibitory cell and and synapse types as well the distribution of synaptic weights in the brains of different species. The emerging high-quality large structural datasets raise the question of what general functional principles can be gleaned from them. Motivated by this question, we developed statistical mechanical theory of learning neural networks that incorporates structural information as constraints. We derived an analytical solution for the memory capacity of the perceptron, basic feedforward model of supervised learning, with constraint on the distribution of its weights. Interestingly. the theory predicts that the reduction in capacity due the constrained weight-distribution is related to the Wasserstein distance between the cumulative distribution function of the constrained weights and that of the standard normal distribution. To test the theoretical predictions, we use optimal transport theory and information geometry to develop an SGD- based algorithm to find weights that simultaneously learn the input-output task and satisfy the distribution constraint. We show that training in our algorithm can be interpreted as geodesic flows in the Wasserstein space probability distributions. We further developed statistical mechanical theory for teacher-student perceptron rule learning and ask for the best way for the student to incorporate prior knowl- edge of the rule (i.e,, the teacher). Our theory shows that is beneficial for the learner to adopt different prior weight distributions during learning, and shows that distributionn-constraneed learning outperforms unconstrained and sign-constrained learning. Our theory and algorithm provide novel strategies for incorporating prior knowledge about weights into learning, and reveal powerful connection between structure and function in neural networks.
Introduction
Learning and memory are thought to take place the microscopic level by modifications of synaptic connections. Unlike learning artificial neural networks, synaptic plasticity in the brain operates under structural biological constraints. Theoretical efforts to incorporate some these constraints have focused largely the degree of connectivity [17 35] and the constraints the sign of the synapses (Excitatory vs. Inhibitory) 4 [6), but few include additional features of synaptic weight distributions observed in the brain More generally, recent large-scale connectomic studies
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

36 59 62 are beginning to provide a wealth of structural information of neuronal circuits at an unprecedented scope and level of precision, which presents a remarkable opportunity for a more refined theoretical study of learning and memory that takes into account these hitherto unavailable structural information.
Perceptron [56] is arguably the simplest model of computation by single neuron and is the fundamental building block for many modern neural networks. Despite the drastic oversimplification, studying the computational properties of (binary and analog) perceptron has been used extensively in computational
neuroscience since its dawn, particularly in the cerebellum (as a model of sensory-mon motor association)
but also in cerebral cortex (for generic associative memory functions) 43 2
16
I8
[5] [4]. Forming
associations is considered an 'atomic' building block for generic cortical functions. and perceptron memory capacity sets a tight bound on the memory capacity in recurrently connected neuronal
circuits with application to cortex and hippocampus 27 55 57 Statistical mechanical analysis predicts that near capacity, an unconstrained perceptron classifying random input-output associations a). In contrast, physiological experiments suggest that biological synapses do not change their excitatory/inhiiiorr identity during learning (but see recent (33). In order to take perceptron a step closer to biological realism, prior work has imposed sign constraints during learning 4 [6. In this case, the predicted weight distribution is delta-fundd delta-function centered at zero plus a half-norma distribution, see Fig (b. However, a wide range of connectomic studies ranging from cortical circuits in animals [38 32 47 74
has normally distributed weights (29 28 21, see [a..
40 9 to human cerebral cortex [47 62] have shown evidence of lognormally distributed synaptic connections. As an example, Fig (c) shows the weight connection distribution in mouse primary auditory cortex (data adapted from (38) D. Possible reasons for the ubiquitous lognormal distributions range from biological
62
structural/deevllopmetall constraints to computational benefits (64 60). Various potential mechanisms
for lognormal distributions have been proposed, from multiplicative gradient updates in feedforward and recurrent networks (34 40 60), to mixture of additive and multiplicative plasticity rules in spiking networks (30), but the majority of these proposals lead not just to lognormal distributions but also to sparsification in the weights. Instead of adding yet another explanation the computational origin of lognormal distribution, here we take the observed weight distribution as a prior on the network structure, and ask for its computational consequences. The goal of the paper is to present for the first time quantitative and qualitative theory of neural network learning performance under non- Gaussian and general weight distributions (not limited to lognormal distributions).
In this paper, we combine two powerful tools: statistical mechanics and optimal transport theory, and present theory of perceptron learning that incorporates the knowledge of both distribution and sign information as constraints, and gives accurate predictions for capacity and generalization error. Interestingly, the theory predicts that the reduction in capacity due the constrained weight- distribution is related to the Wasserstein distance between the cumulative distribution function of the constrained weights and that of the standard normal distribution. Along with the theoretical framework, we also present learning algorithm derived from information geometry that is capable of efficiently finding viable perceptron weights that satisfy desired distribution and sign constraints. This paper is organized as follows: in Section 2. we derive the perceptron capacity for classifying random input-output associations using statistical mechanics, and illustrate our theory with a simple example. In Section [3] we derive our learning algorithm using optimal transport theory, and show that distribution of weights found by the learning algorithm coincide with geodesic distributions on Wasserstein statistical manifold, and therefore training can be interpreted as geodesic flow. In Section[4] we analyze parameterized family of biologically realistic weight distributions, and use our theory to predict the shape of the distribution with optimal parameters. We map out the experimental parameter landscape for the estimated distribution of synaptic weights in mammalian cortex and show that our theory's 's prediction for optimal distribution is close to the experimentally measured value. In Section [ we further develop a statistical mechanical theory for teacher-student perceptron rule learning and ask for the best way for the student to incorporate prior knowledge about the weight distribution of the rule (i.e., the teacher). Our theory shows that it is beneficial for the learner to adopt different prior weight distributions during learning.
2

---

(8)
(b)
0.0
orctraied
08
0.30
5.0
4.30
9.25
2.20
-15
-2.5
Figure 1: Theoretical and empirical synaptic weight distributions. (a)-(b) predicted distribution
following perceptron learning at capacity. (a) Normal distribution when learning unconstrained. (b) A delta-function plus half-normal distribution when learning is sign-constrained. (c) Experimentally measured synaptic weight distribution (mouse primary auditory cortex (8]
2 Capacity
2.1 Learning under weight distribution constraints
We begin by considering canonical learning problem: classifying random input-output associations by a perceptron. In biological memory systems, the heavily correlated sensory data undergoing heavy preprocessing including massive decorrelations, and previous work brain related perceptron modeling 27 16 57 assumes similarly unstructured data. The data consists of ( {4)
where <d is an N-dimentional random vector drawn i.i.d. from a standard normal distribution, p(ç") = (0, 1), and ç* are random binary class labels with p((*) _ + +) + _505"* 1).
The goal is find hyperplane through the origin, described by perceptron weight vector RRN normalized to | | 1|== N.
We call w separating hyperplane when it correctly classifies all the examples with margin >>>>> 0:
S"
Ç"
(I)
We are interested in solutions w to Eqn that obey a prescribed distribution constraint, where q an arbitrary probability density function. We further demand that (w²)o(w) to fix the overall scale of the distribution (since the task is invariant to the overall scale of w). Thus, the goal of learning to find weights that satisfy [ with the additional constraint that the empirical density function ((w) = N E. 8(w w:). formed by the learned weights is similar to q(w), and more (see Section 2.2 below).
g(w).
precisely that converges to q(w) N
Extension of this setup that includes an arbitrary number of populations each satisfying its own prescribed distribution constraints is discussed in Section and in Appendix (..11. Note that the sign constraint a special case this scenario with two synaptic populations: one excitatory and one inhibitory. We further discuss the generalization of this setup to include biased inputs and sparse labels in Appendix A.1.3
2.2 Statistical mechanical theory of capacity
We are interested in the thermodynamic limit where P, - 00. but the load a = N stays O(1). This limit is amenable to mean-field analysis using statistical mechanics.
Following Gardner's seminal work 22 28
we consider the fraction of viable weights that satisfies to all possible weights:
both Eqn and the distribution constraint
f dw T
@ (s" (uuwee - xx 8(1120 12- N)S (g(k) g(k))
||l||
V
I [dwo 12|12- 12--N -N)
3

---

(a)
(b)
(c)
(d)
(standad
2.00
(6(w + 11)
2.73
1.3
p
M
129
3.0
35
1.0
1,00
W2(p,
0.75
0:52
633
6.0
Optimal
0.1
p(a)
Tangert plan 2
Figure 2: An illustration of optimal transport from standard normal distribution A/0 1) to normal distributions with nonzero mean ((VI o².a²), .²).. (a) A schematic of the space (M, W2) of proba- bility distributions. (b) An example optimal transport plan from standard normal, p(z). to normal with a = 0.5, ((w). The optimal transport plan 7 is plotted in between the distributions. T moves p(x) units of probability mass x: to location w, as indicated by the dashed line, and the colors are
chosen to reflect the amount of probability mass to be transported. (c) (V[VI- o2, o²) interpolates between standard normal (o 0) to a d-function at (a 1). (d) Capacity ar 0) as a function of a. Inset shows the W2 distance function of σ.
In Eqn[2 we impose the distribution constraint q = by demanding that in the thermodynamic limit, all Fourier modes of q and / are the same i.e., that (()) dw ((w) ((k)
1 >N esken , in the last equality we have used the definition of empirical distribution. We N perform quenched average over random patterns &" and labels ç". This amounts to calculating
(log V), which can be done using the replica trick 29 28
We focus on solutions with maximum margin capacity (x)(x) of separable patterns given margin K. We proceed by assuming replica symmetry in mean field analysis, which in general might not hold because the constraint q = q is non-convex. For all the results presented the main text, replica symmetry solution supported by numerical simulations. In Appendix A.5 we explore the validity of replica symmetric solutions in the case of strongly bimodal distributions and show that they fail only very close to the binary (Ising) limit.
given load a equivalently, the maximum load
Detailed calculations of the mean-field theory are presented in Appendix A.I.I Our mean-field theory predicts that the reduction in capacity due the distribution constraint proportional the Jacobian of the transformation from ~ ((w) to normally distributed variable xrw) ~ N(0 1),
ddw"
(c(K) = an(k))
dr
where ou (k) ش Dt(k t)²
is the capacity of an unconstrained perceptron, from Gardner 28 and x = 0 reduces to the classical result of on( = 2. The Jacobian factor, (dw /dr x. be written in terms of the constrained distribution's cumulative distribution function (CDF), (w)) and the standard normal CDF P(x)
theory 29
[ [1+ Erf( +
namely,
dw dx = S du)
(4)
Note that since the second moments are fixed to unity, 0 0 ( ) I and it equals 1 iff p 9.
2.3 Geometrical interpretation of capacity
The jacobian factor Eqn can be rewritten
dw dx
1 W2(Q, P)
(5)
2

---

where W& (k 2 in above) is the Wasserstein-k distance, given by
W& (Q (Q. U du (() 1/k
(6)
[In the following, will make frequent of both the probability density function (PDF), and the cumulative distribution function (CDF). We distinguish them by using upper case letters for CDFs, and lower case letters for PDFs.]
The Wasserstein distance measures the dissimilarity between two probability distributions, and the geodesic distance between points on the manifold of probability distributions [41] [25 [0]. Therefore, we can interpret Eqn 3 as predicting that the reduction in memory capacity tracks the geodesic distance we need travel from the standard normal distribution P to the target distribution Q (Fig aa))
We demonstrate Eqn of normal distributions, with the second moment fixed to 1: ((w) (( o², o²), see Fig (c. At a = 1, ((w) is the standard normal distribution and we recover the unconstrained Gardner capacity ao (k 0) 2. As a
Band and 5qn
instructive example. Let's consider a parameterized family
g(w) becomes ô-function
and a.(k) 0 (Fig cc)).
As evident in this simple example, perceptron capacity is strongly affected by its weight distribution. Our theory enables prediction of the shape of the distribution with optimal parameters within parameterized family of distributions. We apply our theory to a family of biologically plausible distributions and compare our prediction with experimentally measured distributions in Section
3
Optimal transport and the DisCo-SGD learning algorithm
predicts the storage capacity for perceptron with a given weight distribution, but it does not specify learning algorithm for finding a solution this non-convex learning problem. Here we present a learning algorithm for perceptron learning with a given weight distribution constraint, This algorithm will also serve test our theoretical predictions. For this purpose, we use optimal transport theory to develop an SGD-base based algorithm that is able to find max-margin solutions that obey the prescribed distribution constraint. Furthermore, we show that training can be interpreted as traveling along the geodesic connecting the current empirical distribution and the target distribution.
Stochastic gradient descent (SGD) on cross-entropy loss has been shown asymptotically converge to max-margin solutions on separable data 663 50 Given data { ) )=)= we use logistic
regression to predict class labels from perceptron weights, Ç" or aww" (1 (1+e-))= and wn the weight at the 1-th update. This defines an SGD update rule
a(2)
w w 8t Σ (5" P.
where the j-summation goes from for full-batch GD and goes from to mini-batch size B for mini-batches SGD (see Appendix A.4 for more details). The theory of optimal transport provides a principled way of transporting each individual weight w/ to a new value so that overall the new set of weights satisfies the prescribed target distribution. In 1-D, the optimal transport plan T has a closed-form solution terms of the current CDF and target CDF Q 66 3: T = Q-1 o P,
where denotes functional composition. We demonstrate the optimal transport map in Fig 2 b) for
the instructive example discussed Section 2.3
In order to apply T* to transport our weights {w:} (omitting superscript t), we form the empirical
CDF Q(w) NN Σm= 1w, Sw. which counts how many weights w, are observed below value Then the new set of weights { satisfying target CDF Q can be written as
i = Q- Q(wx))
(8)
We illustrate Eqn 8 in action in Table bb).
5

---

(a)
(b)
Algorithm Distribution constrained SGD (DisCo SGD)
1.0
Initialive palterus g", labels Ç randomly, and initialize weights ut weros;
while 5* o sgn(w"(") for some do
compute logistic regression labels: (" (== [==o(("eE")) E");
0.8
step along the gradient:
Ep (5* (*)
3 + B Σ w 5" (*)
compute empirical CDF Q(w)) = Ein uusu
step along the optimal tramsport map: wa <- wi
12 i Q to Q(us))
0.4
if crceeds mar iters then break;
1-- (vs)
10
end
11 end
0.0
2
10
1/
Table 1: Disco-SGD algorithm. (a) We perform alternating steps of gradient descent along the cross-entropy loss (Eqn 7p. followed by steps along the optimal transport direction (Eqn2) (b) An
illustration of Equ g For given wa, we first compute its empirical CDF value ((w,), then use the inverse target CDF to transport w; to its new value, ui; Q-1 (Q(wz))
(a)
(b)
(w(w) (www)
6
0.4
5.2
3.5
Figure 3: Intermediate distributions during learning are on the geodesic. (a) The solid histograms are the intermediate distribution P: at different training time from the DisCo-SGD algorithm, the dashed lines are geodesic distributions p- with the same W2 distance the target distribution Q. From right to left the training time advances, and the distributions transform further away from the 8- 5-function initialization, and approach the target distribution (a lognormal, in this example). (b) The geodesic time т as a function of the training time l. Location of the crosses correspond to the distributions shown in (a).
However, performing such a one-step projection strongly interferes with the cross-entropy objective, and numerically often results in solutions that not perfectly classify the data. Therefore, would be beneficial to have an incremental update rule based on Eqn
w +8T + w/ + 87 (10; - w.),
where we have used a different update time 7 to differentiate with the cross-entropy update time SGD'
We present our complete algorithm in Table Ta), which we named (DisCo-SGD) algorithm. In the DisCo-SGD algorithm, we perform alternating updates on Eqn 7qn7aand and Eqn D and identify 8t and 8T as learning rates and 7/. Note that in logistic regression, the norm of the weight vector ||w||| is known increase with training and the max-margin solution is only recovered at ||||| + oo. In contrast, imposing a distribution constraint fixes the norm. Therefore, to allow a variable norm, in Table we include trainable parameter / in our algorithm serve the norm of the weight vector. This algorithm allows us to reliably discover linearly separable solutions obeying the prescribed weight distribution Q.
Interestingly, Eqn EqnnItae takes a similar form to geodesic flows in Wasserstein space. Given samples {wx]
drawn from the initial distribution P and (w:} drawn from the final distribution Q, samples { {wf] from intermediate distributions P, along the geodesic can be calculated as wT) (1-7)wy +T where subscript (i) denotes ascending order (see more in Appendix A.2 For intermediate perceptron

---

(a)
Parameter space
(b)
Capucity &
[c)
(d)
10
simulation theory
Target
Sol.
Oct
0.6
1.0
0.6
Errot
p(w)
6
0.4
0.6
exporimental
2
0.2
simulation
0.2
0.2
theory
0.0
0.0
0.00
0.50
1.00
0.00
0.50
1.00
1.50
2,00
0.0
0.2
0.4
0.6
Figure Biolesically-realiticc distribution and parameter landscape. (a) Capacity (normalized by the optimal value in the landscape) function of the lognormal parameters d and dr Experimental value is shown in green with error bars, and optimal capacity is shown in red. (b)-(d) (theory from Eqn TO and simulations from DisCo-SGD): (b) Determination of capacity; (c) Max-margin x at different load a, which is the same as (x()); (d) Example weight distribution obtained in simulation.
weights wn found by our algorithm, we can compute its empirical distribution pt and compare with theoretical distribution P, along the geodesic with the same W2 distance to the target distribution (see Appendix(A.2 for how to calculate p,). ). In Fig 3 a), we show that indeed the empirical distributions Pt agree with the geodesic distributions py geodesic time T(t) (Fig 3 (a)). The relation between the
geodesic time т and the SGD update time is shown Fig b). The interplay between the cross-
entropy objective and the distribution constraint is manifested in the rate which the distribution
moves along the geodesic between the initial distribution and the target one.
4 Biologically-realistii distribution (E/I balanced lognormals) and experimental landscape
In order to apply our theory to the more biologically-realee cases, we generalize our theory from single prescribed distribution to arbitrary number of input subpopulations each obeys its own distribution. We consider a perceptron that consists of M synaptic populations w indexed by m, each constrained to satisfy its own weight distribution wi" 4m "m (w""). We denote the overall weight
vector (w" M
m= E N2,, where the total number of weights N > Nm- In this
case, the capacity Eqn is generalized to (See Appendix A.1.2 for detailed derivation):
M Σ
du
= =
Jm
(10)
dx
where 9m = N is the fraction of weights in this population. Eqn. 10 allows us to investigate the parameter space of capacity with biologically-raaistte distributions and compare with the experi- mentally measured values. In particular, we are interested the case with two synaptic populations that models the excitatory/inhiiioryy synpatic weights of biological neuron, hence, m F, T. We model the excitatory/nnhiiitory synaptic weights as drawn from two separate lognormal distributions (In maa)""
V exp {-
(Inf--a)" and w!!
(gi = 1- gE): w
exp
20
Vinarw!
20
We also demand that the mean synaptic weights satisfy the E/T balance condition 77
71
68
69
57
[8] gø () gi (w') ) as often observed in cortex connectomic experiments 15
73
51
53
With the E/I balance condition and fixed second moment, the capacity is function of the lognormal parameters og and σ, In FigHTa) we map out the 2d parameter space of og and σ, using Eqn I0 and find that the optimal choice of parameters which yields the maximum capacity solution is close to the experimentally measured values in a recent connectomic studies in mouse primary auditory cortex (38].
In order to test our theory's validity on this estimated distribution of synaptic weights, we perform DisCo-SGD simulation with model parameters FE and a, fixed to their experimentally measured values. Both the capacity (Fig 4b), max-margin xs at different load (Fig(ec)) and the empirical weights found by the algorithm (Fig are in good agreement with our theoretical prediction.
7

---

5 Generalization performance
5.1 Distribution-coonstrannedd learning as circuit inference
A central question in computational neuroscience is how underlying neural circuits determine its computation. Recently, thanks to new parallelized functional recording technologies, simultaneous recordings of the activity of hundreds of neurons in response an ensemble of inputs are possible [2). An interesting challenge is to infer the structural connectivity from the measured input- output activity patterns, It is interesting to ask how are these stimuli-responss relations related to the underlying structure of the circuit [34] 39). In the following, we try to adress this circuit reconstruction task in simple setup where student perceptron tries to learn from a teacher perceptron [6]] (2) In this setup, the teacher is considered to be the underlying ground-trun -tuth neural circuit. The student is attempting to infer the connection weights of this ground-truth circuit by observing series of input-output relations generated by the teacher. After learning is completed, one can assess the faithfulness of the inference by comparing the teacher and student. The teacher-student setup is also 37/ In this case since
a well well-known 'toy model' for studying generalization performance the learning data are generated by the teacher, the overlap between teacher and student determines the generalization performance of the learning. Here we ask to what extent prior knowledge of the teacher weight distribution helps in learning the rule and how this knowledge can be incorporated in learning. A similar motivation may arise in other contexts, in which there is a prior knowledge about the weight distribution of an unknown target linear classifier.
Let's consider the teacher perceptron, WP E TR drawn from some ground-truth distribution pt. Given random inputs 4 with p({"") = =N 1), we generate labels by Ç" = sgn(w,, / moo;:|++7") (y)", where 11t is input noise and 17th ~ N/0, a²). We task the student perceptron w, to find the max-margin linear classifier for data (5", (") >===1 max is u & x:| a/llw/l Let's define the teacher-student overlap as
Wg
Wt
R
(11)
which is a measure the faithfulness of the circuit inference. The student's generalization error is then related to the overlap by E, = 1/T arccos (R/vI+ + o²) [61
As baseline, let's first consider a totally uninformed student (without any structural knowledge of the teacher), learning from a teacher with given (in particular non- non-Gaussian) weight distribution. In this case, we can determine the overlap R (Eqn 11 as function of load a by solving the replica symmetric mean field coniisenss equations as [6] (2 An example of such learning for lognormal teacher distribution is shown in Fig 5 ("unconstrained") for the noiseless case (a 0). Note that in the presence of noise in the labels (a of 0), a is bounded by (a(c)) since max-margin learning of separable data is assumed. The case with nonzero is presented in Appendix A.3.4. In this unconstrained case, the student's 's weight distribution evolves from a Gaussian for low a to one which increasingly resembles the teacher distribution for large (Fig (b)).
Next, we consider student with information about the signs of the individual teacher weights. We can apply this knowledge as a constraint and demand that the signs of individual student weights agree with that of the teacher's. ss The additional sign- constransss require modification of replica calculation in [61 [2], which we present in Appendix A.3.I Surprisingly, we find both analytically and numerically that if the teacher weights are not too sparse, the max-margin solution generalizes poorly: after a single step of learning (with random input vectors), the overlap, R, drops substantially from its initial value (see 'sign- constranndd in F(g)) The source of the problem is that, due to the sign constraint, max-margin training with few examples yields a significant mismatch between the student and teacher weight distributions. After only a few steps of learning, half of the student's weights are set to zero, and the student's distribution, p(w.) = 1/28(0) + 1/v 2n exp(-w3/4), deviates significantly from the teacher's 's distribution (see more in Appendix A.3.3). The discrepancy between the teacher and student weight distributions therefore suggest that we should incorporate distributionncoonstraat into learning.
8

---

(a)
(b)
(c)
students
teasher
103
1.5
1,79
2.5
5,1
1.0
dig- constranedd prior
0.0 0.0
0.5
200 25
-4
0.0
Figure 5: Compare different learning paradigms. (a) Teacher-student overlap R. equivalently the generalization eπror Eg = 1/x arccos R. as a function load a in different learning paradigms. Dashed lines are from theory, and dots are from simulation. Note that there is initial drop of the overlap in sign-constrained learning due to sparsification of weights. (b)-(c) The darker color curves correspond to larger a, and dashed line is teacher distribution (same in both cases). (b) Distribution of an unconstrained student evolves from normal distribution toward the teacher distribution. (c)
Optimal student prior evolves from d-function toward the teacher distribution.
5.2 Distribuution-constraineed learning outperforms unconstrained and sign-constrained learning
Let's consider the case that the student weight are constrained to some prior distribution qa (w,). while the teacher obeys distribution pe (wy)).fr an arbitrary pair qe, Pt. We can write down the Gardner volume V, for generalization as the capacity case (Eqn()):
f dw,, TT θ (sgn
(. ||mdll
w.- + mpt)
||m,11 - -x)] o(11wa112- N78 (π(k) - (())))
duv 50 (122,1112 NN)
(12)
To obtain ensemble average of system over different realizations of the training set, perform the quenched average of log V over the patterns 5" and teacher w. and consider the thermodynamic limit of N. P + oo and a N stays O(1). We use the replica trick similar to [61 [22]. Overlap R (Eqn can be determined as a function of load a by solving the replica symmetric mean field self-consistency equations in Appendix A.3.2 In this cettriuttion-constrane numerical simulations with DisCo-SGD algorithm (Table to find such weights and compare with the predictions of our theory.
setting, we can perform
Now ask if the student has a prior on the teacher's weight distribution pt, whether incorporating this knowledge in training will improve generalization performance. One might be tempted to conclude that the optimal prior distribution the student should adopt is always that of the teacher's, i.e., 4s = Pt- We call this learning paradigm 'fixed prior', and show that its generalization performance better than that of the unconstrained and sign-constrained case (Fig(5ja)). However, instead of using a fixed prior for the student, we can in fact choose the optimal prior distribution P at different load a. This presents new learning paradigm we called 'optimal prior'. In Fig a), we show that choosing optimal priors at different a achieves the overall best generalization performance compared with all other learning paradigms. For given parameterized family of distributions, our theory provides way to analytically obtain the optimal prior pt, as function of (Fig Note that unlike the unconstrained case (Fig the optimal prior starts from a 8-function at zero a, and asymptotically approaches the teacher distribution pt as a CX.
5J),,
6 Summary and Discussion
We have developed a statistical mechanical framework that incorporates structural constraints (sign and weight distribution) into perceptron learning. The synaptic weights in our perceptron learning satisfy two key biological constraints: (1) individual synaptic signs are not affected by the learning task (2) overall synaptic weights obey prescribed distribution. These constraints may arise also in

---

neuromorphic devices [3 67]. Under the replica-symmetry assumption, we derived novel form of iestribution-constraneet perceptron storage capacity, which admits simple geometric interpretation of the reduction in capacity in terms of the Wasserstein distance between the standard normal distribution and the imposed distribution. To numerically test our analytic theory, we used tools from optimal transport and information geometry to develop an SGDDbassed algorithm, DisCo-SGI, in order to reliably find weights that satisfy such prescribed constraints and correctly classify the data, and showed that training with the algorithm can be interpreted as geodesic flows in the Wasserstein space of distributions. It would be interesting to compare our theory and algorithm to 77 where the Wasserstein distance is used to the biologically realistic case of of excitatory/inhiitooy lognormal distributions that are observed in the cortex, and found experimentally meeeunnrrllly--muurreeddde parameters close to the optimal parameter values predicted by our theory. We further studied input-output rule learning where the target rule is defined in terms of weighted sum of the inputs, and asked what extent prior knowledge of the target distribution may improve generalization performance. Using the teacher-student perceptron learning setup, we showed analytically and numerically that distribution constrained learning substantially enhances the generalization performance. In the context of circuit inference, distribution constrained learning provides novel and reliable way to recover the underlying circuit structure from observed input-output neural activities. In summary, our work provides new strategies of incorporating knowledge about weight distribution in neural learning and reveals powerful connection between structure and function in neural networks. Ongoing extensions of the present work include weight distribution constraints in recurrent and deep architectures well as testing against additional connectomic databases.
objective for training generative models. We applied our theory
Acknowledgments
This paper is dedicated to the memory of Mrs. Lily Safra, great supporter of brain research. The authors would like to thank Madhu Advani, Haozhe Shan, and Julia Steinberg for very helpful discussions. W.Z. acknowledger support from the Swartz Program in Theoretical Neuroscience at Harvard and the NIH grant NINDS (IU19NS 104653).. B.S. acknowledges support from the Stanford Graduate Fellowship. D.D.L. acknowledger support from the NIH grant NINDS (IU19NSI044) 104653). H.S. acknowledges support from the Swartz Program in Theoretical Neuroscience at Harvard, the NIH grant NINDS (ELIINSS14655)) and the Gatsby Charitable Foundation.
References
[1] Misha B Ahrens, Michael Orger, Drew N Robson, Jennifer Li, and Philipp Keller. Whole-brain functional imaging at cellular resolution using light-sheet microscopy. Nature methods, 10(5):413 420, 2013. 5.I
[2] James S Albus. A theory of cerebellar function. Mathematical biosciences, 10(1-2):25-61, 1971.
[3] Luigi Ambrosio and Nicola Gigli. A user's guide to optimal transport. In Modelling and optimisation of flows on networks, pages 1-155. Springer, 2013.
A.2
[4] Daniel J Amit, KY Michael Wong, and Colin Campbell. Perceptron learning with sign- constrained weights. Journal of Physics A: Mathematical and General, 22(12):2039, 1989.
[5] Jeffrey S Anderson, Matteo Carandini, and David Ferster. Orientation tuning of input con- ductance, excitation, and inhibition in cat primary visual cortex. Journal of neurophysiology. 84(2):909-926, 2000.
[6] Chris Angeloni and Maria N Geffen. Contextual modulation of sound processing in the auditory cortex. Current opinion in neurobiology, 49:8-15, 2018.
10

---

[7] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223, PMLR, 2017. 6
A.2
[8] Bassam V Atallah and Massimo Scanziani. Instantaneous modulation of gamma oscillation frequency by balancing excitation with inhibition. Neuron, 62(4):566-577 577, 2009.
[9] Michael Avermann, Christian Tomm, Celine Mateo, Wulfram Gerstner, and Carl CH Petersen. Microcircuits of excitatory and inhibitory neurons in layer 2/3 of mouse barrel cortex. Journal of neurophysiology, 107(11):116-3134 2012. A.1.2
[10] Carlo Baldassi, Alfredo Braunstein, Nicolas Brunel, and Riccardo Zecchina. Efficient supervised learning in networks with binary synapses. Proceedings of the National Academy of Sciences, 104(2):1107--11066
2007.
A.5.I
[11] Boris Barbour, Nicolas Brunel, Vincent Hakim, and Jean-Pierre Nadal. What can we learn from synaptic weight distributions? TRENDS in Neurosciences, 30(12):622-629 2007. [
[12] Antal Berényi, Zoltán Somogyvári, Anett J Nagy, Lisa Roux, John D Long, Shigeyoshi Fujisawa, Eran Stark, Anthony Leonardo, Timothy Harris, and György Buzsáki. Large-scale, high- density (up to 512 channels) recording of local circuits in behaving animals. Journal of neurophysiology, 132-)111311199, 2014. 5.T
[13] M Bouten, Reimers, and B Van Rompaey. Learning in the hypercube: stepping stone the binary perceptron. Physical Review E, 58(2):2378, 1998. A.5.1 A.5.I
[14] Guy Bouvier, Johnatan Aljadeff, Claudia Clopath, Célian Bimbard, Jonas Ranft, Antonin Blot, Jean-Pierre Nadal, Nicolas Brunel, Vincent Hakim, and Boris Barbour. Cerebellar learning using perturbations. Elife, 7:e31599, 2018.
[15] Nicolas Brunel. Is cortical connectivity optimized for storing information? Nature neuroscience, 19(5):749-755, 2016.
[
[16] Nicolas Brunel, Vincent Hakim, Philippe Isope, Lean-Pierre Nadal, and Boris Barbour. Optimal information storage and the distribution of synaptic weights: perceptron versus purkinje cell. Neuron, 434557555- 2004. 2.1
[17] György Buzsáki and Kenji Mizuseki. The log-dynamic brain: how skewed distributions affect network operations. Nature Reviews Neuroscience, 15(4):264-278, 2014.
[
[18] Julio Chapeton, Tarec Fares, Darin LaSota, and Armen Stepanyants. Efficient associative memory storage in cortical circuits of inhibitory and excitatory neurons. Proceedings of the National Academy of Sciences, 10965)):E36144E366
2012. A.1.2
[19] Jing Chen, Yifan Chen, Hao Wu, and Dinghui Yang. The quadratic wasserstein metric for earthquake location. Journal of Computational Physics, 373:188-209: 2018. A.2
[20] Yifan Chen and Wuchen Li. Optimal transport natural gradient for statistical manifolds with continuous sample space. Information Geometry, 3(1):1-32, 2020. 2.3 A.2
[21] Thomas M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers, EC-14(3):32-
334, 1965. [
[22] Andreas Engel and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001. 5.1 5.1 5.2 A.3.I
[23] Bjorn Engquist and Brittany D Froese. Application of the wasserstein metric to seismic signals. arXiv preprint urXiv:/311.4555 2013. A.2
[24] Bjom Engquist, Brittany D Froese, and Yunan Yang. Optimal transport for seismic full waveform inversion. arXiv preprint 1602v: 000000000040 2016. A.2
[25] Alessio Figalli and Cédrie Villani. Optimal transport and curvature, In Nonlinear PDE's and Applications, pages 171-217. Springer, 2011. 2.3
A.2

---

[26] Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learning with a wasserstein loss. Advances in neural information processing systems, 28, 2015.
A.2
[27] Elizabeth Gardner. Maximum storage capacity in neural networks. EPL (Europhysics Letters),
4(4))481, 1987. 2
[28] Elizabeth Gardner. The space of interactions in neural network models. Journal of physics A:
Mathematical and general, 21(1): 1988. T 2.2
[29] Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models.
Journal of Physics A: Mathematical and general, 21(1):271, 1988.
2.2 2.2
[30] Matthieu Gilson and Tomoki Fukai. Stability versus neuronal specialization for stdp: long-tail weight distributions solve the dilemma. PloS one, 6(10):a2533, 201
[31] Geonhui Han, Chuljun Lee, Jae-Eun Lee, Jongseon Seo, Myungjun Kim, Yubin Song, Young- Ho Seo, and Daeseok Lee. Alternative negative weight for simpler hardware implementation of synapse device based neuromorphic system. Scientific reports, 11(1):1-7, 2021. 6
[32] Carl Holmgren, Tibor Harkany, Björn Svennenfors, and Yuri Zilherter. Pyramidal cell com- munication within local networks in layer 2/3 of rat neocortex. The Journal of physiology, 551(1):19-153,, 2003,
[ A.1.2
[33] SeulAh Kim, Michael L Wallace, Mahmoud El-Rifai, Alexa R Knudsen, and Bernardo L Sabatini. Co-packaging of opposing neurotransmitters individual synaptic vesicles in the central nervous system. Neuron, 2022. g
[34] Jyrki Kivinen and Manfred K Warmuth. Exponentiated gradient versus gradient descent for linear predictors. information and computation, 132(1):1-633 1997.
[35] Alexei A Koulakov, Tomáš Hromádka, and Anthony M Zador. Correlated connectivity and the distribution of firing rates in the neocortex. Journal of Neuroscience, 29 29(12):38853664 2009.
[36] Michael Kunst, Eva Laurell, Nouwar Mokayes, Anna Kramer, Fumi Kubo, António M Fernan- des, Dominique Förster, Marco Dal Maschio, and Herwig Baier. A cellular-resolution atlas of the larval zebrafish brain. Neuron, 103(1):21-38, 2019.
[37] Sebastian Lee, Sebastian Goldt, and Andrew Saxe. Continual learning in the teacher-student setup: Impact of task similarity. In International Conference on Machine Learning, pages 6109-6119 19. PMLR, 2021. 5.1
[38] Robert B Levy and Alex D Reyes. Spatial profile of excitatory and inhibitory synaptic connec- tivity in mouse primary auditory cortex. Journal of Neuroscience, 32(16):5609-561, 2012.
A.1.2 A.4
[39] Jian K Liu, Helene M Schreyer, Arno Onken, Fernando Rozenblit, Mohammad Khani, Vidhyasankar Krishnamoorthy, Stefano Panzeri, and Tim Gollisch. Inference of neuronal func- tional circuitry with
non-negative matrix factorization. Nature communications,
8(1):1-14, 2017. 5.1
[40] Yonatan Loewenstein, Annerose Kuras, and Simon Rumpel. Multiplicative dynamics underlie the emergence of the log-normal distribution of spine sizes in the neocortex in vivo. Journal of Neuroscience, (1(2)))9981-945))) 2011.
[41] John Lott. Some geometric calculations on wasserstein space. arXiv preprint math/0612562. 2006.
2.3
A.2
[42] Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with teacher-student model. Advances in Neural Information Processing Systems, 34: 14:11117-18111,
2021. 5.]
12

---

[43] David Marr. A theory of cerebellar cortex. The Journal of Physiology, 202(2):437, 1969.
[44] Tambet Matüsenn, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems, (1nmnn,,11))))) 3109):3732-3740, 2019.
5.1
[45] Ludovic Métivier, Romain Brossier, Quentin Mérigot, Edouard Oudet, and Jean Virieux. Mea- suring the misfit between seismograms using an optimal transport distance: Application to full waveform inversion. Geophysical Supplements to the Monthly Notices of the Royal Astronomical Society, 205(():344-337) 2016.
A.2
[46] Ludovic Métivier, Romain Brossier, Quentin Merigot, Edouard Oudet, and Jean Virieux. An optimal transport approach for seismic tomography: Application to 3d full waveform inversion. Inverse Problems, (2(11)::1111111111))))))1) 1 2016. A.2
[47] Gábor Molnár, Szaboles Oláh, Gergely Komlósi, Miklós Füle, János Szabadics, Csaba Varga, Pál Barzó, and Gábor Tamás. Complex events initiated by individual spikes in the human cerebral cortex. PLoS biology, 6(9):e222, 2008. [ A.1.2
[48] Gianluigi Mongillo, Simon Rumpel, and Yonatan Loewenstein. Inhibitory connectivity defines the realm of excitatory plasticity. Nature neuroscience, 21(10):143-1470, 2018.
[49] Grégoire Montavon,
Müller, and Marco Cuturi. Wasserstein training of restricted holtzmann machines. Advances in Neural Information Processing Systems, 29, 2016. A.2
[50] Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3051-3059. PMLR, 2019. 3
[51] Michael Okun and Ilan Lampl. Instantaneous correlation of excitation and inhibition during ongoing and sensory-evoked activities. Nature neuroxcience, 11(5)5335-337 2008.
[52] RW Penney and D Sherrington. The weight-space the binary perceptron. Journal of Physics A: Mathematical and General, 26(22):6173, 1993. A.5. A.5.1
[53] Cindy Poo and Jeffry Isaacson. Odor representations in olfactory "sparse coding, global inhibition, and oscillations. Neuron, 62(())850-88) 2009.
[54] Esteban Real, Hiroki Asari, Tim Gollisch, and Markus Meister. Neural circuit inference from function to structure. Current Biology, 27(2):189-198, 2017.
5.1
[55] Edmund T Rolls, Alessandro Treves, and Edmund Rolls. Neural networks and bruin function, volume 572. Oxford university press Oxford, 1998. !
[56] Frank Rosenblatt. The perceptron: probabilistic model for information storage and organiza- tion in the brain. Psychological review, 65(6):386, 1958.
[57] Ran Rubin, LF Abbott, and Haim Sompolinsky. Balanced excitation and inhibition are required for high-capacity, noise-robust neuronal selectivity. Proceedings of the National Academy of Sciences, 1 (14(44)::99666-99333 2017. 2.1
[58] Maziar Sanjabi, Jimmy Ba, Meisam Razaviyayn, and Jason D Lee. On the convergence and robustness of training gans with regularized optimal transport. Advances in Neural Information Processing Systems, 31, 2018. [
[59] Louis K Scheffer, C Shan Xu, Michal Januszewski, Zhiyuan Lu, Shin yhin-ya Takemura, Kenneth J Hayworth, Gary B Huang, Kazunori Shinomiya, Jeremy Maitlin-Shepard Stuart Berg, et al. A connectome and analysis of the adult drosophila central brain. Elife, 9te57443, 2020.
[60] Gabriele Scheler. Logarithmic distributions prove that intrinsic learning is hebbian. F1000Research, 6, 2017.
[61] Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A, 45(8):6056, 1992. 5.1 5.1 5.2 A.3.I
13

---

[62] Alexander Shapson-Coe, Michat Januszewski, Daniel R Berger, Art Pope. Yuelong Wu, Tim Blakely, Richard Schalek, Peter Li, Shuohong Wang, Jeremy Saitti-Shepardd et al. A connectomic study of petascale fragment of human cerebral cortex. BioRxiv, 2021.
[63] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2222-2878, 2018.7
[64] Jun-nosuke Teramae and Tomoki Fukai. Computational implications of lognormally distributed synaptic weights. Proceedings of the IEEE, 102(4):500-55) 2014. [
[65] Alex M Thomson, David C West, Yun Wang, and A Peter Bannister. Synaptic connections and small circuits involving excitatory and inhibitory neurons in layers 2-5 of adult rat and cat neocortex: triple intracellular recordings and biocytin labelling in vitro. Cerebral cortex,
12(9):936-99) 2002. A.1.2
[66] Matthew Thorpe Introduction to optimal transport. Lecture Notes, 2019.
A.2
[67] Son Ngoc Truong and Kycong-Sik Min. New memristor-based crossbar arry architecture with 50-% area reduction and 48-% power saving for matrix-vecter multiplication of analog neuromorphic computing. JSTS: Journal of Semiconductor Technology and Science, 14(3):356- 363, 2014. 6
[68] Misha Tsodyks and Terrance Sejnowski. Rapid state switching in balanced cortical network models. Network: Computation in Neural Systems, 6(2):111, 1995. 4
[69] C Van Vreeswijk and H Sompolinsky. Irreullar activity in large networks of neurons. In Les Houches, volume 80, pages 341-406. Elsevier, 2005,
[70] Carl Van Vreeswijk and Haim Sompolinsky. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science, 274(5293):1244-7722 1996. 4
[71] Carl van Vreeswijk and Haim Sompolinsky. Chaotic balanced state in model of cortical circuits. Neural computation, 10(6):132--133 1998.
[72] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009. A.2
[73] Michael Wehr and Anthony M Zador. Balanced inhibition underlies tuning and sharpens spike timing in auditory cortex. Nature, 426(6965):442-4444 446, 2003. 4
[74] Jian-Ming Yang. Jing Zhang. Xiao-Juan Chen, Hong- Kong-YYn Geng, Mao Ye, Nicholas C Spitzer, Jian-Hong Luo, Shu-Min Duan, and Xiao-Ming Li. Development of gaba circuitry of fast- spiking basket internuurons the medial prefrontal cortex of erbb4-mutant mice. Journal of Neuroscience, 19724 2013. T. A.1.2
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
14

---

(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 2.2 and Appendix
A.5
(c) Did you discuss any potential negative societal impacts of your work? [No] We do not
see any potential negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.1 A.1
A.3
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See Section 3
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section A..4
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] Our theory concerns the thermodynamic limit where the observables are dominated by their typical values and fluctuations are small.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you'ne using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "vQazy60F-WI", "reviewer": "Reviewer_42gB", "summary": "In this paper, the authors:\n- provide theory for memory capacity (at a given classification margin) for perceptrons with distribution-constrained weights, and that capacity loss from optimum can be phrased as the Wasserstein distance between the target distribution and standard normal\n- design a novel learning algorithm that account for both classification loss and distribution matching (via optimal transport)\n- apply to a perceptron with balanced E & I input populations\n- investigate the optimal prior for a student neuron in a task where the student neuron is asked to match the teacher's weight distribution given only the teacher's input-output response", "questions": "As stated above, the question of how task optimization and weight distribution matching can be simultaneously achieved, given experimental observations of log-normally distributed weights, is a very important question. However, the paper starts with this motivation, but essentially solves the problem for a single perceptron (or, uni-dimensional output logistic regression) with random inputs. I understand that assumptions are necessary for such theoretical work, which I believe is of high quality. But the massive reduction in scope feels almost like a bait and switch (though I'm sure the authors do not intend for this to be the case). At the very least, some investigation on a system that relaxes one or several of the assumptions should be done, even if the theory does not end up so clean:\n\n- would the theory / simulation results hold for non-random inputs? Basically in no scenario does the real brain have to perform classification on perceptually random inputs, so at the minimum, a toy experiment on structured inputs would be justified (MNIST 0 vs 1, for example).\n\n- similarly, the paper is motivated by neural \"networks\", so how do the results hold up for more than a single neuron? For higher dimensional outputs? Do compensation of weights for different neurons violate the theory?\n\n- I don't see how the results of Figure 5 is relevant to the overall story. Perhaps more justification or clarity in writing needs to be provided\n\n- in the learning algorithm, why does distribution matching have to occur separately from weight optimization? I suppose it fits nicely with the optimal transport theory, but simply optimizing a loss function that combines task performance and, e.g., DKL or Wasserstein distance between the weight distributions, simultaneously would be naively reasonable, and no less biologically implausible (since individual weights do not know of the other weights and certainly does not actively converge to some target lognormal distribution). ", "limitations": "Some discussion of future works, but very little discussion of limitations in section 6. No discussion of societal impact (but none necessary)", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "originality: in my limited knowledge, I do not know of other related works that investigate distribution-constrained weights of a network from a theoretical / statistical mechanics point of view\n\nquality: the proof itself (section 2.2) is beyond me, but at a superficial level, the main result (Fig 2-4) seem solid and that simulations support the theoretical prediction. I don't really get how the teacher-student learning paradigm is relevant, however. It seems to be a bit of a contrived setting: even though it makes sense from a decoding perspective (as a neuroscientist trying to infer the weights of a neuron given only its input and outputs), no biological neuron would be required to do this as the student.\n\nclarity: the work is clearly presented, with high quality and thoughtful visualizations, as well as an easy-to-follow high level narrative. \n\nsignificance: the high level motivation of distribution-matched learning is clearly stated, and of significant interest to the neuroscience community at large. However, the amount of assumptions / simplifications required to arrive at the actual results dampens my enthusiasm (expanded below)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_RcMveE7PL", "reviewer": "Reviewer_8wH5", "summary": "The paper develops a theoretical framework for weight distribution-constrained perceptron learning using replica method. They compute capacity based on the proposed distribution, and develop a modification to SGD to satisfy the weight distribution constraints. Their theory is verified in simulations, and it allows to compute the optimal weight prior in the student-teacher learning case. Moreover, its predictions match the experimentally observed weight distributions in mouse primary auditory cortex.", "questions": "### General suggestions\nSec. 5: it'd be nice to see a discussion on why you're comparing the auditory cortex to a perceptron in the first place. Does the data from [27] come from a specific subarea known to perform just classification? If it's the whole auditory cortex, why do you think it's comparable?  \n\nLines 99-100: worth noting that the number of synapses per neuron is at most 1k-10k, depending on the area. The large N limit with many memories presumably makes the number of synapses infinite too. (It's a small point that doesn't invalidate any of the results; it's just worth keeping in mind when doing capacity calculations, in my opinion.)\n\nSign constrained in Fig. 5a: could you fix the drop of the blue line by reducing the initial learning rate? \n\n### Small corrections\n\nTable 1 is really a figure… \n\nLine 121: reiterate that it holds only when the variance of w is one. Otherwise, it’s confusing.\n\nLine 233: put parenthesis for arccos\n\nLine 241: capitalize Gaussian\n\nAppendix placeholders should be removed from the main pdf\n\n**Appendix**\n\nFigs. 5-6: separation, not seperation in the titles\n\nEq. 2: $\\lambda$ should be $\\rho$\n\nEq. 3: better switch index i to something else (you already have the imaginary unit)\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. Capacity calculations are rigorous and in good agreement with simulations.\n2. Student-teacher results on optimal priors are interesting and non-trivial.\n3. The match with mouse data for E/I networks is really interesting, although it's not clear to me why the auditory cortex should look like a big perceptron. \n\n### Weaknesses\n1. The results of this paper might be hard really to extend to multi-layer network (which is not surprising)\n2. ...and also hard/tedious to extend to other distributions, as it would require a replica calculation (although some ground work for that is done in the appendix)\n\nThese are not crucial problems, however.\n\n### Summary\nI think it's an interesting paper with a solid theoretical results, good experiments and an interesting connection to real neural data; 8 (strong accept).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YGuNj87GtFJ", "reviewer": "Reviewer_hTnp", "summary": "The authors compare the capacity of a perceptron with normally distributed weights to the capacity of a perceptron with weights distributed according to a specified distribution. They show analytically that the difference in capacity is a function of the Wasserstein distance between the normal distribution and the specified distribution. Based on their analysis, the authors derive an algorithm for optimizing the weights of a perceptron subject to a specified weight distribution, which they analyze using tools from optimal transport theory. The authors consider a perceptron whose positive/negative weights are constrained to have log-normal distributions and show that the optimal parameters of the log-normal distributions are close to experimentally observed parameters from connectomics data. Finally, they apply their theory to a student-teacher network and compute the optimal prior on the student network weights given knowledge of the distribution of the teacher network weights.", "questions": "Given the discrepancy between experimentally observed weight distributions and the theoretically optimal distribution for the perceptron, why is the natural problem (from the perspective of a computational neuroscientist) to understand the performance of the perceptron under a constrained weight distribution?\n\nDoes your learning algorithm suggest how a biological circuit would enforce the distribution constraint (specifically, the optimal transport map step)?\n\nIs there prior work along these lines?\n\nMinor questions:\n\nWhy are Eq (4) and Eq (5) equivalent? Because the moments of the distributions are fixed to 1? I think this could be clarified.\n\nIn Eq (7) and Algorithm 1, what is $\\mu$ summed over? The current batch? It current reads more like GD than SGD.", "limitations": "I do not foresee any direct negative societal impact of their work.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As far as I can tell, this work is original (though this is not my specific area of expertise) and their technical arguments are clear (though I did not read the appendices carefully).\n\nMy main concern is with the motivation of the paper. In the second paragraph of the introduction, the authors first cite work stating that near capacity, an unconstrained perceptron classifying random input-output associations has normally distributed weights. Then they cite work showing that connectomics data indicates that weight distributions are not normally distributed, but rather closer to log-normally distributed. From my point of view, the natural conclusion from these theoretical and experimental works is that the perceptron is the wrong model with which to understand the log-normal weight distribution. A potential question is then: \"What is the simplest task that is optimized when the weights are log-normal?\" However, the authors come to an alternative conclusion: \"The discrepancy between theory and current experimental result... call for a theory of perceptron learning that incorporates the knowledge of global structural information.\" Why is this the natural conclusion? Are the authors assuming that the log-normal distribution of the weights is a constraint that is not due to the learning task? I think this needs to be clarified.\n\nThis being said, I believe the results are interesting and worthwhile from a theoretical perspective, but the connections to brain function are less clear. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7EO7OD_cSl", "reviewer": "Reviewer_X5tG", "summary": "The manuscript presents a generalization of traditional perceptron analysis, under constraints over the marginal distribution of weights, showing that perceptron capacity is negatively affected, the more so as the target weight distribution deviated from the unit normal.  Furthermore, a SGD-type algorithm is provided for the solution ", "questions": "- what does this have to do with biological weight statistics: do you view connectomics data as a source of additional constraints for learning? and if so how does one resolve the apples vs oranges issue of the rest of the statistics being mismatched (see above)?\n- i did not understand what exactly do the 'experimental measurements' correspond to in fig4a. please clarify. what does the notion of perceptron capacity is supposed to correspond to in mouse auditory cortex?\n- what brain process does the student teacher (a perceptron version of mimic task) setup correspond to? \n- isn't it trivially expected that an accurate prior (true weight distribution) should improve performance in this task?\n- how is the 'optimal prior' defined exactly?\n- is there any relevance of these constraints for machine learning practice?\n- is there any benefit of log-normality of weight distributions in this framework?", "limitations": "No issues", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- interesting capacity analysis in that it combines classic tools (Gardner) with optimal transport\n- optimization algorithm that goes with it: standard logistic regression combined with steps ensuring marginal statistics match the target distribution(s) as close as possible\n\nWeaknesses:\n- although the math seems sound enough, the link to biology remains fuzzy throughout, at even the broadest level of why is perceptron capacity a useful measure for anything biological, what the student teacher setup is supposed to correspond to, or most importantly the reasons for nonnormality in biological weight distributions\n- i found  the arguments on the discrepancy between the marginal statistics of weight strengths in perceptrons trained on a family of random classification tasks and biological weight distributions and the need to unify the two to be fundamentally suspicious; given the discrepancy in computational tasks (input statistics, nature of ecologically relevant tasks, etc), all the heterogeneity in biological units that is in no way modeled in the theory, plus constraints on the learning processes themselves it is not clear at all what weight constrained perceptrons have to do with the data. Also the directionality of the argument seems to flip from the marginal weight distributions being a pre-defined constraint or an explanandum. More generally, the introduction was poorly written at if left unclear what the point of the exercise is exactly.\n- accessibility: significant parts of the results text require the reader to be quite familiar with statistical mechanics in general and the results of Gardner on perceptron capacity in particular; this limits somewhat readability for what i'd consider to be the typical neurips reader. Variables are not always clearly defined, which does not help either (e.g. line 104 'k').\n-  main text figures were rather uninformative, the numerical vs analytical key comparisons were relegated to the appendix.\n- section 5.2 is very difficult to follow, with not enough information provided to be able to judge the reasonableness of the claims ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept