Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors:
- provide theory for memory capacity (at a given classification margin) for perceptrons with distribution-constrained weights, and that capacity loss from optimum can be phrased as the Wasserstein distance between the target distribution and standard normal
- design a novel learning algorithm that account for both classification loss and distribution matching (via optimal transport)
- apply to a perceptron with balanced E & I input populations
- investigate the optimal prior for a student neuron in a task where the student neuron is asked to match the teacher's weight distribution given only the teacher's input-output response
strengths_and_weaknesses: originality: in my limited knowledge, I do not know of other related works that investigate distribution-constrained weights of a network from a theoretical / statistical mechanics point of view

quality: the proof itself (section 2.2) is beyond me, but at a superficial level, the main result (Fig 2-4) seem solid and that simulations support the theoretical prediction. I don't really get how the teacher-student learning paradigm is relevant, however. It seems to be a bit of a contrived setting: even though it makes sense from a decoding perspective (as a neuroscientist trying to infer the weights of a neuron given only its input and outputs), no biological neuron would be required to do this as the student.

clarity: the work is clearly presented, with high quality and thoughtful visualizations, as well as an easy-to-follow high level narrative. 

significance: the high level motivation of distribution-matched learning is clearly stated, and of significant interest to the neuroscience community at large. However, the amount of assumptions / simplifications required to arrive at the actual results dampens my enthusiasm (expanded below)
questions: As stated above, the question of how task optimization and weight distribution matching can be simultaneously achieved, given experimental observations of log-normally distributed weights, is a very important question. However, the paper starts with this motivation, but essentially solves the problem for a single perceptron (or, uni-dimensional output logistic regression) with random inputs. I understand that assumptions are necessary for such theoretical work, which I believe is of high quality. But the massive reduction in scope feels almost like a bait and switch (though I'm sure the authors do not intend for this to be the case). At the very least, some investigation on a system that relaxes one or several of the assumptions should be done, even if the theory does not end up so clean:

- would the theory / simulation results hold for non-random inputs? Basically in no scenario does the real brain have to perform classification on perceptually random inputs, so at the minimum, a toy experiment on structured inputs would be justified (MNIST 0 vs 1, for example).

- similarly, the paper is motivated by neural "networks", so how do the results hold up for more than a single neuron? For higher dimensional outputs? Do compensation of weights for different neurons violate the theory?

- I don't see how the results of Figure 5 is relevant to the overall story. Perhaps more justification or clarity in writing needs to be provided

- in the learning algorithm, why does distribution matching have to occur separately from weight optimization? I suppose it fits nicely with the optimal transport theory, but simply optimizing a loss function that combines task performance and, e.g., DKL or Wasserstein distance between the weight distributions, simultaneously would be naively reasonable, and no less biologically implausible (since individual weights do not know of the other weights and certainly does not actively converge to some target lognormal distribution). 
limitations: Some discussion of future works, but very little discussion of limitations in section 6. No discussion of societal impact (but none necessary)
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The manuscript presents a generalization of traditional perceptron analysis, under constraints over the marginal distribution of weights, showing that perceptron capacity is negatively affected, the more so as the target weight distribution deviated from the unit normal.  Furthermore, a SGD-type algorithm is provided for the solution 
strengths_and_weaknesses: Strengths:
- interesting capacity analysis in that it combines classic tools (Gardner) with optimal transport
- optimization algorithm that goes with it: standard logistic regression combined with steps ensuring marginal statistics match the target distribution(s) as close as possible

Weaknesses:
- although the math seems sound enough, the link to biology remains fuzzy throughout, at even the broadest level of why is perceptron capacity a useful measure for anything biological, what the student teacher setup is supposed to correspond to, or most importantly the reasons for nonnormality in biological weight distributions
- i found  the arguments on the discrepancy between the marginal statistics of weight strengths in perceptrons trained on a family of random classification tasks and biological weight distributions and the need to unify the two to be fundamentally suspicious; given the discrepancy in computational tasks (input statistics, nature of ecologically relevant tasks, etc), all the heterogeneity in biological units that is in no way modeled in the theory, plus constraints on the learning processes themselves it is not clear at all what weight constrained perceptrons have to do with the data. Also the directionality of the argument seems to flip from the marginal weight distributions being a pre-defined constraint or an explanandum. More generally, the introduction was poorly written at if left unclear what the point of the exercise is exactly.
- accessibility: significant parts of the results text require the reader to be quite familiar with statistical mechanics in general and the results of Gardner on perceptron capacity in particular; this limits somewhat readability for what i'd consider to be the typical neurips reader. Variables are not always clearly defined, which does not help either (e.g. line 104 'k').
-  main text figures were rather uninformative, the numerical vs analytical key comparisons were relegated to the appendix.
- section 5.2 is very difficult to follow, with not enough information provided to be able to judge the reasonableness of the claims 
questions: - what does this have to do with biological weight statistics: do you view connectomics data as a source of additional constraints for learning? and if so how does one resolve the apples vs oranges issue of the rest of the statistics being mismatched (see above)?
- i did not understand what exactly do the 'experimental measurements' correspond to in fig4a. please clarify. what does the notion of perceptron capacity is supposed to correspond to in mouse auditory cortex?
- what brain process does the student teacher (a perceptron version of mimic task) setup correspond to? 
- isn't it trivially expected that an accurate prior (true weight distribution) should improve performance in this task?
- how is the 'optimal prior' defined exactly?
- is there any relevance of these constraints for machine learning practice?
- is there any benefit of log-normality of weight distributions in this framework?
limitations: No issues
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper develops a theoretical framework for weight distribution-constrained perceptron learning using replica method. They compute capacity based on the proposed distribution, and develop a modification to SGD to satisfy the weight distribution constraints. Their theory is verified in simulations, and it allows to compute the optimal weight prior in the student-teacher learning case. Moreover, its predictions match the experimentally observed weight distributions in mouse primary auditory cortex.
strengths_and_weaknesses: ### Strengths
1. Capacity calculations are rigorous and in good agreement with simulations.
2. Student-teacher results on optimal priors are interesting and non-trivial.
3. The match with mouse data for E/I networks is really interesting, although it's not clear to me why the auditory cortex should look like a big perceptron. 

### Weaknesses
1. The results of this paper might be hard really to extend to multi-layer network (which is not surprising)
2. ...and also hard/tedious to extend to other distributions, as it would require a replica calculation (although some ground work for that is done in the appendix)

These are not crucial problems, however.

### Summary
I think it's an interesting paper with a solid theoretical results, good experiments and an interesting connection to real neural data; 8 (strong accept).
questions: ### General suggestions
Sec. 5: it'd be nice to see a discussion on why you're comparing the auditory cortex to a perceptron in the first place. Does the data from [27] come from a specific subarea known to perform just classification? If it's the whole auditory cortex, why do you think it's comparable?  

Lines 99-100: worth noting that the number of synapses per neuron is at most 1k-10k, depending on the area. The large N limit with many memories presumably makes the number of synapses infinite too. (It's a small point that doesn't invalidate any of the results; it's just worth keeping in mind when doing capacity calculations, in my opinion.)

Sign constrained in Fig. 5a: could you fix the drop of the blue line by reducing the initial learning rate? 

### Small corrections

Table 1 is really a figure… 

Line 121: reiterate that it holds only when the variance of w is one. Otherwise, it’s confusing.

Line 233: put parenthesis for arccos

Line 241: capitalize Gaussian

Appendix placeholders should be removed from the main pdf

**Appendix**

Figs. 5-6: separation, not seperation in the titles

Eq. 2: $\lambda$ should be $\rho$

Eq. 3: better switch index i to something else (you already have the imaginary unit)

limitations: The authors adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors compare the capacity of a perceptron with normally distributed weights to the capacity of a perceptron with weights distributed according to a specified distribution. They show analytically that the difference in capacity is a function of the Wasserstein distance between the normal distribution and the specified distribution. Based on their analysis, the authors derive an algorithm for optimizing the weights of a perceptron subject to a specified weight distribution, which they analyze using tools from optimal transport theory. The authors consider a perceptron whose positive/negative weights are constrained to have log-normal distributions and show that the optimal parameters of the log-normal distributions are close to experimentally observed parameters from connectomics data. Finally, they apply their theory to a student-teacher network and compute the optimal prior on the student network weights given knowledge of the distribution of the teacher network weights.
strengths_and_weaknesses: As far as I can tell, this work is original (though this is not my specific area of expertise) and their technical arguments are clear (though I did not read the appendices carefully).

My main concern is with the motivation of the paper. In the second paragraph of the introduction, the authors first cite work stating that near capacity, an unconstrained perceptron classifying random input-output associations has normally distributed weights. Then they cite work showing that connectomics data indicates that weight distributions are not normally distributed, but rather closer to log-normally distributed. From my point of view, the natural conclusion from these theoretical and experimental works is that the perceptron is the wrong model with which to understand the log-normal weight distribution. A potential question is then: "What is the simplest task that is optimized when the weights are log-normal?" However, the authors come to an alternative conclusion: "The discrepancy between theory and current experimental result... call for a theory of perceptron learning that incorporates the knowledge of global structural information." Why is this the natural conclusion? Are the authors assuming that the log-normal distribution of the weights is a constraint that is not due to the learning task? I think this needs to be clarified.

This being said, I believe the results are interesting and worthwhile from a theoretical perspective, but the connections to brain function are less clear. 
questions: Given the discrepancy between experimentally observed weight distributions and the theoretically optimal distribution for the perceptron, why is the natural problem (from the perspective of a computational neuroscientist) to understand the performance of the perceptron under a constrained weight distribution?

Does your learning algorithm suggest how a biological circuit would enforce the distribution constraint (specifically, the optimal transport map step)?

Is there prior work along these lines?

Minor questions:

Why are Eq (4) and Eq (5) equivalent? Because the moments of the distributions are fixed to 1? I think this could be clarified.

In Eq (7) and Algorithm 1, what is $\mu$ summed over? The current batch? It current reads more like GD than SGD.
limitations: I do not foresee any direct negative societal impact of their work.
review_title: null

Meta-review: metareview: Reviewers appreciate the novel weight-distribution contrained algorithm, in spite of reservations about potential impact in comp neuro or ML remain.
confidence: Less certain
Decision: Accept