Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: UPDATED SCORE AFTER REBUTTAL.

The pioneering work of Pennington et al. used free probability to predict the spectrum of the input-output Jacobian. The conditioning of this matrix is known to be important for training stability. The authors generalize those results to sequences of width ratios that need not be identical. Despite knowing the self-consistent equation for the spectrum, obtaining numerical predictions are known to be numerically challenging. The authors propose a new method, which is guaranteed to converge, to numerically solve for the spectrum. Experiments are also performed to show that the 90th percentile of the spectrum is highly correlated with the loss of a fully connected neural network trained on MNIST.
strengths_and_weaknesses: Strengths
- To my knowledge, the lilypad algorithm is an original contribution even in the free probability community and not merely free probability applied to ML. The speed-ups provided over the algorithm of Pennington et al. are significant. Combined with the convergence guarantee the lilypad algorithm appears preferable.
- The paper is clear and easy to read. In particular, the overview of free probability is nice and an accessible introduction.
- It's great to see open sourced code released with the paper.

Weaknesses
- The paper's significance is hurt by only considering the MNIST dataset in the experiments. It should be easy to try CIFAR10 for example to check whether the correlation still holds. This is important given how easy MNIST is as a benchmark.
- Similarly only FC architectures are considered. Even the activation function is unchanged.
- The main contributions are interesting mathematically but they might not be of great interest to the NeurIPS community, given the limitations above in the more practical experiments.
questions: - Why is there a line of points in Figure 2 at $x\approx-1.6$?
- Correlation for a random sample of architectures is interesting, however, there is still a question of whether this metric could be used iteratively in some kind of hyperparameter search.
- There have been other works using free probability to study ML. Can this algorithm be used for the equations in any of those cases?
- Were other functionals of the spectrum than the 90th percentile considered?
limitations: No concerns about negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents an extension of the work by Pennington et al. 2018 for the spectrum of the Jacobian in deep fully connected neural networks at initialization. The result presented in the paper requires the numerical computation of the S-transform and the authors presented a method that out-performs the proposals from Pennington et al. 2018. 
strengths_and_weaknesses: Strengths:
* The paper presents an improvement over the results of Pennington et al. 2018 to more realistic settings.

Weaknesses:
* The improvement is slight. Although I agree with the authors that the settings of Pennington et al. are far from the DNN used in applications, the same critique applies to their setup: they only consider dense layers. 
* I couldn't understand whether they found situations where their generalisation leads to architectures that outperform the one identifiable using Pennington's result. 
* The section on the empirical results shows tests on the MNIST dataset. However, on this dataset even a single-layer network can achieve more than 90% accuracy, I honestly find that section not significant unless they show results on simple but more complex datasets (for instance fashion MNIST). 
* Overall I found the paper hard to read because it follows a rather unconventional structure. I would suggest shuffling and re-adapting some subsections of section 1 in order to be closer to other papers in ML conferences. For instance, without the intension of forcing this order in particular, the following order seems more natural to me:
   Introduction ->  Randomness -> Modelling -> Contributions -> Structure of the paper -> (maybe this part could go in section 3) Framework -> intuition (this doesn't seem necessary) 
* It would be good to add a "related work" section. The work from Pennington is from 2018, I believe progress has been made since then and probably they should be mentioned. 
* Connected with the assumption made in section 3 on the equivalence to Gaussian random variables, I would point out the work below in supports "Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mezard, Lenka Zdeborova Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, PMLR 145:426-471, 2022."
questions: Please see the section "Strengths And Weaknesses"
limitations: The limitations have not been addressed. I commented on this point in the section "Strengths And Weaknesses".
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper deals with the multiplicative case of the free rectangular convolution and its application to neural networks. 

The multiplicative case of the free rectangular convolution, contrarily to the additive one, comes down, as it is noticed i Section 2 of the paper, to the square case (ie to the case of square matrices). The numerical computation algorithm the authors give was a missing pice in this pipeline and it is valuable.

I thus recommend acceptation.
strengths_and_weaknesses: This paper deals with the multiplicative case of the free rectangular convolution and its application to neural networks. 

The multiplicative case of the free rectangular convolution, contrarily to the additive one, comes down, as it is noticed i Section 2 of the paper, to the square case (ie to the case of square matrices). The numerical computation algorithm the authors give was a missing pice in this pipeline and it is valuable.

I thus recommend acceptation.
questions: .
limitations: .
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work extends the free probability theory in the rectangular setup and gives a new method for computing the spectral density of the Jacobian of a general feed-forward neural network with theoretical guarantees. In addition, it also empirically demonstrates that the accuracy of a feed-forward neural network after training is highly correlated to higher quantiles of the theoretically computed spectral distribution.
strengths_and_weaknesses: Strengths:
- This work is well-motivated: focusing on understand the spectrum of the Jacobian for varying architectures and initializations, providing an effective computation method, and preventing unnecessary tuning of neural networks based on these results.
- On the theoretical side, the authors generalize previous result to rectangular matrices and provide an associate multiplicative convolution operation, which appears novel.
- On the computation side, they present a new method for computing the spectrum leveraging the Newton lilypads, which not only providing theoretical guarantees of always choosing the correct branch but also much faster comparing to existing methods.
- To demonstrate the usefulness of the proposed method, the authors show that the accuracy of a feed-forward neural network after training is highly correlated to higher quantiles of the theoretically computed spectral distribution.

Weaknesses: the main weakness of this paper is the lack of sufficient evidence to support the claims of solving the neural network tuning problem. 
- As suggested in the title and the Intro, a main focus of this work is trying to "estimate the performance of an architecture before training", thus providing "a solution to the problem of tuning neural networks". However, there is only indirect evidence that the correlations between the accuracy on the test set and the 90th percentile of the spectral distribution is considerably high for various 4-layer MLPs trained on the MNIST benchmark. This is indeed intriguing, but it is far from a solution to network tuning. The choice of 90th percentile seems arbitrary. Is the 90th quantile the best summary statistics of the spectral density for estimating model performance? What about other quantiles? Or this particular quantile alone is sufficient to quantify the model performance?
- There is lack of discussion of how to apply this metric, 90th percentile of the theoretically computed spectral distribution for example, in practice. Also, how to avoid the vanishing and explosive regimes? It would also be nicer to provide more results on multiple benchmarks.
- In Section 1.1, the authors argued that "smaller networks with more spread-out $\nu_J$ perform better than larger counterparts". But I could not find any supporting evidence for this. In addition, the quantitative results are only concerned with the 90th quantile with little discussion/summarization about the spread-ness of the whole spectral distribution. 
- There is existing work discussing the relationship between trainability and generalization, [1] for example. It argues that "trainability and generalization are distinct notions that are, at least in this case, at odds with one another. Indeed, good conditioning of the NTK (which is a necessary condition for training) seems necessarily to lead to poor generalization performance. It will be interesting to see whether these results carry over in shallower and narrower networks." The last question seems can be answered from this work. It would be interesting if the authors could discuss more on this.

Typo:
- In Line 186, it should be $M_{J^TJ}$ instead of $M_J$.

[1] L. Xiao, J. Pennington, and S. Schoenholz. Disentangling trainability and generalization in deep neural networks. ICML 2020.
questions: - How is the 90th quantile selected? What about other quantiles or other summary statistics? Is a single quantile alone sufficient to quantify the model performance?
- In Line 167, it states that "$N_{l-1}/N_l \rightarrow \lambda_l \ge 0$". Should this be "$\ge$ 0" or "> 0"?
limitations: The authors addressed the limitation in the final paragraph of the manuscript. Regarding the potential societal impact of this work, I do not see there is any.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors considered the problem of computing the limiting eigenvalue/singular value distribution of the input-output Jacobian matrix of a fully connected "deep" neural network model having random weights, in the limit where the network widths go to infinity.

The contribution of this paper is threefold:
1. extend the analysis proposed by Pennington et al. in a sequence of papers to the setting of non-square weight matrices by introducing the tool of "rectangular multiplicative free convolution"; and
2. propose a more accurate and faster computational scheme to solve the limiting distribution (see Figure 1), with theoretical guarantees; and 
3. apply these theoretical and computational results to empirically show that the above eigenvalue distribution is closely connected to the accuracy of networks, at least in some cases (see Figure 2).


strengths_and_weaknesses: **Strengths**: Despite having some knowledge of the theory of large dimensional random matrices, I am not an expert in free probability theory and it is less evident for me to evaluate the contribution of this work.

The first contribution of this work is of significant practical interest, and the second contribution of this work is, to the best of my knowledge, long believed to be challenging in the field, and the proposed result is worth publishing.

With that said, I strongly recommend that:
1. the Area Chairs and Senior Area Chairs could find an external reviewer for this paper (I can propose a few names if that helps) to better evaluate the (theoretical and computational) contribution of this work; and
2. the authors submit a longer and more detailed version of this paper to a stats/applied math journal since the proposed approaches are of independent interest.

**Weaknesses**: The paper, in its current form, is NOT easily accessible to a large AI/ML audience and should be carefully revised and better organized: for instance, most of the contents in the introduction are in fact "preliminaries", and to introduce the concept of multiplicative free convolution the authors discussed the case of square weight matrices but the scenario considered in this paper is in fact "varying width profiles". In short, the paper can be reorganized to be made clear.

questions: Some minor issues/remarks:
* The claim "stability is a good proxy for performance" in the introduction should be supported by theoretical, empirical, and/or related previous efforts
* Please provide some references for the term "edge of chaos" in line 44, which may not be familiar to all the readers.
* Please provide some references for the discussions in the paragraph AFTER line 47.
* line 74: I do NOT understand the claim "In the limit, asymptotic freeness will also hold": this is in fact an ASSUMPTION, right? And the notion of "asymptotic freeness" only makes sense asymptotically? Or did I misunderstand something?
* Just a side remark, but I believe it is of interest to mention after Equation (7) whether the operator of multiplicative free convolution is "exchangeable".
* The authors constantly mentioned in the paper "Pennington et al.", but I personally believe it is more clear and more precise to cite the exact paper and the sequence of papers that this work is building upon.
* I wonder if the conclusion drawn from Figure 2 remains valid on more involved ML problems such as CIFAR10/100 and ImageNet image classification.
* line 155: is not associative, i.e., for ...
* line 188: I believe ReLU is more commonly used than ReLu.
* Please provide a few references for the paragraph from line 212 to 217 for better readability.
limitations: The authors have adequately addressed the technical limitations of their approach.
This work is mainly theoretical and I do not envision any potential negative social impact of this work.
review_title: null

Meta-review: metareview: The main contribution of this paper is an algorithm to compute the spectral density of the free multiplicative convolution which appears in the expression of the Jacobian of neural networks. Some concerns have been raised regarding the impact on a general ML audience (reviewers PUn2 and 1DWD), and about the presentation of the material (especially abstract and introduction, see review GQUV). I share part of these concerns. At the same time, the proposed “Newton lilypads” algorithm not only provides a speed-up over prior work by Pennington et al., but it appears to be an original and interesting contribution in free probability beyond its ML application. Therefore, I ultimately agree with the reviewers, who have reached a consensus of accepting this paper. The novelty aspect of this paper will make it an interesting addition to the NeurIPS 2022 technical program. As a final note, I would like to strongly encourage the authors to include in the camera ready the additional experiments on FashionMNIST and CIFAR10, as well as the discussions related to the feedback from the reviewers (including, if possible, the suggested title/abstract change).

confidence: Certain
Decision: Accept