Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper generalizes the Tensor Programs framework, which computes infinite-width limits of deep networks / differentiable computation graphs under the assumption that weights are randomly initialized from independent Gaussians with variance 1/n, to the case of iid non-Gaussian initializations. 
strengths_and_weaknesses: The result in this work is an incremental extension of the Tensor Program framework to non-Gaussian weight initializations, and is clearly not intended as a standalone exposition of the framework. Unfortunately I have only a vague familiarity with the existing Tensor Program line of work and so can only provide superficial comments, not critique the result in any detail.

Significance: this result contributes to the theory of infinite-width limits of neural nets, which is certainly of general interest and has motivated interesting new methods that may be practical in some settings, e.g., for hyperparameter search. The specific contribution of this paper is incremental, and the result (although apparently nontrivial) does not seem particularly surprising, so its broader significance seems limited at best. Does this extension 'deserve' to be its own top-conference paper, rather than (say) an appendix to the general framework? I don't feel qualified to judge, but given the community's recent interest in this line of work I'm willing to give it some benefit of the doubt.

Quality and clarity: the paper is generally well-written. Technical conditions and results are clearly stated, as are the limitations and the relationship to previous results. I did not attempt to check the proof (section 4) but it appears well-motivated, structured and carefully reasoned; I believe it is likely correct. 

Originality: the proof technique appears to follow a similar approach used in other recent work (Chen and Lam 2021), but its application and extension within the Tensor Programs framework is novel and, I think, nontrivial.
questions: nits:
- line 24: 'applications of Law of Large Numbers' doesn't parse, I would say 'applications of *the* Law ...'
- line 41: typo 'identially' -> identically

limitations: The technical conditions and limitations of the result are clearly articulated and discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors generalize the Master Theorem of Tensor Programs (TP) by demonstrating that non-Gaussian initializations result in many of same consequences of TP including (perhaps most interestingly) the NNGP correspondence (infinitely wide neural networks behave as Gaussian processes). Differences between this result and the original Gaussian initialized TP include the following: requirement of smooth nonlinearities, and weaker convergence (in probability versus almost surely). Much of the manuscript is dedicated towards proof. 
strengths_and_weaknesses: Strengths:
-The formalism introduced by TP was (and still is) of both theoretical and practical use (say, for hyperparameter tuning). Generalizing it even further is beneficial for the DL community. 
-Proof of the non-Gaussian Master Theorem is thoroughly detailed. 
-Nearly all of the consequences of the Gaussian Master Theorem are obtained. 

Weakness:
-Notation is cumbersome. This is a challenging paper, indeed. Tensor Programs I (G. Yang) expends serious effort towards the development of NETSOR to aid the architectural universality result, but this paper is at times a pure exercise in analysis. Any type of diagram or visual to accompany the proof of Theorem 2 would have been nice. 
-Requirement of smooth nonlinearities (although this is addressed, several times over). 
questions: 1. Why do the authors consider it useful to note that the semi-circle distribution and Marcenko-Pastur laws for non-Gaussian matrices fall out of Theorem 2? Going through tensor programs is not necessarily easier than existing proofs. What perspective/insights might deriving these results from TP have to offer? 
limitations: Limitations and societal impacts are discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper extends the Tensor Program to the non-Gaussian case, which justifies a previous hyperparameter tuning method for non-Gaussian weight initialization.
strengths_and_weaknesses: Since I don't have a background in Tensor Programs, I can only comment on writing.

The proposed theory helps to explain some previous practices in learning neural networks; however, it does not suggest new techniques nor has empirical verification. Moreover, the paper is generally dense and dry, therefore challenging for non-specialists to follow. 
questions: The authors may consider moving the proofs to the appendix and using the space to explain more about Tensor Program and its application.

For example, in Section 2, readers may want to know how to convert a practical convolutional networks (with weight sharing convolutions and batch normalization) to Tensor Program. Since the authors emphasize on universality, it is also informative to discuss whether the scheme is applicable in graph neural networks, where the central limit theorem may not hold given sparsity.

Moreover, in Section 3, it is useful to discuss how to use the theoretical result for hyperparameter tuning, instead of citing pror works. Lastly, since the result only hold for infinite-width case, it is helpful to include some empirical study to show the gap between theory and practice.
limitations: Not applicable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work continues in the spirit of Tensor Program series of papers. Starting from a theoretical
result showing that nearly any neural architecture with Gaussian weight initialisation in the limit
of infinite width exhibits Gaussian Process behaviour, this paper extends this theorem to apply where
weights have non-Gaussian initialisation. This requires assuming that all matrices (A) used have iid
entries from a Gaussian distribution, all bias vectors also come from an iid Gaussian distribution, and
that all non-linearities be polynomially-smooth.
  
strengths_and_weaknesses:   Originality:

  This work is very original and I am not aware of any other theoretical contributions of this sort
  in the literature. 

  Technical Quality:

  The work is technically sound and although I did not carefully inspect the proofs I don't have
  significant reasons to doubt the results. Although there are not experiments in the paper, given
  the theoretical nature of the contribution I don't think it's necessary.
  
  Clarity:

  The work relies heavily on familiarity with previous Tensor Programs papers for background and motivation
  which makes the paper and its contribution less self-contained. But given some familiarity, the paper was
  straightforward to read and follow.
  
  Significance:

  The paper offers to me seems a fairly modest contribution. With the restrictions needed to not require Gaussian
  weights it's unclear why simpler extensions could not have been done. For example, Appendix section A. states that
  images of smooth polynomial maps can be used to get around the Gaussian initial bias and layer weights.
  Why could the same trick not been used in original formulation?

questions:   The paper offers to me seems a fairly modest contribution. With the restrictions needed to not require Gaussian
  weights it's unclear why simpler extensions could not have been done. For example, Appendix section A. states that
  images of smooth polynomial maps can be used to get around the Gaussian initial bias and layer weights.
  Why could the same trick not been used in original formulation?

limitations: I believe the authors have adequately addressed the limitations and potential negative societal impacts of the work.
review_title: null

Meta-review: metareview: This submission is borderline.  Reviewers were generally in consensus --- all felt that the theoretical contribution is sound and non-trivial, but delivers a relatively minor addition to the Tensor Programs framework.  I fully agree with this perspective, and similarly to all reviewers, recommend that the paper be accepted and the NeurIPS community will be the judge of how significant its contribution is.
confidence: Less certain
Decision: Accept