Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The main result of the paper is a derivation of the exact characterization of priors in function space through Meijer G-functions. The results are obtained for Gaussian priors, ReLU and linear activation functions. The article also provides a solid analysis of the obtained characterization. 

**Advantages**: 
* It is the first accurate description of hidden units for standard feed-forward neural networks among the concurrent work mentioned in the paper. 
* The results are in line with other works on the heavy-tailed nature of hidden units and the Gaussian process limit. 
* There is a study on the role of width and depth demonstrating how wider networks lead to more Gaussian-like distributions. 
* Authors also introduced generalized He priors that enable to tune the variance induced in the function space. 


**Limitations**: 
* Only linear and ReLU functions 
* Only Gaussian priors 
* Without the bias term 

main_review: 
**Suggestions**: 
* While there is a study on the depth limit, I think it is important to talk about the deep propagation papers and Edge of Chaos: Schoenholz et al. “Deep information propagation”. In addition,  Hayou et al “On the impact of the activation function on deep neural networks training” argue that smooth activation functions lead to better information propagation. Do you think it is possible to achieve similar results based on the obtained depth limit?
* I would like to see a discussion on the limitations of the setting and the Meijer-G functions approach such as if it is possible to add a bias term or to work with other activation functions (e.g. smooth function, see the previous paragraph). 
* Lee et al. “Finite Versus Infinite Neural Networks: an Empirical Study” might be a relevant citation about the performance of finite-width networks over infinite-width counterparts 
* Aitchison. (2020) “Why bigger is not always better: on finite and infinite neural networks.” might be added to the discussion

**Clarity**:

The paper is well-written and enjoyable to read despite the technicality of the results.

**Significance**: 

I think the results are significant and will help to extend the line of research on function space description of Bayesian neural networks. 

**Conclusion**:

The results are obtained for a simple model and seems hard to generalize to other settings or priors but they are very solid. I recommend publication. 
 



**Minor**:

* “In-line” -> in line 
* References: titles should be properly capitalised: Gaussian, NNGP, NTK, Meijer, Bayesian,… The conferences names should be unified. The reference of Novak et al is mentioned twice. 

Appendix:
* “Based in” -> on
* “We state and proof”, “we proof” 
* “For both distribution”
* “Well known” -> well-known 
* “Is significantly reduce” 

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Leveraging the Meijer-G function, the authors characterise the predictive priors of fully-connected (finite-width) feedforward induced by independent standard Gaussian weight priors. They show that they can recover existing results concerning infinite width and infinite depth networks, and are also able to tackle different limiting regimes. 
main_review: The paper is very clearly written and the technical tools and derivations are introduced in a logical order. Although I was not intimately familiar with the Meijer-G function, I was able to follow the argument of the authors by reading the paper from start to finish without having to understand the details. I like, for example, that the authors started with the linear case, giving the readers the intuition needed to begin to understand the ReLU case.

Questions:
1. One future direction the authors mention is extending to other architectures, in particular, conv nets. What about other activation functions? Lemma C.1 especially leverages the piecewise linear nature of ReLU activations on line 616 of the Appendix. Can you also extend this to Leaky ReLU activations? What about Heaviside step activations?
2. Unless I missed it, I don't think there was any insight into the posterior that is induced. Is there anything you can see here?
3. This work exclusively applies to the Bayesian setup. What breaks in the NTK setup? I'm guessing the issue arises from the fact that the NTK is no longer close to constant for finite-width. 
4. One part of the paper that confuses me is all of section 6. The authors derive the variance at the output of a finite-width network (equation (24)) by leveraging their previous results. As far as I can tell, this can be derived without the earlier parts of the paper, but perhaps I am missing something. The ReLU function is absolutely homogeneous, ReLU(|a|x) = |a|ReLU(x) for any real number a. Then, factor out the standard deviations from each of the weight matrices so that they are standard Gaussians. The result is a scalar product of all the standard deviations with a random variable with variance (1/2)^L, the same as the result obtained by the authors.
5. Is it possible to sample from this prior directly, without using a neural network?
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Uses the Meijer-G function to analyse the properties of finite-width Bayesian neural networks.  Provides additional proofs of the infinite-width limit, in addition to results for the infinite depth limit.  Shows that finite networks are heavy tailed and that this increases with depth.
main_review: I really like the paper, the while the maths are a bit heavy and will be unfamiliar to most researchers, they do a fantastic job giving a clear high-level description of    what the Meijer-G function is and why it is useful to understand Bayesian neural networks.

My key issue surrounds references to past work.  The authors appear to be unfamiliar with a line of work that began with Aitchison (2019) "Why bigger is not always better:  on finite and infinite neural networks".  This work explicitly address the question of why infinite networks perform worse than finite networks (which is mentioned here in  the related work).  It concludes that finite networks differ from infinite networks principally in the existence of flexibility in the Gaussian process kernel, with deeper, narrower networks having more flexibility.  Flexibility/variability in the kernel naturally implies heavy-tailed-ness of the output distribution.  But more importantly, the view in terms of representations gives us much more insight into what the flexibility in finite nets is actually used for.  This work was continued by Aitchison el al. (2020) "Deep kernel processes", which uses these insights to develop a new family of flexible, deep kernel methods that are in some cases equivalent to deep neural networks, and give considerable  insights into representation learning in practically relevant deep networks.

There are a number of most likely contemporaneous works (potentially submitted to this conference).  While the authors may not have been aware of many of these (I realise that some may have appeared after the submission date to the conference), it is important to discuss the interrelationships between these works in the "Related Work" section.  These papers include:
* https://arxiv.org/abs/2104.11734
* https://arxiv.org/abs/2106.06529
* https://arxiv.org/abs/2106.00651
* https://arxiv.org/abs/2106.04013

Very happy to increase the score if these issues are resolved!

The "Predictive variance" section is a bit confusing.  It just seems to tell us how to set the variance of the weights of an infinite width in the network such that the output variance is a prescribed value e.g. tells us that a standard Gaussian prior gives extremely high output variance.  Unless I'm missing something, this has been obvious for quite a long time, and was the motivation behind the introduction of this the He normalisation.
limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper adds to the large and important body of work of characterising the prior in function space of a deep learning model implied by the architecture and the prior over the parameters. The paper focuses on Gaussian priors with relu and identity activation functions.

All the reviewers liked the paper. They found it clearly written and appreciated that the paper made an effort to convey the high level results despite the paper using quite heavy machinery (the Meijer-G function) to derive finite network width results. The discussion was substantial and led to a good consensus among reviewers and authors.

The paper adds substantial new insights into an important area of research in deep learning. 

Accept. :-)
Decision: Accept (Spotlight)