Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Layer outputs of neural networks with Gaussian priors are proved to be Gaussian processes at infinite width. The paper is focused on obtaining a characterization of layer outputs at finite width. Based on the NNGP kernel resulting in the wide regime, the main result is the introduction of perturbative corrections to the average of this kernel. These corrections appear in finite-width neural networks and don’t exist in the limit. 

**Contributions**: 
* Updated description of Yaida [11] on the difference between finite and infinite wide neural networks 
* Feature learning interpretation 

**Limitations**:
* Not rigorous 
* Works on very simple setting but no additional experiments 
* Not evident how to use the results
main_review: **Questions**: 

* What is the difference with the kernel used in the paper and introduced in Lee et al. “Deep neural networks as Gaussian processes”? I guess the regime when widths of layers tend to infinity sequentially can be easily addressed in the paper?
* “As a possible complication, one could imagine a situation in which the activation distributions are ‘rotated’ during inference without changing the kernels. However, Yang and Hu [8] have shown that this cannot happen in the infinite-width limit…” Could you please precise the place in the manuscript? 
* The authors cite paper [16] in related works about the heavy-tailed nature of the prior densities over the outputs. I wonder why not cite other works that stated heavy tails like “Precise characteristics..” (the concurrent work of [16]) and Vladimirova et al. “Understanding priors…”

**Quality**:

I find the paper and the appendix well-written and I believe that mathematics is correct. 

**Significance**: 

The results follow recent results of Yaida [11] and can be well-placed in the literature. However, I would like to see applications of these corrections (at least possible). 

**Conclusion**:

The paper has a solid theory. I recommend publication even if it misses the application part. 

**Minor**: 
* “Pertubative” 
* “The second class of corrections are” 
* missed comma in (A.2)
* I would reintroduce the notations in the appendix as well, like $E$ for error function, $n$, etc 

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper shows analytically and numerically that the leading finite-width corrections to the average hidden layer kernels of any fully-connected feedforward networks with linear readout and least-squares loss have a largely prescribed form :  these assumptions fix the dependency of the correction on the target outputs, the size of the correction scale with the ratio of the number of outputs to the typical hidden layer width. Authors explicitly compute the leading finite-width corrections for deep linear networks 	and networks with a single nonlinear hidden layer . Theyshow how certain directions in the output kernel matrix can be enhanced or suppressed in the hidden layer representations depending on the structure of the input similarities 

This is equivalent to considering the equilibrium distribution of Langevin sampling of the parameters at inverse temperature β. Moreover, by tuning β, one can then adjust whether the posterior is dominated by the prior (β ≪ 1) or the likelihood (β ≫ 1).  

Authors conjecture, non-rigorously, that these results should extend to Bayesian neural network with different architectures 

main_review: This is a very interesting paper, of particular interest as, despite the numerous critics of infinite-width neural networks low representational power, their learned features are not well characterized.   

Authors derive the leading finite-width corrections to the average hidden layer kernels of any fully-connected feedforward networks with linear readout and least-squares loss rigorously. I find the high- and low-temperature limits of the leading correction as well as the recovery of Aitchison implicit recurrence very compelling. 

This is a first step towards understanding the features learned by Infinite width BNNs, but conjecturing that results  should extend to other architectures is an overstatement to me. Neural Networks behave in a very complex way and most related papers rely on strong assumptions on the shape of the networks to make any conclusions. The analysis of the wide “narrow bottleneck” is interesting and aims at generalizing the results but is not enough.

For better readability, Figure 1 should be updated. It is not clear that authors obtain the predicted 1/n decay with increasing width and the linear scaling with the depth, as the axis are not correctly labeled. What do the markers represent? Why do error bars don’t appear? Moreover, both figures can be made bigger. 

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Work analyzing representation learning using finite-width corrections to Bayesian neural networks.  Specifically, they look at deep linear networks and networks with one nonlinear hidden layer.

main_review: Work presenting insightful results on neural network theory in simple cases.  The methods are clearly developed in the Appendix.  I have no technical concerns.  There are a couple of points that I'd be interested to see discussed further (but this can easily be done in a revision):

Possibly the key weakness of this work is that it uses simplistic deep linear networks, or networks with one nonlinear layer.  Of course, this allows the authors to come up with simple, interpretable results that give solid insight into the underlying processes.  But it would be worth presenting the case that these corrections are relevant in "real world" cases (i.e. deep nonlinear networks).

Likewise, the empirical findings of Aitchison (2019) that upper-layer kernels are radically different from the infinite limit.  I'd be very interested to know whether the results of Aitchison (2019) differ from the perturbative approach, and whether that tells us anything about "how far" these low-order perturbations can take you from the underlying infinite limit.

Appendix is very clear!  The "Preliminary Technical Results" section is great.

Good literature review, and extensive connections to existing literature.

While Zavatone-Veth and Pehlevan (2021) appears in the Related Work, I wasn't at all clear about the relationship between the papers (at least from the discussion there).  Would being _really_ clear about the high-level and technical relationship.  At a high-level, my guess would be that this papers looks at representation learning, while that paper focuses on the distribution of a single output variable?  But it would also be useful to sketch the technical relationship, if any!
limitations_and_societal_impact: The idea that the paper is "non-quantitative" is a bit strong!
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of leading-order perturbations to average feature kernels of Bayesian finite-but-wide neural networks (NNs). The paper is mainly of theoretical contribution and the main result establishes a conjecture for the leading order corrections for general NN architectures under linear readout and squared loss. The authors then delve into limiting situations of tempered posteriors before analysing two tractable cases: deep linear NNs and single-hidden layer non-linear NNs. Experimental evidence on toy settings support their claims.
main_review: *Originality*: This is to my knowledge the first paper to study the leading-order pertubations to average feature kernels of general wide BNNs, and certainly I believe Conjecture 1 is original. The case of deep linear networks is closely linked to the works of Aitchison [9] and Li and Sompolinsky [15], and the authors detail this comparison in Appendix E, though the authors stress that their contributions of leading-order corrections extend these previous works.

*Quality*: I believe the submission is technically sound, and written in a fair way without overstating their claims (which I commend). However, the work seems a bit light and incomplete and thus is slightly unsatisfying. Indeed, the main contribution is a non-rigorous conjecture and the analytic cases considered do not give me a great deal of insight into the inner workings of feature learning in BNNs (which is a very interesting question). 

Some of the questions/thoughts I have which I believe would improve the quality of this work if addressed:
- How can you make conjecture 1 rigorous?
- Do you get posterior concentration of the feature kernels with large p? I realise you say this is left for future work in  line 277-278, but this is quite a crucial question I think for a Bayesian perspective of wide NNs
- Related to above, can we say anything about other moments of the finite-width corrections to feature kernels besides the posterior mean?
- I realise that analytic tractability restricts you to deep linear and single hidden-layer, but unfortunately that means we don't see any notion of 'hierarchical' feature learning (i.e. simpler features being learned in the early layers), which intuitively is important for (B)NNs.
- Is it wrong to think of having n_d outputs per input, as having a dataset size of $p\times n_d$ instead of p? In which case makes the paragraph from lines 130-134 slightly unsurprising. Indeed I would expect the second term in eq 8 to scale as $O(p\times n_d)$, but the dependence on p isn't clear?
- Can the authors empirically demonstrate that your theory holds in more practically relevant settings. E.g. using the whole MNIST dataset as opposed to 100 points. What about CNNs with many channels?

*Clarity* The paper is well written and organised. 

*Significance* The question of feature learning (or lack of it) in wide (B)NNs is interesting to me and I believe to the community as a whole. The results in this submission are a new angle towards understanding the interplay between the infinite-width 'lazy' GP limit and the finite-width 'feature-learning' regime, and in that sense are significant. Although see my comments in *Quality* about questions that I wish were addressed in this submission. 

Missed citations:
- https://arxiv.org/abs/2106.06615 concurrent work to Zavatone-Veth and Pehlevan [6]
- https://arxiv.org/abs/2006.10541 prove limiting posterior converges weakly to GP limit.

Minor points:
- the notation p for dataset size is confusing.
- The conclusion focuses in on the NNGP kernel but this seems like you're selling yourself short as you also discuss earlier layers' kernels.
- The expectation with respect to \mathcal{W} e.g. in Eq 8 doesn't seem to be defined. I take it means the prior over weights?
limitations_and_societal_impact: The authors adequately address the limitations of their work in the conclusion section. They do not discuss potential negative societal impacts though I do not believe there are any.
review_title: null

Meta-review: This paper analyzes the leading-order corrections to the average feature kernels in Bayesian neural networks, and conjectures a universal form for the result. The analysis and conclusions should be of interest to theorists in the NeurIPS community, and I recommend acceptance.


Decision: Accept (Poster)