Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper provides a first order approximation of the variance of a one-layer neural network. This approximation is composed of 5 terms:

- $\mathbb V^a$ is the finite-width equivalent of the predictive variance of an infinite width NTK, and depends on functional noise at initialization
- $\mathbb V^c$ is an approximation of the predictive variance of a linearly-trained network, and depends only on the kernel noise.
- $\mathbb V^i$ is the predictive variance of an ensemble with a NTK kernel and a data-specific prior
- $\mathbb V^{cor}$ and $\mathbb V^{res}$ are two terms that are negligible in empirical experiments.

Based on this decomposition, the authors define ensembles that depend only on one or another source of noise ($f^c$ and $f^a$), and show that isolating these sources of noise can improve OOD detection.
strengths_and_weaknesses: # Strengths
- Understanding the degree to which the variance of a deep ensemble is indicative of an ensemble's certainty in making a prediction is a crucial question as deep ensembles are deployed on real-world applications.
- The authors push through a detailed decomposition of the variance of linearly trained neural networks, then generalize this decomposition under several assumptions to neural networks trained with gradient descent and SGD. 
- The authors analyze the terms in the proposed decomposition empirically across a variety of experimental settings (different training modalities, datasets, and dataset samples).

# Weaknesses
- I found the empirical section of this paper difficult to put into context with the previous decomposition (cf. "Questions" section below).
- I am not sure I follow the author's claim that isolating unfavorable noise improves OOD detection.  This appears to be the case in certain experiments but not all (e.g., Table 1 for the CNN on Cifar-10), and it also seems like the "unfavorable" noise depends on the experiment setting (Table 1, linear models on CNN: $f^{lin-c}$ does better for SVHN but worse for LSUN).
- I would've liked to see $\mathbb V^i$ included in the empirical analysis, since it is the third major linear term in the approximation to the full variance. 

questions: - How would one extend the decomposition of Prop. 2.2 in the presence of additional sources of noise? Would an approach such as [1] be sufficient?
- How are the AUROC values computed? Are they computed based on the (pointwise) predictive variances of the different ensembles?
- Line 219: "Interestingly, $\mathbb V[f^{lin}(x)]$ depicts a superposition of the three isolated variances." Is this not a validation of the decomposition of Prop 2.2? Why is this surprising?
- In cases where we have $\mathbb V[f^{lin}] \le \mathbb V[f^{lin-c}] + \mathbb V[f^{lin-a}]$, is this due to $\mathbb V_i$? Does $\mathbb V_i$ factor into defining favorable/unfavorable sources of noise?
- Are there any difficulties in obtaining high-quality estimates of the different variance terms? Are the empirical estimates unbiased?


[1] Understanding Double Descent Requires A Fine-Grained Bias-Variance Decomposition, Adlam & Pennington, 2020
limitations: The authors discuss different limitations of their work, mostly based around assumptions made about the behavior of NNs (e.g., 2.3). These assumptions are discussed in detail, evaluated empirically, and put in the general context of this work.

In my mind, the two major limitations of this work are the following:
- The interplay between the two sources of noise (initialization & kernel noise) is not analyzed empirically, despite appearing with equal importance in the decomposition of Prop. 2.2. 
- I found the conclusions of the empirical analysis to be somewhat unclear. I believe the discussion could be improved by the authors stating early on which hypothesis they seek to verify, and how that hypothesis is or isn't validated by the experimental results. In the paper's current state, I find the conclusions to be somewhat unclear (which source of noise is conducive to OOD detection? Is a single source of noise sufficient in general?).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work is focused on decomposing the sources of variance in neural network predictions - the predictive variance of neural networks is an important quantity that allows for neural ensembles (and other approximate bayesian models) to detect OOD samples without seeing OOD data. However, this quantity is poorly understood - this work proposes to take a close, principled look at neural network predictive variance by studying _linear_ neural networks. Linear neural network predictions can be represented via a kernel expression, allowing a theoretical decoupling of model predictions and (noisy) gradient descent.

The work largely comprises of 2 main sections. The first section (section 2) contains the main theoretical results of the work, and begins by introducing linear neural networks and infinitely wide neural networks. The work then builds up a theoretical decomposition of variance terms in the predictive distribution of an ensemble of neural networks. To do this, the authors begin by analyzing an ensemble of infinitely wide neural networks, whose kernels are deterministic, demonstrating that it's predictive variance can explained by the _functional noise_ over different initializations. The authors then extend this analysis to _finite_ width linear networks, for one and more layers - under some assumptions, the authors show that the predictive variance of finite-width linear networks can be described largely by two noise sources: the _functional noise_ as described above, and the _kernel noise_ stemming from the variance over different kernels in the finite-width regime. Finally, the authors provide evidence that the discrepancy between a linearly trained model and it's non-linear, gradient descent counterpart, can be bounded on the order of it's layer-width, justifying the approximation of non-linear ensembles with linear models.

The second section (section 3) validates the authors hypothesis via a set of empirical experiments. The first set of experiments utilizes tractable linear models for which distinct components of predictive variance can be explicitly removed. Here the authors use a  toy dataset to provide an intuitive visualization of how each isolated component of the variance behaves. Next the authors study 2 real-world tasks, CIFAR-10 and MNIST, using a more complex linear model (both a CNN and MLP model), examining both ensemble performance and ensemble OOD detection rates. They find that different noise components have different levels of OOD detection - in some cases, the model with only kernel noise can achieve comparable or better performance in OOD detection than the full model. They then extend their empirical analysis to non-linear models - using a clever trick around model initialization, the authors derive two variants of gradient-descent models which isolate either functional initialization noise or initial kernel noise. They find that the isolated components of predictive variance in neural ensembles trained with (S)GD mirror those of the linearly trained models on OOD detection performance. Finally, the authors train fully modern neural networks, using residual connections, batch-norm, and cross-entropy loss on CIFAR-10, leveraging the same techniques to isolate each noise source. They show that, in this setting, isolating noise sources can have a significant effect on OOD performance - in particular, _centered_ models which remove initialization noise, outperform standard models.
strengths_and_weaknesses: Strengths:
- The predictive variance of neural networks, and it's utility to OOD detection is of broad interest to the field. 
- Leveraging linear networks (and their kernels) to get a theoretical intuition about the different sources of variance in neural ensembles is a clever and novel approach to understanding the sources of variance, and their effects.
- The theoretical analysis provided in this work is very interesting, and provides the foundation for future work aiming to understand predictive variance in neural models.
- The final empirical result is relatively strong - namely, they show that a particular source of noise (kernel noise) is more effective at identifying OOD examples than standard neural models in modern neural architectures trained with SGD.

Weaknesses:
- The biggest weakness of this work lies in the empirical analysis and the takeaways from this analysis:
	- Overall, the benefits of each isolated noise source are not consistent. It is not clear what effect of each noise source has on OOD detection - in some cases isolated kernel noise outperforms isolated functional noise, and in other cases the opposite occurs.
	- The "big takeaway" of the role of each noise source in OOD detection largely seems to rely on the single result of the fully fleshed out neural network on CIFAR-10. Only is this model is there a clear story on the importance of isolated kernel noise.
	- In other words, the paper cannot empirically show us why the proposed decomposition is practically useful other than that it has _some effect_ on OOD detection.
- Lastly, I think the structure of the paper could be improved:
	- For instance, much of the theoretical analysis lacks intuition around key quantities (e.g. the matrix Q is referenced all over the paper, and it would help the readability of the formulas to provide some intuition as to what this matrix represents).
	- Figures and tables are placed distant from their corresponding sections - additionally, sometime the corresponding figure or table is not referenced in the main text (e.g. Table 1 is not referenced in section 3.2.2).
questions: - The experiment in Figure 1 is a nice visualization, but the intuition it provides is not tied into later results - why is the distance or angle between datapoints a useful interpretation of functional or kernel noise? If it is not relevant to the next set of experiments due to data dependency then what is the toy experiment showing us?  
limitations: yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies ensembles of linearly trained deep networks, for which a decomposition of the variance is proposed into two main components:
 - V_a results from the variance of the initial value f_theta_0
 - V_c results from the variance of the initial kernel Theta_0
and other components that decay faster as the width of the network is grown.

It is also shown that this decomposition still hold when training non-linearly one hidden layer networks under stability assumption of the NTK during training. It is conjectured (and empirically somehow observed) that the difference in prediction between linear and non-linear training is small enough for the above decomposition to still be relevant in deeper networks.

Experiments on toy datasets enable visualizing the different sources of variance. Experiments on more difficult (MNIST, CIFAR10) datasets show that conclusions vary across setups. A method is proposed to isolate sources of variance in actual setups.
strengths_and_weaknesses: *Originality*: To the best of my knowledge (though I am not so familiar with recent literature), the idea of studying ensemble of ensembles of linearly trained networks is new and sound. The results linking infinite-width models to linearly-trained finite networks are new, in the line of works around the NTK formalism.

*Quality*: The paper is well organised, the experiments and figures are relevant to the discussion. I had some comments about the experiments (see questions below)

*Clarity*: The material is well presented. The theoretical results are clearly discussed, as well as the assumptions. I however was not sure what is the main message to get from the experiments on large networks/datasets. This is probably related to the latter, but I did not understand what are the "practical ways to eliminate noisy sources" claimed to be proposed in the abstract.

*Significance:*: The study of ensembles of neural nets is of particular relevance as many recent papers use ensembles of deep nets. The decomposition between variance of the initial function and variance of the initial kernel is of interest for understanding the training dynamics of neural networks.
questions:  - it is not clear that the y scale in figure 2 and 5 is a log scale from just looking at the figure (and not mentioned in the text).
 - in figure 2, contrary to what is mentioned in the text, it looks that the decay of V_c is slightly slower than 1/h (thus suggesting that V_c does not decay in O(1/h)). It is difficult to see because of the log-log scale but it looks to be closer to 1/h^{3/4}
 - in figure 5 in appendix, I did not understand how many steps of GD were run for this experiment. I am guessing that it uses very few steps since the dataset if very small (N=100), and I am expecting the results to be different on actual setups. What do you think?
limitations: I did not find a discussion about potential negative societal impact, nor do I need that such paper should have one.

The limitations of the setups (linearly trained networks, small datasets) are discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to understand finite-width neural networks in terms of linearly trained finite width networks. The authors propose to decompose the variance of ensembles of linearly trained, finite width neural networks into different sources (of noise), and study empirically the implications of this decomposition for understanding actual finite-width networks trained with gradient descent and stochastic gradient descent. The work builds from [1][2] which proposes to approximate a sufficiently wide network $f_\theta$ with its linearly trained counterpart $f_lin$, and in the infinite with limit with the fully trained network $f_\inf$.
The paper decompose the variance of $f_\inf$  and $f_lin$, and interprets different variance terms arising in ensembles of linear predictors in terms of kernel and functional noise (ln 140), which affects the predictive variance. The derivation relies on certain independence assumptions that do not hold in practice.
The paper additionally claims ways to eliminate sources of noise which improve OOD detection in ensembles. The results comparing the networks which only include disentangled noise sources show no consensus in how/if the proposed sources of noise affect the total variance. The claims of improving OOD detection after reducing sources of noise coming from the results don’t take into account that the results lie within the confidence interval (cf. Table 2).
strengths_and_weaknesses: Strengths:
This paper proposes to understand finite-width neural networks in terms of linearly trained finite width networks. The paper decompose the variance of ensembles of linearly trained, finite width neural networks into different sources (of noise). The work builds from  [1][2] which proposes to approximate a sufficiently wide network $f_\theta$ with its linearly trained counterpart $f_lin$, and in the infinite with limit with the fully trained network $f_\inf$. This is an interesting perspective for analyzing deep ensemblesl

Weaknesses:
- The paper decompose the variance of ensembles of $f_\inf$  and $f_lin$.  The paper claims the variance for the ensemble of $f_lin$  includes two components which correspond to the functional initialization noise and the other to the kernel noise (ln 140), and that this noise affects the predictive variance. During the derivation it is assumed that the that the functional initialization noise is decorrelated from the $f(x)$ (ln 133). The authors mentioned that this is future work but this is a central assumption in this work.
- The results comparing the networks which only include disentangled noise sources show no consensus in how/if the proposed sources of noise affect the total variance. Could we add a Fig 2 which includes the baseline, total V?
- The claim of improving OOD detection after reducing sources of noise coming from the results in Table 2. In Table 2, we can see that the comparisons lie within each other's confidence intervals, so it is unclear why we can claim a model is better than another. 
- Some sections need additional clarification, i.e. the description of the variance components in paragraph ln 147 to 160 is not clear to me. Given that this paper is offering a new perspective on the variance decomposition a clear description of each component would be greatly beneficial. I had to come back to this twice. The same holds for the implications of the results in Fig 1. 
- The assumptions and implications of the propositions, in particular 2.1 and 2.3, could be clearer.  Proposition 2.3 can be true for a single network, is this correct? This proposition is in the deep ensemble section 2.2.
questions: - what is it the implication of assuming the functional initialization noise is decorrelated from the $f(x)$ in proposition 2.1?
- In line 140, why is it obvious that Va corresponds to the inf width model?
- What are the implications of Fig 1, if we cannot draw any conclusions from the shape/location of the data points, as the authors mentioned this changes per model/dataset?
- Could we add a Fig 2 which includes the baseline, total V? this would be useful to better understand the scale of different V. 

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This manuscript uses the concepts of Neural Network Gaussian Processes and Neural Tangent Kernels to analyze the sources of prediction variance from neural networks.  By making several reasonable assumptions, they can break the predictive variance into the superposition of several distinct terms.  Through this mathematical analysis and accompanying empirical studies, they demonstrate the practical consequences of this decomposition on several different neural networks.  Finally, they demonstrate that these decompositions can be used to build a greater intuitive understanding on out-of-distribution data.
strengths_and_weaknesses: This manuscript clearly presents its theoretical arguments and accompanies them with empirical studies that confirm the findings.  Given the large focus on robustness to out-of-distribution data in recent years, this type of decomposition can help practitioners think about potential ways to understand and improve their models.  I believe that this decomposition is original, and has high quality and significance.

The final empirical studies, though, are underexplained and a bit confusing.  It is not clearly described enough what prediction tasks are being performed (simply “out-of-distribution” tasks) and what approach is being used to determine out-of-distribution samples.  This needs to be much more clearly described in a revised manuscript.  The claims about performance are not refined enough. They claim that “These observations highlight the particular inductive bias of each variance term for OOD detection.” However, it is not clearly described enough how these inductive biases relate to the particular predictive performance.  The analysis would be much enhanced with clearer statements about exactly how these inductive biases relate to performance.
questions: How exactly were OOD tasks performed?  Update: This has been largely clarified after the rebuttal.

I do not fully understand the claims about the relationship between each variance term and OOD on data-specific performance.  Can you please elaborate on your claims of how the results highlight and explain these relationships?  Update: The response really highlights the challenges of interpretation.  Further claims and exploration are beyond the scope of this work, but I would encourage the authors to continue examining this question.
limitations: I believe that the authors have been forthright and fair in their descriptions of the limitations of their theoretical analysis.
review_title: null

Meta-review: metareview: The paper proposes to decompose the predictive variance of deep ensembles, into different sources of noise, by studying linearly trained finite width neural networks. 

The reviewers found the paper well-written, the theoretical analysis interesting, and the experimental evaluation adequate to justify the key claims.

I also appreciate the detailed author rebuttal to the reviewers’ questions. During the discussion, all the reviewers leaned towards acceptance. I encourage authors to address remaining comments in the final manuscript.

I recommend accept. Nice work!
confidence: Certain
Decision: Accept